Ingest pod logs
The following configuration example shows how you configure a Collector instance to fetch logs from all Kubernetes pods. It also shows how to enrich the logs with Kubernetes metadata in order to automatically link OpenTelemetry services to pods and attach the logs to the Kubernetes services and pods.
Prerequisites
- A deployed ActiveGate for Kubernetes API monitoring
- One of the following Collector distributions with the Filelog receiver and the Kubernetes Attributes processor
- The Collector deployed on each node
- The API URL of your Dynatrace environment
- An API token with the relevant access scope
- Kubernetes configured for the required role-based access control
- Kubernetes Secrets set up as shown in Collector deployment
Demo configuration
This sample configuration uses the same Kubernetes enrichment approach as the use case at Enrich from Kubernetes.
In addition to the Collector configuration, be sure to also update your Kubernetes configuration for the following components:
- Service account: Specify the same service account name used in the RBAC file (see entries for Helm, Operator)
- Mounted volumes: Specify the file system volumes where your Kubernetes host keeps the relevant log files (see entries for Helm, Operator)
- Mount paths: Specify the file system paths, to which the previously configured volumes should be mounted within the container (see entries for Helm, Operator)
receivers:filelog:include:- /var/log/pods/*/*/*.log- /var/log/containers/*.loginclude_file_name: falseinclude_file_path: truestart_at: endoperators:- id: container-parsertype: containerprocessors:k8sattributes:extract:metadata:- k8s.pod.name- k8s.pod.uid- k8s.deployment.name- k8s.statefulset.name- k8s.daemonset.name- k8s.cronjob.name- k8s.namespace.name- k8s.node.name- k8s.cluster.uidpod_association:- sources:- from: resource_attributename: k8s.pod.name- from: resource_attributename: k8s.namespace.name- sources:- from: resource_attributename: k8s.pod.ip- sources:- from: resource_attributename: k8s.pod.uid- sources:- from: connectionexporters:otlphttp:endpoint: ${env:DT_ENDPOINT}headers:Authorization: "Api-Token ${env:DT_API_TOKEN}"service:pipelines:traces: nullmetrics: nulllogs:receivers: [filelog]processors: [k8sattributes]exporters: [otlphttp]
Validate your settings to avoid any configuration issues.
Kubernetes configuration
Configure the following rbac.yaml
file with your Kubernetes instance, to allow the Collector to use the Kubernetes API with the service-account authentication type.
apiVersion: v1kind: ServiceAccountmetadata:labels:app: collectorname: collector---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:name: collectorlabels:app: collectorrules:- apiGroups:- ''resources:- 'pods'- 'namespaces'verbs:- 'get'- 'watch'- 'list'- apiGroups:- 'apps'resources:- 'replicasets'verbs:- 'get'- 'list'- 'watch'- apiGroups:- 'extensions'resources:- 'replicasets'verbs:- 'get'- 'list'- 'watch'---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:name: collectorlabels:app: collectorroleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: collectorsubjects:- kind: ServiceAccountname: collectornamespace: default
Components
For our configuration, we configured the following components.
Receiver
Under receivers
, we specify the filelog
receiver as active receiver component for our Collector instance.
The Filelog receiver supports a number of configuration parameters, which enable you to customize its behavior. For the example, we use the following:
include
—Specifies the path pattern of the files we want to ingest.start_at
—Specifies if the receiver should read from the beginning of the file or, for the most recent entries only, the end.operators
—Configures thecontainer
operator, which automatically parses each log entry.
Processor
Under processors
, we specify the k8sattributes
processor with the following parameters:
extract
—Specifies which information should be extracted.pod_association
—Specifies how the pod information is linked to attributes.
Exporter
Under exporters
, we specify the default otlphttp
exporter and configure it with our Dynatrace API URL and the required authentication token, as set up and configured under Kubernetes Secrets.
Service pipeline
Under service
, we assemble our receiver, processor, and exporter objects into pipelines for traces, metrics, and logs. These pipelines allow us to send OpenTelemetry signals via the Collector instance and have them automatically enriched with additional Kubernetes-specific details.