Ingest pod logs

The following configuration example shows how you configure a Collector instance to fetch logs from all Kubernetes pods. It also shows how to enrich the logs with Kubernetes metadata in order to automatically link OpenTelemetry services to pods and attach the logs to the Kubernetes services and pods.

Prerequisites

A deployed ActiveGate for Kubernetes API monitoring
One of the following Collector distributions with the Filelog receiver and the Kubernetes Attributes processor
The Collector deployed on each node
The API URL of your Dynatrace environment
An API token with the relevant access scope
Kubernetes configured for the required role-based access control
Kubernetes Secrets set up as shown in Collector deployment

Demo configuration

This sample configuration uses the same Kubernetes enrichment approach as the use case at Enrich from Kubernetes.

You also need to verify that the configured paths were successfully mounted within Kubernetes and are accessible to the Collector instance.

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
      - /var/log/containers/*.log
    include_file_name: false
    include_file_path: true
    start_at: end
    operators:
      - id: container-parser
        type: container

processors:
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.cluster.uid
    pod_association:
      - sources:
        - from: resource_attribute
          name: k8s.pod.name
        - from: resource_attribute
          name: k8s.namespace.name
      - sources:
        - from: resource_attribute
          name: k8s.pod.ip
      - sources:
        - from: resource_attribute
          name: k8s.pod.uid
      - sources:
        - from: connection

exporters:
  otlphttp:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [k8sattributes]
      exporters: [otlphttp]

Configuration validation

Validate your settings to avoid any configuration issues.

Kubernetes configuration

Configure the following rbac.yaml file with your Kubernetes instance, to allow the Collector to use the Kubernetes API with the service-account authentication type.

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: collector
  name: collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: collector
  labels:
    app: collector
rules:
  - apiGroups:
      - ''
    resources:
      - 'pods'
      - 'namespaces'
    verbs:
      - 'get'
      - 'watch'
      - 'list'
  - apiGroups:
      - 'apps'
    resources:
      - 'replicasets'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
  - apiGroups:
      - 'extensions'
    resources:
      - 'replicasets'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: collector
  labels:
    app: collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: collector
subjects:
  - kind: ServiceAccount
    name: collector
    namespace: default

Components

For our configuration, we configured the following components.

Receiver

Under receivers, we specify the filelog receiver as active receiver component for our Collector instance.

The Filelog receiver supports a number of configuration parameters, which enable you to customize its behavior. For the example, we use the following:

include—Specifies the path pattern of the files we want to ingest.
start_at—Specifies if the receiver should read from the beginning of the file or, for the most recent entries only, the end.
operators—Configures the container operator, which automatically parses each log entry.

Processor

Under processors, we specify the k8sattributes processor with the following parameters:

extract—Specifies which information should be extracted.
pod_association—Specifies how the pod information is linked to attributes.

Exporter

Under exporters, we specify the default otlphttp exporter and configure it with our Dynatrace API URL and the required authentication token, as set up and configured under Kubernetes Secrets.

Service pipeline

Under service, we assemble our receiver, processor, and exporter objects into pipelines for traces, metrics, and logs. These pipelines allow us to send OpenTelemetry signals via the Collector instance and have them automatically enriched with additional Kubernetes-specific details.