Ingest pod logs

The following configuration example shows how you configure a Collector instance to fetch logs from all Kubernetes pods. It also shows how to enrich the logs with Kubernetes metadata in order to automatically link OpenTelemetry services to pods and attach the logs to the Kubernetes services and pods.

Prerequisites

Demo configuration

Kubernetes configuration

This sample configuration uses the same Kubernetes enrichment approach as the use case at Enrich from Kubernetes.

In addition to the Collector configuration, be sure to also update your Kubernetes configuration for the following components:

  • Service account: Specify the same service account name used in the RBAC file (see entries for Helm, Operator)
  • Mounted volumes: Specify the file system volumes where your Kubernetes host keeps the relevant log files (see entries for Helm, Operator)
  • Mount paths: Specify the file system paths, to which the previously configured volumes should be mounted within the container (see entries for Helm, Operator)
receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
      - /var/log/containers/*.log
    include_file_name: false
    include_file_path: true
    start_at: end
    operators:
      - id: container-parser
        type: container


processors:
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.cluster.uid
    pod_association:
      - sources:
        - from: resource_attribute
          name: k8s.pod.name
        - from: resource_attribute
          name: k8s.namespace.name
      - sources:
        - from: resource_attribute
          name: k8s.pod.ip
      - sources:
        - from: resource_attribute
          name: k8s.pod.uid
      - sources:
        - from: connection


exporters:
  otlphttp:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"


service:
  pipelines:
    traces: null
    metrics: null
    logs:
      receivers: [filelog]
      processors: [k8sattributes]
      exporters: [otlphttp]
Configuration validation

Validate your settings to avoid any configuration issues.

Kubernetes configuration

Configure the following rbac.yaml file with your Kubernetes instance, to allow the Collector to use the Kubernetes API with the service-account authentication type.

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: collector
  name: collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: collector
  labels:
    app: collector
rules:
  - apiGroups:
      - ''
    resources:
      - 'pods'
      - 'namespaces'
    verbs:
      - 'get'
      - 'watch'
      - 'list'
  - apiGroups:
      - 'apps'
    resources:
      - 'replicasets'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
  - apiGroups:
      - 'extensions'
    resources:
      - 'replicasets'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: collector
  labels:
    app: collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: collector
subjects:
  - kind: ServiceAccount
    name: collector
    namespace: default

Components

For our configuration, we configured the following components.

Receivers

Under receivers, we specify the filelog receiver as active receiver component for our Collector instance.

The Filelog receiver supports a number of configuration parameters, which enable you to customize its behavior. For the example, we use the following:

  • include—Specifies the path pattern of the files we want to ingest.
  • start_at—Specifies if the receiver should read from the beginning of the file or, for the most recent entries only, the end.
  • operators—Configures the container operator, which automatically parses each log entry.

Processors

Under processors, we specify the k8sattributes processor with the following parameters:

  • extract—Specifies which information should be extracted.
  • pod_association—Specifies how the pod information is linked to attributes.

Exporters

Under exporters, we specify the default otlphttp exporter and configure it with our Dynatrace API URL and the required authentication token, as set up and configured under Kubernetes Secrets.

Service pipelines

Under service, we assemble our receiver, processor, and exporter objects into pipelines for traces, metrics, and logs. These pipelines allow us to send OpenTelemetry signals via the Collector instance and have them automatically enriched with additional Kubernetes-specific details.

Related topics