Ingest pod logs

The following configuration example shows how you configure a Collector instance to fetch logs from all Kubernetes pods. It also shows how to enrich the logs with Kubernetes metadata in order to automatically link OpenTelemetry services to pods and attach the logs to the Kubernetes services and pods.

Prerequisites

Demo configuration

Kubernetes configuration

This sample configuration uses the same Kubernetes enrichment approach as the use case at Enrich from Kubernetes.

In addition to the Collector configuration, be sure to also update your Kubernetes configuration for the following components:

  • Service account: Specify the same service account name used in the RBAC file (see entries for Helm, Operator)
  • Mounted volumes: Specify the file system volumes where your Kubernetes host keeps the relevant log files (see entries for Helm, Operator)
  • Mount paths: Specify the file system paths, to which the previously configured volumes should be mounted within the container (see entries for Helm, Operator)
receivers:
  # configure the filelog receiver to access the pod and container logs
  # from the mounted volumes
  filelog:
    include:
      - /var/log/pods/*/*/*.log
      - /var/log/containers/*.log
    include_file_name: false
    include_file_path: true
    start_at: end
    operators:
      - id: container-parser
        type: container


processors:
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.cluster.uid
    pod_association:
      - sources:
        - from: resource_attribute
          name: k8s.pod.name
        - from: resource_attribute
          name: k8s.namespace.name
      - sources:
        - from: resource_attribute
          name: k8s.pod.ip
      - sources:
        - from: resource_attribute
          name: k8s.pod.uid
      - sources:
        - from: connection


  transform:
    error_mode: ignore
    trace_statements:
      - context: resource
        statements: &k8s-statements
          - set(attributes["dt.kubernetes.workload.kind"], "statefulset") where IsString(attributes["k8s.statefulset.name"])
          - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.statefulset.name"]) where IsString(attributes["k8s.statefulset.name"])
          - set(attributes["dt.kubernetes.workload.kind"], "deployment") where IsString(attributes["k8s.deployment.name"])
          - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.deployment.name"]) where IsString(attributes["k8s.deployment.name"])
          - set(attributes["dt.kubernetes.workload.kind"], "daemonset") where IsString(attributes["k8s.daemonset.name"])
          - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.daemonset.name"]) where IsString(attributes["k8s.daemonset.name"])
          - set(attributes["dt.kubernetes.cluster.id"], attributes["k8s.cluster.uid"]) where IsString(attributes["k8s.cluster.uid"])
    log_statements:
      - context: resource
        statements: &k8s-statements


exporters:
  otlphttp:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"


service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [k8sattributes,transform]
      exporters: [otlphttp]
Configuration validation

Validate your settings to avoid any configuration issues.

Kubernetes configuration

Configure the following rbac.yaml file with your Kubernetes instance, to allow the Collector to use the Kubernetes API with the service-account authentication type.

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: collector
  name: collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: collector
  labels:
    app: collector
rules:
  - apiGroups:
      - ''
    resources:
      - 'pods'
      - 'namespaces'
    verbs:
      - 'get'
      - 'watch'
      - 'list'
  - apiGroups:
      - 'apps'
    resources:
      - 'replicasets'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
  - apiGroups:
      - 'extensions'
    resources:
      - 'replicasets'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: collector
  labels:
    app: collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: collector
subjects:
  - kind: ServiceAccount
    name: collector
    namespace: default

If you are running the Collector on GKE Autopilot, you need the following adjustments in the configuration:

  • Deployment mode: On GKE Autopilot, the Collector needs to be deployed as a DaemonSet to be able to access the pod log files on the host. For details on deploying the Collector as a DaemonSet, see Deployment instructions.
  • Volume mount: GKE requires volume mounts to /var/log/pods to be read-only, otherwise the collector will not be able to access the log files within that directory.

Below is an example configuration for a Collector DaemonSet with the required volume mounts for gathering the pod logs:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dynatrace-otel-collector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dynatrace-otel-collector
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: dynatrace-otel-collector
        app.kubernetes.io/name: dynatrace-otel-collector
    spec:
      serviceAccountName: collector
      tolerations:
      # these tolerations are to have the daemonset runnable on control plane nodes
      # remove them if your control plane nodes should not run pods
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
        - args: ["--config", "/conf/otel-collector-config.yaml"]
          env:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
          image: ghcr.io/dynatrace/dynatrace-otel-collector/dynatrace-otel-collector:v0.19.0
          name: otel-collector
          resources:
            limits:
              memory: 512Mi
          volumeMounts:
            - name: dynatrace-otel-collector-config
              mountPath: /conf
              # read-only volumeMount for the directory containing the pod logs
            - name: logs
              mountPath: /var/log
              readOnly: true
      volumes:
        - configMap:
            name: dynatrace-otel-collector-config
            items:
              - key: otel-collector-config
                path: otel-collector-config.yaml
          name: dynatrace-otel-collector-config
          # hostPath volume containing the pod logs of the respective node the collector instance is running on
        - hostPath:
            path: /var/log
          name: logs

In the Collector configuration defined in otel-collector-config.yaml, only the /var/log/pods directory is required to be read by the filelog receiver, and the /var/log/containers directory can be removed from the list of observed directories, as it is not used by GKE. This results in the following change of the filelog receiver in the otel-collector-config.yaml:

...
receivers:
  filelog:
    include:
      # only include the /var/log/pods directory here, as the /var/log/containers directory is not used on GKE Autopilot
      - /var/log/pods/*/*/*.log
    include_file_name: false
    include_file_path: true
    start_at: end
    operators:
      - id: container-parser
        type: container
...

Components

For our configuration, we configured the following components.

Receivers

Under receivers, we specify the filelog receiver as active receiver component for our Collector instance.

The Filelog receiver supports a number of configuration parameters, which enable you to customize its behavior. For the example, we use the following:

  • include—Specifies the path pattern of the files we want to ingest.
  • start_at—Specifies if the receiver should read from the beginning of the file or, for the most recent entries only, the end.
  • operators—Configures the container operator, which automatically parses each log entry.

Processors

Under processors, we specify the k8sattributes processor with the following parameters:

  • extract—Specifies which information should be extracted.
  • pod_association—Specifies how the pod information is linked to attributes.

Exporters

Under exporters, we specify the default otlphttp exporter and configure it with our Dynatrace API URL and the required authentication token, as set up and configured under Kubernetes Secrets.

Service pipelines

Under service, we assemble our receiver, processor, and exporter objects into pipelines for traces, metrics, and logs. These pipelines allow us to send OpenTelemetry signals via the Collector instance and have them automatically enriched with additional Kubernetes-specific details.

Related topics