Stream Kubernetes logs with Fluent Bit

This page provides instructions for deploying and configuring Fluent Bit in your Kubernetes environment for log collection.

Prerequisites

Setup security context constraints (SCC) properly if you use OpenShift.
Helm is required. Use Helm version 3.
Egress traffic must be allowed from the namespace in which Fluent Bit is installed (dynatrace-fluent-bit), to Dynatrace.
For workload enrichment, Dynatrace Operator version 1.1.0+ is required.

Customize Fluent Bit configuration

Follow the step-by-step guide to prepare the configuration for Fluent Bit.

Copy the sample values.yaml file and open it with your preferred editor.

openShift:
  # set to true for OpenShift
  enabled: false

securityContext:
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true
  # uncomment the line below for OpenShift
  #privileged: true

rbac:
  nodeAccess: true

config:
  inputs: |

    [INPUT]
        Name tail
        Tag kube.*
        Path /var/log/containers/*.log
        DB /fluent-bit/tail/kube.db
        DB.Sync Normal
        multiline.parser cri
        Mem_Buf_Limit 15MB
        Skip_Long_Lines On

  filters: |

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser Off
        K8S-Logging.Exclude Off
        Labels Off
        Annotations On
        Use_Kubelet On
        Kubelet_Host ${NODE_IP}
        tls.verify Off
        Buffer_Size 0

    # Only include logs from pods with the annotation
    #[FILTER]
    #    Name grep
    #    Match kube.*
    #    Regex $kubernetes['annotations']['logs.dynatrace.com/ingest'] ^true$

    # Only include logs from specific namespaces, remove the whole filter section to get all logs
    #[FILTER]
    #    Name grep
    #    Match kube.*
    #    Logical_Op or
    #    Regex $kubernetes['namespace_name'] ^my-namespace-a$
    #    Regex $kubernetes['namespace_name'] ^my-namespace-b$

    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under kubernetes
        Add_prefix kubernetes.

    [FILTER]
        name nest
        match kube.*
        operation lift
        nested_under kubernetes.annotations
        add_prefix kubernetes.annotations.

    [FILTER]
        Name nest
        Match kube.*
        Operation nest
        Nest_under dt.metadata
        Wildcard kubernetes.annotations.metadata.dynatrace.com/*

    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under dt.metadata
        Remove_prefix kubernetes.annotations.metadata.dynatrace.com/

    [FILTER]
        Name modify
        Match kube.*

        # Map data to Dynatrace log format
        Rename time timestamp
        Rename log content

        Rename kubernetes.host k8s.node.name
        Rename kubernetes.namespace_name k8s.namespace.name
        Rename kubernetes.pod_id k8s.pod.uid
        Rename kubernetes.pod_name k8s.pod.name
        Rename kubernetes.container_name k8s.container.name

        Add k8s.cluster.name ${K8S_CLUSTER_NAME}
        Add k8s.cluster.uid ${K8S_CLUSTER_UID}

        # deprecated, but still in use
        Add dt.kubernetes.cluster.name ${K8S_CLUSTER_NAME}
        Add dt.kubernetes.cluster.id ${K8S_CLUSTER_UID}

        Remove_wildcard kubernetes.

  outputs: |

    # Send data to Dynatrace log ingest API
    [OUTPUT]
        Name http
        Match kube.*
        host ${DT_INGEST_HOST}
        port 443
        tls On
        tls.verify On
        uri /api/v2/logs/ingest
        format json
        allow_duplicated_headers false
        header Authorization Api-Token ${DT_INGEST_TOKEN}
        header Content-Type application/json; charset=utf-8
        json_date_key timestamp
        json_date_format iso8601
        log_response_payload false

daemonSetVolumes:
  - hostPath:
      path: /var/lib/fluent-bit/
    name: positions
  - hostPath:
      path: /var/log/containers
    name: containers
  - hostPath:
      path: /var/log/pods
    name: pods

daemonSetVolumeMounts:
  - mountPath: /fluent-bit/tail
    name: positions
  - mountPath: /var/log/containers
    name: containers
    readOnly: true
  - mountPath: /var/log/pods
    name: pods
    readOnly: true

podAnnotations:
  dynatrace.com/inject: "false"
#  Uncomment this to collect Fluent Bit Prometheus metrics
#  metrics.dynatrace.com/path: "/api/v1/metrics/prometheus"
#  metrics.dynatrace.com/port: "2020"
#  metrics.dynatrace.com/scrape: "true"

envWithTpl:
  - name: K8S_CLUSTER_UID
    value: '{{ (lookup "v1" "Namespace" "" "kube-system").metadata.uid }}'

env:
  - name: K8S_CLUSTER_NAME
    value: "{ENTER_YOUR_CLUSTER_NAME}"
  - name: DT_INGEST_HOST
    value: "{your-environment-id}.live.dynatrace.com"
  - name: DT_INGEST_TOKEN
    value: "{ENTER_YOUR_INGEST_TOKEN}"
  - name: NODE_IP
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: status.hostIP

Get a Dynatrace API token with the logs.ingest (Ingest Logs) scope for the DT_INGEST_TOKEN environment variable.
Update the K8S_CLUSTER_NAME, DT_INGEST_HOST, and DT_INGEST_TOKEN environment variables in the values.yaml file. Use the same cluster name that you have configured in Dynatrace for K8S_CLUSTER_NAME, and specify your SaaS or Managed endpoint as DT_INGEST_HOST.
optional Adapt the filter section in the values.yaml file to target specific namespaces or pods, as described in the Fluent Bit Filter section for details.
optional Be sure to remove or mask any sensitive information in the logs.
Save the file.

Install and configure Fluent Bit with Helm

Add the fluent repository to your local Helm repositories

helm repo add fluent https://fluent.github.io/helm-charts

Update the Fluent Bit repository
```
helm repo update
```

Install Fluent Bit with the prepared configuration

helm install fluent-bit fluent/fluent-bit -f values.yaml --create-namespace --namespace dynatrace-fluent-bit

View ingested logs

Ingested logs are accessible at the cluster, namespace, workload, and pod levels. They can be inspected on the object's details page by selecting Explorer in the Kubernetes app and selecting a cluster, namespace, workload, or pod from the list. The Logs tab displays logs either as a graph or a list.

Pod logs

Limitations

GKE Autopilot is not supported.
fluentbit.io/parser and fluentbit.io/exclude annotations are disabled by default.

Troubleshooting

Check that Fluent Bit pods are running

kubectl get pods -n dynatrace-fluent-bit

NAME               READY   STATUS              RESTARTS    AGE
fluent-bit-5jzlr   0/1     CrashLoopBackOff    1 (7s ago)  11s
fluent-bit-8zfr4   1/1     Running             0           38s
fluent-bit-qxjzh   1/1     Running             0           39s

If pods are in an error state, then the helm values file might contain errors. Check logs of the non-running pods for details.

kubectl logs fluent-bit-5jzlr -n dynatrace-fluent-bit

Check Fluent Bit health and metrics

Fluent Bit metrics give you insights into how the logs are being collected (fluentbit_input_*), filtered (fluentbit_filter_*) and sent to Dynatrace (fluentbit_output_*).

Find the node on which the pod you are troubleshooting is running.

kubectl get pod pod-with-logs -o wide -n dms

NAME            READY   STATUS    RESTARTS   AGE   IP           NODE                       NOMINATED NODE   READINESS GATES
pod-with-logs   1/1     Running   0          31m   10.28.2.41   some-node-782e86b8-mnoz    <none>           <none>

Find the Fluent Bit pod that runs on the same node.

kubectl get pods -o wide -n dynatrace-fluent-bit

NAME               READY   STATUS    RESTARTS   AGE   IP           NODE                       NOMINATED NODE   READINESS GATES
fluent-bit-5jzlr   1/1     Running   0          30m   10.28.3.44   some-node-782e86b8-zdb1    <none>           <none>
fluent-bit-8zfr4   1/1     Running   0          30m   10.28.4.23   some-node-782e86b8-mkjw    <none>           <none>
fluent-bit-qxjzh   1/1     Running   0          30m   10.28.2.42   some-node-782e86b8-mnoz    <none>           <none>

Set up Fluent Bit pod metrics port forwarding to your localhost.

kubectl port-forward fluent-bit-qxjzh 2020:2020 -n dynatrace-fluent-bit

Check the health endpoint.

curl http://127.0.0.1:2020/api/v1/health

ok

Examine the metrics.

fluentbit_output_proc_* metrics indicate how many logs are being ingested
fluentbit_* metrics give you more insights into what happens before that

curl http://127.0.0.1:2020/api/v2/metrics | grep fluentbit_output_proc

2024-06-11T07:05:37.257418778Z fluentbit_output_proc_records_total{name="http.0"} = 767
2024-06-11T07:05:37.257418778Z fluentbit_output_proc_bytes_total{name="http.0"} = 359630

When fluentbit_output_errors_total or fluentbit_output_retries_failed_total metrics indicate problems, a potential reason is that you have reached log monitoring limits.