Stream Kubernetes logs with Fluent Bit
This page provides instructions for deploying and configuring Fluent Bit in your Kubernetes environment for log collection.
Prerequisites
- Setup security context constraints (SCC) properly if you use OpenShift.
- Helm is required. Use Helm version 3.
- Egress traffic must be allowed from the namespace in which Fluent Bit is installed (
dynatrace-fluent-bit
), to Dynatrace. - For workload enrichment, Dynatrace Operator version 1.1.0+ is required.
Customize Fluent Bit configuration
Follow the step-by-step guide to prepare the configuration for Fluent Bit.
-
Copy the sample
values.yaml
file and open it with your preferred editor.openShift:enabled: falsesecurityContext:capabilities:drop:- ALLreadOnlyRootFilesystem: truerbac:nodeAccess: trueconfig:inputs: |[INPUT]Name tailTag kube.*Path /var/log/containers/*.logDB /fluent-bit/tail/kube.dbDB.Sync Normalmultiline.parser criMem_Buf_Limit 15MBSkip_Long_Lines Onfilters: |[FILTER]Name kubernetesMatch kube.*Merge_Log OnKeep_Log OffK8S-Logging.Parser OffK8S-Logging.Exclude OffLabels OffAnnotations OnUse_Kubelet OnKubelet_Host ${NODE_IP}tls.verify OffBuffer_Size 0# Only include logs from pods with the annotation#[FILTER]# Name grep# Match kube.*# Regex $kubernetes['annotations']['logs.dynatrace.com/ingest'] ^true$# Only include logs from specific namespaces, remove the whole filter section to get all logs#[FILTER]# Name grep# Match kube.*# Logical_Op or# Regex $kubernetes['namespace_name'] ^my-namespace-a$# Regex $kubernetes['namespace_name'] ^my-namespace-b$[FILTER]Name nestMatch kube.*Operation liftNested_under kubernetesAdd_prefix kubernetes.[FILTER]name nestmatch kube.*operation liftnested_under kubernetes.annotationsadd_prefix kubernetes.annotations.[FILTER]Name nestMatch kube.*Operation nestNest_under dt.metadataWildcard kubernetes.annotations.metadata.dynatrace.com/*[FILTER]Name nestMatch kube.*Operation liftNested_under dt.metadataRemove_prefix kubernetes.annotations.metadata.dynatrace.com/[FILTER]Name modifyMatch kube.*# Map data to Dynatrace log formatRename time timestampRename log contentRename kubernetes.host k8s.node.nameRename kubernetes.namespace_name k8s.namespace.nameRename kubernetes.pod_id k8s.pod.uidRename kubernetes.pod_name k8s.pod.nameRename kubernetes.container_name k8s.container.nameAdd k8s.cluster.name ${K8S_CLUSTER_NAME}Add k8s.cluster.uid ${K8S_CLUSTER_UID}# deprecated, but still in useAdd dt.kubernetes.cluster.name ${K8S_CLUSTER_NAME}Add dt.kubernetes.cluster.id ${K8S_CLUSTER_UID}Remove_wildcard kubernetes.outputs: |# Send data to Dynatrace log ingest API[OUTPUT]Name httpMatch kube.*host ${DT_INGEST_HOST}port 443tls Ontls.verify Onuri /api/v2/logs/ingestformat json_streamallow_duplicated_headers falseheader Authorization Api-Token ${DT_INGEST_TOKEN}header Content-Type application/json; charset=utf-8json_date_key timestampjson_date_format iso8601log_response_payload falsedaemonSetVolumes:- hostPath:path: /var/lib/fluent-bit/name: positions- hostPath:path: /var/log/containersname: containers- hostPath:path: /var/log/podsname: podsdaemonSetVolumeMounts:- mountPath: /fluent-bit/tailname: positions- mountPath: /var/log/containersname: containersreadOnly: true- mountPath: /var/log/podsname: podsreadOnly: truepodAnnotations:dynatrace.com/inject: "false"# Uncomment this to collect Fluent Bit Prometheus metrics# metrics.dynatrace.com/path: "/api/v1/metrics/prometheus"# metrics.dynatrace.com/port: "2020"# metrics.dynatrace.com/scrape: "true"envWithTpl:- name: K8S_CLUSTER_UIDvalue: '{{ (lookup "v1" "Namespace" "" "kube-system").metadata.uid }}'env:- name: K8S_CLUSTER_NAMEvalue: "{ENTER_YOUR_CLUSTER_NAME}"- name: DT_INGEST_HOSTvalue: "{your-environment-id}.live.dynatrace.com"- name: DT_INGEST_TOKENvalue: "{ENTER_YOUR_INGEST_TOKEN}"- name: NODE_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.hostIP -
Get a Dynatrace API token with the
logs.ingest
(Ingest Logs) scope for theDT_INGEST_TOKEN
environment variable. -
Update the
K8S_CLUSTER_NAME
,DT_INGEST_HOST
, andDT_INGEST_TOKEN
environment variables in thevalues.yaml
file. Use the same cluster name that you have configured in Dynatrace forK8S_CLUSTER_NAME
, and specify your SaaS or Managed endpoint asDT_INGEST_HOST
. -
optional Adapt the filter section in the
values.yaml
file to target specific namespaces or pods, as described in the Fluent Bit Filter section for details. -
optional Be sure to remove or mask any sensitive information in the logs.
-
Save the file.
Install and configure Fluent Bit with Helm
-
Add the fluent repository to your local Helm repositories
helm repo add fluent https://fluent.github.io/helm-charts -
Install Fluent Bit with the prepared configuration
helm install fluent-bit fluent/fluent-bit -f values.yaml --create-namespace --namespace dynatrace-fluent-bit
Install and configure Fluent Bit with Kubernetes app
Latest Dynatrace
Go to Kubernetes and select Add cluster in the upper-right corner.
Log analytics with Fluent Bit
If the selected distribution is not GKE Autopilot, Log analytics with Fluent Bit will be enabled by default.
You can restrict log monitoring to specific resources if desired.
-
Namespaces input field accepts multiple comma-separated names, which must conform to RFC 1123 DNS labels.
-
Pod Annotation input requires a single, valid key=value pair. If specified, log monitoring will only be applied to pods with the specified annotation.
If both filters are set, logs will only be ingested for pods in the specified namespaces that also have the specified pod annotation.
Configure cluster
In this step, Cluster Name input is required. It must:
- Use only lowercase alphanumeric characters and hyphens (
-
) Start with an alphabetic character and end with an alphanumeric character
Be no longer than 63 characters.
Install Fluent Bit
This step is only visible if Log analytics with Fluent Bit is enabled.
- Generate a token or enter an existing token with
Ingest logs
scope enabled. - Download the
fluent-bit-values.yaml
file. The download button is enabled when at least Cluster name and Fluent Bit token are set. The filters are optional. Execute the shell commands to complete the installation.
Uninstall Fluent Bit
Uninstall Fluent Bit from your Kubernetes environment using the following command:
helm uninstall fluent-bit
View ingested logs
Ingested logs are accessible at the cluster, namespace, workload, and pod levels. They can be inspected on the object's details page by selecting Explorer in the Kubernetes app and selecting a cluster, namespace, workload, or pod from the list. The Logs tab displays logs either as a graph or a list.
Limitations
GKE Autopilot is not supported.
fluentbit.io/parser
andfluentbit.io/exclude
annotations are disabled by default.