Stream Kubernetes logs with OneAgent Log Module

Dynatrace provides integrated Log management and analytics for your Kubernetes environments by either running the OneAgent Log module or integrating with log collectors such as Fluent Bit, Dynatrace OpenTelemetry Collector, Logstash, or Fluentd.

On this page you learn how you can use the OneAgent Log module to ingest logs from Kubernetes.

Deployment options for Kubernetes log monitoring

Dynatrace provides a flexible approach to Kubernetes observability where you can pick and choose the level of observability you need for your Kubernetes clusters. The Dynatrace Operator manages all the necessary components to get the data into Dynatrace for you. This also applies to collecting logs from Kubernetes containers. Depending on the selected observability option, the Dynatrace Operator configures and manages the Log module to work with or without a OneAgent on the node.

Prerequisites

Auto-discovery of Kubernetes container logs

OneAgent Log module automatically discovers logs written to the stdout/stderr streams through containerized applications running in pods. Under the covers, these log streams are stored as files on the Kubernetes nodes, where the OneAgent Log module can pick those files up and stream them to Dynatrace. The log source attribute for these logs in Dynatrace is set to Container Output. The attribute log.iostream defines the log stream the log entries were written to, for example, stdout or stderr.

OneAgent Log module does not discover logs written to the container filesystem (as opposed to stdout/stderr). In this case, you can use a log shipper to read the logs from the container filesystem and write them to stdout/stderr for the OneAgent Log module to pick them up.

Log enrichment with Kubernetes metadata

OneAgent Log module decorates the ingested logs with the following Kubernetes metadata: k8s.cluster.name, k8s.cluster.uid, k8s.namespace.name, k8s.workload.name, k8s.workload.kind, dt.entity.kubernetes_cluster, k8s.pod.name, k8s.pod.uid, k8s.container.name, dt.entity.kubernetes_node.

See metadata enrichment for Kubernetes to learn more.

Control log ingest with Kubernetes metadata

You can control logs from Kubernetes ingestion with log ingest rules in Dynatrace. You can configure these rules at the Kubernetes cluster level to allow cluster-specific log ingestion. The rules use matchers for Kubernetes metadata and other common log entry attributes to determine which logs are to be ingested. Standard log processing features from OneAgent, including sensitive data masking, timestamp configuration, log boundary definition, and automatic enrichment of log records, are also available and enabled here.

Use the following recommended matching attributes when configuring log ingestion from Kubernetes.

Log ingest rule hierarchy

Log ingest rules can be defined on environment scope but also on more fine-grained level like Kubernetes cluster. The matching hierarchy is as follows:

Matching occurs in a predefined hierarchy and rules are executed from top to bottom. This means that if a rule above on the list matches certain log data, then the lower ones will be omitted. Items matched in the higher-level configurations are overwritten in the lower-level configurations if they match the same log data. If no rule is matched, the file is not sent.

Consult the Configuration scopes for the three scopes of the configuration hierarchy.

Use cases

Explore the following use cases for log ingestion from Kubernetes environments using Dynatrace. By configuring log ingestion with different matchers, you can control which logs are captured in the system. The use cases below offer guidance on configuring Dynatrace to capture logs based on your specific monitoring needs, whether it's from a particular namespace, container, or other criteria.

For detailed instructions on how to configure log ingestion, see Log ingest rules.

Ingest all logs from a specific namespace

You can now analyze the logs in the log viewer or notebooks after fitering the proper namespace. You can also find the logs in context in the Kubernetes application by selecting the Logs tab.

Ingest logs from a specific namespace and container

You can now analyze the logs in the log viewer or notebooks after fitering the proper namespace and container. You can also find the logs in context in the Kubernetes application by selecting the Logs tab.

Ingest all Kubernetes logs excluding specific namespaces

Ingest error logs from a given Kubernetes cluster and namespace

REST API

You can use the Settings API to manage your log ingest rules:

To check the current schema version for log ingest rules, list all available schemas and look for the builtin:logmonitoring.log-storage-settings schema identifier.

Log ingest rule objects can be configured for the following scopes:

To create a log ingest rule using the API:

Examples

The examples that follow show the results of various combinations of rules and matchers.

Example 1: Ingest all logs from a specific namespace

This task requires setting one rule with one matcher.

[{
  "schemaId": "builtin:logmonitoring.log-storage-settings",
  "scope": "tenant",
  "value": {
    "enabled": true,
    "config-item-title": "All logs from kube-system namespace",
    "send-to-storage": true,
    "matchers": [
      {
        "attribute": "k8s.namespace.name",
        "operator": "MATCHES",
        "values": [
          "kube-system"
        ]
      }
    ]
  }
}]

Example 2: Send logs from a specific namespace and containers with content containing 'ERROR'

This task requires setting one rule with three matchers.

[{
  "schemaId": "builtin:logmonitoring.log-storage-settings",
  "scope": "tenant",
  "value": {
    "enabled": true,
    "config-item-title": "Error logs from kube-proxy and kube-dns containers",
    "send-to-storage": true,
    "matchers": [
      {
        "attribute": "k8s.namespace.name",
        "operator": "MATCHES",
        "values": [
          "kube-system"
        ]
      },
      {
        "attribute": "k8s.container.name",
        "operator": "MATCHES",
        "values": [
          "kubedns",
          "kube-proxy"
        ]
      },
      {
        "attribute": "log.content",
        "operator": "MATCHES",
        "values": [
          "*ERROR*"
        ]
      }
    ]
  }
}]

Example 3: Ingest all Kubernetes logs excluding specific namespaces on a specific host group scope

This task requires setting two rules.

[{
  "schemaId": "builtin:logmonitoring.log-storage-settings",
  "scope": "HOST_GROUP-1D91E46493049D07",
  "value": {
    "enabled": true,
    "config-item-title": "Exclude logs from kube-system namespace",
    "send-to-storage": false,
    "matchers": [
      {
        "attribute": "k8s.namespace.name",
        "operator": "MATCHES",
        "values": [
          "kube-system"
        ]
      }
    ]
  }
},{
  "schemaId": "builtin:logmonitoring.log-storage-settings",
  "scope": "HOST_GROUP-1D91E46493049D07",
  "value": {
    "enabled": true,
    "config-item-title": "All Kubernetes logs",
    "send-to-storage": true,
    "matchers": [
      {
        "attribute": "k8s.namespace.name",
        "operator": "MATCHES",
        "values": [
          "*"
        ]
      }
    ]
  }
}]

Frequently asked questions

For more ingest related FAQs, please consult the Log ingest rules page.