Metadata enrichment of all telemetry originating from Kubernetes

7-min read

Prerequisites

Operator version 1.6.0+, ActiveGate version 1.321+, OneAgent version 1.321+

Dynatrace Operator is installed and running in your Kubernetes cluster.
A valid DynaKube is applied to your cluster.
Metadata enrichment is enabled.

Use cases

Enhance your metrics, logs, trace data, events and entities with additional information using Kubernetes annotations and labels.
Enhance your metrics, logs, trace data, events and entities with additional information using OpenTelemetry environment variables
Enriched data can be used for defining access control to users, or for solving cost allocation in DPS.
Enriched data can be used for pipeline routing, bucket segmentation, segmentation, and filtering.

Security context and cost allocation

In Dynatrace, you can set up policy boundaries for fine-grained restrictions on the data level. By default, you can use k8s.namespace.name and k8s.cluster.name, but sometimes this is not enough and you need a more fine grained way to set up your boundaries.

You might already have defined such boundaries for yourself and defined them as Kubernetes labels or annotations. This feature enables you to use these at the source for your security context in Dynatrace. If you have not done so already, we recommend to either use cluster or namespace name or set up a dedicated annotation for your Kubernetes workloads that serves as your security context.

Similarly Dynatrace provides a solution for cost allocation in DPS. You might already have the necessary data like department and product available in your existing Kubernetes labels or annotations. Even if not you might find it very convenient to setup up cost allocation as a Kubernetes annotation or label, which is what Dynatrace recommends. This feature then enables you to use these labels and annotations as the means to solving cost allocation in DPS.

The following attributes are supported:

Primary Grail Tags

To streamline tasks like bucket selection, segmentation, filtering, and problem routing, Dynatrace allows you to enrich your telemetry data using existing Kubernetes labels or annotations. These tags are made available as domain-specific fields, such as k8s.namespace.label.your_key or k8s.namespace.annotation.your_key.

We are working on automatically copying the domain-specific keys to primary_tags.<key>, enhancing usability and simplifying the tagging process. Primary Grail tags have already been added to the Semantic Dictionary.

Which data will be enriched

Data

Primary Grail Tags

Security Context

Cost Allocation

OneAgent Metrics

JMX/PMI Metrics collected via OneAgent

Planned

Service Metrics

Kubernetes platform metrics

Planned

OTLP Metrics

Planned

Metrics collected by OpenTelemetry Collector

Planned

Logs collected by OpenTelemetry Collector

Logs collected by OneAgent Log Module

Logs collected by FluentBit

Planned

Kubernetes entities

n/a

Planned for the new Smartscape

n/a

Process entities

Planned for the new Smartscape

n/a

Service entities

Planned for the new Smartscape

n/a

OneAgent Events

Kubernetes Events

Planned

Enrichment options

Depending on the specific use case, the following enrichment options are supported:

We recommend this option, as it is the only option that enriches all signals, including Kubernetes platform metrics, events, and entities—unlike environment variables or manual pod annotations. Use Kubernetes enrichment rules to leverage your existing namespace labels and annotations. The tenant configuration is applied to all Kubernetes clusters by default. However, you can override it for specific clusters if needed.

Hint: If you setup your rules before you deploy your Dynakube, you don't need to wait 45 minutes for the rules to be propagated.

Go to Kubernetes App > Namespace > Select your namespace to have an overview of your existing namespace labels.
Go to Settings > Cloud and virtualization > Kubernetes telemetry enrichment.
Select Add rule.
Select Annotation or Label in the Metadata type dropdown.
Enter the namespace annotation/label key in the Source field, following Kubernetes conventions:
For primary Grail tag enrichment, turn on Use as Primary Grail Tag.
For remapping, turn off Use as Primary Grail Tag and choose a value from the Target dropdown.
Select Save changes.
After creating or modifying rules, allow up to 45 minutes for the changes to take effect. Once this time has passed, restart your pods.
Navigate to your data and verify that the metadata is successfully enriched.

This works automatically for OneAgent and OpenTelemetry code-change scenarios.

This option is intended for scenarios where namespace labels or annotations cannot be used as a source. If both methods are present, manual annotations take precedence.

Unlike the settings-based approach, manually added pod annotations do not provide full enrichment. They will not enrich Kubernetes metrics, Kubernetes events, or entities. For Oneagent metrics and service metrics to be enriched with these attributes, they must follow the convention k8s.namespace.<label>/<annotation>.<key>: <value>.

For comprehensive enrichment, the settings-based approach is recommended.

You might create the following annotations at the pod level:

metadata:
  annotations:
    metadata.dynatrace.com/dt.security_context: sre
    metadata.dynatrace.com/dt.cost.costcenter: it_services
    metadata.dynatrace.com/dt.cost.product: fin_app
    metadata.dynatrace.com/k8s.namespace.label.domain: finance

The following attributes will enrich the data:

dt.security_context: sre
dt.cost.costcenter: it_services
dt.cost.product: fin_app
k8s.namespace.label.domain: finance

OpenTelemetry setup

For OTLP setups without OneAgent, additional steps for signal enrichment are required. This can be achieved either by modifying your code to parse metadata files provided by the operator or by using environment variables.

This option is suitable for standalone OTLP setups without OneAgent injection. For optimal results, enrich your OTLP telemetry by parsing Dynatrace metadata files and adding the metadata directly in your code, as outlined in Enrich ingested data with Dynatrace-specific dimensions. You can find code samples in our OpenTelemetry section, like here for Java. This approach provides enrichment comparable to the OneAgent case.

If modifying your code isn't feasible, you can use the OTEL_RESOURCE_ATTRIBUTES environment variable for enrichment. However, this method has limitations: configuration can be complex, and certain properties, like k8s.container.name and Primary Grail Tags, must be set as static strings.

Store DynaKube name

DYNAKUBE="dynakube" # set this to the name of your DynaKube / kubectl get dynakube -n dynatrace

Get k8s.cluster.uid

K8S_CLUSTER_UID="$(kubectl get dynakube -o jsonpath='{.status.kubeSystemUUID}' -n dynatrace $DYNAKUBE)"

Get k8s.cluster.name

K8S_CLUSTER_NAME="$(kubectl get dynakube -o jsonpath='{.status.kubernetesClusterName}' -n dynatrace $DYNAKUBE)"

Get Kubernetes entity dt.entity.kubernetes_cluster

DT_ENTITY_KUBERNETES_CLUSTER="$(kubectl get dynakube -o jsonpath='{.status.kubernetesClusterMEID}' -n dynatrace $DYNAKUBE)"

Create config map in the target namespace

kubectl create configmap dynatrace-metadata \
  --from-literal K8S_CLUSTER_UID=$K8S_CLUSTER_UID \
  --from-literal K8S_CLUSTER_NAME=$K8S_CLUSTER_NAME \
  --from-literal DT_ENTITY_KUBERNETES_CLUSTER=$DT_ENTITY_KUBERNETES_CLUSTER \
  --namespace <YOUR_NAMESPACE>

Adapt your Kubernetes pod specification by adding the following environment variables. You can include these in your Kubernetes Deployment or Pod manifest.

Primary Grail tags and k8s.container.name cannot be set via downward API. It needs to be provided as a static string.

envFrom:
- configMapRef:
    name: dynatrace-metadata
    optional: false
env:
- name: K8S_CONTAINER_NAME
  value: "" # replace with actual container name
- name: K8S_POD_NAME
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.name
- name: K8S_POD_UID
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.uid
- name: K8S_POD_NAMESPACE
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.namespace
- name: K8S_WORKLOAD_KIND
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/k8s.workload.kind'] # only works when metadata enrichment is enabled
- name: K8S_WORKLOAD_NAME
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/k8s.workload.name'] # only works when metadata enrichment is enabled
- name: K8S_NODE_NAME
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: spec.nodeName
- name: DT_SECURITY_CONTEXT # only works when automatic security context enrichment is configured
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/dt.security_context']
- name: DT_COST_PRODUCT # only works when automatic cost product enrichment is configured
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/dt.cost.product']
- name: DT_COST_COSTCENTER # only works when automatic cost center enrichment is configured
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/dt.cost.costcenter']

This example shows all our recommended attributes. Remove attributes that are not in use.

- name: OTEL_RESOURCE_ATTRIBUTES
  value: k8s.cluster.name=$(K8S_CLUSTER_NAME),k8s.cluster.uid=$(K8S_CLUSTER_UID),k8s.node.name=$(K8S_NODE_NAME),k8s.workload.name=$(K8S_WORKLOAD_NAME),k8s.workload.kind=$(K8S_WORKLOAD_KIND),k8s.pod.name=$(K8S_POD_NAME),k8s.pod.uid=$(K8S_POD_UID),k8s.namespace.name=$(K8S_POD_NAMESPACE),k8s.container.name=$(K8S_CONTAINER_NAME),dt.entity.kubernetes_cluster=$(DT_ENTITY_KUBERNETES_CLUSTER),dt.security_context=$(DT_SECURITY_CONTEXT),dt.cost.costcenter=$(DT_COST_COSTCENTER),dt.cost.product=$(DT_COST_PRODUCT)

Limitations

Limit of 20 rules per configuration scope.
Primary Grail tags do not yet enrich Kubernetes metrics or events.
After creating or modifying rules, allow up to 45 minutes for the changes to take effect. Once this time has passed, restart your pods.
Manually set metadata.dynatrace.com pod annotations take precedence.
Manually added attributes (anything other than dt.security_context, dt.cost.costcenter, or dt.cost.product) do not enrich any Kubernetes metrics or Kubernetes events.
The settings-based approach does not work in conjunction with manually applied dedicated pod annotations. Using both simultaneously may cause conflicts, leading to unexpected behavior.

Troubleshooting

Verify the rule definition

Confirm that each rule points to the correct metadata type (label vs. annotation).
Ensure the source key in the rule exactly matches the key that exists on the namespace.

Check that the source metadata really exists

Open the namespace in the Dynatrace Kubernetes app and look for the expected labels/annotations.
Alternatively, run
```
kubectl get namespace <name> -o yaml
```

and inspect the metadata.labels and metadata.annotations sections.

Validate that metadata enrichment is turned on

The feature works only if metadataEnrichment is enabled in your DynaKube configuration.
If you specify a namespaceSelector in the DynaKube, make sure it matches the namespace you are testing.

Confirm that enrichment reached the pods

Inspect any pod in the namespace:
```
kubectl get pod <pod-name> -o yaml
```
Look for annotations that start with metadata.dynatrace.com/…. Their presence means the metadata is enriched.

Examples

Rules in builtin:kubernetes.generic.metadata.enrichment

"rules":
  [
    {
      # rule #1
      "type": "Annotation",
      "source": "metadata.example.com/team",
      "target": "dt.security_context"
    },
    {
      # rule #2
      "type": "Label",
      "source": "department",
      "target": "dt.cost.costcenter"
    },
    {
      # rule #3
      "type": "Label",
      "source": "app/name",
      "target": "dt.cost.product"
    }
    {
      # rule #4
      "type": "Label",
      "source": "domain",
      "primaryGrailTag": "true"
    }
  ]

Your existing namespace labels and annotations:

metadata:
  annotations:
    metadata.example.com/team: sre
  labels:
    department: it_services
    app/name: fin_app
    domain: finance

The Operator will create pod annotations:

metadata:
  annotations:
    metadata.dynatrace.com:|
      {
        "dt.security_context": "sre",
        "dt.cost.costcenter": "it_services",
        "dt.cost.product": "fin_app",
        "k8s.namespace.label.domain": "finance"
      }

The following attributes will be enriched on the data:

dt.security_context: sre
dt.cost.costcenter: it_services
dt.cost.product: fin_app
k8s.namespace.label.domain: finance

Metadata enrichment of all telemetry originating from Kubernetes

Prerequisites

Use cases

Security context and cost allocation

Primary Grail Tags

Which data will be enriched

Enrichment options

Use settings to use existing labels and annotations (recommended)

Use dedicated Dynatrace metadata pod annotations

OpenTelemetry setup

Enrich via code changes (recommended)

Enrich via environment variable

Create a config map for cluster level attributes

Set K8s attributes via downward API on Pod

Add attributes to `OTEL_RESOURCE_ATTRIBUTES`

Limitations

Troubleshooting

Verify the rule definition

Check that the source metadata really exists

Validate that metadata enrichment is turned on

Confirm that enrichment reached the pods

Examples

Metadata enrichment of all telemetry originating from Kubernetes

Prerequisites

Use cases

Security context and cost allocation

Primary Grail Tags

Which data will be enriched

Enrichment options

Use settings to use existing labels and annotations (recommended)

Use dedicated Dynatrace metadata pod annotations

OpenTelemetry setup

Enrich via code changes (recommended)

Enrich via environment variable

Create a config map for cluster level attributes

Set K8s attributes via downward API on Pod

Add attributes to `OTEL_RESOURCE_ATTRIBUTES`

Limitations

Troubleshooting

Verify the rule definition

Check that the source metadata really exists

Validate that metadata enrichment is turned on

Confirm that enrichment reached the pods

Examples

Related topics