Try it free

Enrich Kubernetes telemetry with primary Grail fields and tags

  • Latest Dynatrace
  • How-to guide
  • 8-min read

Dynatrace Operator version 1.10+ OneAgent version 1.333+ ActiveGate version 1.343+

This page describes the new tagging model for Latest Dynatrace with some capabilities still rolling out.

  • Coming from classic auto-tagging? See Classic vs. latest to understand how your existing setup maps to the new model.
  • Already using Kubernetes telemetry enrichment? Follow the migration steps to adopt the new tagging and enrichment capabilities.

Dynatrace enriches telemetry from Kubernetes workloads with primary Grail fields and tags. You can use the same metadata for segments, pipeline routing, bucket assignment, Grail permissions, and cost allocation, consistently across logs, metrics, spans, events, and Smartscape entities.

If you operate Kubernetes workloads as an SRE or application owner, this page shows you how to make the labels and annotations you already maintain work for filtering, routing, and access control in Dynatrace. Data arrives in Dynatrace already tagged, with no post-processing rules required. The same metadata appears on every signal the workload emits.

For general guidance on primary Grail fields and tags, see Primary tags.

Enrichment guidance

Choose one of the options below. Try each in order and stop at the first one that covers your use case.

Prerequisites

All enrichment options require metadataEnrichment to be enabled in your DynaKube:

spec:
metadataEnrichment:
enabled: true

By default, enrichment applies to all namespaces.1 If you configure a namespaceSelector, only matching namespaces receive enrichment. Excluded namespaces won't have complete primary Grail fields or tags populated on their telemetry. For complete coverage, prefer omitting the selector or configuring it to match all monitored namespaces.

1

Except kube-*, openshift-*, and the Dynatrace Operator namespace. See Configure monitoring for namespaces and pods for details.

1. Out-of-the-box primary Grail fields

Dynatrace automatically populates k8s.cluster.name and k8s.namespace.name on all telemetry from Kubernetes workloads, with no configuration required. If filtering, routing, and access control at the cluster or namespace level cover your use case, no additional setup is needed.

2. Promote existing namespace labels or annotations

If your namespaces already carry labels or annotations that represent the context you want, such as team ownership, environment, cost center, security boundary, use central configuration to promote them to primary Grail tags or fields. No changes to workload manifests are required.

Each central configuration rule selects a source, for example, a namespace label, a namespace annotation, or a static literal value, and a target, such as a primary field or a primary tag.

How the source value reaches telemetry depends on the rule type:

  • Custom: The configured source is treated as a static literal and written directly to the target primary field or primary tag.
  • Namespace label or annotation: The source is treated as a key, and the corresponding value is written to telemetry as a Kubernetes tag or primary Grail tag, for example, k8s.namespace.label.<key> or primary_tags.<key>. Primary tags are written on the edge, without requiring a separate OpenPipeline processing step. You can configure two rules for the same source if you want to populate both Kubernetes and primary Grail tag targets with the same value.

Configuration changes to central rules can take up to 15 minutes to take effect. You might need to restart affected pods. Rules created before you deploy your DynaKube are picked up immediately.

To set up central configuration

  1. In Dynatrace, go to Settings and search for Ingest Enrichment Config.
  2. Select New rule.
  3. Set Rule type to the appropriate Kubernetes source, for example Kubernetes namespace label or Kubernetes namespace annotation.
  4. Enter the Key. A label or annotation key from your namespace.
  5. Select the Dimension to write the value to.
  6. Select Create.

For a full reference of form fields and available dimensions, see Configure central enrichment rules.

3. Enrich with dedicated metadata.dynatrace.com annotations

If your namespaces don't carry suitable existing labels or annotations, or if you need pod-level granularity that central configuration can't provide, add dedicated metadata.dynatrace.com/primary_tags.<key>:<value> annotations directly to your Kubernetes manifests to set primary Grail tags. The same convention also accepts the supported primary fields dt.security_context, dt.cost.costcenter, and dt.cost.product. Any other metadata.dynatrace.com/<key> annotation is ignored for enrichment on signals.

Pod or namespace annotation example
metadata:
annotations:
# Primary tags:
metadata.dynatrace.com/primary_tags.team: payments
metadata.dynatrace.com/primary_tags.environment: production
# Supported primary fields:
metadata.dynatrace.com/dt.security_context: confidential
metadata.dynatrace.com/dt.cost.costcenter: it_services
metadata.dynatrace.com/dt.cost.product: fin_app

You can place these annotations at two scopes:

  • Namespace-level: Annotations on a Namespace apply to every workload and pod inside that namespace. Use this when the team, environment, or security context is defined at the namespace boundary and is shared by everything that runs there.
  • Pod-level: Annotations on a Pod (typically through the pod template of a Deployment, StatefulSet, or DaemonSet) apply only to that pod. Use this when you need to override the namespace defaults for a specific workload, or when different workloads in the same namespace need different metadata.

When the same key is set at both scopes, the pod-level value wins for that pod.

4. Configure attributes in the DynaKube

You can define additional resource attributes directly in your DynaKube. Dynatrace Operator propagates these attributes to all telemetry signals without requiring you to modify namespaces, workloads, or pods. This is useful when you want to attach a fixed tag to all signals from a cluster, such as a team identifier, at the DynaKube level.

Three DynaKube sections control which signals receive which attributes:

FieldScope

.spec.resourceAttributes

Applied to all signals: OneAgent injection, OTLP exporter injection, standalone log monitoring, and ActiveGate.

.spec.oneAgent.<mode>.additionalResourceAttributes

Applied to OneAgent signals only. Takes precedence over duplicate keys in .spec.resourceAttributes.

.spec.otlpExporterConfiguration.additionalResourceAttributes

Applied to OTLP telemetry of configured exporters only. Takes precedence over duplicate keys in .spec.resourceAttributes.

DynaKube snippet showcasing attribute configuration

Attribute keys prefixed with primary_tags.* become primary Grail tags. The primary fields dt.security_context, dt.cost.costcenter, and dt.cost.product are also supported.

apiVersion: dynatrace.com/v1beta6
kind: DynaKube
metadata:
name: dynakube
namespace: dynatrace
spec:
apiUrl: https://<environment-id>.live.dynatrace.com/api
resourceAttributes:
primary_tags.team: "alpha"
oneAgent:
cloudNativeFullStack:
additionalResourceAttributes:
aws.account.id: "000000000000"
primary_tags.team: "charly"
namespaceSelector:
matchLabels:
my.app.com/oneagent: "true"
otlpExporterConfiguration:
additionalResourceAttributes:
aws.account.id: "000000000000"
primary_tags.team: "charly"
namespaceSelector:
matchLabels:
my.app.com/otel: "true"
signals:
metrics: {}
traces: {}
logs: {}

Dynatrace Operator propagates the merged attributes as follows:

  • OneAgent injection: Dynatrace Operator writes attributes to dt_node_metadata.properties for OneAgents, and to dt_metadata.properties, dt_metadata.json, and the metadata.dynatrace.com pod annotation for OneAgent injected pods.
  • OTLP exporter injection: Dynatrace Operator appends attributes to OTEL_RESOURCE_ATTRIBUTES and the metadata.dynatrace.com pod annotation for injected pods.
  • Standalone log monitoring and ActiveGate: Dynatrace Operator applies .spec.resourceAttributes directly. Mode-specific additionalResourceAttributes does not affect these components.
Conflicting keys with concurrent OneAgent and OTLP injection

When both OneAgent injection and OTLP exporter injection are active on the same pod, both write to the shared metadata.dynatrace.com JSON annotation. Having the same key defined in both sections will result in undefined behavior. You must use distinct keys across OneAgent and OTLP additionalResourceAttributes when both are active on the same pod to avoid unpredictable behavior.

Attribute key sanitization

For pod injection use cases (OneAgent injection and OTLP exporter injection), Dynatrace Operator propagates attributes as Kubernetes pod annotations in the form metadata.dynatrace.com/<key>. Kubernetes annotation key suffixes must consist of valid DNS label characters, so Dynatrace Operator sanitizes attribute keys by replacing any invalid characters before writing them to annotations.

Dynatrace Operator validates attribute keys and reports the following:

  • Warning: The key contains characters Dynatrace Operator replaces during sanitization - Dynatrace Operator renames the key but still writes the annotation.
  • Error: The sanitized key is an empty string - Dynatrace Operator drops the key.
  • Error: Two keys produce the same sanitized value - the result is a collision.
  • Error: The sanitized key exceeds 63 characters - this violates the Kubernetes annotation name-segment limit for metadata.dynatrace.com/<key>.

For the full parameter reference, see the DynaKube API reference.

5. OpenPipeline (last resort)

If none of these options apply, derive or assign primary tags at ingest with OpenPipeline and the processing stage. Use this as a fallback for environments where source-side or central enrichment isn't possible.

Precedence

When the same key is set at multiple levels, the most specific definition wins:

  1. Pod-level annotation (most specific)
  2. Namespace-level annotation
  3. DynaKube resource attributes
  4. Central configuration rule (least specific)

Coverage by signal type

The following table shows where primary tags and the special fields (dt.security_context, dt.cost.costcenter, dt.cost.product) land, by enrichment option.

SignalCentral configurationDedicated annotations (pod or namespace)

Distributed traces powered by Grail 1

OneAgent metrics 1

JMX/PMI metrics collected via OneAgent 1

Service metrics 1

Kubernetes platform metrics

Kubernetes events

Smartscape Kubernetes entities

Prometheus metrics scraping

Logs collected by OneAgent log module 1

Logs collected by FluentBit

Spans, metrics, and logs collected by the OpenTelemetry Collector

OneAgent events 1

1

OneAgent-based enrichment requires the Cloud Native Full Stack or Application Monitoring deployment mode. Classic Full Stack is not supported.

Query enriched data in Grail

Primary Grail fields and tags appear as top-level fields and can be queried with DQL.

Filter logs by namespace and team
fetch logs
| filter k8s.namespace.name == "checkout" AND primary_tags.team == "payments"
Cost allocation aggregation
fetch bizevents
| filter dt.cost.costcenter == "it_services"
| summarize sum(value), by: {dt.cost.product}
Security context filtering
fetch logs
| filter dt.security_context == "confidential"
| filter k8s.cluster.name == "prod-eu-1"

Central configuration setup checks

The following checks apply to the central configuration setup described in Promote existing namespace labels or annotations.

  • Verify the rule definition. Confirm that each rule points to the correct metadata type Label, Annotation, or Literal, and that the source key exactly matches the key on the namespace.
  • Check that the source metadata exists. Open the namespace in the Kubernetes app and look for the expected labels and annotations, or run kubectl get namespace <name> -o yaml and inspect the metadata.labels and metadata.annotations sections.
  • Validate that metadata enrichment is turned on. The feature works only if metadataEnrichment is allowed in your DynaKube. If you specify a namespaceSelector, make sure it matches the namespace you're testing.
  • Confirm that enrichment reached the pods. Run kubectl get pod <pod-name> -o yaml on a pod in the namespace and look for annotations starting with metadata.dynatrace.com/. Their presence means the metadata reached the pod.

Limitations

  • Dynatrace Classic platform is not supported.

Related topics

  • Primary Grail fields and tags
  • Organize your data with primary Grail fields and tags
  • Enrich OneAgent telemetry with primary Grail fields and tags
  • Set up Grail permissions for telemetry from Kubernetes and Kubernetes workloads
  • Set up Cost Allocation for Kubernetes deployments
  • Global field reference
  • Primary Grail fields
  • Plan your tagging strategy
  • Best practices for enriching primary Grail fields and tags
Related tags
Dynatrace Platform