Try it free

Enrich Kubernetes telemetry with primary Grail fields and tags

  • Latest Dynatrace
  • How-to guide
  • 8-min read

Dynatrace Operator version 1.10+ OneAgent version 1.333+ ActiveGate version 1.343+

If you're currently using Dynatrace Classic auto-tagging, see Dynatrace Classic versus Latest Dynatrace to understand how your existing setup maps to the new model, or go straight to the upgrade guide for step-by-step instructions.

Dynatrace enriches telemetry from Kubernetes workloads with primary Grail fields and tags. You can use the same metadata for segments, pipeline routing, bucket assignment, Grail permissions, and cost allocation, consistently across logs, metrics, spans, events, and Smartscape entities.

If you operate Kubernetes workloads as an SRE or application owner, this page shows you how to make the labels and annotations you already maintain work for filtering, routing, and access control in Dynatrace. Data arrives in Dynatrace already tagged, with no post-processing rules required. The same metadata appears on every signal the workload emits.

For general guidance on primary Grail fields and tags, see Primary tags.

Enrichment guidance

Choose one of the options below. Try each in order and stop at the first one that covers your use case.

Prerequisites

All enrichment options require metadataEnrichment to be enabled in your DynaKube:

spec:
metadataEnrichment:
enabled: true

By default, enrichment applies to all namespaces (except kube-*, openshift-*, and the Dynatrace Operator namespace—for details, see Configure monitoring for namespaces and pods). If you configure a namespaceSelector, only matching namespaces receive enrichment. Excluded namespaces won't have complete primary Grail fields or tags populated on their telemetry. For complete coverage, prefer omitting the selector or configuring it to match all monitored namespaces.

1. Out-of-the-box primary Grail fields

Dynatrace automatically populates k8s.cluster.name and k8s.namespace.name on all telemetry from Kubernetes workloads, with no configuration required. If filtering, routing, and access control at the cluster or namespace level cover your use case, no additional setup is needed.

2. Promote existing Kubernetes annotations and labels

If your namespaces, workloads, or pods already carry labels or annotations that represent the context you want, such as team ownership, environment, cost center, security boundary, use central configuration to promote them to primary Grail tags or fields. No changes to workload manifests are required.

Each central configuration rule selects a source, for example, a namespace label, a workload annotation, a pod label, or a static literal value, and a target, such as a primary field or a primary tag.

How the source value reaches telemetry depends on the rule type:

  • Custom: The configured source is treated as a static literal and written directly to the target primary field or primary tag.
  • Namespace, workload, or pod label or annotation: The source is treated as a key, and the corresponding value is written to telemetry as a Kubernetes tag or primary Grail tag, for example, k8s.namespace.label.<key>, k8s.workload.annotation.<key>, or primary_tags.<key>. Primary tags are written on the edge, without requiring a separate OpenPipeline processing step. You can configure two rules for the same source if you want to populate both Kubernetes and primary Grail tag targets with the same value.
    • If a rule maps a label or annotation directly to the Primary tag target, no OpenPipeline primary Grail tag step is needed.
    • If a rule forwards the label or annotation with the Kubernetes tag target, you need an OpenPipeline primary Grail tag rule to turn the forwarded value into a primary_tags.<key> tag. See Promote forwarded Kubernetes tags to primary tags in OpenPipeline.

Configuration changes to central rules can take up to 15 minutes to take effect. You might need to restart affected pods. Rules created before you deploy your DynaKube are picked up immediately.

To set up central configuration

  1. In Dynatrace, go to Settings and search for Ingest Enrichment Config.

  2. Select New rule.

  3. Set Rule type to the appropriate Kubernetes source, for example Kubernetes namespace label, Kubernetes workload annotation, or Kubernetes pod label.

    Rule types and targets have version requirements:

    • Namespace rule types require Dynatrace Operator version 1.10+ ActiveGate version 1.343+.
    • Workload and pod rule types require Dynatrace Operator version 1.11+ ActiveGate version 1.347+.
    • The Primary tag target requires Dynatrace Operator version 1.10+ (1.11+ for workload and pod rule types), Dynatrace version 1.348+, and ActiveGate version 1.345+. In earlier versions, a rule can't write primary tags, so no primary_tags.<key> field appears on your telemetry. Select the Kubernetes tag target instead and promote the forwarded tag to a primary tag in OpenPipeline.
  4. Enter the Key. A label or annotation key from your Kubernetes resource.

  5. Select the Target to write the value to.

  6. Select Create.

For a full reference of form fields and available targets, see Configure central enrichment rules.

Promote forwarded Kubernetes tags to primary tags in OpenPipeline

You need this step only if a central configuration rule forwards a label or annotation with the Kubernetes tag target. This applies if your environment runs a version earlier than the versions required for the Primary tag target, or if you want to combine Kubernetes tags with tags from other sources in a single rule.

If a rule maps a label or annotation directly to the Primary tag target, skip this step.

Forwarded Kubernetes tags arrive in OpenPipeline as, for example, k8s.namespace.label.<key>, k8s.workload.annotation.<key>, or k8s.pod.label.<key>, which are not primary tags. To make them primary_tags.* tags that you can use uniformly across Dynatrace, review your OpenPipeline primary Grail tag rules.

Define the target primary_tags.<key> once and list the source fields to read from, in order. The first source with a non-null value is used to set the target. You can also add the equivalent fields from other cloud providers to the same rule, so a single primary_tags.team definition works across Kubernetes, AWS, Azure, and Google Cloud.

For the full setup, see Set primary Grail tag rules in OpenPipeline.

3. Enrich with dedicated metadata.dynatrace.com annotations

If your Kubernetes resources don't carry suitable information in existing labels or annotations, add dedicated metadata.dynatrace.com/primary_tags.<key>:<value> annotations directly to your Kubernetes manifests to set primary Grail tags. The same convention also accepts the supported primary fields dt.security_context, dt.cost.costcenter, and dt.cost.product. Any other metadata.dynatrace.com/<key> annotation is ignored for enrichment on signals.

Pod, workload, or namespace annotation example

This block goes under metadata.annotations on a Namespace, on a Pod, or on the workload object's own metadata (not its pod template):

metadata:
annotations:
# Primary tags:
metadata.dynatrace.com/primary_tags.team: payments
metadata.dynatrace.com/primary_tags.environment: production
# Supported primary fields:
metadata.dynatrace.com/dt.security_context: confidential
metadata.dynatrace.com/dt.cost.costcenter: it_services
metadata.dynatrace.com/dt.cost.product: fin_app

You can place these annotations at three levels:

  • Namespace-level: Annotations on a Namespace apply to every workload and pod inside that namespace. Use this when the team, environment, or security context is defined at the namespace boundary and is shared by everything that runs there.
  • Workload-level: Dynatrace Operator version 1.11+ ActiveGate version 1.347+ Annotations on the workload object itself (Deployment, StatefulSet, DaemonSet, and other recognized workload kinds) apply to every pod created by that workload. Use this when the metadata belongs to the workload as a whole but shouldn't apply to everything else in the namespace.
  • Pod-level: Annotations on a Pod (typically through the pod template of a Deployment, StatefulSet, or DaemonSet) apply only to that pod. Use this when you need to override the workload or namespace defaults for a specific pod, or when different pods of the same workload need different metadata.

When the same key is set at multiple levels, the pod-level value wins over the workload-level value, and the workload-level value wins over the namespace-level value.

4. Configure attributes in the DynaKube

You can define additional resource attributes directly in your DynaKube. Dynatrace Operator propagates these attributes to all telemetry signals without requiring you to modify namespaces, workloads, or pods. This is useful when you want to attach a fixed tag to all signals from a cluster, such as a team identifier, at the DynaKube level.

Three DynaKube sections control which signals receive which attributes:

FieldScope

.spec.resourceAttributes

Applied to all signals: OneAgent injection, OTLP exporter injection, standalone log monitoring, and ActiveGate.

.spec.oneAgent.<mode>.additionalResourceAttributes

Applied to OneAgent signals only. Takes precedence over duplicate keys in .spec.resourceAttributes.

.spec.otlpExporterConfiguration.additionalResourceAttributes

Applied to OTLP telemetry of configured exporters only. Takes precedence over duplicate keys in .spec.resourceAttributes.

DynaKube snippet showcasing attribute configuration

Attribute keys prefixed with primary_tags.* become primary Grail tags. The primary fields dt.security_context, dt.cost.costcenter, and dt.cost.product are also supported.

apiVersion: dynatrace.com/v1beta6
kind: DynaKube
metadata:
name: dynakube
namespace: dynatrace
spec:
apiUrl: https://<environment-id>.live.dynatrace.com/api
resourceAttributes:
primary_tags.team: "alpha"
oneAgent:
cloudNativeFullStack:
additionalResourceAttributes:
aws.account.id: "000000000000"
primary_tags.team: "charly"
namespaceSelector:
matchLabels:
my.app.com/oneagent: "true"
otlpExporterConfiguration:
additionalResourceAttributes:
aws.account.id: "000000000000"
primary_tags.team: "charly"
namespaceSelector:
matchLabels:
my.app.com/otel: "true"
signals:
metrics: {}
traces: {}
logs: {}

Dynatrace Operator propagates the merged attributes as follows:

  • OneAgent injection: Dynatrace Operator writes attributes to dt_node_metadata.properties for OneAgents, and to dt_metadata.properties, dt_metadata.json, and the metadata.dynatrace.com pod annotation for OneAgent injected pods.
  • OTLP exporter injection: Dynatrace Operator appends attributes to OTEL_RESOURCE_ATTRIBUTES and the metadata.dynatrace.com pod annotation for injected pods.
  • Standalone log monitoring and ActiveGate: Dynatrace Operator applies .spec.resourceAttributes directly. Mode-specific additionalResourceAttributes does not affect these components.
Conflicting keys with concurrent OneAgent and OTLP injection

When both OneAgent injection and OTLP exporter injection are active on the same pod, both write to the shared metadata.dynatrace.com JSON annotation. Having the same key defined in both sections will result in undefined behavior. You must use distinct keys across OneAgent and OTLP additionalResourceAttributes when both are active on the same pod to avoid unpredictable behavior.

Attribute key sanitization

Before merging resource attributes into these targets, Dynatrace Operator sanitizes each key: it replaces any character outside a-zA-Z0-9-_. and strips leading or trailing non-alphanumeric characters.

Dynatrace Operator validates attribute keys and reports the following:

  • Warning: The key contains characters Dynatrace Operator replaces during sanitization - Dynatrace Operator renames the key but still applies the attribute.
  • Error: The sanitized key is an empty string - Dynatrace Operator drops the key.
  • Error: Two keys produce the same sanitized value - the result is a collision.
  • Error: The sanitized key exceeds 63 characters.

For the full parameter reference, see the DynaKube API reference.

5. OpenPipeline (last resort)

If none of these options apply, derive or assign primary tags at ingest with OpenPipeline and the processing stage. Use this as a fallback for environments where source-side or central enrichment isn't possible.

Precedence

When the same key is set by multiple sources, the most specific scope takes precedence within each source (pod over workload over namespace). Across sources, the priority from highest to lowest is:

  1. metadata.dynatrace.com/<key> annotations
  2. DynaKube resource attributes
  3. Central enrichment rules

Coverage by signal type

The following table shows where primary tags and the special fields (dt.security_context, dt.cost.costcenter, dt.cost.product) land, by enrichment option.

SignalCentral configurationDedicated annotations (pod, workload, or namespace)

Distributed traces powered by Grail 1

OneAgent metrics 1

JMX/PMI metrics collected via OneAgent 1

Service metrics 1

Kubernetes platform metrics

Kubernetes events

Smartscape Kubernetes entities

Prometheus metrics scraping

Logs collected by OneAgent log module 1

Logs collected by FluentBit

Spans, metrics, and logs collected by the OpenTelemetry Collector

OneAgent events 1

1

OneAgent-based enrichment requires the Cloud Native Full Stack or Application Monitoring deployment mode. Classic Full Stack is not supported.

What to expect after configuration

Primary Grail fields and tags are added to the telemetry that your workloads emit. They aren't added as properties of Kubernetes entities, such as namespaces or workloads, in the entity details.

Dynatrace version 1.348+ After the data is enriched, you can use primary Grail fields and tags in the Infrastructure Observability apps, including Kubernetes (new) Kubernetes:

  • Filter by primary Grail fields and tags in the filter bar and facets.
  • See primary tags in a dedicated column in list views.
  • Use the primary tag built-in filters on ready-made dashboards.

Query enriched data in Grail

Primary Grail fields and tags appear as top-level fields and can be queried with DQL.

Filter logs by namespace and team
fetch logs
| filter k8s.namespace.name == "checkout" AND primary_tags.team == "payments"
Cost allocation aggregation
fetch bizevents
| filter dt.cost.costcenter == "it_services"
| summarize sum(value), by: {dt.cost.product}
Security context filtering
fetch logs
| filter dt.security_context == "confidential"
| filter k8s.cluster.name == "prod-eu-1"

Central configuration setup checks

The following checks apply to the central configuration setup described in Promote existing Kubernetes annotations and labels.

  • Verify the rule definition. Confirm that each rule points to the correct metadata type Label, Annotation, or Literal, and that the source key exactly matches the key on the namespace, workload, or pod.
  • Check that the source metadata exists. Open the namespace, workload, or pod in the Kubernetes app and look for the expected labels and annotations, or run kubectl get <resource> <name> -o yaml, for example kubectl get namespace <name> -o yaml or kubectl get deployment <name> -o yaml, and inspect the metadata.labels and metadata.annotations sections.
  • Validate that metadata enrichment is turned on. The feature works only if metadataEnrichment is allowed in your DynaKube. If you specify a namespaceSelector, make sure it matches the namespace you're testing.
  • Confirm that enrichment reached the pods. Run kubectl get pod <pod-name> -o yaml on a pod in the namespace and look for annotations starting with metadata.dynatrace.com/. Their presence means the metadata reached the pod.

Limitations

  • Changes in central enrichment configuration builtin:ingest.enrichment.config or OpenPipeline primary Grail tag rules builtin:openpipeline.primary-grail-tag can take up to 15 minutes to propagate.
  • A newly enriched primary tag key can take up to 24 hours to appear in the filter dropdowns of Dynatrace apps. This is temporary. The data is correctly enriched and can be queried and filtered with DQL.
  • Dynatrace Classic platform is not supported.

Related topics

  • Primary Grail fields and tags
  • Organize your data with primary Grail fields and tags
  • Enrich OneAgent telemetry with primary Grail fields and tags
  • Set up Grail permissions for telemetry from Kubernetes and Kubernetes workloads
  • Set up Cost Allocation for Kubernetes deployments
  • Set primary Grail tag rules in OpenPipeline
  • Metadata enrichment of all telemetry originating from Kubernetes
  • Global field reference
  • Primary Grail fields
  • Plan your tagging strategy
  • Best practices for enriching primary Grail fields and tags
Related tags
Dynatrace Platform