Metadata enrichment of all telemetry originating from kubernetes workloads

Prerequisites

Use cases

Automatic enrichment of telemetry data with information about Kubernetes cluster, namespace, and workload

When following the Dynatrace deployment models with enabled metadata enrichment, all your telemetry (metrics, logs, and trace data) will be automatically enriched with the following Kubernetes metadata information:

Metrics that are about Kubernetes nodes, pods, and containers, as well as host and process metrics, additionally get the following information:

This enables you to always associate any kind of telemetry with the underlying Kubernetes deployment and infrastructure.

Security context and cost allocation

In Dynatrace you can set up policy boundaries, for fine-granular restrictions on the data level. Per default you can use k8s.namespace.name and k8s.cluster.name, but sometimes this is not enough and you need a more fine grained way to set up your boundries. You might already have defined such boundaries for yourself and defined them as Kubernetes labels or annotations. This feature enables you to use these at the source for your security context in Dynatrace. If you have not done so already Dynatrace does recommend to either use cluster or namespace name or setup a dedicated annotation for your Kubernetes workloads that serves as your security context.

Similarly Dynatrace provides a solution for cost allocation in DPS. You might already have the necessary data like department and product available in your existing Kubernetes labels or annotations. Even if not you might find it very convenient to setup up cost allocation as a Kubernetes annotation or label, which is what Dynatrace recommends. This feature then enables you to use these labels and annotations as the means to solving cost allocation in DPS.

The following attributes are supported:

If annotations with the same keys are specified in both the namespace and pod, the pod annotations will take precedence.

Prerequisites

Operator version 1.3.0+, ActiveGate version 1.305+, OneAgent version 1.303+

Which data will be enriched

Enrichment options

Depending on the specific use case, the following enrichment options are supported:

Use existing labels and annotations

You can enrich your data using existing Kubernetes namespace labels and annotations via Kubernetes enrichment rules. You can define up to 20 rules on a tenant or Kubernetes cluster level that define the labels or annotations for your security context or cost allocation.

In order for all data enrichment to work the Dynatrace Operator stores the values of the namespace annotations and labels on pods as special annotations using the following conventions:

annotations:
  metadata.dynatrace.com/<target>: <value> # created by the Operator

The pod annotation will be used to enrich the Dynatrace data with attributes: <target>: <value>

Example

Rules

"rules":
  [
    {
      # rule #1
      "type": "Annotation",
      "source": "metadata.example.com/team",
      "target": "dt.security_context"
    },
    {
      # rule #2
      "type": "Label",
      "source": "department",
      "target": "dt.cost.costcenter"
    },
    {
      # rule #3
      "type": "Label",
      "source": "app/name",
      "target": "dt.cost.product"
    }
  ]

Namespace

metadata:
  annotations:
    metadata.example.com/team: sre
  labels:
    department: it_services
    app/name: fin_app

Pod

The Operator will create pod annotations:

metadata:
  annotations:
    metadata.dynatrace.com/dt.security_context: sre         # created by the Operator from metadata.example.com/team namespace annotation by rule #1
    metadata.dynatrace.com/dt.cost.costcenter: it_services  # created by the Operator from department namespace label by rule #2
    metadata.dynatrace.com/dt.cost.product: fin_app         # created by the Operator from app/name namespace label by rule #3

Attributes on the telemetry data

The following attributes will enrich the data:

# come from pod annotations with the metadata.dynatrace.com prefix
dt.security_context: sre
dt.cost.costcenter: it_services
dt.cost.product: fin_app

Use dedicated dynatrace metadata pod annotations

You can enrich your data using dedicate pod annotations. You might create the following annotations at the pod level

metadata:
  annotations:
    metadata.dynatrace.com/dt.security_context: sre
    metadata.dynatrace.com/dt.cost.costcenter: it_services
    metadata.dynatrace.com/dt.cost.product: fin_app
    metadata.dynatrace.com/my.own.attribute: my_value

The following attributes will enrich the data:

dt.security_context: sre
dt.cost.costcenter: it_services
dt.cost.product: fin_app
my.own.attribute: my_value

Use OpenTelemetry environment variables

You can also choose to enrich your data via OpenTelemetry environment variable (without using Dynatrace Operator).

How it works

You can use the OTEL_RESOURCE_ATTRIBUTES environment variable to configure attributes.

Both OneAgent and OpenTelemetry will enrich your data with the attributes defined in OTEL_RESOURCE_ATTRIBUTES.

See the Resource semantic conventions for details.

Example

Pod

spec:
  containers:
      env:
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "dt.security_context=sre,dt.cost.costcenter=it_services,dt.cost.product=fin_app"

Attributes on the telemetry data

The following attributes will enrich the data:

dt.security_context: sre
dt.cost.costcenter: it_services
dt.cost.product: fin_app