Metadata enrichment of all telemetry originating from Kubernetes workloads

Prerequisites

Use cases

Automatic enrichment of telemetry data with information about Kubernetes cluster, namespace, and workload

When following the Dynatrace deployment models with enabled metadata enrichment, all your telemetry (metrics, logs, and trace data) will be automatically enriched with the following Kubernetes metadata information:

Metrics that are about Kubernetes nodes, pods, and containers, as well as host and process metrics, additionally get the following information:

This enables you to always associate any kind of telemetry with the underlying Kubernetes deployment and infrastructure.

Security context and cost allocation

In Dynatrace you can set up policy boundaries, for fine-granular restrictions on the data level. Per default you can use k8s.namespace.name and k8s.cluster.name, but sometimes this is not enough and you need a more fine grained way to set up your boundries. You might already have defined such boundaries for yourself and defined them as Kubernetes labels or annotations. This feature enables you to use these at the source for your security context in Dynatrace. If you have not done so already Dynatrace does recommend to either use cluster or namespace name or setup a dedicated annotation for your Kubernetes workloads that serves as your security context.

Similarly Dynatrace provides a solution for cost allocation in DPS. You might already have the necessary data like department and product available in your existing Kubernetes labels or annotations. Even if not you might find it very convenient to setup up cost allocation as a Kubernetes annotation or label, which is what Dynatrace recommends. This feature then enables you to use these labels and annotations as the means to solving cost allocation in DPS.

The following attributes are supported:

If annotations with the same keys are specified in both the namespace and pod, the pod annotations will take precedence.

Prerequisites

Operator version 1.3.0+, ActiveGate version 1.305+, OneAgent version 1.303+

Which data will be enriched

Enrichment options

Depending on the specific use case, the following enrichment options are supported:

Use existing labels and annotations

You can enrich your data using existing Kubernetes namespace labels and annotations via Kubernetes enrichment rules. You can define up to 20 rules on a tenant or Kubernetes cluster level that define the labels or annotations for your security context or cost allocation.

In order for all data enrichment to work the Dynatrace Operator stores the values of the namespace annotations and labels on pods as special annotations using the following conventions:

annotations:
  metadata.dynatrace.com/<target>: <value> # created by the Operator

The pod annotation will be used to enrich the Dynatrace data with attributes: <target>: <value>

Example

Rules

"rules":
  [
    {
      # rule #1
      "type": "Annotation",
      "source": "metadata.example.com/team",
      "target": "dt.security_context"
    },
    {
      # rule #2
      "type": "Label",
      "source": "department",
      "target": "dt.cost.costcenter"
    },
    {
      # rule #3
      "type": "Label",
      "source": "app/name",
      "target": "dt.cost.product"
    }
  ]

Namespace

metadata:
  annotations:
    metadata.example.com/team: sre
  labels:
    department: it_services
    app/name: fin_app

Pod

The Operator will create pod annotations:

metadata:
  annotations:
    metadata.dynatrace.com/dt.security_context: sre         # created by the Operator from metadata.example.com/team namespace annotation by rule #1
    metadata.dynatrace.com/dt.cost.costcenter: it_services  # created by the Operator from department namespace label by rule #2
    metadata.dynatrace.com/dt.cost.product: fin_app         # created by the Operator from app/name namespace label by rule #3

Attributes on the telemetry data

The following attributes will enrich the data:

# come from pod annotations with the metadata.dynatrace.com prefix
dt.security_context: sre
dt.cost.costcenter: it_services
dt.cost.product: fin_app

Use dedicated dynatrace metadata pod annotations

You can enrich your data using dedicate pod annotations. You might create the following annotations at the pod level

metadata:
  annotations:
    metadata.dynatrace.com/dt.security_context: sre
    metadata.dynatrace.com/dt.cost.costcenter: it_services
    metadata.dynatrace.com/dt.cost.product: fin_app
    metadata.dynatrace.com/my.own.attribute: my_value

The following attributes will enrich the data:

dt.security_context: sre
dt.cost.costcenter: it_services
dt.cost.product: fin_app
my.own.attribute: my_value

Use OpenTelemetry environment variables

You can also choose to enrich your data via OpenTelemetry environment variable.

How it works

You can use the OTEL_RESOURCE_ATTRIBUTES environment variable to configure attributes.

Both OneAgent and OpenTelemetry will enrich your data with the attributes defined in OTEL_RESOURCE_ATTRIBUTES.

See the Resource semantic conventions for details.

DYNAKUBE="dynakube" # set this to the name of your DynaKube / kubectl get dynakube -n dynatrace

K8S_CLUSTER_UID="$(kubectl get dynakube -o jsonpath='{.status.kubeSystemUUID}' -n dynatrace $DYNAKUBE)"

K8S_CLUSTER_NAME="$(kubectl get dynakube -o jsonpath='{.status.kubernetesClusterName}' -n dynatrace $DYNAKUBE)"

DT_ENTITY_KUBERNETES_CLUSTER="$(kubectl get dynakube -o jsonpath='{.status.kubernetesClusterMEID}' -n dynatrace $DYNAKUBE)"

kubectl create configmap dynatrace-metadata \
  --from-literal K8S_CLUSTER_UID=$K8S_CLUSTER_UID \
  --from-literal K8S_CLUSTER_NAME=$K8S_CLUSTER_NAME \
  --from-literal DT_ENTITY_KUBERNETES_CLUSTER=$DT_ENTITY_KUBERNETES_CLUSTER

envFrom:
- configMapRef:
    name: dynatrace-metadata
    optional: false
env:
- name: K8S_CONTAINER_NAME
  value: "" # replace with actual container name
- name: K8S_POD_NAME
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.name
- name: K8S_POD_UID
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.uid
- name: K8S_POD_NAMESPACE
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.namespace
- name: K8S_WORKLOAD_KIND
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/k8s.workload.kind'] # only works when metadata enrichment is enabled
- name: K8S_WORKLOAD_NAME
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/k8s.workload.name'] # only works when metadata enrichment is enabled
- name: K8S_NODE_NAME
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: spec.nodeName
- name: DT_SECURITY_CONTEXT # only works when automatic security context enrichment is configured
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/dt.security_context']
- name: DT_COST_PRODUCT # only works when automatic cost product enrichment is configured
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/dt.cost.product']
- name: DT_COST_COSTCENTER # only works when automatic cost center enrichment is configured
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.annotations['metadata.dynatrace.com/dt.cost.costcenter']

- name: OTEL_RESOURCE_ATTRIBUTES
  value: k8s.cluster.name=$(K8S_CLUSTER_NAME),k8s.cluster.uid=$(K8S_CLUSTER_UID),k8s.node.name=$(K8S_NODE_NAME),k8s.workload.name=$(K8S_WORKLOAD_NAME),k8s.workload.kind=$(K8S_WORKLOAD_KIND),k8s.pod.name=$(K8S_POD_NAME),k8s.pod.uid=$(K8S_POD_UID),k8s.namespace.name=$(K8S_POD_NAMESPACE),k8s.container.name=$(K8S_CONTAINER_NAME),dt.entity.kubernetes_cluster=$(DT_ENTITY_KUBERNETES_CLUSTER),dt.security_context=$(DT_SECURITY_CONTEXT),dt.cost.costcenter=$(DT_COST_COSTCENTER),dt.cost.product=$(DT_COST_PRODUCT)