Enrich from Kubernetes

The following configuration example shows how you configure a Collector instance to enrich OTLP request with additional Kubernetes metadata.

Demo configuration

1receivers:
2  otlp:
3    protocols:
4      grpc:
5      http:
6
7processors:
8  resource:
9    attributes:
10    - key: dt.kubernetes.cluster.id
11      from_attribute: k8s.cluster.uid
12      action: insert
13  k8sattributes:
14    auth_type: "serviceAccount"
15    passthrough: false
16    extract:
17      metadata:
18        - k8s.pod.name
19        - k8s.pod.uid
20        - k8s.deployment.name
21        - k8s.namespace.name
22        - k8s.node.name
23        - k8s.cluster.uid
24    pod_association:
25      - sources:
26        - from: resource_attribute
27          name: k8s.pod.name
28        - from: resource_attribute
29          name: k8s.namespace.name
30
31exporters:
32  otlphttp:
33    endpoint: $DT_ENDPOINT/api/v2/otlp
34    headers:
35      Authorization: "Api-Token $DT_API_TOKEN"
36
37service:
38  pipelines:
39    traces:
40      receivers: [otlp]
41      processors: [k8sattributes, resource]
42      exporters: [otlphttp]
43    metrics:
44      receivers: [otlp]
45      processors: [k8sattributes, resource]
46      exporters: [otlphttp]
47    logs:
48      receivers: [otlp]
49      processors: [k8sattributes, resource]
50      exporters: [otlphttp]

Prerequisites

A deployed ActiveGate for Kubernetes API monitoring
Contrib distribution or a custom Builder version with the Kubernetes Attributes and Resource processors
Collector deployed in agent mode
The API URL of your Dynatrace environment
An API token with the relevant access scope
Kubernetes configured for the required role-based access control

Kubernetes configuration

Configure the following rbac.yaml file with your Kubernetes instance, to allow the Collector to use the Kubernetes API with the service-account authentication type.

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  labels:
5    app: otelcontribcol
6  name: otelcontribcol
7---
8apiVersion: rbac.authorization.k8s.io/v1
9kind: ClusterRole
10metadata:
11  name: otelcontribcol
12  labels:
13    app: otelcontribcol
14rules:
15  - apiGroups:
16      - ""
17    resources:
18      - events
19      - namespaces
20      - namespaces/status
21      - nodes
22      - nodes/spec
23      - nodes/stats
24      - nodes/proxy
25      - pods
26      - pods/status
27      - replicationcontrollers
28      - replicationcontrollers/status
29      - resourcequotas
30      - services
31      - endpoints
32    verbs:
33      - get
34      - list
35      - watch
36  - apiGroups:
37      - "events.k8s.io"
38    resources:
39      - events
40    verbs:
41      - watch
42  - apiGroups:
43      - apps
44    resources:
45      - daemonsets
46      - deployments
47      - replicasets
48      - statefulsets
49    verbs:
50      - get
51      - list
52      - watch
53  - apiGroups:
54      - extensions
55    resources:
56      - daemonsets
57      - deployments
58      - replicasets
59    verbs:
60      - get
61      - list
62      - watch
63  - apiGroups:
64      - batch
65    resources:
66      - jobs
67      - cronjobs
68    verbs:
69      - get
70      - list
71      - watch
72  - apiGroups:
73      - autoscaling
74    resources:
75      - horizontalpodautoscalers
76    verbs:
77      - get
78      - list
79      - watch
80---
81apiVersion: rbac.authorization.k8s.io/v1
82kind: ClusterRoleBinding
83metadata:
84  name: otelcontribcol
85  labels:
86    app: otelcontribcol
87roleRef:
88  apiGroup: rbac.authorization.k8s.io
89  kind: ClusterRole
90  name: otelcontribcol
91subjects:
92  - kind: ServiceAccount
93    name: otelcontribcol
94    namespace: default

Components

For our configuration, we configured the following components.

Receiver

Under receivers, we specify the standard otlp receiver as an active receiver component for our Collector instance.

This is mainly for demonstration purposes. You can specify any other valid receiver here (for example, zipkin).

Processor

Under processors, we specify the k8sattributes processor with the following parameters:

auth_type—Specifies to use a service account to obtain the necessary data.
passthrough—Set to false, in order not to forward IP address related information. Should be true when running as gateway.
extract—Specifies which information should be extracted.
pod_association—Specifies how the pod information is linked to attributes.

We also configure the resource processor to have the Kubernetes cluster identifier automatically added as resource attribute.

Exporter

Under exporters, we specify the default otlphttp exporter and configure it with our Dynatrace API URL and the required authentication token.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.

DT_ENDPOINT contains the base URL of your ActiveGate (for example, https://{your-environment-id}.live.dynatrace.com)
DT_API_TOKEN contains the API token

Service pipeline

Under service, we assemble our receiver, processor, and exporter objects into pipelines for traces, metrics, and logs. These pipelines allow us to send OpenTelemetry signals via the Collector instance and have them automatically enriched with additional Kubernetes-specific details.