Scrape and enrich Kubernetes data with Prometheus
The following configuration example shows how you configure a Collector instance to scrape data from an existing Kubernetes Prometheus setup, enrich the data with additional information provided by the Kubernetes attributes processor, and eventually import everything to Dynatrace.
Demo configuration
1receivers:2 prometheus:3 config:4 scrape_configs:5 - job_name: 'istiod'6 kubernetes_sd_configs:7 - role: endpoints8 namespaces:9 names:10 - istio-system11 relabel_configs:12 - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]13 action: keep14 regex: istiod;http-monitoring15 - source_labels: [__meta_kubernetes_endpoint_address_target_name]16 action: replace17 target_label: "k8s_pod_name"18 - source_labels: [__meta_kubernetes_namespace]19 action: replace20 target_label: "k8s_namespace_name"2122processors:23 resource:24 attributes:25 - key: dt.kubernetes.cluster.id26 from_attribute: k8s.cluster.uid27 action: insert28 cumulativetodelta: {}29 k8sattributes:30 auth_type: "serviceAccount"31 passthrough: false32 extract:33 metadata:34 - k8s.pod.name35 - k8s.pod.uid36 - k8s.deployment.name37 - k8s.namespace.name38 - k8s.node.name39 - k8s.cluster.uid40 pod_association:41 - sources:42 - from: resource_attribute43 name: k8s.pod.name44 - from: resource_attribute45 name: k8s.namespace.name4647exporters:48 otlphttp:49 endpoint: ${DT_ENDPOINT}api/v2/otlp50 headers:51 Authorization: "Api-Token ${DT_API_TOKEN}"5253service:54 pipelines:55 metrics:56 receivers: [prometheus]57 processors: [k8sattributes, cumulativetodelta, resource]58 exporters: [otlphttp]
Prerequisites
A deployed ActiveGate for Kubernetes API monitoring
- Contrib distribution or a custom Builder version with the Prometheus receiver and the Kubernetes Attributes processor
- The API URL of your Dynatrace environment
- An API token with the relevant access scope
Components
For our configuration, we configure the following components.
Receiver
Under receivers
, we specify the prometheus
receiver as active receiver component for our Collector instance and configure the job istiod
, under scrape_configs
, for a service discovery configuration of endpoints within the provided namespace.
For a full list of configuration parameters, see the Prometheus documentation.
Processor
Under processors
, we specify the following three processors:
resource
, to add the cluster identifier as resource attribute.cumulativetodelta
, to convert the cumulative Prometheus metrics to delta aggregation.k8sattributes
, to enrich the request with Kubernetes-specific data.
Exporter
Under exporters
, we specify the default otlphttp
exporter and configure it with our Dynatrace API URL and the required authentication token.
For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint
and Authorization
.
DT_ENDPOINT
contains the base URL of your ActiveGate (for example,https://{your-environment-id}.live.dynatrace.com
)DT_API_TOKEN
contains the API token
Service pipeline
Under service
, we assemble our receiver, processor, and exporter objects into a metrics pipeline, which will execute the Prometheus jobs, enrich the data with Kubernetes-specific information, and ingest it into Dynatrace.