Monitor Prometheus metrics

Prometheus is an open-source monitoring and alerting toolkit which is popular in the Kubernetes community. Prometheus scrapes metrics from a number of HTTP(s) endpoints that expose metrics in the OpenMetrics format. See the list of available exporters in the Prometheus documentation.

Prometheus Metric Type Ingest

Dynatrace is able to ingest Prometheus metrics Counter, Gauge, Histogram and Summary in Kubernetes using Prometheus exporters and makes them available for charting, alerting and analysis.

Counter

The Prometheus counter metric¹ is a monotonically increasing value typically used for measurements that can only increase or remain constant. Dynatrace uses its COUNT metric type, which uses delta encoding² in order to reduce data redundancy, for data ingestion. As such, the value displayed in Dynatrace does not reflect the actual counter value, but instead its change, or delta, over observations.

This method results in a counter metric appearing one minute delayed in contrast to a Gauge metric if scraping for both metrics started simultaneously. For details, see the metric ingestion protocol reference.

Gauge

In contrast to Counter, the gauge metric¹ stores a single numerical value which can increase and decrease and is typically used for measured values such as current memory usage or number of users online. In Dynatrace, the GAUGE metric type will be used for data ingest.

Histogram

Histograms¹ provide visual insights into the distribution and frequency of numerical data. For a base metric name of <basename>, Dynatrace will ingest the data according to the following table.

Additional flexibility and control is provided via the builtin:histogram-metrics settings schema. This schema allows to configure the ingestion of <basename>_bucket{le="<upper inclusive bound>"} metrics.

Summary

Like a Histogram, the summary metric¹ samples observations. In contrast to the histogram metric, a summary's buckets are represented by φ-quantiles where 0 ≤ φ ≤ 1. For a base metric name of <basename>, Dynatrace will ingest the data according to the following table.

Unsupported

The following types are currently not supported by Dynatrace:

Prerequisites

Annotate Prometheus exporter pods

Dynatrace collects metrics from any pods that are annotated with a metrics.dynatrace.com/scrape property set to true in the pod definition. This functionality applies to all pods across the entire Kubernetes cluster, regardless of whether the pod is running in a namespace that matches the Dynakube's namespace selector.

Depending on the actual exporter in a pod, you might need to set additional annotations to the pod definition in order to allow Dynatrace to properly ingest those metrics.

Enable metrics scraping required

Set metrics.dynatrace.com/scrape to true to enable Dynatrace to collect Prometheus metrics exposed for this pod.

Metrics port required

By default, Prometheus metrics are available at the first exposed TCP port of the pod. Set metrics.dynatrace.com/port to the respective port.

To determine the port value, see Default port allocations for a list of common ports for known exporters.

Path to metrics endpoint optional

Use metrics.dynatrace.com/path to override the default (/metrics) Prometheus endpoint.

HTTP/HTTPS optional

Set metrics.dynatrace.com/secure to true if you want to collect metrics that are exposed by an exporter via HTTPS. The default value is false, because most exporters expose their metrics via HTTP.

Filter metrics optional

Use metrics.dynatrace.com/filter to define a filter that allows you to include ("mode": "include") or exclude ("mode": "exclude") a list of metrics. If no filter annotation is defined, all metrics are collected. The filter syntax also supports the asterisk (*). This symbol allows you to filter metrics keys that begin with, end with, or contain a particular sequence, such as:

This example shows how to use the filter syntax in a pod definition with annotations:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  annotations:
    metrics.dynatrace.com/scrape: 'true'
    metrics.dynatrace.com/path: '/path/to-metrics'
    metrics.dynatrace.com/port: '9001'
    metrics.dynatrace.com/secure: 'false'
    metrics.dynatrace.com/filter: |
      {
        "mode": "include",
        "names": [
            "redis_db_keys",
            "*insights*",
            "*bytes"
            ]
      }
spec:
  containers:
  - name: mycontainer
    image: myregistry/myimage:mytag

Client authentication optional

Requirements: Add the permissions to access secrets and configmaps for the dynatrace-kubernetes-monitoring ClusterRole.

Some systems require extra authentication before Dynatrace can scrape them. For such cases, you can set the following additional annotations:

The required certificates/keys are automatically loaded from secret/configmaps specified in the annotation value.
The schema for the annotation values is <configmap|secret>:<namespace>:<resource_name>:<field_name_in_data_section>.

For example, the annotations could look as follows:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  annotations:
    metrics.dynatrace.com/scrape: 'true'
    metrics.dynatrace.com/path: '/path/to-metrics'
    metrics.dynatrace.com/port: '9001'
    metrics.dynatrace.com/secure: 'false'
    metrics.dynatrace.com/tls.ca.crt: 'configmap:kubernetes-config:etcd-metric-serving-ca:ca-bundle.crt'
    metrics.dynatrace.com/tls.crt: 'secret:kubernetes-config:etcd-metric-client:tls.crt'
    metrics.dynatrace.com/tls.key: 'secret:kubernetes-config:etcd-metric-client:tls.key'
spec:
  containers:
  - name: mycontainer
    image: myregistry/myimage:mytag

Ingesting metrics from exporters requiring client authentication is only possible with an ActiveGate deployed inside the monitored cluster and the Kubernetes connection settings configured to monitor the local Kubernetes API endpoint.

HTTP - Basic authentication optional

For systems that require basic HTTP authentication before scraping, you can apply the following additional annotations.

The following example shows two secrets created in the default namespace – one for a username and one for a password. The aforementioned annotations are then used on a pod, with the secrets referenced in the annotation values.

apiVersion: v1
kind: Secret
metadata:
  name: user-secret
data:
  username: bXktdXNlcm5hbWUtc2VjcmV0Cg==
---
apiVersion: v1
kind: Secret
metadata:
  name: password-secret
data:
  password: bXktcGFzc3dvcmQtc2VjcmV0Cg==

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  annotations:
    metrics.dynatrace.com/scrape: 'true'
    metrics.dynatrace.com/path: '/path/to-metrics'
    metrics.dynatrace.com/port: '9001'
    metrics.dynatrace.com/secure: 'false'
    metrics.dynatrace.com/http.auth.basic.username: 'secret:default:user-secret:username'
    metrics.dynatrace.com/http.auth.basic.password: 'secret:default:password-secret:password'
spec:
  containers:
  - name: mycontainer
    image: myregistry/myimage:mytag

Ingesting metrics from exporters requiring basic HTTP authentication is only possible with an ActiveGate deployed inside the monitored cluster and the Kubernetes connection settings configured to monitor the local Kubernetes API endpoint.

Role-based access control (RBAC) authorization for metric ingestion

Exporter pods such as node-exporter, kube-state-metrics, and openshift-state-metrics require RBAC authorization. For these pods, add the annotation:

metrics.dynatrace.com/http.auth: 'builtin:default'

This annotation applies the token from the dynatrace-kubernetes-monitoring service account as an authorization header for requests to the exporter.

Ingesting metrics from exporters requiring RBAC authorization is only possible with an ActiveGate deployed inside the monitored cluster and the Kubernetes connection settings configured to monitor the local Kubernetes API endpoint.

For more information on how to annotate pods, see annotations best practices.

Annotate Kubernetes services

Requirements: Add the permission to access services for the dynatrace-kubernetes-monitoring ClusterRole (not needed for Dynatrace Operator users, as this is enabled by default in clusterrole-kubernetes-monitoring.yaml).

You can also annotate services instead of pods. Pods corresponding to the Kubernetes services are automatically discovered via the service label selector, causing scraping of all pods belonging to the service.

For more information on how to annotate services, see Annotation best practices.

Annotation best practices

There are multiple ways to place annotations on pods or services. See below to decide which approach fits your scenario best.

Recommended if you have full control

If you have full control over the pod template or service definition, we recommend adding the annotations by editing these files. This is the most reliable way to ensure persistency of annotations. We recommend editing the pod template over editing the service definition, as this requires fewer permissions (for example, if you don't have access to services).
Pro: Annotations are persistent, so they don't need to be recreated if a pod is removed.

Options if you don't have full control

If you don't have full control over the pod template, you have the following options:

View metrics on a dashboard

Metrics from Prometheus exporters are available in Data Explorer for custom charting. Select Create custom chart and select Try it out in the top banner. For more information, see Data Explorer.

You can simply search for metric keys of all available metrics and define how you’d like to analyze and chart your metrics. After that you can pin your charts on a dashboard.

Metric alerts

You can also create custom alerts based on the Prometheus scraped metrics. Go to Settings > Anomaly detection > Metric events and select Add metric event. In the Add metric event page, search for a Prometheus metric using its key and define your alert. For more information, see Metric events for alerting.

Limitations

The current limitations of the Prometheus metrics integration are as follows:

Multiple exporters in a pod

Multiple exporters currently aren't supported; you can only select the exporter that is being used with the metrics.dynatrace.com/port annotation.

Number of pods, metrics, and metric data points

This integration supports a maximum of

Monitoring methods

There are two distinct methods of monitoring technologies:

These two methods serve different contexts, work independently from each other, and don't share the same metrics.

Monitoring consumption

If you have DPS licensing, you can get more information about your environment's custom metric consumption from our licensing documentation.

If you have Classic licensing, Prometheus metrics in Kubernetes environments are subject to DDU consumption.

Troubleshoot

To troubleshoot Prometheus integration issues, download the Kubernetes Monitoring Statistics extension. For more information, see the community article on How to troubleshoot missing Prometheus metrics.