Try it free

Scrape Prometheus metrics with the OTel Collector (simplified)

  • How-to guide
  • 4-min read

This page describes how to configure the OpenTelemetry Collector to scrape Prometheus endpoints and forward the resulting metrics to Dynatrace. It focuses on a simplified setup where you have a static or simple set of endpoints and don't need auto-scaling or redundancy.

When to use this approach

Use a single Collector when one or more of the following scenarios apply.

  • You scrape a small or moderate number of endpoints (up to a few dozen).
  • The list of targets is mostly static or driven by simple service discovery (static_configs, single Kubernetes namespace, file-based discovery).
  • A single Collector instance has enough CPU and memory headroom for the expected load.

A single Collector eventually runs out of headroom. If you don't yet need the standard architecture, you can run multiple independent simplified Collectors, each scraping a distinct set of targets (for example, one per namespace or team). However you need to partition targets yourself, and there's no shared auto-scaling.

When to move to the standard deployment

Move to the standard deployment with the standard architecture using Target Allocator when you notice one or more of the following signals.

  • Throughput ceiling: The Collector can't keep up with the volume of scraped data, even after you increase its CPU and memory.
  • Scrape duration approaches the scrape interval: A scrape takes almost as long as the interval itself, so the Collector can't reliably finish one cycle before the next begins.
  • memory_limiter trips or the pod is OOM-killed: The Collector repeatedly hits its memory limit and drops data or restarts.
  • Target count outgrows manual partitioning: You have more targets than one Collector can handle, and splitting them across multiple independent Collectors has become a maintenance burden.
  • You need auto-scaling or redundancy: A single Collector is a single point of failure and can't scale horizontally with load.

The standard deployment adds automatic target discovery through Prometheus Operator CRDs, horizontal auto-scaling, and redundancy across the scraper and gateway pools.

When you switch, you need to convert your static scrape_configs into ServiceMonitor or PodMonitor CRDs; see Migrate from a single Collector.

Prerequisites

This use case assumes that you have:

  • A Prometheus scrape target exposing port 8888.
  • One of the following Collector distributions with the Prometheus receiver, the metric start time processor, and the cumulative-to-delta processor:
    • The Dynatrace OTel Collector
    • OpenTelemetry Contrib
    • A custom Builder version
  • The Dynatrace API endpoint URL to which the data should be exported.
  • An API token with the relevant access scope (only required for SaaS and ActiveGate).

See Collector Deployment and Collector Configuration on how to set up your Collector with the configuration below.

Resource sizing

For Collectors deployed as pods in Kubernetes, we recommend the following resource limits for ingesting up to 1 million data points per minute (DPM). To ingest more data, consider using the standard deployment model.

  • CPU: 2 cores
  • Memory: 8 GiB

These values assume the Collector is configured according to the demo configuration, which includes the prometheus receiver, metric_start_time and cumulativetodelta processors, and otlp_http exporter. Additional processors will increase these requirements.

Demo configuration

This configuration requires Dynatrace Collector v0.41.0 or later. The example pipeline below uses the metric_start_time processor, which adds start timestamps to metrics, and the cumulativetodelta processor, which converts the metrics to delta temporality.

receivers:
  prometheus:
     config:
       scrape_configs:
       - job_name: 'node-exporter'
         scrape_interval: 60s
         static_configs:
         - targets: ['prometheus-prometheus-node-exporter:9100']
       - job_name: opentelemetry-collector
         scrape_interval: 60s
         static_configs:
         - targets:
            - 127.0.0.1:8888


processors:
  metric_start_time:
  cumulativetodelta:
    max_staleness: 25h


exporters:
  otlp_http:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"


service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [metric_start_time, cumulativetodelta]
      exporters: [otlp_http]
Configuration validation

Validate your settings to avoid any configuration issues.

Cumulativetodelta processor recommendation

It is recommended to set the max_staleness parameter of the cumulativetodelta processor to a value higher than how often the Collector receives metrics (e.g., how often metrics via OTLP are received, or how long the Prometheus scrape interval is). This ensures that no references to abandoned metric streams accumulate in memory over time.

Components

For our configuration, we configure the following components.

Receivers

Under receivers, we specify the prometheus receiver as active receiver component for our Collector instance. We configure the receiver with the two jobs node-exporter and opentelemetry-collector under scrape_configs, to fetch data from the specified hosts once a minute (scrape_interval: 60s).

For a full list of configuration parameters, see the Prometheus documentation.

Processors

Under processors, we specify the cumulativetodelta processor to convert the metrics emitted by the Prometheus receiver to their delta aggregation format.

In Dynatrace Collector v0.41.0+, we specify the metric_start_time processor to add start timestamps to the metrics. This is required to properly convert the metrics to delta temporality.

Exporters

Under exporters, we specify the default otlp_http exporter and configure it with our Dynatrace API URL and the required authentication token.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.

  • DT_ENDPOINT contains the base URL of the Dynatrace API endpoint (for example, https://{your-environment-id}.live.dynatrace.com/api/v2/otlp)
  • DT_API_TOKEN contains the API token

Service pipeline

Under service, we assemble our receiver, processor, and exporter objects into a metrics pipeline, which will execute the Prometheus jobs, convert their metrics to delta values, and ingest the data into Dynatrace.

Limits and limitations

Metrics are ingested using the OpenTelemetry protocol (OTLP) via the Dynatrace OTLP API and are subject to the API's limits and restrictions. For more information see:

  • OpenTelemetry metrics limitations
  • Dynatrace metrics mapping

To avoid data duplication, make sure that only one OTel Collector scrapes a given target (for example, Kafka broker or Prometheus endpoint).

If you run multiple OTel Collector replicas, configure each one with a different target. This prevents duplicate metrics and unnecessary ingest costs.

The Target Allocator automatically distributes the Prometheus targets among a pool of OTel Collectors.

Related topics

  • Scrape Prometheus metrics with the OTel Collector (standard)
  • Enrich ingested data with Dynatrace-specific fields
  • Enrich OTLP requests with Kubernetes data
  • Prometheus data source
Related tags
Application Observability