Compute histogram summaries with the OTel Collector

• How-to guide
• 3-min read
• Published Apr 08, 2024

This page describes how to ingest histograms via the OTel Collector, which can help you to reduce costs associated with ingestion.

You can also use the OTLP ingest API to ingest histograms directly, without additional data manipulation.

The following configuration example shows how to use the Collector to compute and ingest summaries of histogram buckets, such as the overall sum of all values in the bucket and their total count.

Prerequisites

See Collector Deployment and Collector Configuration on how to set up your Collector with the configuration below.

Demo configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  transform:
    metric_statements:
      - context: metric
        statements:
          # Get count from the histogram. The new metric name will be <histogram_name>_count
          - extract_count_metric(true) where type == METRIC_DATA_TYPE_HISTOGRAM

          # Get sum from the histogram. The new metric name will be <histogram_name>_sum
          - extract_sum_metric(true) where type == METRIC_DATA_TYPE_HISTOGRAM
          # convert the <histogram_name>_sum metrics to gauges.
          - convert_sum_to_gauge() where IsMatch(name, ".*_sum")

  filter:
    metrics:
      metric:
        # Drop metrics of type histogram. The _count and _sum metrics will still be exported.
        - type == METRIC_DATA_TYPE_HISTOGRAM

exporters:
  otlp_http:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [transform,filter]
      exporters: [otlp_http]

Components

For our configuration, we configure the following components.

Receivers

Under receivers, we specify the standard otlp receiver as active receiver component for our Collector instance and configure it to accept OTLP requests on gRPC and HTTP.

Processors

Under processors, we configure the following two processor instances:

With this configuration, the Collector drops histogram metrics and creates in their place two new metrics for the sum and item count of each respective histogram.

Exporters

Under exporters, we specify the default otlp_http exporter and configure it with our Dynatrace API URL and the required authentication token.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization:

Service pipelines

Under service, we assemble our receiver and exporter objects into a metric pipeline and enable the two processors by referencing them under processors.

Limits and limitations

Metrics are ingested using the OpenTelemetry protocol (OTLP) via the Dynatrace OTLP API and are subject to the API's limits and restrictions. For more information see: