Migrate from the Dynatrace OpenTelemetry metrics exporter to OTLP metrics exporter

The following sections describe how to migrate from the Dynatrace OpenTelemetry metrics exporter (now deprecated) to the OTLP HTTP exporter and highlight their differences and how to address them.

Why migrate to the OTLP HTTP exporter

Background

When the OpenTelemetry metrics signal was not yet stable, Dynatrace made it possible to ingest OpenTelemetry metrics by offering its own metric exporters in several programming languages. These exporters, which rely on the consolidated Dynatrace metric ingestion protocol, bridged the gap while the OpenTelemetry components were being stabilized.

Since then, the OpenTelemetry metrics signal has been declared stable and Dynatrace has added support for native OpenTelemetry protocol (OTLP) ingest for traces, metrics, and logs, making the Dynatrace custom metric exporters no longer needed.

Advantages of switching to the official OTLP HTTP exporter:

  • You can send metrics from all OpenTelemetry-supported programming languages, which wasn't possible before, as the Dynatrace metrics exporter wasn't available in all of them.
  • The complexity of your OpenTelemetry Collector configuration/pipelines is reduced. With a single OTLP HTTP exporter configuration, you can send all the signals to Dynatrace.

What is changing

With the stabilization of the OpenTelemetry metrics signal and the release of the Dynatrace OTLP metrics ingest API, the Dynatrace custom metrics exporters are now obsolete. As all language SDKs offer an OTLP exporter, it is no longer necessary to use the Dynatrace OpenTelemetry metrics exporter.

As a result:

  • The Dynatrace OpenTelemetry metrics exporters are deprecated and no longer recommended.
  • The exporters are still available but, after the end of 2023, no support, updates, or compatibility with newer OTel versions will be provided.

Recommendation

If you are using the Dynatrace OpenTelemetry metrics exporter, we recommend that you switch to the official OTLP HTTP exporter available from the OpenTelemetry project.

  • If you're using the Dynatrace OpenTelemetry metrics exporter in your applications, go to Migrate applications for instructions.
  • If you're using the Dynatrace OpenTelemetry metrics exporter in the OpenTelemetry Collector, go to Migrate collector configuration for instructions.

Get the new ingest URL

Before you start the migration, see OpenTelemetry metrics ingest API to get the URL for the OTLP metrics ingest API.

You can use the same token for the OTLP metrics ingest API. The required scope is still Ingest metrics (metrics.ingest).

Migrate applications

This section describes how to migrate your applications from the Dynatrace OpenTelemetry metrics exporter to the OTLP HTTP exporter.

1. Install and configure the OTLP HTTP exporter

Install the OTLP HTTP exporter package in your application. You can use the OpenTelemetry Registry to discover the packages.

After installing the exporter package, check the OpenTelemetry Integration walk-throughs for instructions on how to configure the OTLP HTTP exporter.

You can now uninstall the Dynatrace OpenTelemetry metrics exporter from your application.

2. Configure the endpoint and token

Apart from configuring the endpoint and token directly via code (as seen in the samples), the OTLP exporter can also be configured via environment variables.

Although the environment variables are the same for all OpenTelemetry SDK languages, the way they are consumed and configured in the SDKs changes. Consult the SDK documentation of your programming language to learn how to configure the OTLP exporter using environment variables.

Endpoint

  • When using the signal-agnostic environment variable, OTEL_EXPORTER_OTLP_ENDPOINT, use the OTLP metrics ingest API URL without the /v1/metrics suffix.

  • When using the metric-specific environment variable, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, use the full OTLP metrics ingest API URL.

Token

The Dynatrace token is transmitted via the Authorization HTTP header. Like the endpoint, the OpenTelemetry specification defines one environment variable that is signal-agnostic (OTEL_EXPORTER_OTLP_HEADERS) and one that is specific to metrics (OTEL_EXPORTER_OTLP_METRICS_HEADERS).

A benefit of using the metric-specific environment variable is that you isolate the tokens and reduce the required scope to the specific signal.

Regardless of the header environment variable you choose, you can set the value as in this example:

Authorization=Api-Token dt0c01.abc123.abcdefghij1234567890

For details on setting headers via the environment variable, see the OpenTelemetry documentation.

If your language supports configuring the OTLP exporter completely from environment variables (for example, Java), you also need to set the OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf, as Dynatrace only supports ingesting OTLP/protobuf data via HTTP.

3. Configure the exporter to generate metrics with delta aggregation temporality

Dynatrace only supports ingesting metrics with delta aggregation temporality.

To learn more about metric aggregation temporality and how to configure the OTLP exporter to produce delta metrics, see Aggregation temporality.

4. Attribute type support

When using the Dynatrace OpenTelemetry metrics exporter, only attributes of type string are sent to Dynatrace.

With the Dynatrace OTLP metrics ingest API, apart from string attributes, you can now also send attributes of type boolean and integer.

Attributes of type double, bytes, array, and map are currently not supported by Dynatrace and are dropped upon ingest.

Additional features

The following sections describe how to migrate the additional features available in the Dynatrace OpenTelemetry metrics exporter to the OTLP exporter.

Automatic endpoint and token configuration

When using the Dynatrace OpenTelemetry metrics exporter together with Dynatrace OneAgent, the endpoint and token were automatically detected and configured.

This is a Dynatrace-specific feature and is not available in the OTLP exporter. See Configure the endpoint and token to learn how to configure the endpoint and token.

Histogram support

Dynatrace version 1.300+

Starting with Dynatrace version 1.300, histograms ingestion is supported directly via OTLP ingest API. For Exponential Histogram, Dynatrace ingests the histogram's min|max|sum|count but doesn't ingest the buckets.

Metric Key Prefix

The automatic metric prefix is a Dynatrace-specific feature and is not available in the OTLP exporter.

In this case, you need to modify your metric definitions and add the prefix to them explicitly.

If you are using the OpenTelemetry Collector, an alternative is to use the Metrics Transform Processor to add the prefix to your metrics.

Default Dimensions

Although this feature does not exist in the OTLP exporter, the same result can be achieved in different ways:

  • Configure the default dimensions as resource or instrumentation scope attributes.

    After configuring the default attributes on the resource and/or instrumentation scope, you need to configure them in Dynatrace so they are added to all your metrics automatically. To learn more, see Configure resource and scope attributes to be added as dimensions.

    Resource and instrumentation scope attributes that are not configured as part of the allow list in Dynatrace are dropped upon ingest.

  • Configure the default dimensions via the OTEL_RESOURCE_ATTRIBUTES environment variable.

  • Configure the default dimensions in the OpenTelemetry Collector Metrics Transform Processor.

Automatically enrich metrics with Dynatrace OneAgent metadata

When using the Dynatrace OpenTelemetry metrics exporter together with Dynatrace OneAgent, metrics are automatically enriched with Dynatrace OneAgent metadata. This is a Dynatrace-specific feature and is not available in the OTLP exporter.

However, you can get the same result by accessing the OneAgent metadata and adding them as resource attributes.

See Enrich ingested data with Dynatrace-specific dimensions for instructions on how to enrich ingested metrics with Dynatrace-specific dimensions.

Migrate Collector configuration

This section describes how to migrate your Collector configuration from the Dynatrace OpenTelemetry metrics exporter to the OTLP HTTP exporter.

1. Configure the OTLP HTTP exporter

The OTLP HTTP exporter is part of the core OpenTelemetry Collector distribution, so there is no need to install any extra modules.

The OTLP HTTP exporter in the Collector can be used to send all signals (traces, metrics, and logs).

2. Configure the endpoint and token

The endpoint and token are required when using the OTLP HTTP exporter in the Collector.

Endpoint

The endpoint can be configured using two different properties:

  • Property endpoint

    The endpoint property refers to the base URL where data will be sent to. When using this property, the Collector will append the appropriate suffix (e.g., /v1/metrics) depending on the type of data being exported.

    If using this approach, use the OTLP metrics ingest API URL without the /v1/metrics suffix.

    exporters:
      otlphttp:
        endpoint: "https://{your-environment-id}.live.dynatrace.com/api/v2/otlp"

  • Property metrics_endpoint

    The metrics_endpoint refers to the signal-specific URL to which data will be sent. When using this property, the collector only sends metrics to the specified endpoint. This option is useful when the endpoints for the different signals are different.

    If using this approach, use the full OTLP metrics ingest API URL.

    exporters:
      otlphttp:
        metrics_endpoint: "https://{your-environment-id}.live.dynatrace.com/api/v2/otlp/v1/metrics"

    If metrics_endpoint is set in the configuration, the endpoint property is ignored for metrics.

Token

The Dynatrace token is transmitted via the Authorization HTTP header. The OTLP HTTP exporter in the collector offers a key-value list property that can be used to configure the Dynatrace token:

exporters:
  otlphttp:
    endpoint: "https://{your-environment-id}.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token dt0c01.abc123.abcdefghij1234567890"

Only one Dynatrace API token can be specified in the OTLP HTTP exporter in the Collector. Be sure to generate a token with all the scopes necessary for all signals you are exporting.

For the full list of available settings in the OTLP HTTP exporter, see the configuration sample in the OpenTelemetry Collector repository.

3. Make sure metrics have delta aggregation temporality

Dynatrace only supports ingesting metrics with delta aggregation temporality.

If your applications are already producing metrics with delta temporality (as explained in Configure the exporter to generate metrics with delta aggregation temporality), you don't need to do anything in your Collector configuration.

An alternative is to use the Cumulative to Delta processor as part of your Collector pipeline.

Caution: To be able to convert cumulative metrics to delta, the Collector needs to keep the previous values of metrics. In a multiple-instance Collector deployment, this conversion can produce inconsistent data unless it can be guaranteed that metrics from the same source are always processed by the same Collector instance.

For this reason, we recommend that you configure the OTLP exporter in your applications to output delta metrics to the Collector.

4. Attribute type support

When using the Dynatrace OpenTelemetry metrics exporter in the Collector, all attributes were converted to string by using the Collector's internal transformation logic.

This logic transformed the attribute types as follows:

  • boolean, integer, double: converted to their string representation
  • bytes: converted to a base64 string
  • array and map: converted to a JSON string

If you are recording attributes of type double, bytes, array, or map, when migrating to the OTLP exporter in the Collector you must use the Attributes Processor in your pipeline so the attributes continue to be exported to Dynatrace as before.

processors:
  attributes/convert:
    actions:
      - key: my.double.attribute
        action: convert
        converted_type: string
      - key: my.byte.attribute
        action: convert
        converted_type: string
      - key: my.array.attribute
        action: convert
        converted_type: string
      - key: my.map.attribute
        action: convert
        converted_type: string

Attributes of types double, bytes, array, and map are currently not supported by Dynatrace.

If you are recording attributes of these types and you don't configure the attributes processor in your Collector pipeline, these attributes are dropped upon ingest.

Additional features

The following sections describe how to migrate the additional features available in the Dynatrace OpenTelemetry metrics exporter to the OTLP exporter.

Automatic Endpoint and Token configuration

When using the Dynatrace OpenTelemetry metrics exporter together with Dynatrace OneAgent, the endpoint and token were automatically detected and configured. This is a Dynatrace-specific feature and is not available in the OTLP exporter.

See Configure the endpoint and token to learn how to configure the endpoint and token in the Collector.

Histogram support

Dynatrace version 1.300+

Starting with Dynatrace version 1.300, histograms ingestion is supported directly via OTLP ingest API. For Exponential Histogram, Dynatrace ingests the histogram's min|max|sum|count but doesn't ingest the buckets.

Metric key prefix

The metric key prefix feature in the Dynatrace OpenTelemetry metrics exporter can be replaced by the Metrics Transform Processor in the Collector by using the "rename metric" feature:

# Add a prefix "myprefix" to a "system.cpu.usage" metric
include: system.cpu.usage
action: update
new_name: myprefix.system.cpu.usage_time

You can also rename multiple metrics by using a combination of regex/substitution. See the Rename multiple metrics using Substitution sample in the Collector repository for an example.

Default dimensions

The default dimensions feature in the Dynatrace OpenTelemetry metrics exporter can be replaced by the Metrics Transform Processor in the Collector by using the "add label" feature:

# Add a label `my.label` with value `my.value` to the 'system.cpu.usage' metric
include: system.cpu.usage
action: update
operations:
  - action: add_label
    new_label: my.label
    new_value: my.value

You can also add labels to multiple metrics by using a regex to select the metrics. See the Add a label to multiple metrics sample in the Collector repository for an example.

Resource to telemetry conversion

Although this feature does not exist in the OTLP exporter, the Dynatrace OTLP metrics ingest API enables you to define a list of allowed resource and instrumentation scope attributes that are then automatically added as dimensions to all ingested metrics.

For details, see Configure resource and scope attributes to be added as dimensions.

Related topics