Enable the OpenTelemetry Span Sensor for OneAgent

  • How-to guide
  • 5-min read

In addition to the application-side configuration, several Dynatrace-specific settings let you control how OpenTelemetry data is used in Dynatrace. These can be configured at Settings Settings > Collect and capture > General monitoring settings > OneAgent features.

To learn how to send OpenTelemetry data to a Dynatrace OneAgent, see Use OneAgent with OpenTelemetry data.

Prerequisites

Monitoring frameworkVersions
OpenTelemetry1.0 - 1.31, 1.4 - 1.541
OpenTracing0.33, 0.32, 0.31
1
Supported in AWS Lambda.

To enable OpenTelemetry Java

  1. Go to Settings Settings > Collect and capture > General monitoring settings > OneAgent features.
  1. Find and enable OpenTelemetry (Java).

Existing tracers are replaced and will no longer work after you enable OpenTelemetry Java.

Attribute redaction

The OneAgent code module's OpenTelemetry Span Sensor automatically captures all OpenTelemetry attributes. If you want to prevent the accidental storage of personal data, you can exclude specific attribute keys for which the values must not be persisted. By omitting attributes containing personal data, you can meet your organization's privacy requirements and control the scope of stored monitoring data.

To configure attribute storage and masking settings for your environment

  1. Go to Settings.
  2. Select Server-side service monitoring > Attribute capturing.
  3. optional To change the default OpenTelemetry attribute persistence, go to Preferences.
    • To store all attributes except the ones in the Blocked attributes list, select Allow all attributes
    • To block all attributes except the ones in the Allowed attributes list, select Block all attributes

    Only one setting preference is possible.

  4. Add an attribute name to the attribute list.
    1. On the Attribute capturing page, select Blocked attributes or Allowed attributes.

      Allowed attributes list Dynatrace recommends a few basic attributes to generally be included, such as service.name or service.version. For ease of use, Dynatrace comes with a default configuration that can be adjusted.

    2. Select Add item to add a new key to the attribute list and enter the key.
    3. Select Save changes.
  5. Perform the following actions to mask a stored attribute value.
    1. On the Attribute capturing page, select Attribute data masking.
    2. Select Add item to add a new key to the masked attributed list.
    3. Enter a stored value key and select an option from the Masking dropdown list. To learn more about masking options, see OpenTelemetry traces.
    4. Select Save changes.

You can then find the attribute key on the Distributed traces page on the Summary tab.

Trace search limitations

Resource attributes

Searching by resource attribute is limited to the service name: filter by Service name on the Distributed traces page.

Span attributes

Searching by span attribute is limited to the span name: filter by Request on the Distributed traces page.

How the Span Sensor works

For more information about the OneAgent code module's OpenTelemetry Span Sensor, see Detect OpenTelemetry spans using the OneAgent code module's OpenTelemetry Span Sensor.

Entry points

To avoid possible conflicts with existing PurePath distributed traces, OneAgent ingests by default only spans with a span kind of Server or Consumer. This usually is not an issue, as instrumentation libraries typically configure the appropriate span kind, however something to take into account if your application fully uses manual instrumentation.

This behavior can be customized with an entry point rule. To do that, in Dynatrace go to Settings Settings > Collect and capture > OpenTelemetry > Span entry points and create a new rule with the appropriate action and matcher entry.

Span hierarchy

Depending on your setup, you may experience "flat span enrichment". This is when spans are displayed in Dynatrace as a flat list instead of a tree hierarchy. While this generally is the default behavior with OneAgent ingestion of OpenTelemetry traces, the hierarchy may still reflect actual span relations as defined by the instrumentation, depending on the involved OneAgent code modules and their support for the instrumented technologies.

Leaf spans

When merging OpenTelemetry spans into OneAgent sensor traces, make sure that OpenTelemetry spans are leaf spans and not in-between OneAgent spans.

Attribute capturing

As OneAgent ingests spans already upon their creation, not all eventual attributes may be already present at the initial ingest. Any attributes added at a later point are highlighted in Dynatrace with an initial value not set note and cannot be used for span capture rules, as they were not yet available when the rules were being evaluated.

Context propagation

When ingesting OpenTelemetry traces automatically with the OneAgent Span Sensor, there is a difference between the context propagation of OpenTelemetry traces and OneAgent traces.

While propagation of OpenTelemetry traces may be already handled properly by your application, it is also important to consolidate them with the OneAgent-specific trace. This can be achieved with a context propagation rule. To configure this, in Dynatrace go to Settings Settings > Collect and capture > OpenTelemetry > Span context propagation and create a context propagation rule with a Propagate action and a matcher entry for the span in question (for example, based on the span name or instrumentation library).

Try to avoid trace consolidation for technologies already covered natively by OneAgent sensors. Merging such OpenTelemetry spans into a OneAgent trace may lead to undefined states.

Export to third-party backends while using OneAgent

While OpenTelemetry traces are always exported to other backends as is, a small data adjustment takes place when your OneAgent-instrumented application starts a fresh OpenTelemetry trace. This applies only to new traces and not when a trace is continued via context propagation.

In that case, OneAgent may have already created a new trace object when OpenTelemetry was initialized. If these two traces (with separate IDs) are not reconciled, telemetry data might be duplicated or fragmented. To mitigate this and still keep Dynatrace PurePath traces consistent, OneAgent uses the following approach:

  • The OpenTelemetry trace ID takes precedence in exports to third parties
  • On the Dynatrace backend, the PurePath trace ID is assigned instead

To enable correlation between these two IDs, Dynatrace creates additional span links for each span, linking to the OpenTelemetry trace.

The ID rewrite applies only to newly started traces (not context propagation) and to the OpenTelemetry SDKs for Go, Java, JavaScript, PHP, and Python but not to .NET.

Limitations

Java

  • OneAgent version 1.294 and earlier OneAgent replaces installed global OpenTelemetry SDK components TracerProvider, Propagator, and ContextManager. Therefore, with OpenTelemetry Java enabled, traces are no longer seen by this SDK or exported to backends like Jaeger.
  • OneAgent version 1.259+ To avoid duplicates, OneAgent will ignore spans from some automatic instrumentation libraries.
  • When OneAgent and OpenTelemetry sensors are both present for the same technology, you may experience additional overhead.

Go

  • OneAgent can only instrument Tracer implementation of the default OpenTelemetry SDK.
  • When both OneAgent and OpenTelemetry sensors are present for the same technology, you may experience the following limitations:
    • Duplicate nodes in distributed traces
    • Additional overhead

Node.js

  • In versions earlier than 1.261, OneAgent replaces installed global OpenTelemetry SDK components TracerProvider, Propagator, and ContextManager. Therefore, with OpenTelemetry Node.js enabled, traces are no longer seen by this SDK or exported to backends like Jaeger.
  • When OneAgent and OpenTelemetry instrument the same module (such as HTTP or GRPC), you may experience the following limitations:
    • Duplicate nodes in distributed traces
    • Disconnected distributed traces
    • Additional overhead

PHP

Python

  • When both OneAgent and OpenTelemetry sensors are present for the same technology, you may experience the following limitations:
    • Duplicate nodes in distributed traces
    • Disconnected distributed traces
    • Additional overhead

All languages

  • OneAgent captures OpenTelemetry resource attributes only if they are provided via the OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES environment variables. When using the OpenTelemetry trace ingest API, this limitation doesn't apply.
  • You can't create request attributes (commonly used for trace searching and filtering) based on OpenTelemetry resource attributes.
  • OneAgent truncates attribute values exceeding 4,096 characters.

Prevention of span duplication in Java

OneAgent version 1.259+

To avoid possible span duplicates for areas covered by OpenTelemetry and OneAgent, OneAgent skips spans from the following automatic instrumentation Java libraries if OneAgent is configured to instrument your Java application and ingest OpenTelemetry spans.

Such spans are skipped only by OneAgent. Exports to third parties (for example, other backends or the Collector) remain unaffected.

io.opentelemetry.akka-http-10.0

io.opentelemetry.apache-dbcp-2.0

io.opentelemetry.apache-httpasyncclient-4.1

io.opentelemetry.apache-httpclient-2.0

io.opentelemetry.apache-httpclient-4.0

io.opentelemetry.apache-httpclient-4.3

io.opentelemetry.apache-httpclient-5.0

io.opentelemetry.async-http-client-1.9

io.opentelemetry.async-http-client-2.0

io.opentelemetry.c3p0-0.9

io.opentelemetry.cassandra-3.0

io.opentelemetry.cassandra-4.0

io.opentelemetry.cassandra-4.4

io.opentelemetry.cxf-jaxrs-3.2

io.opentelemetry.google-http-client-1.19

io.opentelemetry.grpc-1.6

io.opentelemetry.http-url-connection

io.opentelemetry.java-http-client

io.opentelemetry.jaxrs-1.0

io.opentelemetry.jaxrs-1.0-common

io.opentelemetry.jaxrs-2.0-annotations

io.opentelemetry.jaxrs-2.0-common

io.opentelemetry.jaxrs-2.0-cxf-3.2

io.opentelemetry.jaxrs-2.0-jersey-2.0

io.opentelemetry.jaxrs-2.0-resteasy-3.0

io.opentelemetry.jaxrs-2.0-resteasy-3.1

io.opentelemetry.jaxrs-3.0-annotations

io.opentelemetry.jaxrs-3.0-jersey-3.0

io.opentelemetry.jaxrs-3.0-resteasy-6.0

io.opentelemetry.jaxrs-annotations-2.0

io.opentelemetry.jaxrs-annotations-3.0

io.opentelemetry.jaxrs-client-1.1

io.opentelemetry.jaxrs-client-2.0

io.opentelemetry.jaxrs-client-2.0-resteasy-3.0

io.opentelemetry.jaxws-2.0

io.opentelemetry.jaxws-2.0-axis2-1.6

io.opentelemetry.jaxws-2.0-cxf-3.0

io.opentelemetry.jaxws-2.0-metro-2.2

io.opentelemetry.jaxws-cxf-3.0

io.opentelemetry.jaxws-common

io.opentelemetry.jaxws-jws-api-1.1

io.opentelemetry.jdbc

io.opentelemetry.jedis-1.4

io.opentelemetry.jedis-3.0

io.opentelemetry.jedis-4.0

io.opentelemetry.jersey-2.0

io.opentelemetry.jetty-11.0

io.opentelemetry.jetty-8.0

io.opentelemetry.jetty-httpclient-9.2

io.opentelemetry.jms-1.1

io.opentelemetry.jms-3.0

io.opentelemetry.jsp-2.3

io.opentelemetry.kafka-clients

io.opentelemetry.kafka-clients-0.11

io.opentelemetry.kafka-clients-2.6

io.opentelemetry.kafka-streams-0.11

io.opentelemetry.lettuce-5.1

io.opentelemetry.liberty

io.opentelemetry.liberty-20.0

io.opentelemetry.mongo-3.1

io.opentelemetry.netty-3.8

io.opentelemetry.netty-4.0

io.opentelemetry.netty-4.1

io.opentelemetry.okhttp-2.2

io.opentelemetry.okhttp-3.0

io.opentelemetry.orcale-ucp-11.2 (sic!)

io.opentelemetry.rabbitmq-2.7

io.opentelemetry.reactor-kafka-1.0

io.opentelemetry.reactor-netty-1.0

io.opentelemetry.resteasy-3.0

io.opentelemetry.resteasy-3.1

io.opentelemetry.resteasy-6.0

io.opentelemetry.rmi

io.opentelemetry.servlet-2.2

io.opentelemetry.servlet-3.0

io.opentelemetry.servlet-5.0

io.opentelemetry.servlet-javax-common

io.opentelemetry.spring-jms-2.0

io.opentelemetry.spring-jms-6.0

io.opentelemetry.spring-kafka-2.7

io.opentelemetry.spring-rabbit-1.0

io.opentelemetry.spring-rmi-4.0

io.opentelemetry.spring-webflux-5.0

io.opentelemetry.spring-webflux-5.3

io.opentelemetry.spring-ws-2.0

io.opentelemetry.tomcat-10.0

io.opentelemetry.tomcat-7.0

io.opentelemetry.tomcat-jdbc

io.opentelemetry.undertow-1.4

io.opentelemetry.vibur-dbcp-11.0

Prevention of span duplication in PHP

OneAgent version 1.313+

To avoid possible span duplicates for areas covered by OpenTelemetry and OneAgent, OneAgent skips spans from the following automatic instrumentation PHP libraries if OneAgent is configured to instrument your PHP application and ingest OpenTelemetry spans.

Such spans are skipped only by OneAgent. Exports to third parties (for example, other backends or the Collector) remain unaffected.

io.opentelemetry.contrib.php.curl

io.opentelemetry.contrib.php.laravel

io.opentelemetry.contrib.php.mongodb

io.opentelemetry.contrib.php.mysqli

io.opentelemetry.contrib.php.pdo

io.opentelemetry.contrib.php.slim

io.opentelemetry.contrib.php.symfony

io.opentelemetry.contrib.php.wordpress

Related tags
Application Observability