OpenTelemetry traces with OneAgent
Opt-In
Dynatrace OneAgent code modules for Java, .NET, PHP, Node.js, and Go automatically collect OpenTelemetry span data and integrate it into end-to-end distributed traces.
The OneAgent code module enables you to:
Gain insights into third-party libraries or frameworks that aren’t natively covered by OneAgent but which come with OpenTelemetry pre-instrumentation.
Enrich monitoring data with project-specific additions (for example, custom instrumentation that adds business data or the capture of developer-specific diagnostics points).
Stitch together independent, unrelated transactions to extend end-to-end traces (for instance, by adding vendor-neutral custom instrumentation to gain business-process-specific or domain-specific end-to-end transactional insights).
The quality of the OpenTelemetry spans captured by OneAgent depends on the quality of instrumentation provided by the third-party library.
Prerequisites
| Monitoring framework | Versions |
|---|---|
| OpenTelemetry | 1.0 - 1.3, 1.4 - 1.32 |
| OpenTracing | 0.33, 0.32, 0.31 |
Enable OpenTelemetry integration in OneAgent code module
OneAgent automatically taps into traces exposed via OpenTelemetry custom- or pre-instrumentation and sends the telemetry data to the Dynatrace platform.
To enable OpenTelemetry Java
- In the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (Java).
Existing tracers are replaced and will no longer work after you enable OpenTelemetry Java.
To enable OpenTelemetry Go
- In the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (Go).
For OneAgent version 1.217 and earlier, the OpenTelemetry Go Sensor propagates Dynatrace context across processes only if Send W3C Trace Context HTTP headers is enabled.
- In the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Turn on Send W3C trace context HTTP headers.
Existing tracers are not affected by OneAgent OpenTelemetry for Go support.
To enable OpenTelemetry Node.js
- In the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (Node.js).
To enable OpenTelemetry PHP
- In the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (PHP).
Existing tracers are replaced and will no longer work after you enable OpenTelemetry for PHP.
To enable OpenTelemetry .NET
- In the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (.NET).
Existing tracers are not affected by OneAgent OpenTelemetry for .NET support.
See Span settings for all configuration options.
Export to third-party backends while using OneAgent
In earlier versions, OneAgent took control over OpenTelemetry data and disabled other configured exporters. This is no longer the case as of OneAgent version 1.261; exports to third parties are available across all languages supported by OneAgent (Go, Java, JavaScript, .NET, and PHP).
While OpenTelemetry traces are always exported to other backends as is, a small data adjustment takes place when your OneAgent-instrumented application starts a fresh OpenTelemetry trace. This applies only to new traces and not when a trace is continued via context propagation.
In that case, OneAgent may have already created a new trace object when OpenTelemetry was initialized. If these two traces (with separate IDs) are not reconciled, telemetry data might be duplicated or fragmented. To mitigate this and still keep Dynatrace PurePath traces consistent, OneAgent uses the following approach:
The OpenTelemetry trace ID takes precedence in exports to third parties
On the Dynatrace backend, the PurePath trace ID is assigned instead
To enable correlation between these two IDs, Dynatrace creates additional span links for each span, linking to the OpenTelemetry trace.
The ID rewrite applies only to newly started traces (not context propagation) and to the OpenTelemetry SDKs for Go, Java, JavaScript, and PHP, and not to .NET.
Limitations
Java
- In versions earlier than 1.259, OneAgent replaces installed global OpenTelemetry SDK components
TracerProvider,Propagator, andContextManager. Therefore, with OpenTelemetry Java enabled, traces are no longer seen by this SDK or exported to backends like Jaeger. - OneAgent version 1.259+ To avoid duplicates, OneAgent will ignore spans from some automatic instrumentation libraries.
When OneAgent and OpenTelemetry sensors are both present for the same technology, you may experience additional overhead.
- OneAgent currently does not support AOT-compiled applications. Only JIT-compiled applications can be instrumented.
Go
OneAgent can only instrument Tracer implementation of the default OpenTelemetry SDK.
- When both OneAgent and OpenTelemetry sensors are present for the same technology, you may experience the following limitations:
Duplicate nodes in distributed traces
Additional overhead
Node.js
- In versions earlier than 1.261, OneAgent replaces installed global OpenTelemetry SDK components
TracerProvider,Propagator, andContextManager. Therefore, with OpenTelemetry Node.js enabled, traces are no longer seen by this SDK or exported to backends like Jaeger. - When OneAgent and OpenTelemetry instrument the same module (such as HTTP or GRPC), you may experience the following limitations:
Duplicate nodes in distributed traces
Disconnected distributed traces
Additional overhead
All languages
- OneAgent captures OpenTelemetry resource attributes only if they are provided via the
OTEL_SERVICE_NAMEandOTEL_RESOURCE_ATTRIBUTESenvironment variables. When using the OpenTelemetry trace ingest API, this limitation doesn't apply. - You can't create request attributes (commonly used for trace searching and filtering) based on OpenTelemetry resource attributes.
OneAgent truncates attribute values exceeding 4,096 characters.
Prevention of span duplication in Java
OneAgent version 1.259+
To avoid possible span duplicates for areas covered by OpenTelemetry and OneAgent, OneAgent skips spans from the following automatic instrumentation Java libraries if OneAgent is configured to instrument your Java application and ingest OpenTelemetry spans.
Such spans are skipped only by OneAgent. Exports to third parties (for example, other backends or the Collector) remain unaffected.
io.opentelemetry.akka-http-10.0 | io.opentelemetry.apache-dbcp-2.0 | io.opentelemetry.apache-httpasyncclient-4.1 |
io.opentelemetry.apache-httpclient-2.0 | io.opentelemetry.apache-httpclient-4.0 | io.opentelemetry.apache-httpclient-4.3 |
io.opentelemetry.apache-httpclient-5.0 | io.opentelemetry.async-http-client-1.9 | io.opentelemetry.async-http-client-2.0 |
io.opentelemetry.c3p0-0.9 | io.opentelemetry.cassandra-3.0 | io.opentelemetry.cassandra-4.0 |
io.opentelemetry.cassandra-4.4 | io.opentelemetry.cxf-jaxrs-3.2 | io.opentelemetry.google-http-client-1.19 |
io.opentelemetry.grpc-1.6 | io.opentelemetry.http-url-connection | io.opentelemetry.java-http-client |
io.opentelemetry.jaxrs-1.0 | io.opentelemetry.jaxrs-1.0-common | io.opentelemetry.jaxrs-2.0-annotations |
io.opentelemetry.jaxrs-2.0-common | io.opentelemetry.jaxrs-2.0-cxf-3.2 | io.opentelemetry.jaxrs-2.0-jersey-2.0 |
io.opentelemetry.jaxrs-2.0-resteasy-3.0 | io.opentelemetry.jaxrs-2.0-resteasy-3.1 | io.opentelemetry.jaxrs-3.0-annotations |
io.opentelemetry.jaxrs-3.0-jersey-3.0 | io.opentelemetry.jaxrs-3.0-resteasy-6.0 | io.opentelemetry.jaxrs-annotations-2.0 |
io.opentelemetry.jaxrs-annotations-3.0 | io.opentelemetry.jaxrs-client-1.1 | io.opentelemetry.jaxrs-client-2.0 |
io.opentelemetry.jaxrs-client-2.0-resteasy-3.0 | io.opentelemetry.jaxws-2.0 | io.opentelemetry.jaxws-2.0-axis2-1.6 |
io.opentelemetry.jaxws-2.0-cxf-3.0 | io.opentelemetry.jaxws-2.0-metro-2.2 | io.opentelemetry.jaxws-common |
io.opentelemetry.jaxws-jws-api-1.1 | io.opentelemetry.jdbc | io.opentelemetry.jedis-1.4 |
io.opentelemetry.jedis-3.0 | io.opentelemetry.jedis-4.0 | io.opentelemetry.jersey-2.0 |
io.opentelemetry.jetty-11.0 | io.opentelemetry.jetty-8.0 | io.opentelemetry.jetty-httpclient-9.2 |
io.opentelemetry.jms-1.1 | io.opentelemetry.jms-3.0 | io.opentelemetry.jsp-2.3 |
io.opentelemetry.kafka-clients | io.opentelemetry.kafka-clients-0.11 | io.opentelemetry.kafka-clients-2.6 |
io.opentelemetry.kafka-streams-0.11 | io.opentelemetry.lettuce-5.1 | io.opentelemetry.liberty |
io.opentelemetry.liberty-20.0 | io.opentelemetry.mongo-3.1 | io.opentelemetry.netty-3.8 |
io.opentelemetry.netty-4.0 | io.opentelemetry.netty-4.1 | io.opentelemetry.okhttp-2.2 |
io.opentelemetry.okhttp-3.0 | io.opentelemetry.orcale-ucp-11.2 (sic!) | io.opentelemetry.rabbitmq-2.7 |
io.opentelemetry.reactor-kafka-1.0 | io.opentelemetry.reactor-netty-1.0 | io.opentelemetry.resteasy-3.0 |
io.opentelemetry.resteasy-3.1 | io.opentelemetry.resteasy-6.0 | io.opentelemetry.rmi |
io.opentelemetry.servlet-2.2 | io.opentelemetry.servlet-3.0 | io.opentelemetry.servlet-5.0 |
io.opentelemetry.servlet-javax-common | io.opentelemetry.spring-jms-2.0 | io.opentelemetry.spring-jms-6.0 |
io.opentelemetry.spring-kafka-2.7 | io.opentelemetry.spring-rabbit-1.0 | io.opentelemetry.spring-rmi-4.0 |
io.opentelemetry.spring-webflux-5.0 | io.opentelemetry.spring-webflux-5.3 | io.opentelemetry.spring-ws-2.0 |
io.opentelemetry.tomcat-10.0 | io.opentelemetry.tomcat-7.0 | io.opentelemetry.tomcat-jdbc |
io.opentelemetry.undertow-1.4 | io.opentelemetry.vibur-dbcp-11.0 |