OpenTelemetry traces with OneAgent
Dynatrace OneAgent code modules for Java, .NET, PHP, Node.js, and Go automatically collect OpenTelemetry span data and integrate it into end-to-end distributed traces.
The OneAgent code module enables you to:
- Gain insights into third-party libraries or frameworks that aren’t natively covered by OneAgent but which come with OpenTelemetry pre-instrumentation.
- Enrich monitoring data with project-specific additions (for example, custom instrumentation that adds business data or the capture of developer-specific diagnostics points).
- Stitch together independent, unrelated transactions to extend end-to-end traces (for instance, by adding vendor-neutral custom instrumentation to gain business-process-specific or domain-specific end-to-end transactional insights).
The quality of the OpenTelemetry spans captured by OneAgent depends on the quality of instrumentation provided by the third-party library.
Prerequisites
Enable OpenTelemetry integration in OneAgent code module
OneAgent automatically taps into traces exposed via OpenTelemetry custom- or pre-instrumentation and sends the telemetry data to the Dynatrace platform.
Except for Java (where enabled by default), automatic ingestion of OpenTelemetry traces is currently an opt-in feature and needs to be specifically enabled by the user for OneAgent to ingest traces with the following platforms:
- Go
- .NET
- Node.js
- PHP
To enable OpenTelemetry Java
- Go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (Java).
Existing tracers are replaced and will no longer work after you enable OpenTelemetry Java.
Opt-in
To enable OpenTelemetry Go
- Go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (Go).
For OneAgent version 1.217 and earlier, the OpenTelemetry Go Sensor propagates Dynatrace context across processes only if Send W3C Trace Context HTTP headers is enabled.
- Go to Settings > Preferences > OneAgent features.
- Turn on Send W3C trace context HTTP headers.
Existing tracers are not affected by OneAgent OpenTelemetry for Go support.
Opt-in
To enable OpenTelemetry Node.js
- Go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (Node.js).
Opt-in
To enable OpenTelemetry PHP
- Go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (PHP).
Existing tracers are not affected by OneAgent OpenTelemetry for PHP support.
Opt-in
To enable OpenTelemetry .NET
- Go to Settings > Preferences > OneAgent features.
- Find and enable OpenTelemetry (.NET).
Existing tracers are not affected by OneAgent OpenTelemetry for .NET support.
See Span settings for all configuration options.
Differences between OneAgent ingestion and OTLP exports
Both, automatic ingestion using OneAgent and OTLP exports, are about the same underlying goal of sending traces and their spans to Dynatrace, but there are subtle differences between the two approaches. These primarily apply to the following areas:
- How spans are captured and reported to Dynatrace.
- What is considered an entry span and how they are ingested.
- The span hierarchy.
- The processing of any possibly present attributes.
Point of ingestion
One major difference between OneAgent ingestion and OTLP exports is the point of ingestion. When exporting using OTLP, the exporter sends the span information to the backend only once the span has finished.
OneAgent, on the other hand, ingests span information the moment the span is created and OneAgent parsed it. As this span object may still only contain preliminary details, OneAgent will keep updating the backend on any changes to the span taking place throughout the flow of the trace.
Entry points
Another difference is how entry point spans (the first span of a trace) are handled. To avoid possible conflicts with existing PurePath traces, OneAgent ingests by default only spans with a span kind of Server or Consumer. This usually is not an issue, as instrumentation libraries typically configure the appropriate span kind, however something to take into account if your application fully uses manual instrumentation.
This behavior can be customized with an entry point rule. To do that, in Dynatrace go to Settings > Server-side service monitoring > Span entry points and create a new rule with the appropriate action and matcher entry.
Span hierarchy
Depending on your setup, you may also experience "flat span enrichment". This is when spans are displayed in Dynatrace as a flat list instead of a tree hierarchy. While this generally is the default behavior with OneAgent ingestion of OpenTelemetry traces, the hierarchy may still reflect the actual span relations as defined by the instrumention, depending on the involved OneAgent code modules and their support for the instrumented technologies.
When merging OpenTelemetry spans into OneAgent sensor traces, make sure that OpenTelemetry spans are leaf spans and not in-between OneAgent spans.
Attribute capturing
As OneAgent ingests spans already upon their creation, not all eventual attributes may be already present at the initial ingest. Any attributes added at a later point are highlighted in Dynatrace with an initial value not set note and cannot be used for span capture rules, as they were not yet available when the rules were being evaluated.
Context propagation
When ingesting OpenTelemetry traces automatically with the OneAgent span sensor, there is a difference between context propagation of OpenTelemetry traces and OneAgent traces.
While propagation of OpenTelemetry traces may be already handled properly by your application, it is also important to consolidate them with the OneAgent-specific trace. This can be achieved with a context propagation rule. To configure this, in Dynatrace go to Settings > Server-side service monitoring > Span context propagation and create a context propagation rule with a Propagate action and a matcher entry for the span in question (for example, based on the span name or instrumentation library).
Try to avoid trace consolidation for technologies already covered natively by OneAgent sensors. Merging such OpenTelemetry spans into a OneAgent trace may lead to undefined states.
Export to third-party backends while using OneAgent
In earlier versions, OneAgent took control over OpenTelemetry data and disabled other configured exporters. This is no longer the case as of OneAgent version 1.261; exports to third parties are available across all languages supported by OneAgent (Go, Java, JavaScript, .NET, and PHP).
While OpenTelemetry traces are always exported to other backends as is, a small data adjustment takes place when your OneAgent-instrumented application starts a fresh OpenTelemetry trace. This applies only to new traces and not when a trace is continued via context propagation.
In that case, OneAgent may have already created a new trace object when OpenTelemetry was initialized. If these two traces (with separate IDs) are not reconciled, telemetry data might be duplicated or fragmented. To mitigate this and still keep Dynatrace PurePath traces consistent, OneAgent uses the following approach:
- The OpenTelemetry trace ID takes precedence in exports to third parties
- On the Dynatrace backend, the PurePath trace ID is assigned instead
To enable correlation between these two IDs, Dynatrace creates additional span links for each span, linking to the OpenTelemetry trace.
The ID rewrite applies only to newly started traces (not context propagation) and to the OpenTelemetry SDKs for Go, Java, JavaScript, and PHP, and not to .NET.
Limitations
Java
- In versions earlier than 1.259, OneAgent replaces installed global OpenTelemetry SDK components
TracerProvider,Propagator, andContextManager. Therefore, with OpenTelemetry Java enabled, traces are no longer seen by this SDK or exported to backends like Jaeger. - OneAgent version 1.259+ To avoid duplicates, OneAgent will ignore spans from some automatic instrumentation libraries.
- When OneAgent and OpenTelemetry sensors are both present for the same technology, you may experience additional overhead.
Go
- OneAgent can only instrument Tracer implementation of the default OpenTelemetry SDK.
- When both OneAgent and OpenTelemetry sensors are present for the same technology, you may experience the following limitations:
- Duplicate nodes in distributed traces
- Additional overhead
Node.js
- In versions earlier than 1.261, OneAgent replaces installed global OpenTelemetry SDK components
TracerProvider,Propagator, andContextManager. Therefore, with OpenTelemetry Node.js enabled, traces are no longer seen by this SDK or exported to backends like Jaeger. - When OneAgent and OpenTelemetry instrument the same module (such as HTTP or GRPC), you may experience the following limitations:
- Duplicate nodes in distributed traces
- Disconnected distributed traces
- Additional overhead
All languages
- OneAgent captures OpenTelemetry resource attributes only if they are provided via the
OTEL_SERVICE_NAMEandOTEL_RESOURCE_ATTRIBUTESenvironment variables. When using the OpenTelemetry trace ingest API, this limitation doesn't apply. - You can't create request attributes (commonly used for trace searching and filtering) based on OpenTelemetry resource attributes.
- OneAgent truncates attribute values exceeding 4,096 characters.
Prevention of span duplication in Java
OneAgent version 1.259+
To avoid possible span duplicates for areas covered by OpenTelemetry and OneAgent, OneAgent skips spans from the following automatic instrumentation Java libraries if OneAgent is configured to instrument your Java application and ingest OpenTelemetry spans.
Such spans are skipped only by OneAgent. Exports to third parties (for example, other backends or the Collector) remain unaffected.
io.opentelemetry.akka-http-10.0 | io.opentelemetry.apache-dbcp-2.0 | io.opentelemetry.apache-httpasyncclient-4.1 |
io.opentelemetry.apache-httpclient-2.0 | io.opentelemetry.apache-httpclient-4.0 | io.opentelemetry.apache-httpclient-4.3 |
io.opentelemetry.apache-httpclient-5.0 | io.opentelemetry.async-http-client-1.9 | io.opentelemetry.async-http-client-2.0 |
io.opentelemetry.c3p0-0.9 | io.opentelemetry.cassandra-3.0 | io.opentelemetry.cassandra-4.0 |
io.opentelemetry.cassandra-4.4 | io.opentelemetry.cxf-jaxrs-3.2 | io.opentelemetry.google-http-client-1.19 |
io.opentelemetry.grpc-1.6 | io.opentelemetry.http-url-connection | io.opentelemetry.java-http-client |
io.opentelemetry.jaxrs-1.0 | io.opentelemetry.jaxrs-1.0-common | io.opentelemetry.jaxrs-2.0-annotations |
io.opentelemetry.jaxrs-2.0-common | io.opentelemetry.jaxrs-2.0-cxf-3.2 | io.opentelemetry.jaxrs-2.0-jersey-2.0 |
io.opentelemetry.jaxrs-2.0-resteasy-3.0 | io.opentelemetry.jaxrs-2.0-resteasy-3.1 | io.opentelemetry.jaxrs-3.0-annotations |
io.opentelemetry.jaxrs-3.0-jersey-3.0 | io.opentelemetry.jaxrs-3.0-resteasy-6.0 | io.opentelemetry.jaxrs-annotations-2.0 |
io.opentelemetry.jaxrs-annotations-3.0 | io.opentelemetry.jaxrs-client-1.1 | io.opentelemetry.jaxrs-client-2.0 |
io.opentelemetry.jaxrs-client-2.0-resteasy-3.0 | io.opentelemetry.jaxws-2.0 | io.opentelemetry.jaxws-2.0-axis2-1.6 |
io.opentelemetry.jaxws-2.0-cxf-3.0 | io.opentelemetry.jaxws-2.0-metro-2.2 | io.opentelemetry.jaxws-common |
io.opentelemetry.jaxws-jws-api-1.1 | io.opentelemetry.jdbc | io.opentelemetry.jedis-1.4 |
io.opentelemetry.jedis-3.0 | io.opentelemetry.jedis-4.0 | io.opentelemetry.jersey-2.0 |
io.opentelemetry.jetty-11.0 | io.opentelemetry.jetty-8.0 | io.opentelemetry.jetty-httpclient-9.2 |
io.opentelemetry.jms-1.1 | io.opentelemetry.jms-3.0 | io.opentelemetry.jsp-2.3 |
io.opentelemetry.kafka-clients | io.opentelemetry.kafka-clients-0.11 | io.opentelemetry.kafka-clients-2.6 |
io.opentelemetry.kafka-streams-0.11 | io.opentelemetry.lettuce-5.1 | io.opentelemetry.liberty |
io.opentelemetry.liberty-20.0 | io.opentelemetry.mongo-3.1 | io.opentelemetry.netty-3.8 |
io.opentelemetry.netty-4.0 | io.opentelemetry.netty-4.1 | io.opentelemetry.okhttp-2.2 |
io.opentelemetry.okhttp-3.0 | io.opentelemetry.orcale-ucp-11.2 (sic!) | io.opentelemetry.rabbitmq-2.7 |
io.opentelemetry.reactor-kafka-1.0 | io.opentelemetry.reactor-netty-1.0 | io.opentelemetry.resteasy-3.0 |
io.opentelemetry.resteasy-3.1 | io.opentelemetry.resteasy-6.0 | io.opentelemetry.rmi |
io.opentelemetry.servlet-2.2 | io.opentelemetry.servlet-3.0 | io.opentelemetry.servlet-5.0 |
io.opentelemetry.servlet-javax-common | io.opentelemetry.spring-jms-2.0 | io.opentelemetry.spring-jms-6.0 |
io.opentelemetry.spring-kafka-2.7 | io.opentelemetry.spring-rabbit-1.0 | io.opentelemetry.spring-rmi-4.0 |
io.opentelemetry.spring-webflux-5.0 | io.opentelemetry.spring-webflux-5.3 | io.opentelemetry.spring-ws-2.0 |
io.opentelemetry.tomcat-10.0 | io.opentelemetry.tomcat-7.0 | io.opentelemetry.tomcat-jdbc |
io.opentelemetry.undertow-1.4 | io.opentelemetry.vibur-dbcp-11.0 |