OpenTelemetry troubleshooting
General
Setup issues
To resolve issues with your code or the configuration of your application, check the following areas:
- Imports and dependencies: make sure that all necessary packages are up to date, imported into the code, and can be used.
- Exporter: check for errors in your endpoint or access token configuration.
- SDK initialization: make sure that the SDK is being initialized before any library is used.
- Spans: check whether all manually created spans are ended, even in case of exceptions, errors, or other unpredicted behavior.
- Logs: review any logs you might have for error messages.
- Collector: if you're using the Collector, make sure the OTLP/HTTP exporter is used in your pipelines to send data to Dynatrace.
Connection issues
If you encounter connection issues:
- Make sure you have configured the correct URL, matching your environment.
See if you have a network security layer (for example, a firewall or a proxy) blocking connections to port 443.
Make sure you can reach the ActiveGate instance (for example, using ping).
Make sure your system's root SSL certificates are up to date, and apply OS updates if necessary.
When exporting to a local Environment ActiveGate instance, remember that this setup uses a self-signed certificate for the HTTPS endpoint by default. If you encounter SSL issues, do one of the following:
- recommended Add the certificate to your exporter's truststore.
Configure your exporter to ignore the broken certificate chain and continue with the SSL connection. This option is less secure.
- The EEC endpoint is only available with Full-Stack and Infrastructure Monitoring, but not in container environments. Use ActiveGate instead.
- Make sure you've configured the correct
http://localhostURL. - Make sure you're using the correct port (default
14499).
Authentication issues
When you send OTLP requests to ActiveGate and receive a 403 error, this indicates that the provided token either expired or was not configured with the necessary permission scopes.
Verify that the token:
- Has the necessary permission scopes. For details, see Authentication.
Is still within its validity period and did not expire.
If necessary, issue a new token and change your exporter configuration accordingly.
Ingestion issues
If you can't export OTLP data, make sure your exporter is configured:
- To use OTLP over HTTP. OTLP over gRPC is currently not supported.
To use Protobuf encoding. JSON is currently not supported.
- With the correct Dynatrace API URL (ActiveGate or OneAgent) and token (ActiveGate only).
Vertical topology
Vertical topology depends on additional information provided by OneAgent. If OneAgent isn't installed, Dynatrace can't report details on the vertical topology.
If OTLP requests aren't routed via OneAgent (implicitly via automatic ingestion or explicitly via an OTLP export to EEC), make sure you manually enrich your requests with the information from the enrichment files.
Traces
This is most likely caused by an error in context propagation on the code level.
Make sure the network libraries used in your application are covered by applicable instrumentation libraries and, especially if you instrument manually, that the context is properly injected into outbound requests, and extracted from inbound requests.
For details and samples on context propagation for the different supported languages, see Integration walk-throughs.
There are at least two potential causes of missing span attributes:
- If the attribute key is listed but the value is missing, check your privacy settings and make sure that the attribute you want to see is present on the allowlist. To learn how to manage your allowlist, see Attribute redaction.
If the attribute key is also missing, check the attribute definition in your code.
Metrics
Make sure that you're not exceeding the maximum request size of 4 MB.
Make sure that all your metric values are configured for delta temporality (cumulative is not yet supported).
Do your metrics include histograms? These are currently unsupported.
To get more information on how many metrics were ingested and why some were rejected, you can take a look at the specific ingestion metrics. For details, see Dynatrace ingestion metrics.
Dynatrace currently does not support cumulative aggregation. Make sure to configure a delta temporality.
Dynatrace currently does not support histograms.
A "Partial Success" response indicates that only parts of the metrics in your request were valid, while some of the metrics either had to be normalized by Dynatrace or were rejected altogether. For details, see the list of warnings (for normalized metrics) and errors (for rejected metrics) in the response.
For example, verify that your metrics:
- Use the delta temporality and no cumulative aggregation
Are not histograms
Do not contain more than 50 dimensions/attributes
Follow all applicable naming conventions
For details, see OpenTelemetry Metrics Limitations.
Metric attributes are ingested by default. However, additional configuration is needed for resource and scope attributes.
To also ingest resource and scope attributes, you need to make sure the attribute name is on the allowlist and not on the deny list.
Attributes also need to be within the limits described at OpenTelemetry Metrics Limitations.