Cross-environment tracing

In multiple environment scenarios, data pools are separated. If a request affects services that are monitored in different Dynatrace environments, its traces aren't automatically correlated.

To correlate and follow such traces, you can configure a connection between the Dynatrace environments and enable the response header and coordinated sampling switches. The service flow then represents the connection between the connected environments, and distributed traces link to the connected remote environment.

Configure a connection between environments (via UI)

Create an access token

Add the remote environment

Enable the features

Create an access token

With this procedure, you get two items that you need in the other steps:

An access token from the remote environment
The URI of the remote Dynatrace environment

To create an access token in the remote Dynatrace environment

In the Dynatrace menu, go to Access tokens.
Select Generate new token.
Enter a token name.
Find and select the following scopes:
- Look up a single trace (traces.lookup)
- Fetch data from a remote environment (RestRequestForwarding)
Select Generate token.
This generates a token that gives your local environment permission to check the existence of a trace in a remote environment and to pull data from a remote environment.
Select Copy and then paste the token to a secure location.
It's a long string that you need to copy and paste back into Dynatrace later.
While you're signed in to the remote environment, copy the URI of the remote environment (for example, https://dynatrace.mybostonsite.com) from the browser address line.

Add the remote environment

To add the remote Dynatrace environment to the list of available remote environments in your current environment

In the Dynatrace menu, go to Settings.
Select Integration > Remote environments.
Select Connect environment.
Define the remote environment and then select Save changes.
- Name is the name under which the remote environment will be listed in your current Dynatrace environment. This is freeform text. It doesn't affect the remote environment.
- Remote environment URI is the URI you copied from the browser address line in the previous procedure.
- Network scope
  - External: The remote environment is located in another network.
  - Internal: The remote environment is located in the same network.
  - Cluster: The remote environment is located in the same cluster.
  Dynatrace SaaS can only use the External network scope.
- Token is the token you generated in the previous procedure. It needs to include the Look up a single trace scope (traces.lookup).
- Test connection checks the connection from your current environment to the remote environment.

Enable the features

After connecting your environments, enable the following features in all connected environments:

Coordinated sampling

You can increase the consistency of cross-environment tracing with coordinated sampling. When you enable the switch, sampling is coordinated across environments with the help of the W3C trace ID.

For this to work, all connected environments must have both coordinated sampling and W3C trace context propagation enabled.
1. In the Dynatrace menu, go to Settings > Preferences > OneAgent features.
2. Set W3C trace context switch according to your preferences.
  You can send W3C trace context HTTP headers or gRPC headers.
3. Find and turn on Cross-environment tracing - Coordinated sampling.
4. Apply this configuration in each environment of the group that you want to connect.
Response headers

You can enable the identification and tracking of cross-environment calls, including the environment ID and the trace ID in HTTP response headers. If the calling environment receiving the response has the environment configured as a trusted remote environment, the call destination is linked.
1. In the Dynatrace menu, go to Settings and select Preferences > OneAgent features.
2. Find and turn on Cross-environment tracing - Environment and transaction IDs in HTTP response headers.

Requirements

Make sure that a direct network connection between the clusters can be established via HTTP/HTTPs.

Limitations

Cross-environment tracing is limited to traces of requests that can transfer information about response headers and trace context from the receiving environment, such as HTTP or synchronous requests.
IBM z/OS (CICS, IMS, and Java) and AWS Lambda don't support cross-environment tracing.

Troubleshooting

The message Verification failed, please check your settings: Constraints violated. indicates a configuration error.

What to verify:

Name—this is freeform text, so it doesn't matter for functionality, but check it to make sure you are looking at the right environment.
Remote environment URI—copy and paste the contents of this field to a new browser tab and make sure it opens the correct Dynatrace environment.
Token—make sure you copied and pasted the entire token to this field. If you have lost this token, you can't go back and copy it again, but you can create a new one and get the token from it. For details, see Create an access token above.

Configure a connection between environments (via API)

The previous procedures use the Dynatrace web UI. To carry out the equivalent tasks via API, see:

Access tokens API—to create a token in the remote environment
Remote environments API—to create a link to the remote environment from the local environment

Analyze cross-connected remote environments

Once you configure a connection between environments, the following specific analysis options are available.

Service overview

Dynatrace detects requests to connected Dynatrace environments and uses the available information to create an associated Remote environment web service or web request service.

Outgoing calls to connected environments are represented in Smartscape and the service flow.
All the services monitored by the remote environment and contributing to the request are listed on the Remote environments page. To open it, select Open environment on a Remote environment service page.

Service flow

Service flow recognizes remote environment calls. Because the requests are monitored by another Dynatrace environment, a specific drill-down option is available when you select the remote environment call. In the side pane, select Open environment to access the Remote environments page.

Service flow for cross-connected remote environments

Distributed traces

When a request goes to a connected remote environment, Dynatrace displays the remote environment name in the call tree.

To follow the trace in the remote environment
1. Select the remote environment request.
2. In the Summary tab, select Open environment.
If the environments' connection is configured correctly, Dynatrace automatically displays the call Aggregated requests. It aggregates all the data of detected requests on each respective connected remote environment.

If you configured a connection to the remote environment but you can't see requests on it, you can trigger a manual search for the trace in configured environments. In the upper-right corner of the trace page, select Find in remote environments.