Try it free

Distributed Tracing app

  • Latest Dynatrace
  • Explanation
  • 6-min read

Distributed Tracing Distributed Tracing enhances Dynatrace abilities to analyze and filter trace data at both the request and span levels. With advanced filtering options, such as facets and the grouping function, you can easily explore and pinpoint issues. In addition, your selection is preserved across interactions, enabling you to drill down to the single trace information, and to identify the root cause and prevent failures in your environments.

In Dynatrace, go to Distributed Tracing Distributed Tracing to access this app.

  • In the Distributed Tracing welcome view, you can get started discovering the app and getting data into Dynatrace. Alternatively, you can add trace data at any time by selecting Traces and choosing a source. Follow the in-product guidance to continue the configuration for the selected source.
  • The default view Explorer contains all the user-interface elements to analyze your trace data.

Requests and spans

A distributed trace is a collection of spans representing a request's journey through a distributed system.

Requests

The request is the call initiated by a user or system to perform a specific task. It interacts with various services and components within the distributed system.

To view trace data created in your environment in response to an external request, select Requests in the upper-left corner of Distributed Tracing Distributed Tracing.

Spans

Spans are individual operations representing each request interaction with the distributed system.

To view all trace data by single operations, select Spans in the upper-left corner of Distributed Tracing Distributed Tracing.

Filter field

By entering a query in the filter field, you can quickly build DQL-based filtering options.

"Kubernetes namespace" = prod AND Endpoint = /cart/* AND "Response time" >= 5s

You can narrow your results by timeframe or segments.

  • To refresh the result for the selected timeframe, select Refresh.
  • To choose segments, select .

The filter field is automatically modified when you apply other filtering selections, such as facets. To update the results after you change the filter field query, select Update.

We recommend using the dt.smartscape.* attributes for filtering.

If you use the dt.entity.* attributes for filtering, columns, or facets, display names aren't fetched.

Use cases

  • Get results for any key-value pair.
  • Visualize and edit your filtering selection.

Charts

The charts allow you to view your trace data trends and distribution. You can also hide or show the chart panel again at any time.

Distributed Tracing charts

When you select an area on a chart and select Zoom to selection, the filter field and the table results are automatically updated to focus on the selected portion of the trace data.

The charts also highlight the timeframe represented by the table results below them, marked with Table data. This lets you immediately see which portion of the chart your table data corresponds to. Visual indicators appear whenever sampling or limits are applied so that you can assess data quality at a glance.

Time series chart highlights the timeframe represented by the table results in the Distributed Tracing app
Time series chart highlights the timeframe represented by the table results in the Distributed Tracing app

The chart uses sampling for responsiveness. Only the trace list coverage window is highlighted on the chart.

Two chart types are available: time series and histogram. The following table compares them.

Time seriesHistogram

X-axis

Time intervals 1

Response time intervals

Y-axis

Frequency of data points (left-hand side) and response time (right-hand side)

Frequency of data points

Important statistical factors

The legend lists dedicated views for percentiles, averages, and successful and failed requests. Choose an option to view the related trends.

Percentiles and averages are marked with contrast-color vertical lines.

Use cases

Understand how trace data changes over time and identify trends and cyclic behaviors.

Visualize your data's distribution, identify patterns, and spot outliers.

1

Granularity depends on the selected timeframe.

Facets

Facets are quick filters for trace data. They correspond to span attribute key-value pairs detected in your environment and are grouped by facet categories. The most important DQL field IDs are grouped by default in predefined categories. You can define new facet categories and new facets for attributes that are important to you. Each facet category displays the most frequently detected attributes for the current filtering selection.

For details how to manage your facets, see Manage facets in Distributed Tracing app.

As Distributed Tracing Distributed Tracing was migrated from the legacy monitored entity model using dt.entity.xxx attributes to Smartscape 2.0, certain fields in the facets and table results were removed or are no longer supported. For details, see Changes to fields in table results and facets.

Use cases

  • Add new facets to better catalog your trace data.
  • Select facets from the facet list to filter data by them. The filter field is automatically modified according to your selection. Make sure to select Update.
  • Quickly group by attributed keys.
  • Quickly add attributes as columns to the table.

Table results

The table lists the most recent records for the selected timeframe that match the filtering options you applied. It doesn't use sampling, and results are read backwards from the end of the selected timeframe. This means you can reliably find the requests you expect to see, regardless of your environment size.

The number of rows is configurable up to 10,000. To change this limit, select Limit: 5K above the table results and choose the required value.

The timeframe covered by the list is highlighted in the chart with Table data, so you can immediately see which portion of the chart the table data represents.

The table data is available as a list ( ) or grouped by attributes ( ). You can manage columns to display only the attributes you're interested in and exclude noise.

UI elementScope

<column value>

Filter trace data by a column value.

Copy the DQL statement or the DQL API call.

Download the visible table data.

As Distributed Tracing Distributed Tracing was migrated from the legacy monitored entity model using dt.entity.xxx attributes to Smartscape 2.0, certain fields in the table results and facets were removed or are no longer supported. For details, see Changes to fields in table results and facets.

Use cases

  • Compare records and their different attribute values.
  • Filter trace data by a result by selecting the table result. The filter field is automatically modified according to your selection. Make sure to select Update.
  • Reduce noise by hiding unnecessary columns from the table or deselecting an attribute.

Group by

Analyze predetermined important dimensions, such as request count, failure, and percentile contribution, by combining up to 3 attributes that matter the most to you. To group your records by attributes

  • Go to the table and select Group by:. Then select/deselect the attribute you're interested in.
  • Go to the facet list and select > Group by for each attribute key.

Single trace perspective

The single trace perspective offers a detailed view of the trace spans. It loads full end-to-end traces reliably, even in very large environments, with up to 100,000 spans per trace.

  • The waterfall list on the left contains the trace spans, ordered by sequence. For each span, you can see:

    • The corresponding duration bar.

    • The related service; spans associated with the same service are of the same color.

    • The span kind, shown as an icon based on the span attributes below.

      IconSpan kindIcon derived from

      Cloud provider icon

      Calls that have one of the cloud providers (AWS, Azure, GCP)

      aws.region, azure.location, or gcp.region

      Programming language or technology icon

      Call (client or server), internal, or link

      telemetry.sdk.language or dt.agent.module.type; a generic fallback icon is used when neither is present.

      or messaging system icon

      Producer or consumer

      messaging.system

      or database vendor icon

      Database call (client)

      db.system

      Web or RPC call (client or server)

      N/A

    You can understand the sequence and correlation of spans by observing the size and position of the duration bar. Additionally, you can do the following:

    • Search by values (name, endpoint, service, or attribute).
    • View or hide all spans for the selected trace ( Name) or subsequent spans in the trace execution ( next to the span name).
    • View attributes for a span in the details panel by selecting the span name.
    • Explore logs related to the span or the trace (Logs tab). With up to 50,000 log entries, you can thoroughly troubleshoot complex traces linked to a large number of logs.

  • On the right, you can explore and search for attributes for the selected span. Enter a value in the Search details field to view only key or value results matching your search.

To access the single trace perspective, go to the table row of the trace you're interested in and select the trace start time. The single trace perspective opens in the bottom half of the page.

Use cases

  • Focus on the analysis of a single trace.
  • Follow full end-to-end traces, across complex transaction flows, even when spanning different trace IDs.

Exceptions

Use the Exceptions tab in Distributed Tracing Distributed Tracing to understand which exceptions occur across your traces and identify, analyze, and resolve issues related to exceptions.

Through visualizations, aggregated data, and contextual insights, the Exceptions tab can help you understand the root causes of exceptions and their impact on your service.

This feature is designed for SREs, developers, and DevOps teams who need fast, actionable insights into exception patterns and their impact on services.

Exception analysis provides:

  • A clear view of why exceptions occur
  • Advanced filtering by service, endpoint, exception type, and timeframe
  • In-depth analysis and visual trends of exception occurrences
  • Aggregated stack traces to identify root causes
  • Contextual logs linked to exceptions
  • Insights into requests containing exceptions, including sub-traces

Integration with Dynatrace

  • Embedded in Distributed Tracing Distributed Tracing as a new Exceptions tab
  • Integrated into the Dynatrace service-specific drill-down options
  • Replaces the legacy Exceptions Analysis page for DPS customers

Access and navigation

In addition to the Exceptions tab in Distributed Tracing Distributed Tracing, you can analyze exceptions in context through service-specific drill-downs.

To learn more about analyzing exceptions, see Exception analysis.

Changes to fields in table results and facets

As Distributed Tracing Distributed Tracing was migrated from the legacy monitored entity model using dt.entity.xxx attributes to Smartscape 2.0, certain fields in the table results and facets were removed or are no longer supported.

Migrated fields
  • dt.entity.service > dt.smartscape.service
  • dt.entity.service.entity.name > dt.service.name
  • dt.entity.host > dt.smartscape.host
  • dt.entity.host.entity.name > dt.smartscape.host.name
  • dt.entity.process_group_instance > dt.smartscape.process
  • dt.entity.process_group_instance.entity.name > dt.smartscape.process.name
  • dt.entity.container_group_instance > dt.smartscape.container
  • dt.entity.application > dt.smartscape.frontend
  • dt.entity.mobile_application > dt.smartscape.frontend
  • dt.entity.custom_application > dt.smartscape.frontend
  • dt.entity.device_application > dt.smartscape.frontend
  • dt.entity.cloud_application > k8s.workload.name
  • dt.entity.cloud_application_instance > dt.smartscape.k8s_pod
  • dt.entity.cloud_application_namespace > dt.smartscape.k8s_namespace
  • dt.entity.kubernetes_cluster > dt.smartscape.k8s_cluster
  • dt.entity.kubernetes_node > dt.smartscape.k8s_node
  • dt.entity.kubernetes_service > dt.smartscape.k8s_service
  • dt.entity.ec2_instance > dt.smartscape.aws_ec2_instance
  • dt.entity.aws_availability_zone > dt.smartscape.aws_availability_zone
  • dt.entity.aws_availability_zone.entity.name > dt.smartscape.aws_availability_zone.name
  • dt.entity.azure_vm > dt.smartscape.azure_microsoft_compute_virtualmachines
  • dt.entity.azure_vm.entity.name > dt.smartscape.azure_microsoft_compute_virtualmachines.name
  • dt.entity.azure_region > dt.smartscape.azure_location
  • dt.entity.azure_region.entity.name > dt.smartscape.azure_location.name
Fields that stay when ingested in a span

These fields no longer exist in Smartscape 2.0 but can be used if ingested.

  • dt.entity.container_group
  • dt.entity.gcp_zone
  • dt.entity.host_group
  • dt.entity.process_group
  • supportability.external.dt.entity.service
Removed fields

Classic entity lookups no longer exist in Smartscape 2.0.

  • dt.entity.container_group.entity.name

  • dt.entity.ec2_instance.entity.name

  • dt.entity.gcp_zone.entity.name

  • dt.entity.process_group.entity.name

    You can use the dt.smartscape.process attribute and wildcards instead of the removed dt.entity.process_group.entity.name, as a process group is typically a generalization of specific process names.

    For example, suppose the live-BusinessBackend process group consists of several process group instances: live-BusinessBackend-1-WIN1, live-BusinessBackend-1-WIN2, live-BusinessBackend-2-WIN1, and live-BusinessBackend-2-WIN2. To filter traces that land on any of these processes, use the dt.smartscape.process = "live-BusinessBackend-*" query.

To find the Smartscape on Grail DQL equivalents for your classic entity queries, see Differences between classic entities and Smartscape on Grail.

Related tags
Application ObservabilityDistributed TracingDistributed Tracing