Distributed traces concepts

In this article, you will learn about the most important concepts and terminology for distributed tracing in Dynatrace.

Relation between Services and Traces and Spans

Distributed trace

A distributed trace is a sequence of spans, identified by a unique identifier—trace ID. Each distributed trace traverses at least one service and, in a modern microservice environment, typically multiple services. It contains semantically different attributes that make it possible to interpret and understand the collected data for a request.

Use cases

  • Understand how requests propagate across distributed systems and microservices.
  • Use high-quality data generated by distributed systems and microservices for request analysis.
  • Quickly understand how each microservice is performing.
  • Follow Davis AI root cause analysis drill-downs to identify cause-effect relationships between events.

Span

A span is a record of an interaction between a request and a service, triggered by the activity of a request moving through and across multiple services. Spans collect information in the form of attributes, including the name, start and end timestamps, a list of attributes as key/value pairs, the parent's span identifier, and the span kind.

When the activity— parent span—is completed, the next activity passes to its child span. A span without a parent span is called a root span and indicates the start of a trace.

Learn more about span semantic fields.

Span context

The span context is needed to put all spans and events in context with each other. The span context allows a child span to relate to the trace and its parent span. Therefore, the context needs to be propagated within a service (across different threads) but also across services and process boundaries. This typically happens via HTTP headers (like the W3C trace context) or via unique IDs in messaging systems.

Attribute

Attributes are key-value pairs that provide details about a span, request, or resource such as response codes, HTTP methods, and URLs. Via attributes, you can group, query, find, and analyze your traces and spans.

Use cases

Dynatraces uses attribute metadata to

  • Detect and name services.
  • Gather data on the trace context and relationships with other entities for Smartscape topology.
  • Connect log data to traces for Logs or Logs Classic.
  • Understand how the duration of a span is affected by service timings (for example, CPU time, network time, or just waiting for other threads) and analyze which code was executed in the context of the span.

Best practices

If you collect trace data via

Learn more about request attributes and captured attributes semantic fields.

Service

Services are traversed by distributed traces. On horizontally scaled services, specific Service Instances process each span. Services are determined and named based on available attributes or properties that are collected along with the spans.

Use cases

Data collection and context propagation

You can integrate OpenTelemetry and OneAgent to collect trace data—like request status, response time, versions, infrastructure information, and other relevant metadata as attributes. The trace context, including the unique trace ID, is then propagated across your apps and microservices.

Best practices

Before getting started with distributed tracing, understand how setup and trace data collection differs between OpenTelemetry and OneAgent. The following is an overview of the key differences.

OpenTelemetry

OneAgent

Set up

Automatic or manual

Automatic

Capturing

Automatic collection of allowed span attributes.

Automatic collection of several request attributes, including HTTP method, URL, response codes, topology data, and details about the underlying technologies.

Context

Automatically or manually contextualized log entries, depending on the instrumentation library.

Automatically contextualized

  • Log entries produced by prominent log frameworks.
  • Traces in Smartscape and Davis AI.

To get started see

Use OpenTelemetry in combination with OneAgent to enhance your observability coverage, using the best of both.

PurePath® technology

Dynatrace patented PurePath® technology for distributed tracing, since 2006. PurePath® technology combines distributed tracing information with additional insights like user experience information, logs, metrics, topology information, metadata, and even code-level profiling information to provide the highest level of data fidelity and granularity.

Use cases

Analyze data down to code-level detail without ever losing the full context around your environment during drill down, with the highest level of data granularity and fidelity of monitored transactions.