Try it free

Service-related concepts

  • Latest Dynatrace
  • Explanation
  • 4-min read

Service-related concepts, including distributed traces and spans, are central concepts in Dynatrace observability. Understanding these concepts enables effective monitoring and analysis of distributed systems.

Attributes

Attributes are key-value pairs that provide details about spans, requests, or resources (for example, a span name, response codes, HTTP methods, URLs, or failure detection results). They're used to group, query, find, and analyze traces and spans.

Dynatrace uses resource attributes to detect and name services, gather trace context data and entity relationships for Smartscape topology, connect log data to traces, understand span duration impacts from service timings, and analyze executed code. Attribute keys adhere to the Dynatrace Semantic Dictionary.

Spans and distributed traces

Spans are single units of work within a distributed trace. Each span consists of multiple attributes and includes information such as a span ID, span name, start time, duration, span events (for example, exceptions), span kind, and parent span identifier. Spans connect via the parent identifier to build a tree-like structure.

Distributed traces are sequences of spans with an identical trace ID that follow a single path through various services and components. They help you understand request propagation across distributed systems, analyze microservices data, assess microservice performance, and follow Dynatrace Intelligence root cause analysis for cause-and-effect relationships.

Services and service detection

Services represent software modules or applications deployed as Kubernetes workloads, AWS Lambda functions, or processes on VMs.

They are detected based on resource attributes on spans within a distributed trace. All spans coming from the same deployable belong to the same service, for example, all spans produced by the same Kubernetes workload.

Each service exposes metrics for latency, failure rate, and throughput across its endpoints, consumed messages, and invocations. These metrics are also derived from the same spans.

Service Detection v1 (SDv1) is the classic service detection for OneAgent-instrumented processes. It provides detection based on technology-specific service types, with dedicated configurability for each service type. It works best for monolithic applications and application servers; modern cloud-native microservice architectures are better served by Service Detection v2.

Service Detection v2 (SDv2) operates according to a single set of resource attribute-based rules (built-in and user-defined) evaluated against every span of a trace. It works with OpenTelemetry span data, and is available as an Early Access release for Kubernetes and AWS Lambda services monitored by OneAgent.

Endpoints and entry points

Endpoints

Endpoints represent the API entry point of a service.The endpoint metrics (dt.service.request.*) count and measure requests executed against those endpoints. All endpoints are displayed in Services Services for both service detection versions.

For Service Detection v1 (with Enhanced endpoints activated), endpoints are automatically derived and supersede the concept of key requests. For Service Detection v2, endpoints are defined through rules matching span attributes.

Endpoints vs URLs

An endpoint is a logical grouping—for example, GET /orders/{id}—that represents an API instead of an individual request. A URL is the specific path of an individual request, such as /orders/12345 or /orders/67890. Dynatrace aggregates metrics at the endpoint level, providing stable, low-cardinality data for your dashboards, SLOs, and alerts. The underlying URL paths are available in Distributed Tracing.

Naming for HTTP-related endpoints

Dynatrace detects endpoints for HTTP requests using the http.route span attribute.

Most modern frameworks—Spring, Express, Django, ASP.NET Core, Flask, and FastAPI—provide http.route automatically, so your services get the endpoints defined by the application's development team out of the box. Examples are descriptive endpoint names like GET /books/search and POST /orders/checkout.

Web servers and reverse proxies that don't emit http.route (NGINX, Apache, IIS, Kong, and IBM WebSphere Liberty) get a stable name from a derived http.route based on the URL path, where available, and fall back to generic names like GET /* otherwise. For details, check the Automatic naming for services without http.route section for SDv1 and SDv2.

To improve generic endpoint names:

  • SDv1 services: Create request naming rules to define endpoint names based on URL conditions.For details, see Fine-tune endpoint names.
  • SDv2 services: Configure URL pattern matching rules to derive stable endpoint names from raw URL paths.

Entry points

Entry points (Service Detection v1) are the first spans produced when a service is invoked. They mark where tracing activity begins within a service and are used for failure detection configuration, custom service definition, and service flow analysis.

Requests

Requests are individual executions against an endpoint, detected based on span attributes.Each request contributes to the endpoint metrics (dt.service.request.*)—latency, failure rate, and throughput—aggregated at the endpoint level.

What counts as a request differs slightly between service detection versions:

  • Service Detection v1: A request encompasses all spans of a trace that belong to the same service and aren't interrupted by spans of a different service. All workload types—web, messaging, batch, and cron jobs—are treated as requests.
  • Service Detection v2: Requests specifically track calls into a monitored process API based on their endpoints (for example, web requests or RPC/RMI calls). Services without endpoints have no request metrics.

Key requests

Key requests are a legacy way to flag individual requests as requiring special attention, for example, critical business measures or vital technical functionality. SDv1 (with Enhanced endpoints activated) and SDv2 automatically monitor all endpoints, while key requests require manual configuration.

Switch to Enhanced endpoints for SDv1

Instead of defining key requests for SDv1 services, enable the Enhanced endpoints for SDv1 feature. It automatically surfaces all SDv1 endpoints in Services Services, not just key requests.

Mark endpoints with primary Grail tags

To find and filter specific endpoints as a group in dashboards, SLOs, notebooks, and workflows, use OpenPipeline to enrich the related spans with primary Grail tags, for example, primary_tags.environment or primary_tags.business_unit.

Starting with Dynatrace SaaS version 1.343+, primary Grail tags propagate to the endpoint metrics (dt.service.request.*), giving you a consistent filter in dashboards, SLOs, notebooks, and workflows.

Errors, exceptions, and failures

Errors and exceptions are captured as attributes on spans (for example, span events) within distributed traces. They provide details about request processing problems, including an error type, message, stack traces, timestamp, and associated span context. Errors and exceptions are automatically captured from OneAgent-monitored applications and OpenTelemetry instrumentation.

Failures are determined by failure detection, typically based on error and exception data. Note that the existence of errors doesn't necessarily mean the associated request is considered failed. Configuration differs between Service Detection v1 (global or per-service settings) and Service Detection v2 (rule sets based on span attributes).

Related tags
Application Observability