Trace sampling

  • Tutorial
  • 5min

Adaptive Traffic Management automatically starts sampling traces if the amount of ingested traces exceeds the included trace volume. In case of Extended trace ingest the threshold is increased but otherwise it works the same way. If OneAgent capture rate is below 100%, sampling has been applied because the amount of traces that can be captured by OneAgent has exceeded the Full-Stack included trace volume.

You can adjust how OneAgent treats specific HTTP requests and Remote Procedure Call (RPC) requests when sampling is needed by

  • Configuring trace sampling for HTTP and RPC requests.

    If OneAgent is sampling and the capture rate is below 100%, you can advise OneAgent on the importance of specific HTTP and RPC requests in relation to others. By increasing or decreasing the sampling frequency of certain HTTP and RPC requests, HTTP and RPC requests with higher importance are captured more often and vice versa. In this way you can control what the available trace volume is used for without affecting the overall ingested trace volume nor the overall capture rate.

    Adjusting the capture rate is effective only if the OneAgent module that starts the distributed trace is sampling.

  • Excluding traces for specific HTTP and RPC requests from being captured.

    In all environments, there are transactions for which traces are of lower value. You can exclude them from being captured at all. You can turn off tracing for specific HTTP and RPC requests completely at any time, freeing up trace volume for other more important requests.

Prerequisites

  • For HTTP requests trace sampling: OneAgent version 1.281+
  • For RPC requests trace samping: OneAgent version 1.307+

Available levels

You can configure trace sampling

  • For all traces in your environment
  • For traces started within a particular host group, process group, Kubernetes cluster, or Kubernetes namespace (Dynatrace version 1.312+). This is useful if you do not have environment permissions or simply want this to only effect a particular part of your environment.
  1. Go to Settings > Server-side service monitoring.
  2. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.
  1. Go to Deployment Status and then select OneAgents.
  2. On the OneAgent deployment page, turn off Show new OneAgent deployments.
  3. Filter the table by Host group and select the host group you want to configure.

    The Host group property is not displayed when the selected host doesn't belong to any host group.

    This displays the OneAgent deployment page filtered by the selected host group. Each listed host has a Host group: <group name> link, where <group name> is the name of the host group that you want to configure.
  4. Select the host group name in any row.
    As you have filtered by host group, all displayed hosts go to the same host group.
  1. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.
  1. To find and select your process group do one of the following:
    • Go to Technologies & Processes or Technologies & Processes Classic (latest Dynatrace) and choose a technology.
    • Go to Hosts or Hosts Classic (latest Dynatrace) and choose a host.
  2. Find and select your process group.
  3. In the upper-right corner of the process-group overview page, select Settings.
  4. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.
  1. Go to Kubernetes or Kubernetes Classic (latest Dynatrace).
  2. Find your Kubernetes cluster.
  3. On the Kubernetes cluster overview,
    • To configure sampling for the cluster, in the upper-right corner, select More (…) > Settings.
    • To configure sampling for the namespace
      1. Find and select the namespace.
      2. In the upper-right corner of the Kubernetes namespace overview, select More (…) > Settings.
  4. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.

Adjust sampling for web requests

  1. Select Add URL.
  2. Choose how to adjust trace sampling:
    • To never capture traces for the URL, regardless of the available trace volume, turn on Disable tracing for matching HTTP requests.
    • To increase or reduce the frequency of trace capture for the specific URL, set Importance of a specific URL to a different scaling factor.
  3. To match the HTTP request do one or a combination of the following:

    • Enter the URL path and select the comparison condition.

      Use wildcards (**) for URL paths with similar segments.

      For example, in /api/product/**/detail, all values between slashes are ignored; the path applies to both /api/product/1/detail and /api/product/2/detail.

    • Use query parameters.

      1. Go to Query parameters and select Add item.
      2. Enter a query parameter name.
      3. Enter the query parameter value or turn on Query parameter value is undefined.
  4. optional To apply the scaling factor to specific HTTP methods, turn off Any HTTP method and select the HTTP method(s) to target.
  5. Select Save changes.

Use case examples

The following rule reduces the importance by 4 of requests that start with _healthz.

  1. Go to Settings > Server-side service monitoring > Trace sampling for HTTP requests.
  2. Select Add URL.
  3. From Importance of a specific URL, select Reduce capturing by factor 4.
  4. Type the URL path /_healthz.
  5. From Path comparison condition, select Starts with.
  6. Turn on Any HTTP method.
  7. Select Save changes.

The following rule ignores all requests (starts with /) that have OPTIONS as HTTP method for a selected process group.

  1. Go to the process group and select More (…) > Settings > Trace sampling for HTTP requests.
  2. Select Add URL.
  3. Turn on Ignore a specific URL.
  4. Type the URL path /.
  5. From Path comparison condition, select Starts with.
  6. Turn off Any HTTP method.
  7. From the HTTP method list, turn on OPTIONS.
  8. Select Save changes.
  9. optional Create a new rule for each process group that you want to target.
Related tags
Application Observability