Trace sampling

  • Latest Dynatrace
  • Tutorial
  • 5-min read

Adaptive Traffic Management automatically starts sampling traces if the amount of ingested traces exceeds the included trace volume. In case of Extended trace ingest the threshold is increased but otherwise it works the same way. If OneAgent capture rate is below 100%, sampling has been applied because the amount of traces that can be captured by OneAgent has exceeded the Full-Stack included trace volume.

You can adjust how OneAgent treats specific HTTP requests and Remote Procedure Call (RPC) requests when sampling is needed by

  • Configuring trace sampling for HTTP and RPC requests.

    If OneAgent is sampling and the capture rate is below 100%, you can advise OneAgent on the importance of specific HTTP and RPC requests in relation to others. By increasing or decreasing the sampling frequency of certain HTTP and RPC requests, HTTP and RPC requests with higher importance are captured more often and vice versa. In this way you can control what the available trace volume is used for without affecting the overall ingested trace volume nor the overall capture rate.

    Adjusting the capture rate is effective only if the OneAgent module that starts the distributed trace is sampling.

  • Excluding traces for specific HTTP and RPC requests from being captured.

    In all environments, there are transactions for which traces are of lower value. You can exclude them from being captured at all. You can turn off tracing for specific HTTP and RPC requests completely at any time, freeing up trace volume for other more important requests.

  • Configuring adaptive sampling rules.

    You can turn on Trace ingest control at a deployment scope level and change the multiplier for calculating the sampling threshold at that level. For details, see Adjust trace ingest sampling below.

Prerequisites

  • For HTTP requests trace sampling: OneAgent version 1.281+
  • For RPC requests trace samping: OneAgent version 1.307+

Available levels

You can configure trace sampling

  • For all traces in your environment
  • For traces started within a particular host group, process group, Kubernetes cluster, or Kubernetes namespace (Dynatrace version 1.312+). This is useful if you do not have environment permissions or simply want this to only effect a particular part of your environment.
Environment
  1. Go to Settings > Server-side service monitoring.
  2. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.
Host group
  1. Go to Deployment Status and then select OneAgents.
  2. On the OneAgent deployment page, turn off Show new OneAgent deployments.
  3. Filter the table by Host group and select the host group you want to configure.

    The Host group property is not displayed when the selected host doesn't belong to any host group.

    This displays the OneAgent deployment page filtered by the selected host group. Each listed host has a Host group: <group name> link, where <group name> is the name of the host group that you want to configure.
  4. Select the host group name in any row.
    As you have filtered by host group, all displayed hosts go to the same host group.
  1. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.
Process group
  1. To find and select your process group do one of the following:
    • Go to Technologies & Processes (previous Dynatrace) or Technologies Technologies & Processes Classic and choose a technology.
    • Go to Hosts (previous Dynatrace) or Hosts Hosts Classic and choose a host.
  2. Find and select your process group.
  3. In the upper-right corner of the process-group overview page, select Settings.
  4. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.
Kubernetes cluster or namespace
  1. Go to Kubernetes (previous Dynatrace) or Kubernetes Kubernetes Classic.
  2. Find your Kubernetes cluster.
  3. On the Kubernetes cluster overview,
    • To configure sampling for the cluster, in the upper-right corner, select More (…) > Settings.
    • To configure sampling for the namespace
      1. Find and select the namespace.
      2. In the upper-right corner of the Kubernetes namespace overview, select More (…) > Settings.
  4. Choose Trace sampling for HTTP Requests or Trace sampling for RPC Requests.

Adjust trace sampling for web requests

  1. Select Add URL.
  2. Choose how to adjust trace sampling:
    • To never capture traces for the URL, regardless of the available trace volume, turn on Disable tracing for matching HTTP requests.
    • To increase or reduce the frequency of trace capture for the specific URL, set Importance of a specific URL to a different scaling factor.
  3. To match the HTTP request do one or a combination of the following:

    • Enter the URL path and select the comparison condition.

      Use wildcards (**) for URL paths with similar segments.

      For example, in /api/product/**/detail, all values between slashes are ignored; the path applies to both /api/product/1/detail and /api/product/2/detail.

    • Use query parameters.

      1. Go to Query parameters and select Add item.
      2. Enter a query parameter name.
      3. Enter the query parameter value or turn on Query parameter value is undefined.
  4. Optional To apply the scaling factor to specific HTTP methods, turn off Any HTTP method and select the HTTP method(s) to target.
  5. Select Save changes.

Adjust trace ingest sampling

You can use trace ingest control to steer the usage of the Full-Stack Monitoring included trace volume at a settings scope level.

  • Typically, Adaptive Traffic Management distributes the available volume equally to all OneAgents. It achieves this by adjusting the trace ingest rate limit every 15 minutes, ensuring that the overall trace ingest volume remains within the included volume, along with any additional extended trace volume that has been configured.

  • With trace ingest control, you can adjust the trace ingest rate of OneAgents, allowing them to capture traces at a higher or lower rate than normal. This results in a higher trace capture rate for targeted parts of your system. Consequently, Adaptive Traffic Management might lower the trace ingest rate overall on average, so that the resulting overall trace ingest volume stays within the defined bounds

    This control is available now via the Settings API. It will soon also be available in the Settings app directly.

To adjust trace ingest sampling, use the Trace Ingest Control schema provided by the Settings API. The following is general guidance for this setting.

  • Use the schemaId builtin:trace.ingest.control
  • Use the scope (for example, "HOST_GROUP-1A2B3C4D5E6F7G8H")
  • Find or create the objectId for the settings object (use the GET objects request)
  • Set the value of the adaptiveTraceIngestRateFactor setting, with adaptiveSamplingSwitch set to true

For step-by-step details on using the API, see Example 3 and Example 4 in the use cases.

Use case examples

Example 1: Reduce capturing of an often-executed health check request that provides values less than 60 times/minute

The following rule reduces the importance by 4 of requests that start with _healthz.

  1. Go to Settings > Server-side service monitoring > Trace sampling for HTTP requests.
  2. Select Add URL.
  3. From Importance of a specific URL, select Reduce capturing by factor 4.
  4. Type the URL path /_healthz.
  5. From Path comparison condition, select Starts with.
  6. Turn on Any HTTP method.
  7. Select Save changes.
Example 2: Never capture HTTP Options request on web servers for a process group

The following rule ignores all requests (starts with /) that have OPTIONS as HTTP method for a selected process group.

  1. Go to the process group and select More (…) > Settings > Trace sampling for HTTP requests.
  2. Select Add URL.
  3. Turn on Ignore a specific URL.
  4. Type the URL path /.
  5. From Path comparison condition, select Starts with.
  6. Turn off Any HTTP method.
  7. From the HTTP method list, turn on OPTIONS.
  8. Select Save changes.
  9. Optional Create a new rule for each process group that you want to target.
Example 3: Increase the ingest for a crucial Kubernetes cluster to allow a higher capture rate

  1. Go to Kubernetes (previous Dynatrace) or Kubernetes Classic.

  2. Find your Kubernetes cluster and take the scope from the URL (for example, KUBERNETES_CLUSTER-1A2B3C4D5E6F7G8H)

  3. Use POST objects to create a new settings object according to the following:

    • Set adaptiveTraceIngestRateFactor to 50.
      This increases the default trace ingest rate limit 50 times to the default, which results in a higher capture rate while the overall trace ingest volume stays within the defined bounds.
    • Set adaptiveSamplingSwitch to true.
      This enables the new control.
    [
      {
        "schemaId": "builtin:trace.ingest.control",
        "scope": "KUBERNETES_CLUSTER-1A2B3C4D5E6F7G8H",
        "value": {
          "adaptiveSamplingSwitch": true,
          "adaptiveTraceIngestRateFactor": 50
        }
      }
    ]

  4. You should receive a successful response that provides you the objectId.

  5. Optional Use GET effectiveValues to verify the change.

Example 4: Further decrease the ingest for less crucial Kubernetes cluster

  1. Go to Kubernetes (previous Dynatrace) or Kubernetes Classic.

  2. Find your Kubernetes cluster.

  3. Take the scope from the cluster of the URL (for example, KUBERNETES\CLUSTER-H8G7F6E5D4C3B2A1).

  4. Use GET effectiveValues to check the value of adaptiveTraceIngestRateFactor, which is the currently effective adaptive trace ingest rate factor for this scope.

    The response shows that it's currently already reduced to 0.8 for the scope/origin:

    {
      "items": [
        {
          "origin": " KUBERNETES_CLUSTER-H8G7F6E5D4C3B2A1",
          "value": {
            "adaptiveSamplingSwitch": true,
            "adaptiveTraceIngestRateFactor": 0.8
          }
        }
      ],
      "totalCount": 1,
      "pageSize": 100
    }

  5. Use GET objects to find the settings objectId for the scope.

    Add the value "scope" to the fields query parameters.

  6. Use PUT objects/{objectId} to update the existing settings object according to the following:

    • Set adaptiveTraceIngestRateFactor to 0.01.

      This decreases the trace ingest rate limit to one percent of the default, resulting in a very low capture rate while the overall trace ingest volume stays within the defined bounds.

      {
        "value": {
            "adaptiveSamplingSwitch": true,
            "adaptiveTraceIngestRateFactor": 0.01
        }
      }

  7. You should receive a successful response that provides you the objectId.

  8. Optional Use GET effectiveValues to verify the updated value of adaptiveTraceIngestRateFactor.

Related tags
Application Observability