Best practices for avoiding overalerting

• Latest Dynatrace
• Best practices
• 15-min read
• Published Jul 09, 2026

Overalerting—generating a high volume of false-positive or low-value alerts—is one of the most common problems when configuring anomaly detection. Receiving too many false alerts has a number of potential consequences and might lead to:

This page introduces best practices for configuring Anomaly Detection that help you keep your alert volume down to meaningful alarms and your team responsive.

Identify false positive alerts

An alert exists to signal an abnormal state–a state that rarely occurs under healthy operating conditions, in no more than 0.1% of observed time. This threshold gives you a measurable criterion for deciding whether a new alert reflects a genuine problem or is creating noise.

Understand the 0.1% threshold in practice

For a single service monitored continuously:

A well-calibrated alert for a healthy service is typically created once or twice per month, with each alert firing representing a short-lived but valid anomaly. If a service alerts 100 times within 24 hours, the alert is created for a normal operating state of that service rather than an abnormal state.

Why an alert appears too often

When an alert consistently exceeds the 0.1% threshold, there are two possible explanations:

How to use the 0.1% threshold rule

Use the 0.1% threshold rule as a diagnostic during alert reviews: look up the alerting history of each active detector in Anomaly Detection and flag any detector where problem events account for more than 0.1% of the review period as a candidate for reconfiguration or service remediation.

Identify your noisiest detectors with DQL

To get a quick ranked view of which alert configurations are generating the most events–including a direct back-reference to the settings object so you can navigate straight to the detector–run the following query in Notebooks or Dashboards:

fetch dt.davis.events
| filter not in(event.category, {"INFO", "WARNING"})
| summarize alerts = count(), by:{event.name, dt.settings.object_id, dt.smartscape_source.id}
| sort alerts desc

The query excludes informational and warning events to focus on events that have raised problems, groups them by detector name and source entity, and sorts by total alerting count. The dt.settings.object_id field contains the identifier of the anomaly detection configuration that produced each event, which you can use to navigate directly to the offending detector in Anomaly Detection for reconfiguration or removal.

Create the initial configuration with relaxed thresholds

When you create a new anomaly detector, begin with a less sensitive threshold rather than a stricter one. You can increase sensitivity at any point once you understand how your data behaves, but recovering from alert fatigue requires additional time and effort.

To ensure that you fully understand your data behavior before using stricter thresholds, we recommend that you take the following approach:

To learn more about adjusting the sensitivity of built-in anomaly detectors, see Adjust the sensitivity of anomaly detection.

Choose the suitable detection model

Dynatrace offers different anomaly detection models. Choosing the right model for your data pattern reduces false positives before any threshold tuning takes place.

For an overview of all available models, see Anomaly Detection app.

Alert on aggregated data, not on individual dimensions

Splitting an alert on high-cardinality dimensions, such as individual app versions, Kubernetes pods, or HTTP status codes, causes your alert count to scale with the number of dimension values. This leads to:

Example: Fix the frontend error count by app version

Assume that you have configured an anomaly detector for counting frontend error split by app version, which fires a separate alert for each version of the app:

timeseries avg(dt.frontend.error.count), by:{app.short_version}

This configuration fires alerts for any new and old versions, which creates noise by increasing the amount of false positive alerts. To fix the configuration, you should use a query that alerts you on the aggregated error count across all active versions:

timeseries avg(dt.frontend.error.count)

A spike in the aggregated error count is noticeable, and your team can promptly act upon it. The root cause analysis–which specific app version introduced the regression–can follow as a downstream step, such as an agentic investigation triggered by the problem event after the incident has been confirmed.

To learn more about scoping your queries effectively, see Anomaly Detection DQL writing guide and Use segments with custom alerts.

Use the sliding window to prevent sequential alert spam

Without a controlled alert delay, a metric that fluctuates around a threshold can produce dozens of individual alerts within a single hour, each opening and closing every few minutes. This results in a cluttered problem feed and a notification stream that trains your team to ignore alerts.

To avoid cluttering, we recommend that you use the sliding window parameter, setting a condition that a threshold must be breached for a minimum number of consecutive evaluation samples before a problem opens. A reliable starting point is 3 violating samples out of any 5 minutes.

Dynatrace keeps the problem event open until the dealerting criterion is met–by default, the criterion is 5 consecutive non-violating samples. This means threshold fluctuations add new violations to an existing problem instead of opening a new one, which further reduces overall alert volume.

If your alert keeps opening and closing repeatedly, we recommend that you increase the required violation count for the sliding window instead of raising the threshold.

To learn more about configuring the sliding window, see Anomaly detection configuration.

Use large aggregation windows for sparse log pattern signals

Security and fraud detectors often rely on logs rather than metrics. A typical detector looks for a recurring pattern in log lines–failed login attempts, requests from a known bad actor IP address, or repeated access to a sensitive endpoint–and alerts you when the occurrence count crosses an established threshold.

Suppose you want to alert on such patterns. They are, by default, sparse and erratic. A single failed login attempt is an expected background noise, while thousands of failed attempts within an hour might indicate a credential stuffing attack. The problem is that the raw event stream fluctuates heavily from minute to minute, which makes minute-by-minute evaluation unreliable. In such a scenario, you get false positives when a short burst of legitimate retries briefly exceeds a low threshold, but also get missed detections when the attack spreads across a longer time window.

Use a records-based detector with an aggregation window

To avoid redundant noise and increased number of false positives caused by raw event stream fluctuations, we recommend using a records-based custom alert that aggregates the occurrence count over a 60-minute window with a matching 60-minute query execution delay. This means that, once every hour, a custom alert evaluates the total count of matching log events detected during that hour. This provides a more stable and accurate result than the per-minute sampling offered by the timeseries-based custom alert.

Below is the example of the DQL query for a failed login detector:

fetch logs, from: now() - 60m
| filter matchesPhrase(content, "authentication failed") or matchesPhrase(content, "failed login")
| summarize failed_login_count = count()
| filter failed_login_count > 500

With Data type set to Records and Delay set to 60 Minutes, this query runs once every hour. If the total count of matching log lines in the past hour exceeds the threshold, a problem is raised. If the count stays below it, as it would during normal background noise, no problem is opened.

The same pattern of 60-minute delay and 60-minute query window can be applied to other sparse log signals:

To learn more about the records data type and the Delay parameter, see Anomaly detection configuration. For more examples using records-based custom alert, see Monitor daily LLM cost burn rate using records-based custom alerts.

Use built-in event correlation to merge related alerts

Before you adjust any configuration, we encourage you to learn what Dynatrace does automatically by default. Dynatrace Intelligence offers a set of built-in event correlation rules that combine individual alerts into a single problem based on shared topology fields. These rules are applied to every incoming event without any prior setup required.

The universal rule: same Smartscape entity

The same Smartscape entity rule groups all events that share the same dt.smartscape_source.id field value. Since this rule has a matching condition set to true by default, it applies to every event regardless of the entity type. Any two alerts that reference the same Smartscape entity and arrive within the established timeframe are merged into a single problem.

Topology-based rules: merging across related entities

Aside from the universal rule, Dynatrace provides topology-aware rules that merge events of different entity types that share the same source topology node. This mechanism prevents a cascading failure from creating a separate problem for each of the affected entities. Below are some examples of the grouping fields and potential entity types that can be merged, separated by the observability app.

Infrastructure

Kubernetes

Since a Kubernetes environment can have many short-lived pods and frequent rollouts, these rules reduce alert volume and noise: alerts from a pod, its containers, and the node it runs on are all candidates for merging into a single problem.

Additional layers of automatic deduplication

On top of the correlation rules, Dynatrace applies additional deduplication mechanisms that help further reduce the noise and the problem count:

When combined, correlation rules and deduplication layers help you reduce the number of problems a properly configured custom alert produces compared to the raw number of individual alerts raised without the rules and deduplication applied. To learn more about how event correlation and deduplication work, see Event analysis and correlation.

Disable long-running violating alerts

A problem that has been continuously violating the threshold and has remained open for weeks or months signals that the underlying custom alert is misconfigured or monitoring something that no longer exists. Unlike a genuine incident that your team works to resolve using time and effort, a long-running violating alert typically has two characteristics:

Example: synthetic test pointing at a decommissioned application

Suppose your synthetic monitor has been configured to test a web application that has been taken down at some point. In this scenario, the monitor will fail on every execution while continuously consuming monitoring capacity. At the same time, the resulting problem will stay open indefinitely as a constant false positive.

The optimal solution in this case is to disable or delete the custom alert, since it'll automatically:

Long-running open problems are a useful indicator that can help with your regular alert reviews—you can sort open problems by age and treat any problem open for more than a few weeks without an active owner as a candidate for investigation and cleanup.

Use Info and Warning event types to observe your environment without alerting

Some of the monitored conditions provide useful information but aren't impactful enough to raise a problem. An alert you're still calibrating, a metric you're observing to learn trends without committing to an SLO, or a condition that violates regularly but isn't yet actionable—all of these benefit from being captured and stored in Grail without creating a problem.

Dynatrace supports two non-problem-raising event types you can set in the event template of any custom alert:

Both event types have the lowest severity level–SEV-5. For more information about severity levels and their meaning, see Standardized event severity. To learn more about the existing event types, see Event categories.

Setting the event.type value to CUSTOM_INFO or WARNING instead of CUSTOM_ALERT in the event template of a custom alert gives you full observability of the alerting condition. Every violation is recorded in the dt.davis.events field without cluttering your problem feed or creating noise.

Practical uses of the non-problem-raising events

Below are some of the practical uses of the non-problem-raising events:

Calibration period

fetch dt.davis.events
| filter in(event.category, {"INFO", "WARNING"})
| filter event.name == "High failed login rate - observation"
| summarize firing_count = count(), by: { bin(timestamp, 24h) }

Dashboard monitoring for chronic conditions

If an observed condition violates the threshold regularly because an underlying issue is known and tracked elsewhere (for example, in a backlog ticket), use WARNING to keep the signal visible in dashboards and prevent the related problem from being constantly reopened. This will reduce the noise and number of false positive alerts your team receives.

Workflow triggers without Problems

Workflows can be triggered by any Davis event type, including CUSTOM_INFO and WARNING. You can route a non-problem-raising event to a Slack channel or create a Jira ticket without the event appearing in the Problems.

Regularly review and remove alerts that create noise

Alert configurations drift over time. Because of service changes, traffic pattern shifts, and team restructures, an alert that was well-tuned six months ago might be triggered constantly now, or not trigger at all. We recommend that you establish a regular review period (for example, once every quarter) to audit your active configurations in Anomaly Detection. You should focus on the following alerts:

A smaller, well-maintained alert library with a high signal-to-noise ratio produces more accurate and effective results than a large alert library that contains noise-creating configurations and produces a significant number of false positives.