Set up custom alerts based on metrics extracted from logs

Latest Dynatrace
Tutorial
4-min read

Ingested logs can be triggers for opening new Davis problems.

Using a combination of metrics based on logs and custom alerts, you can use the power of different Dynatrace Intelligence data analyzers to address use cases from simple threshold-based alerting to seasonal baselines, for example:

Alert when the average count of matching records exceeds a specific number within a defined time period.
Alert when the value of a metric is abnormal, without setting a static threshold.

Follow this guide to learn more about alerting with metrics based on logs.

If you don't need to set thresholds, you should follow the instructions in Set up alerts based on events extracted from logs.

Prerequisites

You have set up log ingestion.
You are using OpenPipeline.
You have the necessary permissions to configure the custom alert, within Anomaly Detection app.

Steps

In this example we will open a new Davis problem when certain records, which contain a specific phrase, are ingested and exceed a static threshold.

1. Find logs you want to trigger alerts

You can find alerts by opening Logs and using the following DQL query.

fetch logs
| filter matchesPhrase(content, "Dropping data because sending_queue is full")
| sort timestamp desc

If your DQL query uses parse, fieldAdd, or other transformations, you should add a processing rule to set those fields on ingest.
To find which pipeline handles your logs, select a log record in Logs and check the dt.openpipeline.pipelines field. Use that value when selecting a pipeline in OpenPipeline settings.

2. Extract metric in OpenPipeline

Add metric extraction configuration in OpenPipeline.

Open Settings > Process and contextualize > OpenPipeline > Logs and select the Pipelines tab.
Find the pipeline you want to modify, or add a new pipeline.
Select > Edit. The pipeline configuration page appears.
Select Metric extraction tab.
Set
- The metric name and ID.
- The DQL matcher. A matcher sets the condition for the event that is to be extracted. It is a subset of filtering conditions in a single DQL statement.
  
  In Matching condition, use the matcher as shown below.
  matchesPhrase(content, "Dropping data because sending_queue is full")
  If you use Segments or your permissions are set at the record level, you should include those conditions in the matcher.
  
  There are situations when a matcher can't be easily extracted from a DQL statement. In these cases, you can create log alerts for a log event or summary of log data.
Add dimensions. For most logs, you can add automated correlation to entities in Dynatrace Intelligence root cause analysis. To do this, add a dt.source_entity dimension or any other field that contains an entity identifier.

If you added the extraction rule to an existing pipeline, logs are already routed there and no further configuration is needed.

If you created a new pipeline, configure a dynamic route so logs reach it:

Fields that are computed or added during the Processing stage cannot be used in dynamic route matching conditions because routing is evaluated before Processing. This includes fields added by Dynatrace itself, such as dt.entity.service. For a full list of fields with this limitation, see Fields with limits for all configuration scopes.

Go to Settings > Process and contextualize > OpenPipeline > Logs and select the Dynamic routing tab.
Select Dynamic route and set:
- Name: For example, use Dropping data alert route.
- Matching condition: Use the same condition as the metric extraction matcher so the route narrows the data to the records you actually want to process.
  matchesPhrase(content, "Dropping data because sending_queue is full")
- Pipeline: Select the new pipeline.
Select Add.

Routes are evaluated top to bottom and the first matching route wins. If you have other routes that could match the same logs, place this route above them.

3. Configure a custom alert

Go to Anomaly Detection - new Anomaly Detection and create a new custom alert.

This section describes how to create a simple custom alert. If you need to set additional advanced properties and fine-tune your alert, use the Advanced mode.
Use Suggest values to find the right threshold.
Use Preview to get an estimation of how many alerts would have been generated in the last two hours.

Set the scope for your alert.
Use DQL syntax to point the metric you created. To have your alert connected to monitored entity make sure to add by: {dt.source_entity}.
Define the alerting conditions under which a new Davis event will be generated. You can pick different Anomaly Detection analyzers.
Finally set the event details like title and description.

4. Open problem

When the alerting conditions are met you will see a new problem in Problems app - new Problems.

Conclusion

Here's when to use a custom alert with metrics based on logs:

You need to set thresholds or use other machine learning analyzers to trigger alerts.
When you want to alert on anomalies in value coming from a log field like http.response_time.
Metric analyzers are triggered every minute so it’s not a real-time alerting method.
Metric dimensions have low cardinality.

Detected anomalies can trigger automations using simple workflows as described in Create a simple workflow in Dynatrace Workflows.

Set up custom alerts based on metrics extracted from logs

Latest Dynatrace

Tutorial

4-min read

Ingested logs can be triggers for opening new Davis problems.

Alert when the average count of matching records exceeds a specific number within a defined time period.
Alert when the value of a metric is abnormal, without setting a static threshold.

Follow this guide to learn more about alerting with metrics based on logs.

If you don't need to set thresholds, you should follow the instructions in Set up alerts based on events extracted from logs.

Steps

In this example we will open a new Davis problem when certain records, which contain a specific phrase, are ingested and exceed a static threshold.

1. Find logs you want to trigger alerts

You can find alerts by opening Logs and using the following DQL query.

fetch logs
| filter matchesPhrase(content, "Dropping data because sending_queue is full")
| sort timestamp desc

If your DQL query uses parse, fieldAdd, or other transformations, you should add a processing rule to set those fields on ingest.
To find which pipeline handles your logs, select a log record in Logs and check the dt.openpipeline.pipelines field. Use that value when selecting a pipeline in OpenPipeline settings.

2. Extract metric in OpenPipeline

Add metric extraction configuration in OpenPipeline.

Open Settings > Process and contextualize > OpenPipeline > Logs and select the Pipelines tab.
Find the pipeline you want to modify, or add a new pipeline.
Select > Edit. The pipeline configuration page appears.
Select Metric extraction tab.
Set
- The metric name and ID.
- The DQL matcher. A matcher sets the condition for the event that is to be extracted. It is a subset of filtering conditions in a single DQL statement.
  
  In Matching condition, use the matcher as shown below.
  matchesPhrase(content, "Dropping data because sending_queue is full")
  If you use Segments or your permissions are set at the record level, you should include those conditions in the matcher.
  
  There are situations when a matcher can't be easily extracted from a DQL statement. In these cases, you can create log alerts for a log event or summary of log data.
Add dimensions. For most logs, you can add automated correlation to entities in Dynatrace Intelligence root cause analysis. To do this, add a dt.source_entity dimension or any other field that contains an entity identifier.

If you added the extraction rule to an existing pipeline, logs are already routed there and no further configuration is needed.

If you created a new pipeline, configure a dynamic route so logs reach it:

Go to Settings > Process and contextualize > OpenPipeline > Logs and select the Dynamic routing tab.
Select Dynamic route and set:
- Name: For example, use Dropping data alert route.
- Matching condition: Use the same condition as the metric extraction matcher so the route narrows the data to the records you actually want to process.
  matchesPhrase(content, "Dropping data because sending_queue is full")
- Pipeline: Select the new pipeline.
Select Add.

Routes are evaluated top to bottom and the first matching route wins. If you have other routes that could match the same logs, place this route above them.

3. Configure a custom alert

Go to Anomaly Detection - new Anomaly Detection and create a new custom alert.

This section describes how to create a simple custom alert. If you need to set additional advanced properties and fine-tune your alert, use the Advanced mode.
Use Suggest values to find the right threshold.
Use Preview to get an estimation of how many alerts would have been generated in the last two hours.

Set the scope for your alert.
Use DQL syntax to point the metric you created. To have your alert connected to monitored entity make sure to add by: {dt.source_entity}.
Define the alerting conditions under which a new Davis event will be generated. You can pick different Anomaly Detection analyzers.
Finally set the event details like title and description.

4. Open problem

When the alerting conditions are met you will see a new problem in Problems app - new Problems.

Conclusion

Here's when to use a custom alert with metrics based on logs:

You need to set thresholds or use other machine learning analyzers to trigger alerts.
When you want to alert on anomalies in value coming from a log field like http.response_time.
Metric analyzers are triggered every minute so it’s not a real-time alerting method.
Metric dimensions have low cardinality.

Detected anomalies can trigger automations using simple workflows as described in Create a simple workflow in Dynatrace Workflows.

Set up custom alerts based on metrics extracted from logs

Prerequisites

Steps

Conclusion

Related topics

Set up custom alerts based on metrics extracted from logs

Prerequisites

Steps

Conclusion

Related topics