Create anomaly detection metric

In this use case, you need to automate anomaly detection. See how you can extract data from logs, create a processing rule, create a metric, and create an alert that generates a notification if an anomaly occurs.

Scenario

Your logs contain important business data in a raw format. It needs to be extracted and normalized before analysis. The log records that you ingest have the following structure:

"timestamp": "2023-08-10T14:30:23.383000000+02:00",
"content": "AddItemAsync called with userId=04e412f0-c030-4d4b-9c16-72250b071fbd, productId=2ZYFJ3GM2N, quantity=4",
"event.type": "LOG",
"host.name": "gke-myhost-cos-bd5b5ae9-sad319",
"k8s.container.name": "cartservice",

You plan to extract the product identifier and quantity already at ingest time with a log processing rule, and then turn it into a metric with the same dimensions. Then you can track this data via the metric. Finally, you create an alert based on the metric values.

  1. Create a DQL query as an ad hoc analytics task to retrieve the product identifiers and quantities for each transaction.
  2. After you've verified that you have received the correct data, turn that log query into a metric. This will speed up your monitoring, and you can automate anomaly detection and keep your query costs under control.
  3. Create an alert based on the value of that metric.

Build DQL query

To build and run your query

  1. Go to Logs or Logs & Events (latest Dynatrace).
  2. On the Logs and events page, turn on Advanced mode.
  3. Select Copy to clipboard Copy for the code sample below. 
    fetch logs
    | filter matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")
    | parse content, "LD 'productId=' LD:product ', quantity=' INT:quantity"
    | fields timestamp, product, quantity
    | summarize sum(quantity), by:{product, bin(timestamp, 1h)}
    | fieldsRename field:`bin(timestamp, 1h)`, alias:`hour`
  4. Paste the query into the query edit box and select Run query.

This query performs the following actions:

  • Retrieves logs for a Kubernetes container named cartservice that contain the phrase AddItemAsync in the log content.
  • Parses the result for product ID and quantity.
  • Presents these values in new fields product and quantity.
  • Aggregates the quantity field for each product per one-hour interval.
  • Renames the field containing the time for one-hour intervals to hour.

Create metric

To create a metric, you first need to create a processing rule that processes the incoming log data and then creates a metric based on log records matching the specific phrase. With each match, the processing rule extracts product ID and quantity values and places them in their respective fields. This provides you with additional dimensions for this metric (productId and quantity).

Step 1 Create log processing rule

  1. Go to Settings > Log Monitoring > Processing and select Add rule.
  2. In Rule name, give your processing rule a name:
    Product and Quantity from logs
  3. Copy the filter command to the Matcher: 
    matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")
  4. Copy the parse command to the Processor definition: 
    PARSE(content, "LD 'productId=' LD:product ', quantity=' INT:quantity")
  5. Select Save changes

Add log processing rule screen.

This processing rule processes the incoming log data, extracts the product ID and quantity, and places their values in their respective fields.

Step 2 Add log metric

  1. Go to Settings > Log Monitoring > Metrics extraction and select Add log metric.
  2. In Key, append the metric name to the log. metric key: cartservice-product-quantity
  3. Add Matcher.
    Use the DQL function for matching phrases, which is part of the Dynatrace Query Language (DQL): 
    matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")
  4. For the Metrc measurement option, select Attribute value.
  5. For Attribute, enter quantity.
  6. Select Add dimension and enter product.
  7. Select Save changes to create the log metric.

Add log metric screen

Check the result in Data Explorer:

  1. Go to Data Explorer.
  2. Query for metric log.cartservice-product-quantity.
  3. For the space aggregation function, select sum.
  4. For Split by, select product.
  5. Select Run query to view the quantity value for each product graph.
  6. Switch to the advanced mode and copy the metric selector query (log.cartservice-product-quantity:splitBy(product):sum).

Data Explorer screen showing log metric.

Create alert

  1. Go to Settings > Anomaly detection > Metric events and select Add metric event.

  2. For Summary, enter Abnormal product quantity.

  3. In the Query definition section:

    • Select the Metric selector option for Type.
    • Enter log.cartservice-product-quantity:splitBy(product):sum for Metric selector.

    Query definition of Metric event screen.

  4. In the Monitoring strategy section:

    • Select the Auto-adaptive threshold option for Model type.
    • Switch on the Alert on missing data.
    • Leave default value (1) for Number of signal fluctuations option.
    • Select Alert if metric is outside option for Alert condition.

    Monitoring strategy of Metric event screen.

  5. In the Event template section:

    • Enter Abnormal product quantity for Title.
    • Modify the description to fit your needs or leave the default description format.

    Event template of Metric event screen.

  6. Select Save changes to create the metric event.

To check the problem alert created by your anomaly detection metric

  1. Go to Problems.
  2. Search for Abnormal product quantity problems.

Problem tile based on log metric event.