Log processing

Dynatrace Log Management and Analytics can reshape incoming log data for better understanding, analysis, or further processing based on rules that you create.

We recommend using OpenPipeline as a scalable, powerful solution to manage, process, and analyze logs. If you don't have access to OpenPipeline, use the classic log processing pipeline described here.

LMA - Log processing pipeline

Information can be logged in a very wide variety of formats depending on the application or process, operating system or other factors. Log Processing offers a central and flexible way of extracting value from those raw log lines.

Log processing occurs as log data arrives in the Dynatrace SaaS environment and before it is written to disk (stored). By setting log processing rules, you can process the log data once it reaches Dynatrace. After the log data is processed, it is sent to storage and is available for further analysis. This method allows to process log data from all log ingest channels.

For example, you can extract numerical values from log lines with Log Processing, turn these into metrics on the Dynatrace Platform, and include them in dashboards and problem detection.

DDU consumption

Log processing does not affect DDU consumption of log ingest.

Log processing is based on rules that contain a matcher and a processing rule definition.

  • The matcher narrows down the available log data for executing this specific rule.
  • The processing rule is a log processing instruction about how Dynatrace should transform or modify the log data from the matcher.

Log processing rules

Go to Settings > Log Monitoring > Processing to view log processing rules that are in effect, reorder the existing rules, and create new rules. Rules are executed in the order in which they're listed, from top to bottom. This order is critical because a preceding rule may impact the log data that a subsequent rule uses in its definition.

All processing rules that match a log record are applied from top to bottom. An output from one rule is an input for the next one.

Expand Details to examine a rule definition. A log processing rule consists of the following:

  • Rule name
  • Matcher
  • Rule definition

You can turn any rule on or off in the Active column.

Built-in rules

By default, log processing includes many enabled built-in rules responsible for cleaning up or normalizing log data. The name of every built-in rule starts with [Built-in]. You cannot modify these rules directly, but you have the ability to turn them off, copy their definitions and create new rules with your modifications.

Add rule

To create a log processing rule

  1. Select Add processing rule on the Log processing page.

  2. Provide the name for the log processing rule.

  3. Provide a log query in the Matcher section.
    A log search query narrows down the available log data for executing this specific rule. Add a Matcher to your rule by pasting your matcher-specific DQL query.

    Matching based on previous rules is not supported

    The Matcher operates on the initial data set before applying any processing rules. Matching records modified by preceding rules is not supported. For example, the modified field in rule 1 and used for matching in rule 2 will contain the original value for that field and will not use the modified field in rule 1.

  4. Provide the processing rule definition.
    The processing rule definition is a log processing instruction about how Dynatrace should transform or modify your log data.

    The rule definition is created using log processing commands, functions and pattern matching (Dynatrace Pattern Language) that allows you to add, transform or remove incoming log records. This gives you total control over how your log data is presented to Dynatrace log monitoring.

  5. Test the log processing rule.

    1. Provide a log sample
    You can test the rule definition by providing a fragment of the sample log manually in the Paste a log / JSON sample text box. Make sure it's in JSON format. Any textual log data should be inserted into the content field of the JSON.

    2. Run the test
    Select Test the rule and view the result in the Test result text box.

  6. Select Save changes.

Related topics