Log ingest rules (Logs Classic)

Log Monitoring Classic

Dynatrace version 1.252+ OneAgent version 1.243+

Dynatrace allows you to include and exclude specific log sources for your analysis. Using Dynatrace identity and access management (IAM) framework, you can control which user can change configurations on which scope.

The configuration is based on rules that use matchers for hierarchy, log path, and process groups. These rules determine which log files among those detected by OneAgent, either automatically or defined as custom log sources, are ingested.

Log ingest rule

If no wildcard is used in the value, then the matcher looks for an exact fit to the value. If a wildcard is used, the matcher looks for the exact match. For example, the value INFO results in sending only the log data having the exact INFO string, but the value *INFO* (using the wildcards) matches log data that contain the INFO string in its content.

Defined rules can be reordered and are executed in the order in which they appear on the Log storage page.

Matching a list of rules to log data

Matching occurs in a predefined hierarchy and rules are executed from top to bottom. This means that if a rule above on the list matches certain log data, then the lower ones will be omitted. Items matched in the higher-level configurations are overwritten in the lower-level configurations if they match the same log data. If no rule is matched, the file is not sent. The matching hierarchy is as follows:

Configuration scopes

Three hierarchy scopes are supported: host, host group, and tenant. The scope with the least possible set of rules has priority over larger sets.

Host scope

The host scope can be accessed through the Host settings for a specific host.

Host group scope

The host group scope can be accessed via the Host page.

Tenant scope

The tenant scope is available in the settings menu.

List hosts and host groups with overriding rules

The table on Settings > Log Monitoring > Log ingest rules lists all log storage rules that you have set at the tenant level. However, you may want to see where you have set log storage rules for hosts and host groups that override the tenant-level rules.

To list all entities (hosts and host groups) to which more specific log storage rules are applied

Example upload

In this example, we configure the tenant storage upload for c:\inetpub\logs\LogFiles\ex_*.log files in two process groups: IIS (PROCESS_GROUP-3D9D854163F8F07A) and IIS (PROCESS_GROUP-4A7B47FDB53137AE). The log storage rule consists of two matchers: the first matcher finds the process groups and the second matcher matches only for the defined log source.

Example exclude

In this example, we configure the tenant storage upload for all log sources except c:\inetpub\logs\LogFiles\ex_*.log files in a process group IIS (PROCESS_GROUP-4A7B47FDB53137AE).

REST API

You can use the Settings API to manage your log ingest rules:

To check the current schema version for log ingest rules, list all available schemas and look for the builtin:logmonitoring.log-storage-settings schema identifier.

Log ingest rules can be configured for the following scopes:

To create a log ingest rule using the API:

Examples

The examples that follow show the results of various combinations of rules and matchers.

Example 1: Multiple rules

In this example, there are two rules:

Results: access.log is not sent, error.log (of Apache) is sent, and error.log (of other PG) is not sent.

{
  "send-to-storage": false,
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/access.log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 2: Send logs written by Apache and containing 'ERROR'

This task requires setting one rule with two matchers.

{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 3: Send logs written by Apache or containing 'ERROR'

This task requires setting two rules with one matcher each.

{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 4: Send logs written by Apache, and containing 'ERROR' and 'Customer'

This task requires setting one rule with three matchers, with one value each.

{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*"
      ]
    },
    {
      "attribute": "log.content",
      "values": [
        "*Customer*"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 5: Send logs written by Apache, and containing 'ERROR' or 'Customer'

This task requires setting one rule with two matchers: a matcher with the process group value, and a matcher with two content values.

{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*", "*Customer*"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 6: Send logs written by Apache or MySQL

This task requires setting two rules, or one rule with one matcher having two values.
Rules with two matchers will not work here.

Setting two rules:

{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Setting one rule with one matcher having two values:

{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID", "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
}

Example 7: Send all logs

This task requires setting a rule without any matchers.

{
  "send-to-storage": true,
  "matchers": [
  ],
  "enabled": true
}

Example 8: Send all logs except Apache and MySQL logs

This task requires setting two rules.

The rules have to be executed in the order indicated below.

{
  "send-to-storage": false,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID", "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
  ],
  "enabled": true
}

Frequently asked questions