Ingest data from log files

The following configuration example shows how to configure a Collector instance to monitor log files and send their log entries to the Dynatrace backend.

Prerequisites

Demo configuration

receivers:
  filelog:
    include: [ /path/to/file.log ]
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d'
        severity:
          parse_from: attributes.sev


exporters:
  otlphttp/dynatrace:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      "Authorization": "Api-Token ${env:DT_API_TOKEN}"


service:
  pipelines:
    traces: null
    metrics: null
    logs:
      receivers: [filelog]
      processors: []
      exporters: [otlphttp/dynatrace]
Configuration validation

Validate your settings to avoid any configuration issues.

Sample log file

For the demo configuration above, we parse the file file.log with the following format:

1970-01-01 INFO Something routine
1970-01-01 ERROR Some error occurred!
1970-01-01 DEBUG Some details...

Each line starts with an ISO 8601 timestamp, followed by the entry's severity level, and finishes with the log message.

We parse each line into its individual parts with the following regular expression:

^(?P<time>\d{4}-\d{2}-\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$

Apart from the two start (^) and end ($) of line assertions, we have the following named capture groups:

  • (?P<time>\d{4}-\d{2}-\d{2})—Names its capture group time and matches a typical ISO 8601 timestamp.
  • (?P<sev>[A-Z]*)—Names its capture group sev and matches an arbitrary number of Latin uppercase characters.
  • (?P<msg>.*)—Names its capture group msg and matches an arbitrary number of characters.

Components

For our configuration, we use the following components.

Receiver

Under receivers, we specify the filelog receiver as active receiver component for our Collector instance.

The Filelog receiver supports a number of configuration parameters, which enable you to customize its behavior. For our example, we use the following:

  • include—Specifies the path pattern of the files we want to ingest.
  • start_at—Specifies if the receiver should read from the beginning of the file or, for the most recent entries only, the end.
  • operators—Configures the operators we apply to each log entry. For our example, we use the regex_parser operator to extract information using a regular expression.
    • regex—Specifies the actual regular expression. By using named capture groups ((?P<name>)), the receiver makes the captured data available in attributes under the respective name.
    • timestamp—Specifies where to take the entry's timestamp from (the time field of the regular expression) and the date format.
    • severity—Specifies where to take the entry's severity level from (the sev field of the regular expression).

Exporter

Under exporters, we specify the default otlphttp exporter and configure it with our Dynatrace API URL and the required authentication token.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.

Service pipeline

Under service, we eventually assemble our receiver and exporter objects into a traces pipeline, which will continuously monitor the configured files and ingest their entries into Dynatrace using OTLP.

Limits and limitations

Logs are ingested using the OpenTelemetry protocol (OTLP) via the Dynatrace OTLP API and are subject to the API's limits and restrictions. For more information see:

Related topics