Ingest data from log files
The following configuration example shows how to configure a Collector instance to monitor log files and send their log entries to the Dynatrace backend.
Prerequisites
- One of the following Collector distributions with the Filelog receiver:
- The Dynatrace API endpoint URL to which the data should be exported
- An API token with the relevant access scope (only required for SaaS and ActiveGate)
Demo configuration
receivers:filelog:include: [ /path/to/file.log ]start_at: beginningoperators:- type: regex_parserregex: '^(?P<time>\d{4}-\d{2}-\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'timestamp:parse_from: attributes.timelayout: '%Y-%m-%d'severity:parse_from: attributes.sevexporters:otlphttp/dynatrace:endpoint: ${env:DT_ENDPOINT}headers:"Authorization": "Api-Token ${env:DT_API_TOKEN}"service:pipelines:traces: nullmetrics: nulllogs:receivers: [filelog]processors: []exporters: [otlphttp/dynatrace]
Validate your settings to avoid any configuration issues.
Sample log file
For the demo configuration above, we parse the file file.log
with the following format:
1970-01-01 INFO Something routine1970-01-01 ERROR Some error occurred!1970-01-01 DEBUG Some details...
Each line starts with an ISO 8601 timestamp, followed by the entry's severity level, and finishes with the log message.
We parse each line into its individual parts with the following regular expression:
^(?P<time>\d{4}-\d{2}-\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$
Apart from the two start (^
) and end ($
) of line assertions, we have the following named capture groups:
(?P<time>\d{4}-\d{2}-\d{2})
—Names its capture grouptime
and matches a typical ISO 8601 timestamp.(?P<sev>[A-Z]*)
—Names its capture groupsev
and matches an arbitrary number of Latin uppercase characters.(?P<msg>.*)
—Names its capture groupmsg
and matches an arbitrary number of characters.
Components
For our configuration, we use the following components.
Receivers
Under receivers
, we specify the filelog
receiver as active receiver component for our Collector instance.
The Filelog receiver supports a number of configuration parameters, which enable you to customize its behavior. For our example, we use the following:
include
—Specifies the path pattern of the files we want to ingest.start_at
—Specifies if the receiver should read from the beginning of the file or, for the most recent entries only, the end.operators
—Configures the operators we apply to each log entry. For our example, we use the regex_parser operator to extract information using a regular expression.regex
—Specifies the actual regular expression. By using named capture groups ((?P<name>)
), the receiver makes the captured data available inattributes
under the respective name.timestamp
—Specifies where to take the entry's timestamp from (thetime
field of the regular expression) and the date format.severity
—Specifies where to take the entry's severity level from (thesev
field of the regular expression).
Exporters
Under exporters
, we specify the default otlphttp
exporter and configure it with our Dynatrace API URL and the required authentication token.
For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint
and Authorization
.
DT_ENDPOINT
contains the base URL of your ActiveGate (for example,https://{your-environment-id}.live.dynatrace.com/api/v2/otlp
)DT_API_TOKEN
contains the API token
Service pipelines
Under service
, we eventually assemble our receiver and exporter objects into a traces pipeline, which will continuously monitor the configured files and ingest their entries into Dynatrace using OTLP.
Limits and limitations
Logs are ingested using the OpenTelemetry protocol (OTLP) via the Dynatrace OTLP API and are subject to the API's limits and restrictions. For more information see: