Ingest data from log files
The following configuration example shows how to configure a Collector instance to monitor log files and send their log entries to the Dynatrace backend.
Prerequisites
- One of the following Collector distributions with the Filelog receiver
- The API URL of your Dynatrace environment
- An API token with the relevant access scope
Demo configuration
receivers:filelog:include: [ /path/to/file.log ]start_at: beginningoperators:- type: regex_parserregex: '^(?P<time>\d{4}-\d{2}-\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'timestamp:parse_from: attributes.timelayout: '%Y-%m-%d'severity:parse_from: attributes.sevexporters:otlphttp/dynatrace:endpoint: ${env:DT_ENDPOINT}headers:"Authorization": "Api-Token ${env:DT_API_TOKEN}"service:pipelines:logs:receivers: [filelog]exporters: [otlphttp/dynatrace]
Sample log file
For the demo configuration above, we parse the file file.log with the following format:
1970-01-01 INFO Something routine1970-01-01 ERROR Some error occurred!1970-01-01 DEBUG Some details...
Each line starts with an ISO 8601 timestamp, followed by the entry's severity level, and finishes with the log message.
We parse each line into its individual parts with the following regular expression:
^(?P<time>\d{4}-\d{2}-\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$
Apart from the two start (^) and end ($) of line assertions, we have the following named capture groups:
(?P<time>\d{4}-\d{2}-\d{2})—Names its capture grouptimeand matches a typical ISO 8601 timestamp.(?P<sev>[A-Z]*)—Names its capture groupsevand matches an arbitrary number of Latin uppercase characters.(?P<msg>.*)—Names its capture groupmsgand matches an arbitrary number of characters.
Components
For our configuration, we use the following components.
Receiver
Under receivers, we specify the filelog receiver as active receiver component for our Collector instance.
The Filelog receiver supports a number of configuration parameters, which enable you to customize its behavior. For our example, we use the following:
include—Specifies the path pattern of the files we want to ingest.start_at—Specifies if the receiver should read from the beginning of the file or, for the most recent entries only, the end.operators—Configures the operators we apply to each log entry. For our example, we use the regex_parser operator to extract information using a regular expression.regex—Specifies the actual regular expression. By using named capture groups ((?P<name>)), the receiver makes the captured data available inattributesunder the respective name.timestamp—Specifies where to take the entry's timestamp from (thetimefield of the regular expression) and the date format.severity—Specifies where to take the entry's severity level from (thesevfield of the regular expression).
Exporter
Under exporters, we specify the default otlphttp exporter and configure it with our Dynatrace API URL and the required authentication token.
For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.
DT_ENDPOINTcontains the base URL of your ActiveGate (for example,https://{your-environment-id}.live.dynatrace.com/api/v2/otlp)DT_API_TOKENcontains the API token
Service pipeline
Under service, we eventually assemble our receiver and exporter objects into a traces pipeline, which will continuously monitor the configured files and ingest their entries into Dynatrace using OTLP.
Limits and limitations
Logs are ingested using the OpenTelemetry protocol (OTLP) via the Dynatrace OTLP API and are subject to the API's limits and restrictions. For more information see: