Log content autodiscovery

By default, Dynatrace automatically discovers and analyzes new log files, and stores them if selected for ingestion based on log ingest rules. For trial users, everything is already set up, allowing you to start analyzing logs immediately without additional configuration.

Dynatrace automatically discovers all new log files that meet the requirements described below.

Default autodiscovery

Dynatrace automatically discovers, analyzes, and stores (if selected for ingestion) logs every 60 seconds.

Whether your autodiscovered files are stored in Dynatrace depends on the log ingest rules.

By default, the OneAgent log module autodiscovers the following categories of log files:

  • System logs
    On Windows:

    • Windows Security Log
    • Windows Application Log
    • Windows System Log

    On Linux:

    • /var/log/messages
    • /var/log/syslog

  • Log files opened by running processes. For details, see Log autodiscovery requirements

  • IIS Logs (Windows only) - both event logs and plain log files

  • Container logs (Linux only) in Kubernetes, Openshift, and non-instrumented Docker. For details, see Log Monitoring in Kubernetes

  • z/OS logs. For details, see Monitor z/OS logs

If your logs are not ingested, it can be either because the OneAgent Log Enablement is disabled, or because the logs breach a rule concerning OneAgent Log Security.

Attributes selected in Windows event logs

For Windows event logs, Log Monitoring detects the following fields and sends them as custom attributes:

Semantic attribute name

Event property

winlog.level

Event.RenderingInfo.Level

winlog.levelid

Event.System.EventID

winlog.provider

Event.System.Provider.<xmlattr>.Name

winlog.task

Event.System.Task

winlog.opcode

Event.RenderingInfo.Opcode

Autodiscovery requirements

A log file must meet all of the following requirements in order to be autodiscovered:

  • The log file must be opened by an important process.

  • The log file must exist for a minimum of one minute.

    Unsupported timestamps

    Files with an unsupported timestamp are automatically timestamped with the time the file was read.

  • The logs must have a supported character encoding. By default, the supported encoding is UTF-8. Other supported types include UTF-8 BOM and, if the files contain the byte-order mark (BOM), UTF-16LE and UTF-16BE.

    Binary logs

    Binary log files are not detected automatically. You can use custom log sources with Allow binary format option set to ingest Binary log files.

  • The log file must be at least 0.5 KB in size.

  • The log file must have been updated (written to) in the last 7 days.
    Log files that have not been updated in the past 7 days while Log Monitoring is active will not be visible on dashboards.

  • The log file must be in the actual log or logs folder or in its subfolders:

    • Valid path examples:
      c:\log\log_file.txt
      c:\logs\NewFolder\log_file.txt
    • Invalid path example:
      c:\log\NewFolder\NewFolder\log_file.txt

    or the log filename must contain a log string preceded or followed by the period (.) or underscore (_) character:

    • Valid filename examples:
      c:\NewFolder\abc.log
      c:\NewFolder\0865842.log.txt
    • Invalid filename example:
      c:\NewFolder\logfile.txt

  • Log files are ingested only if their timestamp is within the last 24 hours. See Log Management and Analytics default limits for more details.

  • Log entries with timestamps more than 10 minutes ahead of the current time are overridden. See Log Management and Analytics default limits for more details.

Turn off log autodiscovery

If you don't want Dynatrace to automatically discover new log files on a specific monitored host, you can turn off log autodiscovery.

See Configuration scopes to learn about configurations on the tenant, host group, and host.

In the tenant, follow the steps below.

  1. Go to Settings > Log monitoring > Advanced log settings.
  2. Disable each of the settings pertaining to log detection:
    • Detect open log files: This option automatically detects logs written by important processes.
    • Detect system logs:
      • Linux: Detects syslogs, and message logs.
      • Windows: Detects system, application, and security event logs.
    • Detect logs of containerized applications: This option allows the detection of log messages written to the containerized application's stdout/stderr streams. It also detects Kubernetes pod logs.
    • Detect IIS logs: This option allows the detection of logs and event logs written by the Microsoft IIS server.
    • Detect logs on network file systems: This option allows the detection of logs written to mounted network storage drives. This applies only to Linux hosts. For other operating systems, it's always enabled.

Tenant-side configuration is the preferred approach for managing autodetection. However, in specific use cases, the host-side configuration below is considered an option.

  1. Open the log analytics configuration file for editing.
    • On Linux:
      /var/lib/dynatrace/oneagent/agent/config/ruxitagentloganalytics.conf
    • On Windows:
      %PROGRAMDATA%\dynatrace\oneagent\agent\config\ruxitagentloganalytics.conf
  2. Set the following:
    AppLogAutoDetection = false

OneAgent restart is not required.

Limits for your log autodiscovery when using OneAgent

Log files in OneAgent:

  • cannot be deleted earlier than a minute after creation.
  • must be appended (old content is not updated).
  • must have text content.
  • must be opened constantly (not just for short periods of adding log entries).
  • must be opened in write mode.

Built-in autodetector rules

The autodetector has the following additional built-in rules:

OS
Directory pattern
File pattern
Action
All
/Log/
\*xel
Exclude
All
/commitlog/
CommitLog\*log
Exclude
Windows
/CCM/Logs/
\*
Exclude
Windows
/MSSQL/Log/
\*trc
Exclude
Windows
/*MSSQL*/OLAP/Log/
\*trc
Exclude
Windows
MSSQL/DATA/
\*ldf
Exclude
Windows
\*evtx
Exclude
Linux
/var/log/pods/
\*
Exclude
Linux
/var/lib/docker/containers/
\*
Exclude
All
/
*\[-.*]log\[-.*]*
Include
All
/
\*\[-.\_]log
Include
All
/
catalina.out\*
Include
All
/log/
\*
Include
All
/log/*/
\*
Include
All
/logs/
\*
Include
All
/logs/*/
\*
Include
Linux
^/var/log/**/
\*
Include

The custom security rules can only narrow auto-detection but not expand it.

Each custom log source path you add needs to be validated by OneAgent and abide by its security rules (file matching rules). The following security rules are applied on the OneAgent side.

Important

Security rules can't be used to expand the autodetector to detect more files. You can only use them to limit the detection of files. If you need to include a log source that isn't automatically detected, use a custom log source configuration.

See Security rules to learn how to configure security rules for custom log sources to ensure data protection.