Custom log source

Custom log source configuration enables you to manually add log sources that have not been autodetected.

  • Autodiscovery might not identify a log source if a log file is not kept open for writing during a process.
  • It might also fail to find log sources that are not part of any processes or are part of short-lived processes.

In such cases, you can define, configure, and customize log sources to your needs.

The entire process consists of two parts:

  1. Source definition (custom log source configuration), which is described on this page.
  2. Log acquisition (adding logs to storage), which is described on Log ingest rules.

If you need to store your custom logs, you need to complete both steps.

Each custom log source path you add needs to be validated by OneAgent and abide by its security rules. See Security rules for configuration files and examples.

Advantages of custom log source configuration

Compared to the log addition in earlier versions of OneAgent versions, the process now has several improvements.

Custom log source configuration enables you to:

  • Define attributes that will enrich your logs.
  • Define log sources without the process group context (adding a process group is optional). You can add up to three process groups to a log source.
  • Define rules within all three scopes (host, host group, environment).
  • Change security rules through files placed on the host.
  • Add a rule with multiple paths within one process group.
  • Use wildcards in directories.
  • Use a dedicated API.

Supported scopes

Three hierarchy scopes are supported: host, host group, and environment. The narrower a given scope, the higher its priority.

  • Log source rules configured for a host take precedence over log source rules configured for a host group.
  • Log source rules configured for a host group take precedence over log source rules configured for a Dynatrace environment.

Hosts

To configure custom log sources at the host level

  1. Go to Hosts or Hosts Classic (latest Dynatrace) and select your host.
  2. Select More (…) > Settings to open the Host settings page (available only on hosts assigned to a host group).
  3. On the Host settings page, select Log Monitoring > Custom log sources.

Host groups

To configure custom log sources at the host group level

  1. Go to Hosts or Hosts Classic (latest Dynatrace) and select your host.
  2. Expand the Properties and tags section and select the Host group (available only on hosts assigned to a host group).
  3. On the Host group settings page, select Log Monitoring > Custom log sources.

Environment

To configure custom log sources at the environment level

  1. Go to Settings.
  2. On the Settings page, select Log Monitoring > Custom log sources.

Configure log source

  1. Go to the Custom log sources page at the host, host group, or environment level as described above.

  2. Select Add custom log source and add Rule name.

  3. optional Bind your rule to a process group by selecting the process group name from the dropdown menu.

  4. In the Custom log source paths section, select Log source type. There are two source types available:

    • Log path
    • Windows Event Log Each rule is applied to one log source type. You can select either Log path or Windows Event Log.

  5. To define a log path, select Add custom log source path, enter your path (for example, /var/lib/*.log or /var/log/sys.bin), and select Add Path. You can add up to 100 values per log source.

    As in the following example, you can display the log name and path by right-clicking on the chosen event log and selecting Properties:

    Windows event viewer properties screen.

    Windows viewer log properties.

  6. optional Select Show advanced to expand the panel that lets you define the list of attributes which will enrich each log record from the defined log sources. Enter the attribute key or select it from the list, and then enter the attribute value. This is available only if you have selected the Log source type as Log Path.

    When using wildcards in the log path, you may want to distinguish the paths matched by the wildcards. In such cases, you can define attributes that use the whole file path or a part of the path matched by the wildcards.

    To define such an attribute, follow the steps below:

    1. Enter the key or select it from the list.
    2. In the Attribute value field, use the ${N} token, where N denotes the index of the wildcard you refer to, starting from 1. ${0} has a special meaning and expands to the full log file path.

    You can use multiple ${N} tokens in a single attribute and combine them with other characters. For example, worker:${1}-${2}.

    If the ${N} token refers to a wildcard index higher than the number of wildcards in the log path, it won't be replaced, and ${N} will remain in the attribute value. The attribute key must contain only Latin alphanumeric characters (upper or lower case), dots (.), underscores (_), hyphens (-), or colons (:). It must not start with the dt. prefix and must not be any of the following:

    process.technology, log.source, log.content, timestamp, container.name, winlog.eventid, winlog.provider, winlog.opcode, winlog.task, k8s.namespace.name, k8s.container.name, k8s.deployment.name

  7. Select Save changes.

  8. To activate your rule, turn on the Active toggle.

Set up permissions on Network File System (NFS)

When handling logs on NFS, access permissions are strictly enforced. This is true despite the increased capabilities of OneAgent, which allow it to access local files without having to read permissions for the dtuser user.

To allow OneAgent to process and ingest NFS-mounted resources:

  • Every non-local directory along the path needs to have at least the read and execute permissions set.
  • Every log file needs to have the read permission set.

Example log file to add:

/mnt/nfs/logs/app1/test.log
where /mnt/nfs is mounted to an external NFS resource.

In this scenario, both /mnt/nfs/logs and /mnt/nfs/logs/app1 need r-xpermissions for others, and every test.log.* file needs r-- permissions for others as shown below:

$ ls -l /mnt/nfs
drwxr-xr-x 3 1001 1002 4096 Sep 8 17:11 logs
$ ls -l /mnt/nfs/log
drwxr-xr-x 3 1001 1002 4096 Sep 8 17:11 app1
$ ls -l /mnt/nfs/logs/app1
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:22 test.log
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:23 test.log.1
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:24 test.log.2

Log file matching

When configuring a custom log source, follow these rules:

  • Custom log paths must be absolute; relative paths are rejected. An absolute path has the following pattern:
    • For Windows: any letter:\
    • For Linux: Starts with /
  • A Windows Event Log path in Windows Event System must be a relative path.
  • Custom log sources can contain wildcards: # replaces a string of numbers, while * substitutes a string of any characters except for slash (/) or backslash (\). While * can be used both in file names and directories, # can be used only in file names.