Custom log source (Logs Classic)

Log Monitoring Classic

OneAgent version 1.251+ Dynatrace Cluster version 1.254+

In OneAgent version 1.249 and earlier, you need to add log files manually.

Custom log source configuration enables you to manually add log sources that have not been autodetected.

Autodiscovery might not identify a log source if a log file is not kept open for writing during a process.
It might also fail to find log sources that are not part of any processes or are part of short-lived processes.

In such cases, you can define, configure, and customize log sources to your needs.

Starting from OneAgent version 1.251, you can switch to the improved version of manual addition, which is the custom log source configuration. You can opt-in by selecting a banner on the Custom log source configuration page.

The entire process consists of two parts:

Source definition (custom log source configuration), which is described on this page.
Log acquisition (adding logs to storage), which is described on Log ingest rules.

If you need to store your custom logs, you need to complete both steps.

Advantages of custom log source configuration

Compared to the log addition in earlier versions of OneAgent versions, the process now has several improvements.

Custom log source configuration enables you to:

Define log sources without the process group context (adding a process group is optional). You can add up to three process groups to a log source.
Define rules within all three scopes (host, host group, environment).
Change security rules through files placed on the host.
Add a rule with multiple paths within one process group.
Use wildcards in directories.
Use a dedicated API.

Automatically migrate your legacy custom log source configuration: each of your existing rules is migrated to the environment scope with the corresponding process group context set accordingly. The names of migrated rules have the auto-migrated prefix.

Supported scopes

Three hierarchy scopes are supported: host, host group, and environment. The narrower a given scope, the higher its priority.

Log source rules configured for a host take precedence over log source rules configured for a host group.
Log source rules configured for a host group take precedence over log source rules configured for a Dynatrace environment.

Hosts

To configure custom log sources at the host level

Go to Hosts or Hosts Classic (latest Dynatrace).
Find and select your host to display the host overview page.
In the upper-right corner of the host overview page, select More (…) > Settings.

From the host settings, go to Log Monitoring > Custom log sources.

Host groups

To configure custom log sources at the host group level

Go to Hosts or Hosts Classic (latest Dynatrace) and select the host that interests you.
On the host overview page, select Properties and tags.
On the Properties and tags panel, find the Host group property to see the name of the host group to which the selected host belongs.
The Host group property is not displayed when the selected host doesn't belong to any host group.
Select the host group name to list all hosts in that host group. This displays the OneAgent deployment page filtered by the selected host group. Each listed host has a Host group: <group name> link, where <group name> is the name of the host group that you want to configure.
Select the host group name in any row.

In the host group settings, select Log Monitoring > Custom log sources

Environment

To configure custom log sources at the environment level

Go to Settings > Log Monitoring > Custom log sources.

Configure log source

Go to the Custom log sources page at the host, host group, or environment level as described above.
Select Add custom log source and add Rule name.
optional Bind your rule to a process group by selecting the process group name from the dropdown menu.
In the Custom log source paths section, select Log source type. There are two source types available:
- Log path
- Windows Event Log Each rule is applied to one log source type. You can select either Log path or Windows Event Log.
To define a log path, select Add custom log source path, enter your path (for example, /var/lib/*.log or /var/log/sys.bin), and select Add Path. You can add up to 100 values per log source. If you selected Windows Event Log, refer to Windows event example to add a proper log path.
optional Select Show advanced to expand the panel that lets you define the list of attributes which will enrich each log record from the defined log sources. Enter the attribute key or select it from the list, and then enter the attribute value. This is available only if you have selected the Log source type as Log Path.
When using wildcards in the log path, you may want to distinguish the paths matched by the wildcards. In such cases, you can define attributes that use the whole file path or a part of the path matched by the wildcards.
To define such an attribute, follow the steps below:
1. Enter the key or select it from the list.
2. In the Attribute value field, use the ${N} token, where N denotes the index of the wildcard you refer to, starting from 1. ${0} has a special meaning and expands to the full log file path.
You can use multiple ${N} tokens in a single attribute and combine them with other characters. For example, worker:${1}-${2}.
If the ${N} token refers to a wildcard index higher than the number of wildcards in the log path, it won't be replaced, and ${N} will remain in the attribute value. The attribute key must contain only Latin alphanumeric characters (upper or lower case), dots (.), underscores (_), hyphens (-), or colons (:). It must not start with the dt. prefix and must not be any of the following:
```
process.technology, log.source, log.content, timestamp, container.name, winlog.level, winlog.eventid, winlog.provider, winlog.opcode, winlog.task, winlog.keywords, winlog.username, k8s.namespace.name, k8s.container.name, k8s.deployment.name
```
You can define up to 100 attributes per log path. The minimum required version is OneAgent version 1.285.
Select Save changes.
To activate your rule, turn on the Active toggle.

Set up permissions on Network File System (NFS)

When handling logs on NFS, access permissions are strictly enforced. This is true despite the increased capabilities of OneAgent, which allow it to access local files without having to read permissions for the dtuser user.

To allow OneAgent to process and ingest NFS-mounted resources:

Every non-local directory along the path needs to have at least the read and execute permissions set.
Every log file needs to have the read permission set.

Example log file to add:

/mnt/nfs/logs/app1/test.log
where /mnt/nfs is mounted to an external NFS resource.

In this scenario, both /mnt/nfs/logs and /mnt/nfs/logs/app1 need r-xpermissions for others, and every test.log.* file needs r-- permissions for others as shown below:

$ ls -l /mnt/nfs
drwxr-xr-x 3 1001 1002 4096 Sep 8 17:11 logs

$ ls -l /mnt/nfs/log
drwxr-xr-x 3 1001 1002 4096 Sep 8 17:11 app1

$ ls -l /mnt/nfs/logs/app1
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:22 test.log
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:23 test.log.1
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:24 test.log.2

Log file matching

When configuring a custom log source, follow these rules:

Custom log paths must be absolute; relative paths are rejected. An absolute path has the following pattern:
- For Windows: any letter:\
- For Linux: Starts with /
- For NFS: Starts with //hostname/
A Windows Event Log path in Windows Event System must be a relative path.
Custom log sources can contain wildcards:
- # replaces a string of numbers or a hash #. It can only be used in file names.
- * substitutes a string of any characters including the asterisk *. It does not substitute the slash (/) or backslash (\). * can be used both in file names and directories.

Additionally, each custom log source path you add needs to be validated by OneAgent and abide by its security rules (file matching rules). The following security rules are applied on the OneAgent side:

Security rules

Dynatrace's security rules for custom log sources ensure data protection by managing OneAgent access to log sources. By implementing these security measures, Dynatrace ensures defense against unauthorized access and data misuse. You also have the option to add or override these predefined security rules in the configuration file on the host where OneAgent is intalled, allowing for flexibility in adapting to specific security needs.

The rules prohibit log paths in critical system directories (such as /etc, /boot, /proc, and several others), paths containing .ssh, paths with the .pem extension, and paths in directories starting with a dot (indicating hidden directories). Additionally, acceptable log paths must either have a log extension, with certain separators, be located within the first or second level of a log or logs directory, be situated at any level of the /var/log directory, or have the filename catalina.out.

The rules take into account the resolved paths of symbolic links for security matching, emphasizing the importance of the actual file location over the symlink path.

A log path is not in any of the following: /etc, /boot, /proc, /dev, /bin, /sbin, /usr, WindowsRoot:\windows, or WindowsRoot:\winnt. However, Windows|winnt\system32\winevt\Logs is accepted AND
A log path does not contain .ssh AND
A log path does not have the .pem extension AND
A log path is not located in a directory whose name starts with . (for example, /.hidden) AND
A log path must have the log extension separated by ., -, or _ (it can be followed by another extension with the same separator set) OR
- A log path must be located on the first or second level of the log or logs directory OR
- A log path must be located on any level of the /var/log directory OR
- A log path must have the file name catalina.out.

Files with paths that do not fulfill one or more criteria are not accepted. Once the conditions above are met, log file matching takes place. Check the log file matching rules.

If the log file you've configured for ingestion is a symlink, Dynatrace will verify its log path (the path obtained after following the symlink) against security rules.

Override security rules

You can add or override predefined security rules only in the configuration file on the host where OneAgent is installed.

Save your changes as a separate file placed in the OneAgent persistent configuration directory.
- /var/lib/dynatrace/oneagent/agent/config/logmodule on Linux and UNIX
- %PROGRAMDATA%\dynatrace\oneagent\agent\config\logmodule on Windows
  
  Any log file with the .json suffix is allowed in the above directories.
Do not edit the file that contains predefined rules:
- /opt/dynatrace/oneagent/agent/conf on Linux and UNIX
- %PROGRAMFILES%\dynatrace\oneagent\agent\conf on Windows
Rules defined by you under the custom configuration take precedence over the default rules. Additionally, the first matching rule determines whether a path passes the security test. The override configuration file (the one that you save in the persistent configuration directory) format needs to be the same as the format for a file with predefined rules.

Override configuration file

There is a predefined directory pattern that is executed from right to left. For example, /log/ will match /log/file and /var/log/file but not /log/dir/file
Only one directory is matched. For example, /log/*/ will match /log/dir/file but not /log/dir/dir2/file
The [-.\\_] expression in a pattern means that one of the characters provided in the square brackets must be present for a match to occur.

Example override configuration file

The following structure is given in the file:

allowed-log-paths-configuration: Marks the array of the rules.

Each rule consists of three key-value pairs, with the following mandatory keys:

directory-pattern
file-pattern
action

The description of the keys is given below:

directory-pattern: This object specifies the pattern for matching directories. The directory pattern is executed from right to left, for example: /log/ will match /log/file.txt and /var/log/file.txt but not /log/dir/file.txt. The following rules apply:
- A directory is matched by a wildcard *. For example, /log/*/ will match /log/dir/file.txt but not /log/dir/dir2/file.txt.
- ** matches any number of subdirectories. For example, the pattern /log/dir/**/file.txt will match /log/dir/dir1/dir2/dir3/file.txt.
- ^ matches the start of the path. It anchors the pattern to the beginning of the examined path For example, ^/usr/*/ matches paths starting with /usr/, such as /usr/log/file.txt and /usr/local/file.txt, but will not match /some/usr/log/file.txt. For Windows paths, the anchor can also skip the drive letter. For example, the pattern ^/Users/Public/ would match the actual path C:\Users\Public\file.txt. JSON treats \ as an escape character, so when specifying Windows paths, you can use either C:\\Users\\Public or C:/Users/Public but not C:\Users\Public.
You can combine special characters such as *, **, and ^ within a single directory pattern to create more complex matching rules. For example, the pattern ^/log/**/dir/*/*/ will match the path /log/some/deep/dir/and/deeper/file.txt.
file-pattern: This object specifies the pattern for matching files within the directories matched by the directory pattern. This pattern is applied using full match. This means that a pattern such as *.txt will match error.txt but not error.txt.1. To properly detect files that follow rotation patterns, the file pattern must include a wildcard at the end. For example, to match files that rotate from error.txt to error.txt.1, the file pattern should be constructed as *.txt*.
action: This object specifies the action to be taken for the matched file. In this case, EXCLUDE or INCLUDE.

The [-.\\_] expression in square brackets means that one of the characters provided in the square brackets must be present for a match to occur.

An example override configuration file is given below:

{
    "allowed-log-paths-configuration":[
       {
          "directory-pattern":"/",
          "file-pattern":"*.pem",
          "action":"EXCLUDE" // or INCLUDE
       },
       ... your rules ...
    ]
 }

Examples of OneAgent security rules

Each custom log source path you add needs to be validated by OneAgent and abide by its security rules (file matching rules). Some predefined security rules are applied on the OneAgent side. Examples of exclude and include rules for UNIX, Linux, and Windows are listed in the table below.

Operating system	Directory pattern	File pattern	Action
UNIX	`/`	`*.pem`	EXCLUDE
UNIX	`/`	`[-.\\_]log[-.\\_]`	INCLUDE
Linux	`/.ssh/`	*	EXCLUDE
Linux	`/`	`*[-.\\_]log`	INCLUDE
Windows	`/.*/`	*	EXCLUDE
Windows	`/windows/system32/winevt/Logs/`	*	INCLUDE

Security rule lists for UNIX, Linux, and Windows

The full list of security rules for UNIX:

{
  "allowed-log-paths-configuration": [
    {
       "directory-pattern":"/",
       "file-pattern":"*.pem",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"/.ssh/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"/.*/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"/",
       "file-pattern":".*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/etc/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/boot/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/proc/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/dev/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/bin/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/sbin/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/usr/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
      "directory-pattern": "/",
      "file-pattern": "*[-.\\_]log[-.\\_]*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/",
      "file-pattern": "*[-.\\_]log",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/",
      "file-pattern": "catalina.out*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/log/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/log/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/log/*/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/logs/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/logs/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/logs/*/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "^/var/lib/docker/containers/*/",
      "file-pattern": "*.log",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "^/var/log/**/",
      "file-pattern": "*",
      "action": "INCLUDE"
    }
  ]
}

The full list of security rules for Linux:

{
  "allowed-log-paths-configuration": [
    {
       "directory-pattern":"/",
       "file-pattern":"*.pem",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"/.ssh/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"/.*/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"/",
       "file-pattern":".*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/etc/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/boot/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/proc/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/dev/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/bin/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/sbin/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
       "directory-pattern":"^/usr/**/",
       "file-pattern":"*",
       "action":"EXCLUDE"
    },
    {
      "directory-pattern": "/",
      "file-pattern": "*[-.\\_]log[-.\\_]*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/",
      "file-pattern": "*[-.\\_]log",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/",
      "file-pattern": "catalina.out*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/log/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/log/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/log/*/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/logs/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/logs/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "/logs/*/*/",
      "file-pattern": "*",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "^/var/lib/docker/containers/*/",
      "file-pattern": "*.log",
      "action": "INCLUDE"
    },
    {
      "directory-pattern": "^/var/log/**/",
      "file-pattern": "*",
      "action": "INCLUDE"
    }
  ]
}

The full list of security rules for Windows:

{
    "allowed-log-paths-configuration":[
       {
          "directory-pattern":"/",
          "file-pattern":"*.pem",
          "action":"EXCLUDE"
       },
       {
          "directory-pattern":"/.ssh/",
          "file-pattern":"*",
          "action":"EXCLUDE"
       },
       {
          "directory-pattern":"/.*/",
          "file-pattern":"*",
          "action":"EXCLUDE"
       },
       {
          "directory-pattern":"/",
          "file-pattern":".*",
          "action":"EXCLUDE"
       },
       {
          "directory-pattern":"/windows/system32/winevt/Logs/",
          "file-pattern":"*",
          "action":"INCLUDE"
       },
       {
          "directory-pattern":"/winnt/system32/winevt/Logs/",
          "file-pattern":"*",
          "action":"INCLUDE"
       },
       {
          "directory-pattern":"^/windows/**/",
          "file-pattern":"*",
          "action":"EXCLUDE"
       },
       {
          "directory-pattern":"^/winnt/**/",
          "file-pattern":"*",
          "action":"EXCLUDE"
       },
       {
          "directory-pattern":"/",
          "file-pattern":"*[-.\\_]log[-.\\_]*",
          "action":"INCLUDE"
       },
       {
          "directory-pattern":"/",
          "file-pattern":"*[-.\\_]log",
          "action":"INCLUDE"
       },
       {
          "directory-pattern":"/",
          "file-pattern":"catalina.out*",
          "action":"INCLUDE"
       },
       {
          "directory-pattern": "/log/",
          "file-pattern": "*",
          "action": "INCLUDE"
       },
       {
          "directory-pattern": "/log/*/",
          "file-pattern": "*",
          "action": "INCLUDE"
       },
       {
          "directory-pattern": "/log/*/*/",
          "file-pattern": "*",
          "action": "INCLUDE"
       },
       {
          "directory-pattern": "/logs/",
          "file-pattern": "*",
          "action": "INCLUDE"
       },
       {
          "directory-pattern": "/logs/*/",
          "file-pattern": "*",
          "action": "INCLUDE"
       },
       {
          "directory-pattern": "/logs/*/*/",
          "file-pattern": "*",
          "action": "INCLUDE"
       }
    ]
}