Mask sensitive information in logs v1

Legacy Log Monitoring v1

You are viewing documentation for Log Monitoring v1. Dynatrace Log Monitoring v1, is considered a legacy solution.

Log Monitoring v1 will reach the end of life and will be switched off in November 2023.

SaaS environments will be automatically upgraded to LMA or LMC.

We strongly encourage you to switch to the latest Dynatrace Log Monitoring version.

If you are currently using Dynatrace SaaS, upgrade to the latest version of Dynatrace log monitoring.

Your log data contains information that may be considered sensitive. Specific log messages may include user names, email addresses, URL parameters, and other information that you may not want to disclose. Log Monitoring features the ability to mask any information by modifying the configuration file on each of the OneAgents that handles information you consider to be sensitive.

Mask personal data

Each OneAgent provides a ruxitagentloganalytics.conf file where you can set configuration options. To mask personal data in Log Monitoring, a masking rule and a masking rule scope need to be added to the configuration file for each OneAgent. The masking rule defines what data is masked, while the masking rule scope defines to what log files the rule is applied.

The masking is done in the OneAgent. No personal data will be sent to and stored in the Dynatrace server.

For each OneAgent:

  1. Log in to OneAgent and open the Windows command prompt or Linux terminal.

  2. Edit the ruxitagentloganalytics.conf file, where you can set the configuration options:

    • On Linux, the file is located in:
      /var/lib/dynatrace/oneagent/agent/config/

    • On Windows, the file is located in:
      %PROGRAMDATA%\dynatrace\oneagent\agent\config\

  3. Create MaskingRule.
    Append all your masking rules to the ruxitagentloganalytics.conf file.

    This MaskingRule is a prerequisite for defining a scope. Whether you plan to use MaskingRulePGIDScope, MaskingRuleFileScope or both, first you must define the MaskingRule. This rule defines what to search for and how to replace it.

    See MaskingRule

  4. Define the scope where the MaskingRule should be applied.
    Append all your defined scopes to the ruxitagentloganalytics.conf file.

    You can create:

    • MaskingRulePGIDScope
      This scope specifies that the MaskingRule should be applied to the specified process group ID.

      See MaskingRulePGIDScope.

    • MaskingRuleFileScope
      This scope specifies that the MaskingRule should be applied to the specified file or file groups.

      See MaskingRuleFileScope.

    • Both MaskingRulePGIDScope and MaskingRuleFileScope In this case the, MaskingRule rule is applied to each defined scope associated with it in the order in which they were defined.

  5. Save the file.
    The masking configuration will be applied (ready for use) in less than a minute.
    Changes in the configuration don't require OneAgent restart.

Your log files will now have sensitive information masked based on the rules you've defined.

Priority for multiple rules

Each consecutive masking rule is applied to the result of the previous masking rule.

For example, assume that you have created three masking rules:

  1. The first masking rule is applied to the original log files.
  2. The second masking rule is applied to the result of the first masking rule.
  3. The third masking rule is applied to the result of the first and second masking rules.

The result is a log file sequentially processed by masking rules based on the rule order in the configuration file.

Entering log paths

MS Windows system logs

Masking sensitive information works on all Windows-based application logs, but not on MS Windows operating system logs.

The log file paths should be presented in a specific way:

  • The log file path is case sensitive.
    Certain Windows log file paths are presented with the drive letter capitalized. Make sure that your Windows log file path reflects the capitalization of your Windows version.

  • No spaces are accepted between definition components.
    Valid: MaskingRuleFileScope=Username_replace,"C:\logs\errors.log"
    Invalid: MaskingRuleFileScope=Username_replace, "C:\logs\errors.log"

  • Do not use regular expressions or an asterisk wild card (*) to indicate multiple log files. Instead, use the Dynatrace mapped names. For example, access.#.log.

MaskingRule=Username_replace,"Username: {*},",sha1
MaskingRulePGIDScope=Username_replace,0X6714946EC3018AB3
MaskingRuleFileScope=Username_replace,"/var/log/apache2/access.#.log"
MaskingRuleFileScope=Username_replace,"/var/log/apache2/error.log"
MaskingRuleFileScope=Username_replace,"C:\logs\errors.log"

In this example, the masking rule named Username_replace will search for an instance of Username: and, for each occurrence, replace all characters between Username: and , strings with an SHA-1 hash for matching characters.

Next, it will apply the Username_replace rule to all log files in the 0X6714946EC3018AB3 process group.
Then, it will apply that same Username_replace rule to all log files under mapped name access.#.log and specifically to the error.log file.

Finally, it will apply that Username_replace rule to the Windows errors.log file (if OneAgent is operating on a Windows machine).

For this example log file /var/log/apache2/error.log containing the following log lines:

2018-01-12 15:44:12 INFO Successful transaction for Username: Maria Brava in e-trading, Product-Name: KS221
2018-01-12 15:44:12 INFO New worker family started with PID 2332af67
2018-01-12 15:44:12 WARN Authentication failure (Username: Arthur Brand, Times: 3)

The server will transmit the following log file:

2018-01-12 15:44:12 INFO Successful transaction for Username: ************************, Product-Name: KS221
2018-01-12 15:44:12 INFO New worker family started with PID 2332af67
2018-01-12 15:44:12 WARN Authentication failure (Username: ************, Times: 3)

MaskingRule

Syntax: MaskingRule=<name>,"<search_pattern>","<replace_string_or_hash>"

Masking rule name

A masking rule must have a unique name that consists of letters [a-zA-Z], numbers [0-9], and underscore (_) character.

ApacheAccessMethodRule
credit_card_mask
UserNameRule3

Masking rule search pattern

A search pattern consists of any literal string including a {*} wildcard that specifies the part of the string that should be masked. The search pattern must be placed in double quotes (" ") and cannot begin with the {*} wildcard. The {*} wildcard may be used at the end or in the middle of a search pattern.

Input line

Expression

Result

text123_text234_text123_text234

{*}text

not allowed (invalid)

text123text234text123text234

text1{*}text2

text1###text234text1###text234

text123text234text123text234

text{*}

text123text234text123text###

text123text234text123text234

text{*}text1

text###text123text234

search pattern

result

"Username={*}"

All characters following Username= string will be masked.

"Username={*} logged in"

All characters between Username= and logged in strings will be masked.

"CreditCardNo: {*}"

All characters following CreditCardNo: string will be masked.

"CreditCardNo: {*}, Customer:"

All characters between CreditCardNo: and , Customer: strings will be masked.

Masking rule replace string or hash

The replace string can be a fixed phrase or an SHA-1 hash.

The provided string will literally replace the {*} wildcard in the search pattern result. Be sure to put the replace string in double quotes. The length of the replace string is fixed and does not correspond to the length of the pattern matched.

"#########"
"... MASKED ..."
"xxx"

The masking rule ApplicationErrorRule searches for an instance with the Error string and, for each occurrence, replaces all characters between Error and ( with a fixed >>> MASKED <<< string.

MaskingRule=ApplicationErrorRule,"Error {*}(",">>> MASKED <<< "

The masking rule Username_replace searches for an instance with Username: and, for each occurrence, replaces all characters between Username: and , with an SHA-1 hash for found characters.

MaskingRule=Username_replace,"Username: {*},",sha1

The masking rule loginRuleServer1 searches for an instance with Successfully logged in as: ' and, for each occurrence, replaces all characters between Successfully logged in as: ' and ' user with a fixed ##### string.

MaskingRule=loginRuleServer1,"Successfully logged in as: '{*}' user","#####"

The masking rule ApacheAccessRule searches for an instance with 1.0" (notice that double quotes must be escaped with a backslash) and, for each occurrence, replaces all characters between 1.0" and 462 with a fixed Error code string.

MaskingRule=ApacheAccessRule,"1.0\" {*} 462","Error code"

MaskingRulePGIDScope

Syntax: MaskingRulePGIDScope=<name>,<pgid_as_hexadecimal_value>

MaskingRulePGIDScope defines where the linked MaskingRule should be applied based on the specified Process Group ID.

  • The MaskingRulePGIDScope is linked with the MaskingRule through a common name.

    For example, MaskingRulePGIDScope=ApplicationErrorRule,0x6714946EC3018AB3 indicates that the MaskingRule named ApplicationErrorRule should be applied to all log files in the 0x6714946EC3018AB3 process group (the process group is represented as a hexadecimal value).

  • You can have multiple definitions of MaskingRulePGIDScope for the same MaskingRule.

The following scope definitions apply the same UsernameMask MaskingRule but to a different process group:

MaskingRulePGIDScope=UsernameMask,0x6714946EC3018AB3
MaskingRulePGIDScope=UsernameMask,0XB34081EFF9E5F516
  • The process group is represented as a hexadecimal value that must begin with either 0x or 0X to indicate that it is a hex value.

MaskingRuleFileScope

Syntax: MaskingRuleFileScope=<name>,"<path_to_physical_or_mapped_filename>"

MaskingRuleFileScope defines where the linked MaskingRule should be applied based on the specified file or file groups.

  • The MaskingRuleFileScope is linked with the MaskingRule through a common name.

    For example, MaskingRuleFileScope=ApplicationErrorRule,"/var/log/apache2/access.#.log" indicates that the MaskingRule named ApplicationErrorRule should be applied to all access.#.log files.

  • You can have multiple definitions of MaskingRuleFileScope using the same MaskingRule.

The following scope definitions apply the same UsernameMask MaskingRule but to different log files:

MaskingRuleFileScope=UsernameMask,"/var/log/apache2/error.log"
MaskingRuleFileScope=UsernameMask,"/var/log/apache2/mycustom.log"
MaskingRuleFileScope=UsernameMask,"/var/log/apache2/access.#.log"
MaskingRuleFileScope=UsernameMask,"C:\logs\errors.log"

  • The file scope definition can point to a specific log file on the host (for example, /var/log/apache2/error.log) or a mapped name that is used for rotated files (for example, /var/log/apache2/access.#.log). In either case, be sure to place the file path in double quotes.

  • The mapped names that represent rotated files use the same wildcard syntax as the Log viewer.

    For example, the access.#.log mapped name represents files access.1.log, access.2.log, and any consecutive log files matching that pattern.