Sensitive data masking in OneAgent

Dynatrace version 1.253+ OneAgent version 1.243+

Your log data contains information that may be considered sensitive. Specific log messages may include user names, email addresses, URL parameters, and other information that you may not want to disclose. Log Monitoring features the ability to mask any information by modifying the configuration file on each OneAgent that handles information you consider to be sensitive.

You can select the data that needs to be protected by applying a set of masking rules. Within each rule, you can decide what you need to hide and what to replace your hidden content with. If you need to address only specific attributes, such as predefined containers, log sources or process groups, you can achieve it by adding matchers to your rules.

Rule hierarchy

Masking rule execution occurs in a predefined hierarchy, from top to bottom. Each consecutive rule is applied to the result of a preceding rule. The hierarchy is as follows:

Host configuration rules

The host configuration rules can be accessed through the Host settings for a specific host.

Host group configuration rules

The host group configuration rules can be accessed via the Host page.

Environment configuration rules

The tenant scope is available in the settings menu.

Create rule

You can configure sensitive data masking on the host, host group or environment level.

Defined rules can be reordered, and they are executed in the order in which they appear on the Sensitive data masking page.

REST API

You can use the Settings API to manage your sensitive data masking configuration:

To check the current schema version for sensitive data masking configuration, list all available schemas and look for the builtin:logmonitoring.sensitive-data-masking-settings schema identifier. Sensitive data masking configuration objects are available for configuration on the following scopes:

To create a sensitive data masking configuration using the API

[
   {
      "schemaId":"builtin:logmonitoring.sensitive-data-masking-settings",
      "scope":"tenant",
      "value":{
         "config-item-title":"Added from REST API",
         "masking":{
            "expression":"run (\\d+?)",
            "type":"STRING",
            "replacement":"testing"
         },
         "matchers":[
            {
               "attribute":"log.source",
               "operator":"MATCHES",
               "values":[
                  "/var/log/syslog"
               ]
            }
         ]
      }
   }
]

SHA-1 examples

You can mask such data as your credit card or phone number, with or without specifying the capturing group.

Mask credit card number

In this example, you will configure a sensitive data masking rule that targets a credit card number in the following log record:

Username: John Doe, CreditCardNumber: 1234-1234-1234-1234

The rule is further narrowed to the c:\inetpub\logs\LogFiles\ex_*.log files in two process groups: IIS (PROCESS_GROUP-3D9D854163F8F07A) and IIS (PROCESS_GROUP-4A7B47FDB53137AE).

Go to Settings and select Log Monitoring > Sensitive data masking

Username: John Doe, CreditCardNumber: 7e938e089861f3975b38cff3a93cc3aa659f7779

Mask phone number

In this example, you will configure a sensitive data masking rule that targets all phone numbers in the following log record for all log files.

Username: John Doe, PhoneNumber: +48123010100

Go to Settings and select Log Monitoring > Sensitive data masking.

Username: John Doe, 011897d555c81e88f286cbb74c59f4ad99ec2f8d

Advanced SHA-1 examples

In the examples below, you can see how various combinations of sensitive data can be masked. You can use the listed payload JSON files in the REST API, or enter the listed masking rules, matchers, Regex expressions, and attributes directly when creating your rules via Dynatrace web UI.

Mask credit card numbers and emails

To mask all credit card numbers and emails in your content, you need to create two separate rules, each with a different matcher:

{
  "masking": {
    "expression": "(\\d{4}-\\d{4}-\\d{4}-\\d{4})",
    "type": "STRING",
    "replacement": "MaskedCreditCardNumber"
  },
  "matchers": [
  ],
  "enabled": true
},
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
  ],
  "enabled": true
}

Mask Apache logs

To mask logs that are written by Apache AND whose log filename is error.log, you can create one rule with two matchers:

{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error.log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

To mask logs that are written by Apache OR whose log filename is error.log, you need to create two rules with one matcher each:

{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error.log"
      ]
    }
  ],
  "enabled": true
},
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

To mask logs that are written by Apache and whose log filename starts with error AND ends with log, you need to have one rule with three matchers, each matcher having one value.

{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error*"
      ]
    },
    {
      "attribute": "log.source",
      "values": [
        "*log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

To mask logs that are written by Apache and whose log filename starts with error OR ends with log, you need to have one rule with two matchers, one with the process group value, and the second one with two content values, /path/to/error* and *log:

{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error*", "*log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Mask Apache or MySQL logs

To mask logs that are written by Apache or MySQL, you need to have either two rules or one rule with one matcher that has two values.

The scenario with two rules:

{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
},
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

The scenario with one rule with a matcher that has two values:

{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID", "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
}

Regex examples

The common regex formats for sensitive data include:

Unsupported regular expressions

Data masking occurs within the entire expression or a capturing group. An expression has to match the regular expression engine syntax, and it cannot:

Frequently asked questions

Sensitive data masking limits

Be aware of the following limitations to sensitive data masking: