Timestamp/splitting configuration

  • Latest Dynatrace
  • Tutorial
  • 17-min read

Dynatrace allows you to define rules that control log data timestamps.

Timestamp detection

By default, log monitoring automatically detects only the most common and unambiguous subset of date formats supported. For details, see Supported timestamp formats. Each time a timestamp pattern is detected, the line will be treated as the beginning of the log entry. All following lines without a detected timestamp will be treated as a continuation and reported as a single multi-line log record.

You can also control timestamp detection by using the following options from Settings Settings > Log monitoring > Advanced log settings:

  • Detect container time zones: This option enables the automatic detection of the timezone in a container's logs, if the timezone is not explicitly defined or configured.

  • Default timezone for agents: This options enables the default timezone for the agent if more specific configurations are not defined.

  • Timestamp search limit: This option defines the number of characters in every log line (starting from the first character in the line) where the timestamp is searched.

No timestamp detected

When Log Monitoring is unable to determine the time format, it treats each log line as a separate log entry with an automatically assigned timestamp (observation timestamp) using a one-minute time resolution, except for lines starting with whitespaces (space, tab), which are treated as a continuation of an entry.

Timestamp search limit

Regardless of format, the timestamp typically occurs within the first 64 characters of a log entry. However, the timestamp can occur elsewhere, in which case you can raise this limit on the OneAgent configuration page: Log Monitoring > Timestamp/Splitting patterns.

Timestamp rules

Regardless of where it occurs in a log entry, a timestamp may be written in multiple formats. Dynatrace supports some timestamp formats by default, but sometimes multiple formats may fit the incoming log data and match the timestamp to an incorrect timestamp pattern.

Because of this, Log Monitoring also enables you to define a specific date format using timestamp rules that specify what should be considered a timestamp in a log record. These rules contain a timestamp pattern, time zone, and matchers.

  • Pattern—Defines what should be considered a timestamp in your logline.
  • Timestamp search limit—Specifies the count of characters in each log line, measured from the beginning of the line, where the timestamp is searched.
  • Entry boundary—Optional field. Specifies the entry boundary. You need to provide a fragment of the text from the first line of the entry. The pattern is treated literally, which means that there is no support for the asterisk (*) as a wildcard.
  • Timestamps in indented lines—Enable this option if you don't want to parse timestamps in lines that start with whitespace characters.
  • Time zone—Defines the timestamp time zone. Optional if your timestamp pattern includes the timezone indicator (%z).
  • Matcher—Narrows down the range for the rule and applies the timestamp pattern only to matched log entries. Because you can't use the log.content attribute in the timestamp pattern matchers, the highest granularity is a log source. Granularity is at this level because the timestamp pattern is used to split the contents of a log source into separate log records, so it is used before the log.content attribute's value (or any other attributes set on an individual log record's level) is determined.
    • If you create multiple rules matching the same log data, all defined time formats are searched for.
    • If you have at least one rule matching a given log, predefined formats are not applied to it.

Timestamp configuration examples

Consult the timestamp formats below as configuration examples:

  • Timestamp without the default separator: %Y-%m-%d-%H.%M.%S Example: 2024-09-05-12.30.01

  • Using only timestamps from the beggining of the log entry (%^): %^%F %T Example: 2024-09-05 12:30:01

  • Searching for a timestamp with the field name (JSON): "validTimestamp":"%Y-%m-%dT%H:%M:%S" Example: "validTimestamp": "2024-09-05T12:30:01"

  • Timestamp with timezone offset: %m-%d-%Y %H:%M:%S %z Example: 09-05-2024 12:30:01 +01:00

  • Timestamp with timezone name or abbreviation: %m-%d-%Y %H:%M:%S %Z Example: 09-05-2024 12:30:01 UTC

  • Timestamp excluding the year (the current year is used to evaluate the timestamp): %b %t%d-%H:%M:%S Here, %t maches zero or one white space characters. Example: Apr 4-12:30:01 or Apr 14-12:30:01

  • Any timestamp with the myTime.* prefix: myTime%*: %Y-%m-%dT%H:%M:%S Example: myTimeOfCreation: 2024-09-05T12:30:01 You can overwrite the default timezone by defining the timezone without the timestamp pattern.

  • Two digits year format: %m/%d/%y %H:%M:%S %Z Example: 09/05/24 12:30:01 America/Chicago

Multiple timestamp patterns in the same log source

When ingesting log entries, OneAgent parses the log entry for a timestamp. To do this, it uses a list of matcher patterns.

  • When the log entry contains one timestamp, OneAgent evaluates the timestamp against the list of matcher patterns, starting with the first pattern in the list. If the entry’s timestamp matches one of the patterns, OneAgent uses that timestamp as the beginning of the log entry.
  • When the log entry contains two or more timestamps, OneAgent evaluates the timestamp against the list of matcher patterns, starting with the pattern that was matched in the previous log entry.

Therefore, it is possible that even if the log entry’s first timestamp matches the first pattern in the list of matching patterns, OneAgent will actually match the second timestamp—because the previous log entry matched the second pattern. See the code block below for an example.

Log entry 1: Pattern 1, Pattern 2
Log entry 2: Pattern 2
Log entry 3: Pattern 1, Pattern 2

For log entry 3, even if "Pattern 1" appears first in the list of matcher patterns, OneAgent will actually match "Pattern 2".

To ensure OneAgent evaluates only your desired timestamp patterns, carefully select which patterns are in your matching list. In the example above, to guarantee Pattern 1 is always used, remove Pattern 2 from your list. This may result in messages that only contain Pattern 2 timestamps to be dropped.

For example, if your log file contains both %FT%T (2024-01-01T12:30:01) and %F %T %Z (2024-01-01 12:30:01 UTC) patterns, and OneAgent successfully matches the first pattern in a line, it will prioritize that pattern for subsequent lines.

Supported scopes

Four hierarchy scopes are supported: host, host group, and environment.

The hierarchy scopes are merged into one list in the following order:

  1. Host rules
  2. Kubernetes cluster rules
  3. Host group rules
  4. Environment rules

log-storage

The OneAgent receives the merged list (merged lists from its respective hosts, host groups, and environments) with no indication of which scopes are defined.

Host scope

The host scope can be accessed through the Host settings for a specific host.

  1. Go to Hosts Hosts Classic.
  2. Find and select your host to display the host overview page.
  3. In the upper-right corner of the host overview page, select More (…) > Settings.
  1. In the host settings, select Log Monitoring > Timestamp/Splitting patterns.
  2. Configure data masking by adding rules with a set of matchers that specify what should be considered a timestamp in the log record.

Kubernetes cluster scope

The Kubernetes cluster scope can be accessed via the Kubernetes page.

  1. Go to Kubernetes or Kubernetes Classic (latest Dynatrace) and select the cluster that interests you.
  2. Find and select your cluster to display the cluster overview page.
  3. In the upper-right corner of the cluster overview page, select More (…) > Settings.
  4. From the cluster settings, go to Log Monitoring > Timestamp/Splitting patterns.
  5. Configure storage upload by adding rules with a set of attributes that matches the log data to be stored by Dynatrace.

Host group scope

The host group scope can be accessed via the Host page.

  1. Go to Hosts Hosts Classic and select the host that interests you.
  2. On the host overview page, select Properties and tags.
  3. On the Properties and tags panel, find the Host group property to see the name of the host group to which the selected host belongs.

    The Host group property is not displayed when the selected host doesn't belong to any host group.

  4. Select the host group name to list all hosts in that host group. This displays the OneAgent deployment page filtered by the selected host group. Each listed host has a Host group: <group name> link, where <group name> is the name of the host group that you want to configure.
  5. Select the host group name in any row.
  1. In the host group settings, select Log Monitoring > Timestamp/Splitting patterns.
  2. Configure data masking by adding rules with a set of matchers that specify what should be considered a timestamp in the log record.

Environment scope

The environment scope is available in the settings menu.

  1. Go to Settings and select Log Monitoring > Timestamp/Splitting patterns.
  2. Configure data masking by adding rules with a set of matchers that identify your sensitive data.

Create rule

To add a rule (on the host, host group, or environment level) that interprets the incoming log data timestamps

  1. Select Add rule to start configuring your rule.

  2. Rule name
    The name to display for your configuration.

  3. Pattern
    Enter the pattern to be read as a date from the logs. For details on timestamp formats, see Supported timestamp formats and the following list of format specifiers.

    You need to specify at least the month, day, hours, minutes, and seconds, although you can use alternative formats for them. You can include the time zone indicator (%z) or specify the time zone separately in the rule definition.

    Rules without a pattern can override the timezone only for default supported timestamps.

  4. Timestamp search limit

    Use this field to define the number of characters in every log line where timestamp is searched. If you want to ignore timestamps and split logs using the default rules, set this value to 0. Use this field to overwrite the global timestamp search limit (default 64 bytes).

  5. Entry boundary

    Use this field to provide a fragment of the text from the first line of the entry.

  6. Time zone
    Select the time zone to apply to this pattern.
    This setting is not enabled if you have already specified the timezone in the timestamp pattern (%z).

    You can select Local time zone to use the time zone of the host on which the OneAgent is running.

  7. Select Add condition to create a specific match for this rule and narrow down the scope for that rule.

    You can include multiple matchers in one rule. For example, the timestamp configuration rule can be applied to logs from a specific container, namespace, or log source. Multiple matchers with the same attribute use AND logic between matchers, while matchers with multiple values assigned to them use OR logic.

    1

    The minimum required OneAgent version is 1.295.

    2

    Manually or automatically applied tags are not visible to OneAgent.

    3

    The minimum required OneAgent version is 1.289.

    4

    The minimum required OneAgent Log Module version is 1.309.

    5

    The minimum required Dynatrace Operator version is 1.4.2.

  8. Select the matching attribute.

  9. Select Value and, from the Value list, select the detected log data items.

    You can add multiple values to the selected attribute. You can have one matcher that indicates the Log source and matches values /var/log/syslog and Windows Application Log. Use asterisks (*) as wildcards to get a partial match.

  10. Select Save changes.

Rules are executed in the order in which they appear on the Timestamp/Splitting patterns page.

Rule reorder propagation delay

When you change the rule order (to change the order in which they are executed), allow for two or three minutes of propagation time between when you save the change and when the change takes effect.

The Active toggle

Starting with OneAgent version 1.249, you can activate/inactivate your rules by turning on/off the Active toggle. To manage your rules effectively, we recommend that you upgrade your OneAgent to version 1.249. If you have any rules set on the host with OneAgent version earlier than 249, you will not be able to inactivate them, in which case you need to remove such rules by selecting Delete on the rule level or via the REST API.

Rules are executed in the order in which they appear on the Timestamp/Splitting patterns page.

Configuration limits

You can add a maximum of 100 timestamp rules per each scope (host, host group, Kubernetes cluster, or environment).

REST API

You can use the Settings API to manage your timestamp configuration:

  • View schema
  • List stored configuration objects
  • View single configuration object
  • Create new, edit, or remove existing configuration object

To check the current schema version for timestamp configuration, list all available schemas and look for the builtin:logmonitoring.timestamp-configuration schema identifier.

Timestamp configuration objects are available for configuration on the following scopes:

  • environment—configuration object affects all hosts in a given environment.
  • host_group—configuration object affects all hosts assigned to a given host group.
  • host—configuration object affects only the given host.

To create a timestamp configuration using the API

  1. Create an access token with the Write settings (settings.write) and Read settings (settings.read) permissions.

  2. Use the GET a schema endpoint to learn the JSON format required to post your configuration. The timestamp configuration schema identifier (schemaId) is builtin:logmonitoring.timestamp-configuration. Here is an example JSON payload with the timestamp configuration:

    [
      {
        "insertAfter":"uAAZ0ZW5hbnQABnRlbmFudAAkMGUzYmY2ZmYtMDc2ZC0zNzFmLhXaq0",
        "schemaId": "builtin:logmonitoring.timestamp-configuration",
        "schemaVersion": "0.1.0",
        "scope": "tenant",
        "value": {
          "config-item-title": "Added from REST API",
          "date-time-pattern": "%Y-%m-%d %H:%M:%S",
          "timezone": "CET",
          "matchers": [
            {
              "attribute": "dt.entity.process_group",
              "operator": "MATCHES",
              "values": [
                "PROCESS_GROUP-05F00CBACF39EBD1"
              ]
            },
            {
              "attribute": "log.source",
              "operator": "MATCHES",
              "values": [
                "Windows System Log",
                "Windows Security Log"
              ]
            }
          ]
        }
      }
    ]

  3. Use the POST an object endpoint to send your configuration.

Related tags
Log Analytics