Try it free

Monitor log source health with SFM events

  • Latest Dynatrace
  • How-to guide
  • 11-min read

OneAgent version 1.333+

The Log module features a built-in self-monitoring and troubleshooting functionality. The functionality is called Log module self-monitoring events (SFM events).

The Log module provides the list of monitored log sources along with their statuses and metadata. The log source status shows whether a log source is detected correctly and is configured to be ingested, as well as informs about the source type (for example, whether a log source was detected automatically or provided by a user).

The functionality also warns about suspicious issues and malfunctions. In case such an issue appears, the context (for example, the log source name or host ID) is provided along with the information on where to find more help and how to examine and solve the issue. Each type of a malfunction is communicated using a specific SFM event type.

Availability and state

  • The functionality is generally available and turned on by default in OneAgent version 1.339+ and SaaS version 1.340+.

  • The additional performance footprint of the functionality is negligible.

Prerequisites

  • The Log module is deployed with either OneAgent or Dynatrace Operator.

Configure SFM events via API

SFM events are enabled by default. To customize this functionality, update your configuration via the Settings API. Use the builtin:logmonitoring.log-sfm-settings schema ID. If you're familiar with configuring log ingest rules, the same workflow applies here.

Prerequisites for SFM event configuration

  • An access token with settings.write and settings.read permissions.
  • An API client of your choice (the examples below use cURL).

Configuration scopes

You can configure SFM event rules for the following scopes:

  • tenant: The configuration object affects all hosts in a given environment.
  • host_group: The configuration object affects all hosts assigned to a given host group.
  • kubernetes_cluster: The configuration object only affects the given Kubernetes cluster.
  • host: The configuration object only affects the given host.

We recommend configuring the SFM event rules at the highest appropriate scope for easier management. Use fewer, broader rules instead of many small, overlapping rules.

Exclude a particular SFM event type

If you've enabled all SFM events but need to exclude a particular SFM event type, create an additional exclusion SFM event rule.

This exclusion rule must have a higher priority than the rule that enables all SFM events. To add the exclusion rule at the top of the rule list, set insertAfter to an empty value in the payload. If you don't include the insertAfter parameter, the rule is placed at the bottom of the rule list and can be overridden by all other rules within the given configuration scope. See the POST an object parameters for more information.

Follow the instructions in Enable SFM events, but use the following example payload.

[
{
"schemaId": "builtin:logmonitoring.log-sfm-settings",
"scope": "tenant",
"insertAfter": "",
"value": {
"enabled": true,
"config-item-title": "Suppress warning about the absence of timestamp pattern",
"send-to-storage": false,
"matchers": [{ "attribute": "event.type", "operator": "MATCHES", "values": ["timestamp.no_pattern"]}]
}
}
]

By creating this additional rule, you exclude the timestamp.no_pattern SFM event type on the tenant configuration scope.

There can be situations when you don't want to suppress an SFM event type everywhere, but you only want to suppress it for one log source you already know about, while still being notified if the same event occurs elsewhere.

For example, suppose one of your log sources, a.log, has no timestamp in its content. You're aware of that and don't want to be notified about it. However, you still want to be notified about the timestamp.no.pattern event for any other log source.

To do this, create an exclusion rule similar to the one above, but add a second matcher for log.source:

[
{
"schemaId": "builtin:logmonitoring.log-sfm-settings",
"scope": "tenant",
"insertAfter": "",
"value": {
"enabled": true,
"config-item-title": "Suppress 'no timestamp pattern' warning only for a.log",
"send-to-storage": false,
"matchers": [
{ "attribute": "event.type", "operator": "MATCHES", "values": ["timestamp.no_pattern"] },
{ "attribute": "log.source", "operator": "MATCHES", "values": ["*a.log"] }
]
}
}
]

Both matchers must match for the rule to apply, so the event is suppressed only when its type is timestamp.no.pattern and its log source is a.log. The event is still raised for every other log source.

Keep the suppression scope as narrow as needed. A good exclusion rule must match the exact situation which you've already investigated and accepted. If you find yourself adding very broad matchers (for example, suppressing an entire event type across an entire host), confirm that you aren't also hiding events that point to a real issue elsewhere.

Delete all types of SFM events

To delete all SFM event types, use the DELETE an object endpoint. This operation doesn't require a JSON payload.

curl -X DELETE \
"<environment_url>/api/v2/settings/objects/<objectId>" \
-H "Authorization: Api-Token <token>"

To get the required object ID, use the GET objects endpoint.

Create or update SFM event rules

SFM events are enabled by default. If you've deleted your SFM event rules and want to re-enable them, follow these steps.

To create or update SFM event rules:

  1. Use the GET a schema endpoint to learn the JSON format required to post your SFM event configuration.

    curl -X GET \
    "<environment_url>/api/v2/settings/schemas/builtin:logmonitoring.log-sfm-settings" \
    -H "Authorization: Api-Token <token>"

    You can also use the GET objects endpoint to get all SFM event rules available on the specified scope.

    curl -X GET \
    "<environment_url>/api/v2/settings/objects?schemaIds=builtin:logmonitoring.log-sfm-settings&scopes=<tenant/entityId>" \
    -H "Authorization: Api-Token <token>"
  2. Build a payload.json file using the acquired schema as a reference.
    If you want all types of SFM events enabled, you can use the example payload below.

    [
    {
    "schemaId": "builtin:logmonitoring.log-sfm-settings",
    "scope": "tenant",
    "value": {
    "enabled": true,
    "config-item-title": "Enable all SFMs",
    "send-to-storage": true,
    "matchers": []
    }
    }
    ]
  3. Optional To validate the created payload.json payload, use the POST an object endpoint and set validateOnly to true.

    curl -X POST \
    "<environment_url>/api/v2/settings/objects?validateOnly=true" \
    -H "Authorization: Api-Token <token>" \
    -H "Content-Type: application/json" \
    --data @payload.json
  4. Create an SFM event rule using the POST an object endpoint.

    curl -X POST \
    "<environment_url>/api/v2/settings/objects" \
    -H "Authorization: Api-Token <token>" \
    -H "Content-Type: application/json" \
    --data @payload.json
  5. Verify that the rule has been created using the GET objects endpoint.

    curl -X GET \
    "<environment_url>/api/v2/settings/objects?schemaIds=builtin:logmonitoring.log-sfm-settings&scopes=<scope>" \
    -H "Authorization: Api-Token <token>"

Explore the self-monitoring dashboard

Log module SFM events are stored in Grail under dt.system.events. The ready-made Log ingest overview dashboard provides a dedicated view with SFM sections: OneAgent log module for Hosts and OneAgent log module for Kubernetes.

To open the dashboard, go to Dashboards Dashboards and search for Log ingest overview.

To query SFM events with DQL, use the following example:

fetch dt.system.events, from: now() - 24h
| filter event.provider == "Log Module" and event.type != "log_source.status"
| dedup event.id, sort: { timestamp desc }
| filter event.status == "Active"
Log ingest overview dashboard
Log ingest overview dashboard

The following filters apply only to the OneAgent log module sections of the dashboard:

  • HostGroup
  • K8sClusterName
  • K8sNamespaceName

Dashboard features

The OneAgent log modules area of the dashboard contains two sections based on Log module SFM events, one for each deployment type.

OneAgent log module for Hosts

The OneAgent log module for Hosts section helps you identify and investigate ingestion issues on hosts and VMs. It contains the following tiles:

TileDescriptionPurpose

Currently active issues (last 24h)

Active SFM events grouped by status and event type; 24-hour distribution sparkline; count per event type

See which problems are most frequent right now

Self-monitoring events over time

Time series chart of SFM event volume by type over the selected period

Spot trends, spikes, or recurring issues

Host-based log modules events

Detailed event list: start time, event type, affected log source path, and source entity for each active issue

Drill down into specific log files and hosts

Log sources status over time

Time-based chart tracking how the number of log sources in each status category changes

Correlate coverage changes with incidents

Log source status events

Chronological table of log source status events: log source path, source entity, file status, ingest status, and log source origin

Review the full history of log source state changes

OneAgent log module for Kubernetes

The OneAgent log module for Kubernetes section helps you identify and investigate ingestion issues on Kubernetes log sources. It contains the same Currently active issues (last 24h), Self-monitoring events over time, and Log sources status over time tiles as the Hosts section above, plus the following Kubernetes-specific tiles:

TileDescriptionPurpose

Kubernetes-based log modules events

Detailed event list: start time, event type, affected log source, pod name, namespace, and cluster for each active issue

Drill down into specific containers and Kubernetes entities

Log source status events

Chronological table of log source status events: log source, file status, ingest status, pod name, namespace, cluster, and log source origin

Review the full history of log source state changes

SFM events are usually reported within the first 90 seconds after the event occurred. This delay is because events are generated by the Log module during ingest, and it may be longer depending on the network connection between the Log module and your Dynatrace environment.

Use SFM events to resolve log-related issues

The SFM event types and required actions are described below.

log_source.status (info)

Explanation

Provides additional information and is generated for each log source known to the Log module. For each log source, provides context metadata (for example, related process group and host), along with additional string properties.

Additional properties:

  • log.source.file_status: Current monitoring state of a log source (for example, “OK” or “File not exists”).
  • log.source.ingest_status: Log content ingestion status (for example, “Fully ingested”, “Partially ingested”, or “Not ingested”).
  • log.source.origin: Indicates where the log derives from (for example, “Automatically detected” or “Custom log source”).

Required actions

Use the file status and ingest status to decide what to do next.

  • File status OK: No action required.

  • File status Not Exist, but the file is present on the host (this only happens for a custom log source). Check the following:

    • The custom log source file path pattern definition for typos.
    • Whether the Log module operates in the same filesystem in which you defined the custom log source file pattern.
    • Whether the Log module has appropriate privileges to access the file. In particular:
      • On Linux: The Log module (with default settings) has OS capabilities to read each file on local disks, but not on remote ones. For remote files, you need read privileges for the user the Log module runs as (by default, dtuser), and read+execute privileges on each remote directory in the file path.
      • On Windows: The user under which the OneAgent service runs has sufficient privileges to access the log file. This is usually true for local disks, but remote disks may require additional authentication that OneAgent can't pass through.
    • Whether the files matched by the custom file path pattern satisfy security rules. See the ingest.custom_log_source_blocked SFM event type below for details.
  • File status Binary, but the file looks like text. There might still be a portion of binary content inside the file, or there used to be binary content in a log file that is part of the log source, even if such a file has already been removed. Consider the following options:

    • If it's a custom log source, and some files matching the pattern are binary and you don't want to ingest them, adjust the custom log source pattern, or split the custom log source definition into multiple narrower patterns so that those patterns don't match binary files.

    • If it's a custom log source, and some files matching the pattern have occasional binary content but you still want to ingest them, turn on Allow binary files in the custom log source definition.

    • If it's an automatically detected file, create one or more custom log source definitions and adjust them as in the points above.

    • If you don't want to apply any of the above and the binary content no longer exists (because it was in a file that was deleted) and you don't expect it to appear again, you might consider removing the Log module persistence file. The path to the persistence file (Log Analytics persistence) is described in, for example, OneAgent files and disk space requirements on Linux.

      You can safely delete the persistence file only when the Log module isn't running. Deletion might cause some data loss or data duplication around the time of removal.

  • File status Sensitive Masking Timeout: See the ingest.masking_timeout SFM event type below.

  • Ingest status isn't as expected: Review your log ingest rules. Typical mistakes are:

    • Rules are evaluated from top to bottom, and the first matching rule decides whether a log record is ingested.
    • The effective list of rules is a concatenation of the lists from all configuration scopes.
    • The log source matcher matches the log source name, not the log file path. For example, a log source matcher /path/a.2026.log doesn't match a log source named /path/a.#.log, even if a file path /path/a.2026.log exists in the filesystem.
timestamp.no_pattern (warning)

Explanation

The Log module hasn't detected a timestamp in log content of a log source. In this case, the Log module sets a timestamp from the metadata (if available) or the timestamp when the Log module acquires the log record from the log source.

Required actions

When there's no timestamp:

  • Rely on the Log module timestamping. However, when multiline log records are present in the log source, review if custom boundary detection configuration is necessary.
  • Reconfigure log producer to change the log source format.

When the timestamp has a non-standard format, configure non-standard timestamp format.

timestamp.multiple_patterns (info)

Explanation

The Log module has detected multiple timestamp patterns being present simultaneously in a single log source.

Required actions

If it's expected, no action is required.

If the Log module incorrectly considers a part of log message as a log record timestamp, configure timestamp format explicitly.

timestamp.invalid_timezone (error)

Explanation

The Log module suspects that the time zone isn't properly detected or configured. For example, the Log module is configured to use the local time zone (for example, UTC+1), while the given log source writes logs in ETC.

Required actions

The log record timestamps are likely incorrect, and you need to configure the time zone manually.

When working with timestamps, watch for these typical mistakes:

  • Forgetting that the UI shows the value in the timestamp column converted to the browser's time zone when crafting a timestamp rule.
  • Providing a timestamp pattern without a time zone element when you want to read the time zone from log content.
  • Having multiple timestamp rules matching the same log source with the same pattern but different explicit time zones. In this case, the rule selection is undefined, and you should change the configuration to avoid this situation.
data_loss.unable_to_read (error)

Explanation

The Log module was unable to read a log file or a part of it, as the file has been removed, compressed, or moved to another location. This results in data loss.

Required actions

If the log source comes from a containerized application, turning on the Collect all container logs feature flag might resolve the issue.

Otherwise, the Log module supports typical log rotation patterns, but it expects the first rotated file to remain uncompressed and in the same directory. It also expects log files to stay available for a short time after they are written. If files are removed or compressed before the Log module can read them, adjust your log rotation settings so that files are kept longer than the Log module read interval.

The Log module read interval can be unusually long if the Log module experiences performance issues. The most frequent risk factors are:

  • A custom log source definition uses wildcards on directories that effectively match a large number of files (not necessarily log files).
  • The Log module monitors log directories that contain a large number of files (more than 10,000).
  • The Log module monitors slow remote disks.
data_loss.network (error)

Explanation

The Log module was unable to send log records to the Dynatrace environment quickly enough. This results in data loss.

Required actions

The issue might be caused by network connectivity issues between the Log module and the Dynatrace environment. Check your network connectivity health.

If the Log module is configured to ingest a large volume of logs, this exceeds the configured ingest limits or the environment's performance. In this case, reduce the number of ingested logs.

The Log module itself can also be too slow to send log records in time. The most frequent risk factors are:

  • A custom log source definition with wildcards on directories that effectively match a large number of files (not necessarily log files).
  • The Log module monitoring log directories that contain a large number of files (more than 10,000).
  • The Log module monitoring slow remote disks.
ingest.access_flag (warning)

Explanation

The Log module can't ingest log records because the Log Content Access flag (--set-app-log-content-access) is turned off.

Required actions

  • For already installed OneAgents, use onagentctl to unblock log ingest.
  • For newly installed OneAgents, you can enable log ingest during installation.
ingest.masking_timeout (error)

Explanation

The ingestion of a log source stopped because applying sensitive information masking had timed out. See Sensitive data masking in OneAgent for more details.

This issue is also reflected in the log source status SFM event for the log source.

Required actions

Review the masking rules enabled for the log source. There's a high chance the root cause is an improperly written regular expression. For guidance on writing efficient regular expressions, see Regular expressions in Dynatrace. The log source is ingested again after you change the configuration, or after you restart the Log module.

pgi.multiple_pgis (warning)

Explanation

This event type occurs when multiple process group instances (PGIs) are assigned to a single log source. This might sometimes be problematic—for example, some processing rules might expect that a log record is enriched with a single PGI.

In context of a custom log source, it means that multiple PGIs are configured explicitly. See Custom log source for details.

In context of an automatically detected log source, it can mean one of the following:

  • Scenario A: Multiple processes have written to the same log source, and these processes are grouped to different PGIs. Note that this refers to a situation where the PGIs are correctly identified as separate PGIs. The processes might write to the same log source because:
    • Scenario A1: The log source is a "sink" log file that gathers data from multiple applications.
    • Scenario A2: The log source contains multiple log files, where each file belongs to a single PGI, but different files within the log source might belong to different PGIs.
  • Scenario B: One PGI stopped writing to a log file, and shortly afterwards a separate PGI started writing to the same log file (or the PGI ID changed due to a change in process properties or OneAgent configuration). In this case, there's a single log monitoring interval for which the Log module doesn't know whether the log records written in it were written by the "old" or the "new" PGI, and it enriches the log records with both.
  • Scenario C: Process group detection isn't working as expected because processes that belong to the same application have different PGI IDs assigned.

Required actions

In context of a custom log source, review the custom log source rule.

In context of an automatically detected log source:

  • Scenario A1: No action is required.
  • Scenario A2: Provide custom log source rules (at least one rule per PGI involved) that split the log file appropriately.
  • Scenario B: No action is required.
  • Scenario C: Review your process group detection configuration, or use automatic log enrichment.

The custom log source split solution from Scenario A2 can be applied regardless of whether the initial problem occurred for automatically detected or custom log sources.

Automatic log enrichment may also resolve the issue by injecting a proper PGI ID into log content.

pgi.lgi_explosion (error)

Explanation

The Log module automatically detected too many log sources for a single process group instance.

Required actions

If you need to ingest these log sources, do one of the following:

  • Create a custom log source rule with a wildcard pattern that covers all log files written by the process group instance.
ingest.log_source_blocked (warning)

OneAgent version 1.337+

Explanation

Some or all log files detected for a log source have been blocked by security rules.

Required actions

Review the security rules and adjust them to fit your needs. See Security rules for a detailed explanation and instructions on how to alter the security rules.

Related topics

  • OneAgent for logs ingestion
  • Dashboards
  • Settings API
Related tags
Log Analytics