OS services monitoring

Dynatrace provides out-of-the-box availability monitoring of OS services.

You can monitor hosts in full-stack monitoring mode or use lightweight monitoring modes. For more information, see Infrastructure and Discovery monitoring modes.

Requirements

  • Linux systemd version 230+ is required on the monitored host to monitor relations between processes and OS services.
  • Linux systemd version 250+ is required on the monitored host for improved OS services detection performance by less frequent refreshes.

OS services alerting options

Depending on your monitoring requirements, you can choose between basic or advanced alerting of OS services. The Discovery mode allows only basic alerting, while the Full-Stack and Infrastructure monitoring modes also allow advanced alerting.

  • Basic alerting: Provides insight only into the service status. The system will monitor an OS service's current status and alert you when it changes from running to failed.
  • Advanced alerting: Provides access to the service status and the service availability metric. In addition to OS service status changes, you get more granular information about service availability on a per-minute basis. This enables more complex business logic, for example, if you want to be notified if a service is stopped for longer than 10 minutes. You can create alerting rules to monitor availability and trigger an alert when needed.

With the service status property in Smartscape Smartscape, you can query the current status of the service.

fetch `dt.entity.os:service` | fieldsAdd status

Service status - Basic monitoring

If the alert is enabled, events and problems are created when a service status changes, such as when a service goes from running to failed. For more details, refer to Host availability.

Failed service alert

With a service availability metric, you get more detailed information about your service on a per-minute basis. This allows you to monitor the status of the service and create more tailored alerts, such as only alerting if the service has been in a failed state for more than 10 minutes.

timeseries count(dt.osservice.availability),by:{dt.osservice.display_name, dt.osservice.status} | filter dt.osservice.display_name=="apache2"

Service status - Advanced monitoring

Monitor a service

To monitor an OS service, perform the following steps.

In Dynatrace, go to OS services monitoring for the level you are configuring. The level defines priority: settings at the host level override settings at the host-group level, and settings at the host-group level override settings at the environment level.

  1. Go to Hosts or Hosts Classic (latest Dynatrace).
  2. Find and select your host to display the host overview page.
  3. In the upper-right corner of the host overview page, select More (…) > Settings.
  1. In the host settings, select OS services monitoring.
  1. Go to Deployment Status and then select OneAgents.
  2. On the OneAgent deployment page, turn off Show new OneAgent deployments.
  3. Filter the table by Host group and select the host group you want to configure.

    The Host group property is not displayed when the selected host doesn't belong to any host group.

    This displays the OneAgent deployment page filtered by the selected host group. Each listed host has a Host group: <group name> link, where <group name> is the name of the host group that you want to configure.
  4. Select the host group name in any row.
    As you have filtered by host group, all displayed hosts go to the same host group.
  1. In the host group settings, select OS services monitoring.

Go to Settings > Collect and capture > Infrastructure > OS > OS services monitoring.

Based on the service state and the rules, the service monitoring policy defines the way Dynatrace is monitoring your service. By default, Dynatrace comes with Auto-start Windows OS Services and Auto-start Linux OS Services policies for auto-started Windows and Linux services with failed status.

Limits

Note that the default limit of OS Service entities is 100,000 per cluster. In larger environments with many hosts, we recommend creating precise rules that match only the important services for your infrastructure. Creating rules that are too general (for example, matching all services on thousands of hosts) may result in reaching the limit (entity explosion), leading to the disappearance of OS services from Dynatrace. Also, Auto-start Windows OS Services and Auto-start Linux OS Services can be used as a starting point for further refining the policies.

The order of service monitoring policies is important. Policies that are higher on the list will proceed before those on lower positions until they are fulfilled. This allows for the creation of selective alerts or monitoring with minimal policies. For example, if you want to monitor all auto-started services and not just those created by Microsoft, you need to add a policy with disabled alerting and/or monitoring that will verify if the manufacturer is Microsoft.

  1. On OS services monitoring for the level you are configuring based on your OS, select Add policy and define the policy, which is a collection of rules.

  2. System: select your operating system.

  3. Rule name: enter the name that will be displayed in the Summary field.

  4. Monitor: decide whether you want to monitor service availability using the OS service availability (builtin:osservice.availability) metric. If available, the metric sends the service status every 10 seconds. The status is carried by the Service status (dt.osservice.status) dimension.
    Note that the metric consumes data points. For more information, see Metrics powered by Grail.

  5. Alert: decide whether you want alerting for your policy.

  6. OneAgent version 1.257+ Alert if service is not installed: whether you want to receive alerts about OS services that are not installed on the host.

  7. Service status: set the service status for which an alert should be triggered.

    You can use logic operations to monitor the service status. For example, $eq(running) monitors the running service state.

    Available logic operations:

    • $not($eq(paused)) – Matches services that are in state different from paused.
    • $or($eq(paused),$eq(running)) – Matches services that are either in paused or running state.

    These are the service statuses you can monitor. Use one of the following values as a parameter for this condition:

    • running
    • stopped
    • start_pending
    • stop_pending
    • continue_pending
    • pause_pending
    • paused

  8. optional OneAgent version 1.257+ Alerting delay: the number of 10-second measurement cycles for a service to be in configured state before an event is generated.

Next, you need to select which services you want to monitor based on service properties.

  1. Select Add rule.

  2. optional Rule scope: select either OS Service or Host. By default, the OS Service option is selected.

If you selected Host:

  • OneAgent version 1.277+ Custom metadata used for matching:

    • Key specifies the metadata key you want to match

    • Condition in which you can define a string that:

      • Dynatrace version 1.310+ $match(ver*_1.2.?) – Matches string with wildcards. Use * for any number of characters (including zero) and ? for exactly one character.
      • $contains(production) – Matches if production appears anywhere in the host metadata value.
      • $eq(production) – Matches if production matches the host metadata value exactly.
      • $prefix(production) – Matches if production matches the prefix of the host metadata value.
      • $suffix(production) – Matches if production matches the suffix of the host metadata value.

      Available logic operations:

      • $not($eq(production)) – Matches if the host metadata value is different from production.
      • $and($prefix(production),$suffix(main)) – Matches if host metadata value starts with production and ends with main.
      • $or($prefix(production),$suffix(main)) – Matches if host metadata value starts with production or ends with main.

      When including special characters such as brackets ( and ) within your matching expressions, escape these characters with a tilde ~. For example, to match a metadata value that includes brackets, like my(amazing)property, you would write $eq(my~(amazing~)property).

If you selected OS Service, proceed according to your operating system.

  • Service property used for matching:

    • Display name visible for system user.
    • Path to the binary used by OS service.
    • Manufacturer from the binary file of the service.
    • Service Name
    • Startup Type

A monitoring rule may consist of multiple detection rules. All detection rules must be satisfied for the OS Service to match, as a logical AND operation is applied across all specified conditions.

With these properties, we define the services to be monitored based on:

  • Display name visible to a system user

  • Path to the service binary

  • Manufacturer of the service

  • Service name representing the name or ID under which OS service is recognized

  • Condition in which you can define a string that:

    • Starts with – use $prefix qualifier, for example $prefix(ss).
    • Ends with – use $suffix qualifier, for example $suffix(hd).
    • Equals – use $eq qualifier, for example $eq(sshd).
    • Contains – use $contains qualifier, for example $contains(ssh).
    • Dynatrace version 1.310+ Matches – use $match qualifier, for example $match(ip?tables*), where * matches any number of characters (including zero) and ? matches exactly one character.

    Available logic operations:

    • $not($eq(sshd)) – Matches if the service's property value is different from sshd.
    • $and($prefix(ss),$suffix(hd)) – Matches if service's property value starts with ss and ends with hd.
    • $or($prefix(ss),$suffix(hd)) – Matches if service's property value starts with ss or ends with hd.

    When including special characters such as brackets ( and ) within your matching expressions, escape these characters with a tilde ~. For example, to match a property value that includes brackets, like my(amazing)property, you would write $eq(my~(amazing~)property).

With this property we define the services to be monitored based on their startup type.

  • Condition in which you can define a string that:

    • Equals – use $eq qualifier, for example $eq(manual).

    Available logic operations:

    • $not($eq(auto)) – Matches services with startup type different from Automatic.
    • $or($eq(auto),$eq(manual)) – Matches if service's startup type is either Automatic or Manual.

    Use one of the following values as a parameter for this condition:

    • manual for Manual - the service starts only if needed or if you invoke something to start the service.
    • manual_trigger for Manual (Trigger Start) - the service starts along with the startup of another service.
    • auto for Automatic - the service starts automatically.
    • auto_delay for Automatic (Delayed Start) - the service startup is delayed until the system has finished booting.
    • auto_trigger for Automatic (Trigger Start) - the service starts automatically on startup and may be started or stopped due to certain operating system events.
    • auto_delay_trigger for Automatic (Delayed Start, Trigger Start)
    • disabled for Disabled

OneAgent version 1.247+

Dynatrace version 1.247+

optional

  1. Select Add property to specify a custom key-value property for the policy.

    For example, a property with a Key set to custom.message and Value set to The {dt.osservice.name} is with status {dt.osservice.status} (including placeholders {dt.osservice.name} and {dt.osservice.status}) will extract the OS service name and status values once the rule is triggered. If the placeholder substitution fails, both the key and the value will be unavailable.

    os-service-custom-message

    os-service-custom-message-in-event

    For OneAgent version 1.255+, the {dt.osservice.display_name} placeholder is available.

    Additionally, you can utilize specific dt flags in the Key field to tailor the behavior of problem notifications and Davis AI analysis:

    • dt.event.title: Customizes the title of the problem.
    • dt.event.description: Provides a detailed description for the problem.
    • dt.event.allow_davis_merge: Controls Davis AI's decision to merge events based on your settings.

  2. Select Save changes.

Manage monitored OS services

To manage the OS services

  1. In Dynatrace, go to OS services monitoring for the level you are configuring.

    1. Go to Hosts or Hosts Classic (latest Dynatrace).
    2. Find and select your host to display the host overview page.
    3. In the upper-right corner of the host overview page, select More (…) > Settings.
    1. In the host settings, select OS services monitoring.
    1. Go to Deployment Status and then select OneAgents.
    2. On the OneAgent deployment page, turn off Show new OneAgent deployments.
    3. Filter the table by Host group and select the host group you want to configure.

      The Host group property is not displayed when the selected host doesn't belong to any host group.

      This displays the OneAgent deployment page filtered by the selected host group. Each listed host has a Host group: <group name> link, where <group name> is the name of the host group that you want to configure.
    4. Select the host group name in any row.
      As you have filtered by host group, all displayed hosts go to the same host group.
    1. In the host group settings, select OS services monitoring.

    Go to Settings > Monitoring > OS services monitoring.

  2. The OS services you monitor are displayed in a table under the Add policy button.

    • To stop monitoring a listed service, turn the Enabled setting off.
    • To delete a service from the table, select the delete button in the Delete column
    • To view and edit details, select the expand control in the Details column.

Monitor service availability

Dynatrace version 1.243+

OneAgent version 1.243+

The Host overview page contains the OS services analysis section listing the OS services for which any policy (with active alerting or monitoring) is fulfilled.

  1. Go to Hosts or Hosts Classic (latest Dynatrace).
  2. Select any host to go to its overview page.
  3. In the OS services analysis section, select the service name to open the Service overview page.

For more information, see OS services analysis.

Configure at scale using Settings API

You can use the Settings API to configure your service availability monitoring at scale.

  1. To learn the schema, use GET a schema with builtin:os-services-monitoring as the schemaId.
  2. Based on the builtin:os-services-monitoring schema, create your configuration object.
  3. To create your configuration, use POST an object.