Configure Cluster event notifications

How-to guide
11-min read

You can manage the following notifications in a Managed Cluster:

Insufficient disk space on a Managed Cluster node
Triggered when a disk partition on a Managed Cluster node has less disk space than required for a given storage type. In that case, you need to extend your disk or review cluster settings. Otherwise, data retention might be reduced.
Insufficient hardware on a Managed Cluster node
Triggered when a node doesn't have enough CPU cores and RAM to meet recommended requirements. For details, see System requirements.
Metrics storage exceeds supported size (4 TiB)
Triggered when metrics storage for a node is too high. In that case, you need to review your monitoring settings or add additional nodes to the Managed Cluster.
Transaction storage retention period truncation
Triggered when data retention has been automatically reduced to store new data. In that case, Dynatrace recommends that you adjust the target retention time, review monitoring settings, or extend a disk.
Adaptive Load Reduction activity
Triggered when a node is considered overloaded and unable to keep up with processing incoming requests. In that case, you need to review monitoring settings or increase CPU cores and RAM on the node. For details, see Adaptive Load Reduction.

UI workflows

Use the Cluster Management Console (CMC) to configure who receives cluster event notifications and emergency notifications.

Configure email notification recipients

Configure who receives cluster event email notifications in the CMC:

In CMC, go to Settings > Emails > Email notifications.
In Notifications recipients, enter the email addresses that should receive cluster, environment, and account-related notifications. Separate multiple email addresses with commas.
Optional: Turn on Also notify all active users with cluster admin rights.

Configure emergency contacts

Emergency contacts receive notifications from Dynatrace product experts in the event of catastrophic Managed Cluster states or critical security vulnerabilities that must be addressed.

In the CMC, go to Settings > Emails > Email notifications.
On the Email notifications page, under Emergency notifications recipients, enter one or more email addresses separated by commas.
Select Save changes.

API workflows

Use the Cluster API to configure which cluster events trigger email notifications and to manage the notification settings object.

Authenticate API requests

Configure event notification settings

Read configuration schema

Read current configuration

Create notification settings object

Update notification settings

Delete notification settings

Authenticate API requests

To generate a cluster token with the Write settings and Read settings scopes, follow these steps.

Go to Settings > API tokens.
In the Cluster tokens section, select Generate token.
Enter a name for your token and define the Write settings and Read settings access scopes of your cluster API token.
Save and then Copy the token to a secure location.

Configure event notification settings

To configure which cluster events trigger email notifications, send your configuration as a JSON payload. Use the POST an object endpoint with a cluster token that has the appropriate access scopes:

[
  {
    "schemaId": "builtin:cluster-events-notification-settings",
    "scope": "cluster",
    "value": {
                "insufficientDiskSpace": {
                    "sendEmail": true
                },
                "insufficientHardware": {
                    "sendEmail": true
                },
                "insufficientMetricStorage": {
                    "sendEmail": false
                },
                "transactionStorageTruncation": {
                    "sendEmail": false
                },
                "adaptiveLoadReductionActivated": {
                    "sendEmail": false
                }
            }
  }
]

Read configuration schema

To learn the JSON format required to post your configuration, use the GET a schema endpoint with a cluster token that has the appropriate access scopes. The configuration schema identifier (schemaId) is builtin:cluster-events-notification-settings.

Response schema details

{
    "dynatrace": "1",
    "schemaId": "builtin:cluster-events-notification-settings",
    "displayName": "Cluster events notification settings",
    "description": "Configuration of emails notifications for cluster events.",
    "documentation": "Email notifications are sent to recipients defined in Settings - Emails - Email notifications.",
    "version": "0",
    "multiObject": false,
    "maxObjects": 1,
    "allowedScopes": [
        "cluster"
    ],
    "enums": {},
    "types": {
        "SingleEventSettings": {
            "version": "0",
            "versionInfo": "",
            "displayName": "Single event settings",
            "summaryPattern": "",
            "description": "Settings for a single cluster event",
            "documentation": "",
            "properties": {
                "sendEmail": {
                    "displayName": "Send email",
                    "description": "",
                    "documentation": "",
                    "type": "boolean",
                    "nullable": false,
                    "maxObjects": 1,
                    "modificationPolicy": "DEFAULT",
                    "default": true
                }
            },
            "type": "object"
        }
    },
    "properties": {
        "insufficientDiskSpace": {
            "displayName": "Insufficient disk space on a cluster node",
            "description": "Triggered when a disk partition on a cluster node has less disk space than required for a given storage type. In that case, you need to extend your disk or review cluster settings. Otherwise, data retention might be reduced.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "insufficientHardware": {
            "displayName": "Insufficient hardware on a cluster node",
            "description": "Triggered when a cluster node doesn't have enough CPU cores and RAM to meet recommended requirements.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "insufficientMetricStorage": {
            "displayName": "Metrics storage exceeds supported size (4TiB)",
            "description": "Triggered when Metrics storage a cluster node is too high. In that case, you need to review your monitoring settings or add additional nodes to the cluster.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "transactionStorageTruncation": {
            "displayName": "Transaction storage retention period truncation",
            "description": "Triggered when data retention has been automatically reduced to store new data. In that case, it is recommended to adjust target retention time, review monitoring settings or extend a disk.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "adaptiveLoadReductionActivated": {
            "displayName": "Adaptive Load Reduction activity",
            "description": "Triggered when a cluster node has been considered as overloaded and not able to keep up with processing incoming requests. In that case, you need to review monitoring settings or increase CPU cores and RAM on the cluster node.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        }
    }
}

Read current configuration

To check the current configuration, use the GET objects endpoint (/api/cluster/v2/settings/objects?schemaIds=builtin:cluster-events-notification-settings&scopes=cluster) with a cluster token that has the appropriate access scopes.

If these settings have been previously changed, the items list contains a single object. Use the objectId from the list in making subsequent updates.

If the items list is empty, the default value is used (not visible in Dynatrace API):

{
  "value": {
            "insufficientDiskSpace": {
                "sendEmail": true
            },
            "insufficientHardware": {
                "sendEmail": true
            },
            "insufficientMetricStorage": {
                "sendEmail": true
            },
            "transactionStorageTruncation": {
                "sendEmail": false
            },
            "adaptiveLoadReductionActivated": {
                "sendEmail": false
            }
        }
}

Create notification settings object

To create a cluster event notification settings object, use the POST objects endpoint with a cluster token that has the appropriate access scopes. Use the ID of the newly created object (event notification settings) for later notification settings updates.

Example of creating notification settings object

In this example, using a POST API call to the /api/cluster/v2/settings/objects endpoint and builtin:cluster-events-notification-settings schema, you create a notification settings object in your cluster settings:

[
  {
    "schemaId": "builtin:cluster-events-notification-settings",
    "scope": "cluster",
    "value": {
                "insufficientDiskSpace": {
                    "sendEmail": true
                },
                "insufficientHardware": {
                    "sendEmail": true
                },
                "insufficientMetricStorage": {
                    "sendEmail": false
                },
                "transactionStorageTruncation": {
                    "sendEmail": false
                },
                "adaptiveLoadReductionActivated": {
                    "sendEmail": false
                }
            }
  }
]

Update notification settings

There are two methods you can use to update cluster events notification settings once the notification settings object is created. In either case, make sure you have a cluster token that has the appropriate access scopes.

You can use the same POST method that you used to create the settings object (Create notification settings object). The schema doesn't allow duplicate settings objects, so if you attempt to create another settings object, you will overwrite the existing one.

You can modify an existing settings object by making a PUT API call to the /api/cluster/v2/settings/objects/<objectId> endpoint and providing the objectId obtained when creating the initial event notification settings object.

Example of updating the event notification settings object

Make the PUT API call to the /api/cluster/v2/settings/objects/<objectId>

{
  "value": {
            "insufficientDiskSpace": {
                "sendEmail": true
            },
            "insufficientHardware": {
                "sendEmail": true
            },
            "insufficientMetricStorage": {
                "sendEmail": false
            },
            "transactionStorageTruncation": {
                "sendEmail": false
            },
            "adaptiveLoadReductionActivated": {
                "sendEmail": false
            }
        }
}

Delete notification settings

To delete an existing notification settings object, make a DELETE API call to the /api/cluster/v2/settings/objects/<objectId> endpoint. Use a cluster token with the appropriate access scopes and provide the objectId obtained when you created the initial event notification settings object. After deletion, notification behavior falls back to the default: all notifications trigger an email to configured recipients.

Cluster events and email notifications

The following tables list Managed Cluster event notifications by event category.

`ACTIVE_GATE_TOKENS`

Severity	Summary	Email notification	MC notification
WARNING	"ActiveGate Token(s) will expire soon."	Yes	No

`CASSANDRA`

Severity	Summary	Email notification	MC notification
WARNING	"Cassandra node connection lost (`%d` times in the last hour)."	No	Yes
WARNING / INFO	"Cassandra node connection `ADDED/LOST/REACTIVATED`, host ."	No	Yes
INFO	"Cassandra node connection added (`%d` times in the last hour)."	No	Yes
INFO	"Cassandra node connection reestablished (`%d` times in the last hour)."	No	Yes

`CLUSTER_LIFECYCLE`

Severity	Summary	Email notification	MC notification
SEVERE	"License `'%s'` will expire within one day."	Yes	Yes
SEVERE	"License `'%s'` has expired."	Yes	Yes
SEVERE	"Your Dynatrace Managed cluster node `{0}` is undersized!"	Configurable ¹	Yes
SEVERE	"Storage volume for Dynatrace Managed log files is running out of space on `{0}`."	Yes	Yes
SEVERE	"Your Dynatrace Managed cluster should be scaled-out!"	Yes	Yes
SEVERE	"Could not detect `%s` activity on `%s%s`."	No	Yes
SEVERE	"Insufficient system privileges on `%s`."	No	Yes
SEVERE	"Update your Cluster `ActiveGate{0}` immediately. You're using `{1}` `ActiveGate{0}` that `{2}` no longer supported!"	Yes	Yes
SEVERE	"Post-Update data migration failed, please contact Dynatrace Support."	No	Yes
SEVERE	"Dynatrace update `%s` download failed."	No	Yes
SEVERE	"Cassandra backup problem."	No	Yes
SEVERE	"Dynatrace Managed has stopped collecting monitored data…"	No	Yes
SEVERE	"SSL certificate expired."	No	Yes
SEVERE	"License `'%s'` is inactive. Until it is done, you cannot set up any monitoring. Activate your license on Licensing page."	No	Yes
SEVERE	"Failed to import the third-party vulnerability feeds version `%s` into the cluster's Dynatrace Application Security."	No	Yes
SEVERE	"Failed to import metadata file `%s`."	No	Yes
SEVERE	"Default cluster features were changed for non-gov license."	No	Yes
SEVERE	"Cluster traffic control: OneAgent monitoring was disabled on recently connected hosts to avoid cluster overload."	No	Yes
WARNING / SEVERE	"Host is down."	Yes	Yes
WARNING / SEVERE	"ElasticSearch backup problem."	No	Yes
WARNING / SEVERE	"Cluster `'%s'` (`'%s'`) update failed."	No	Yes
WARNING	"WebUI nodes settings change failed."	No	Yes
WARNING	"LDAP connection error."	Yes	Yes
WARNING	"Upcoming update to version `%s` is suspended for the cluster `'%s'` (`'%s'`)."	Yes	Yes
WARNING	"Node is down - `%s`."	Yes, if happened outside of the upgrade procedure	Yes
WARNING	"There is lack of connection to Dynatrace Mission Control."	Yes	Yes
WARNING	"Backup has been disabled because the configuration is no longer supported."	Yes	Yes
WARNING	"Fetching OAuth client credentials failed with status code `{0}`."	No	Yes
WARNING	"Cluster `'%s'` (`'%s'`) failed to determine update status."	No	Yes
WARNING	"Self monitoring download failed on cluster `'%s'`."	No	Yes
WARNING	"License check error."	No	Yes
WARNING	"SSL certificate (`%s`) refresh failed."	No	Yes
WARNING	"Login failed."	No	Yes
WARNING	"Not all tenants from MultiTenant-ActiveGate `%s` are configured with AuthTokens."	No	Yes
WARNING	"Let's Encrypt SSL certificate fetching failure."	No	Yes
WARNING	"Your SSL certificate will expire soon."	No	Yes
WARNING	"Login failed."	No	Yes
WARNING	"User welcome e-mail was not sent."	No	Yes
WARNING	"The Snyk vulnerability feed import failed."	No	Yes
WARNING	"The NVD CVE vulnerability feed import failed."	No	Yes
WARNING	"LDAP connection problems."	No	Yes
WARNING / INFO	"Oidc signature check."	No	Yes
WARNING / INFO	"Billing archive has been successfully downloaded/downloaded with warning/failed."	No	Yes
INFO	"WebUI nodes settings changed successfully."	No	Yes
INFO	"Scheduled update has been resumed for the cluster `'%s'` (`'%s'`)."	Yes	Yes
INFO	"Hardware recommendation are fulfilled on `{0}`."	Configurable ¹	Yes
INFO	"Storage space for Dynatrace Managed log files is sufficient on `{0}`."	Yes	Yes
INFO	"Cluster meets the minimum requirement for the number of nodes."	Yes	Yes
INFO	"Node is up - `%s`."	Yes	Yes
INFO	"Host is up."	Yes	Yes
INFO	"Connection to Dynatrace Mission Control is back again."	Yes	Yes
INFO	"`<processName>` is up on `<hostName>`."	No	Yes
INFO	"Node was restored on ."	No	Yes
INFO	"Dynatrace update `%s` was successfully downloaded."	No	Yes
INFO	"Cluster `'%s'` (`'%s'`) started updating."	No	Yes
INFO	"Cluster update download finished."	No	Yes
INFO	"Cluster `'%s'` update to version `%s` finished."	No	Yes
INFO	"Request for remote access."	No	Yes
INFO	"Successful login to DebugUI."	No	Yes
INFO	"SSL certificate (`%s`) refresh succeeded."	No	Yes
INFO	"Let's Encrypt SSL certificate fetching succeeded."	No	Yes
INFO	"Cluster `'%s'` configuration database initialized successfully."	No	Yes
INFO	"Server `%d` joined cluster `'%s'` (`'%s'`)."	No	Yes
INFO	"Server `%d` left cluster `'%s'` (`'%s'`)."	No	Yes
INFO	"Server `%d` joined cluster `'%s'` (`'%s'`) `%d` times last hour."	No	Yes
INFO	"Server `%d` left cluster `'%s'` (`'%s'`) `%d` times last hour."	No	Yes
INFO	"Successful login."	No	Yes
INFO	"The third-party vulnerability feeds version `%s` were successfully imported into the cluster's Dynatrace Application Security."	No	Yes
INFO	"Failed to import the third-party vulnerability feeds version `%s` into the cluster's Dynatrace Application Security."	No	Yes

`CLUSTER_RUNTIME_SETTING`

Severity	Summary	Email notification	MC notification
SEVERE	"Responsibility cluster nodes override set on node: `%d`."	No	Yes

`CONFIGURATION_AUDIT`

Severity	Summary	Email notification	MC notification
INFO	"Min Agent Version updated to `%d` by Session Replay."	No	Yes
INFO	"Session State Version updated to `v%d` by Session Replay."	No	Yes

`ELASTICSEARCH`

Severity	Summary	Email notification	MC notification
INFO	"Elasticsearch storage service on your Dynatrace Managed cluster might be overloaded!"	No	Yes

`ERROR`

Severity	Summary	Email notification	MC notification
WARNING	"ElasticSearch update transient settings failed."	No	Yes
INFO	"ElasticSearch update transient settings succeeded."	No	Yes

`LOG_EVENT_DROP`

Severity	Summary	Email notification	MC notification
WARNING	"Ingested log data is trimmed."	No	Yes
WARNING	"Log ingest queue is full."	No	Yes
WARNING	"Elasticsearch log queue is full."	No	Yes
WARNING	"Elasticsearch log storing failed."	No	Yes

`MANAGED_INTERNAL`

Severity	Summary	Email notification	MC notification
INFO	"Internal: Cassandra has old files."	No	Yes

`MANAGED_NODE_ADD`

Severity	Summary	Email notification	MC notification
WARNING	"Adding new node is taking more than expected."	No	Yes
INFO	"Adding new node operation has been started."	No	Yes
INFO	"Adding new node precondition check status."	No	Yes
INFO	"Adding new node finished successfully."	No	Yes
WARNING	"Adding new node failure."	No	Yes

`MANAGED_NODE_REMOVAL`

Severity	Summary	Email notification	MC notification
WARNING	"Node removal finished with error `%s` (id=`%d`)."	No	Yes
INFO	"Node removal operation is not allowed `%s` (id=`%d`)."	No	Yes
INFO	"Node removal process started successfully `%s` (id=`%d`)."	No	Yes
INFO	"Node removal process started successfully `%s`."	No	Yes
INFO	"Node removal finished successfully."	No	Yes
INFO	"Node removal failure."	No	Yes

`SECURITY_GATEWAY_LIFECYCLE`

Severity	Summary	Email notification	MC notification
INFO	"ActiveGate (host=`%s`) registered on cluster."	No	Yes
INFO	"ActiveGate (host=`%s`) unregistered on cluster."	No	Yes
INFO	"ActiveGate (host=`%s`) lost connection to cluster."	No	Yes
INFO	"ActiveGate (environment=`%s`, host=`%s`) registered on cluster."	No	Yes
INFO	"ActiveGate (environment=`%s`, host=`%s`) unregistered on cluster."	No	Yes
INFO	"ActiveGate (environment=`%s`, host=`%s`) lost connection to cluster."	No	Yes

`SERVER_LIFECYCLE`

Severity	Summary	Email notification	MC notification
SEVERE	"Transaction storage retention period truncated."	Configurable ¹	Yes
SEVERE	"Insufficient disk space on `%s` on `%s`."	Configurable ¹	Yes
SEVERE	"Long-term Metrics Store size exceeds the maximum acceptable 4 TB on `%s`."	Configurable ¹	Yes
SEVERE	"`<component name>` is down."	No	Yes
SEVERE	"Heap memory: Server `%d` started memory emergency mode."	No	Yes
SEVERE	"Heap memory: Server `%d` triggered a hard memory cleanup action."	No	Yes
SEVERE	"A cluster node can't receive OneAgent traffic."	No	Yes
WARNING	"Server `%d` activated Adaptive Load Reduction."	Configurable ¹	Yes
WARNING	"Long-term Metrics Store size exceeds recommended 2 TB on `%s`."	No	Yes
WARNING	"Node cannot read and write to directory: `'%s'`."	No	Yes
WARNING	"Heap memory: Server `%d` triggered a soft memory cleanup action."	No	Yes
WARNING / INFO	"Disabling OneAgent traffic at "	No	Yes
WARNING / INFO	"Enabling OneAgent traffic at "	No	Yes
INFO	"Failed to import the third-party vulnerability feeds version `%s` into the cluster's Dynatrace Application Security."	Configurable ¹	Yes
INFO	"Server `%d` deactivated Adaptive Load Reduction."	Configurable ¹	Yes
INFO	"Server `%d` shutdown initiated."	No	Yes
INFO	"`<component name>` is up."	No	Yes
INFO	"Server `%d` startup completed. (Version: `%s`)."	No	Yes
INFO	"Heap memory: Server `%d` ended memory emergency mode."	No	Yes
?	"Server time of server `%d` is out of sync. Time difference `%d` milliseconds. Please enable NTP on all cluster nodes."	No	Yes

`TENANT_LIFECYCLE`

Severity	Summary	Email notification	MC notification
SEVERE	"Trial environment expired."	No	Yes
SEVERE	"The node with id `%s` is not properly configured."	No	Yes
SEVERE	"Environment `'%s'` with id `%s` failed to start on server `%d`. See server logs for details."	No	Yes
INFO	"Environment `'%s'` with id `%s` created."	No	Yes
INFO	"Environment `'%s'` with id `%s` updated."	No	Yes
INFO	"Environment `'%s'` with id `%s` removed."	No	Yes

Configurable means that you can configure the notifications via Settings API. Setting id: builtin:cluster-events-notification-settings. To use this REST API, you need to authorize with settings-specific token with settings.read and settings.write permissions.