Davis AI
Davis events represent different types of individual incidents, such as metric-threshold breaches, baseline degradations, or point-in-time events, such as process crashes. Dynatrace also detects and processes informational events such as new software deployments, configuration changes, and other event types.
A Davis problem may be the result of a single event or multiple events, which is often the case in complex environments. To prevent a flood of seemingly unrelated problem alerts for related events in such environments, the Dynatrace AI correlates all events that share the same root cause into a single, trackable problem. This approach prevents event and alert spamming.
Problems have defined lifespans and are updated in real time with all incoming events and findings. Once a problem is detected, it's listed on your problems feed.
Davis events
Every Davis event update is exported to Grail. This includes updates that only refresh the event, which is guaranteed to be at least 1 update every 6 hours.
Events are essential raw data that Davis (the Dynatrace AI engine) considers during automated root-cause analysis to understand the reasons underlying any problems that are detected in your environment. Out of the box, Davis detects more than 80 different built-in system event types, including process crashes, deployment configuration changes, and VM motion events. Using extension points, you can report custom events through OneAgent plugins or via the Dynatrace API.
Davis shows all system events in the context of your data center topology. So you can analyze events in relation to their parent topological components (for example, hosts, processes, or services) and see how they relate to one another.
Query
Davis events can be queried using the following base query.
fetch dt.davis.events
Event
This section contains general event information.
dt.query
timeseries avg(dt.host.cpu.idle)
dt.source_entity
The ID of the entity considered the source of the event. The string needs to be in the format of any MONITORED_ENTITY type. 1
Tags:
entity-id
HOST-E0D8F94D9065F24F
; PROCESS_GROUP_INSTANCE-E0D8F94D9065F24F
dt.source_entity.type
The type of the entity considered the source of the event. The string needs to be in the format of any MONITORED_ENTITY type.
host
; process_group_instance
; cloud:azure:resource_group
event.category
Standard categorization based on the significance of an event (similar to the severity level in the previous Dynatrace).
INFO
; AVAILABILITY
; ERROR
; SLOWDOWN
; RESOURCE_CONTENTION
; CUSTOM_ALERT
; MONITORING_UNAVAILABLE
event.description
Human-readable description of an event.
The current response time (11 s) exceeds the auto-detected baseline (767 ms) by 1,336 %
event.end
The event end timestamp in UTC (given in Grail preferred Linux timestamp nano precision format).
16481073970000
event.group_label
Group label of an event.
Availability
event.id
Unique identifier string of an event, is stable across multiple refreshes and updates.
5547782627070661074_1647601320000
event.kind
Gives high-level information about what kind of information the event contains, without being specific to the contents of the event. Helps to determine the record type of a raw event.
Tags:
permission
DAVIS_EVENT
event.name
The human readable display name of an event type.
CPU saturation
; User action duration degradation
event.provider
Source of the event, for example the name of the component or system that generated the event.
Tags:
permission
AVAILABILITY
; LOG_EVENTS
; METRIC_EVENTS
; SYNTHETIC
; EVENTS_REST_API_INGEST
; KUBERNETES_ANOMALY_DETECTION
; AGENT_LOCAL_REST_API_INGEST
event.start
The event start timestamp in UTC (given in Grail preferred Linux timestamp nano precision format). This is different from the timestamp even for the first record, as event sources require some time to analyze the underlying data, so the time when an event update is created (timestamp) differs from the time when the event started (event.start)
16481073970000
event.status
Status of an event as being either Active or Closed.
Active
event.status_transition
An enum that shows the transition of the above event state.
CREATED
; UPDATED
; REFRESHED
; TIMED_OUT
; RECOVERED
event.type
The unique type identifier of a given event.
Tags:
permission
APPLICATION_SLOWDOWN
; AVAILABILITY_EVENT
timestamp
The time (UNIX Epoch time in nanoseconds) when the event originated, typically when it was created by the source. If no original timestamp is available, it will be populated at ingest time. Required for all events. In case of a correlated event (e.g. ITIL events) this time could be different from the event.start time, as this time represents the actual timestamp when the "update" for the event was created.
1649822520123123123
Value of this attribute will be based on one of dt.entity.<type>
attributes value. That means that both attributes dt.source_entity and corresponding dt.entity.<type>
will be set to the same ID.
Davis config fields
This section contains fields that influence the Davis routine.
dt.davis.analysis_time_budget
The time budget (in seconds) that the Davis® engine is granted before it must raise a problem. The analysis time budget can be set per event and controls the balance of sending out alerts early and granting the AI analysis enough time to finish its analysis. The trade-off of a short analysis budget is that the root cause and event merge analysis is limited or even skipped. For example, the time budget of 0 seconds means that the event raises a problem and sends the alert immediately, without any analysis.
dt.davis.analysis_trigger_delay
The time delay (in seconds) before the trigger of Davis® analysis. For example, the delay of 0 seconds triggers a Davis® problem and the root cause analysis immediately. The trigger delay can be used to hold the analysis until all the relevant root cause data has arrived to Dynatrace. For example, it might be beneficial for cloud integrations or log integrations that report data in different schedules - you can delay the analysis until data from all sources is available. Note that while longer delays means more data is available for root cause analysis, it also delays alerts delivery.
dt.davis.is_entity_remapping_allowed
This flag defines whether the remapping of the target entity is enabled (true) or disabled (false). If the remapping is enabled, Dynatrace can map the event to an entity extracted from the event metadata. If the remapping is disabled or the extraction is not possible, Dynatrace maps the event to the entity specified in the event configuration (for example, a specific host) or to the global environment entity.
dt.davis.is_frequent_issue_detection_allowed
The flag controls whether the Davis® engine should consider suppressing frequent events (true) or bypassing frequent issues check (false).
dt.davis.is_merging_allowed
This flag controls whether the Davis® engine is allowed to merge this event into a larger problem (true) or if a new problem must be created (false).
dt.davis.is_problem_suppressed
This flag controls whether the Davis® engine suppresses the problem from showing up in the UI and sending notifications.
dt.davis.is_rootcause_relevant
This flag controls whether the Davis® engine should include this event within the root cause analysis (true) or if it is not (false) relevant.
dt.davis.preferred_entity_type
The preferred entity type for remapping. You can find possible values in Dynatrace UI under Settings > Topology model > Generic types. If the remapping (dt.event.allow_entity_remapping) is enabled, this property defines the entity type to which the event should be mapped. If no entity of the preferred type is extracted, no remapping is applied.
my.custom.entity.type
dt.davis.timeout
The event timeout period (in minutes). Various event sources use this event property to keep an event active by regularly refreshing an initial event. The timeout defines how fast the event source must refresh an event to keep it active. To keep the event active, the event source must send the refresh within the timeout period. If no refresh is sent, the event is automatically closed by Dynatrace after the timeout period. Note that metric sources use their own configurable de-alerting windows to close events. Setting the timeout shorter than the de-alerting window will force events to close and increase the risk of false-positive alerts.
Davis system fields
This section contains fields that the Davis routine sets.
dt.davis.disable_merging_reason
Short explanation for why an event should not be merged with other events into the same problem. This is usually set when 'dt.davis.is_merging_allowed' is set to true. If merging is disallowed due to a custom configuration, this field contains the value "Set by event reporter". Additionally, there are scenarios where the Davis® engine may automatically disable merging, typically in cases of problem suppression.
Set by event reporter
; Monitored dimensions limit violated
dt.davis.entity_remapping_failure_info
Provides additional information in case remapping an event onto an entity that was extracted from the event metadata failed.
No entity of type 'myPreferredEntityType' could be extracted from the provided event properties.
dt.davis.impact_level
The impact level of the event.
Application
; Environment
; Infrastructure
; Services
dt.davis.is_frequent_event
Indicates if the event was found to frequently happen. Can only be true if dt.davis.is_frequent_issue_detection_allowed is not set to false.
dt.davis.mute.status
Status describing if the event is muted. It is also set to muted when the event is part of a problem that is manually closed.
MUTED
dt.davis.mute.user
User id of the user who muted the event.
donald_duck@gmail.com
dt.davis.suppress_problem_reason
A short description of the suppression reason.
Due to manual problem close with id P-2307288 by user donald
maintenance.is_under_maintenance
Indicates if the event is within a maintenance window.
Settings data
This section contains information on the main settings object that contributes to the event.
dt.settings.object_id
The object ID of a settings value. This corresponds to the 'objectId' field/parameter in the Settings API.
vu9U3hXa3q0AAAABACFidWlsdGluOnJ1bS51c2VyLWV4cGVyaWVuY2Utc2NvcmUABnRlbmFudAAGdGVuYW50ACRhMzZmYmYwMy00NDY1LTNlNTYtOTZiOS1kOWMzOGQ3MzU1NmO-71TeFdrerQ
dt.settings.schema_id
The schema ID of a settings schema, as used in the Settings APIs.
builtin:problem.notifications
; app:dynatrace.jenkins:connection
dt.settings.scope_id
The ID of the scope that a settings object is persisted on. This corresponds to the 'scope' field/parameter in the Settings API.
environment
; HOST-EFAB6D2FE7274823
Environmental data
This section contains information on entities.
affected_entity_ids
A list of all entities that are directly affected. Each element in the list represents a unique entity.
[HOST-1234567890ABCDEF]
affected_entity_types
A distinct list of entity types corresponding to the entities listed in 'affected_entity_ids'. The order of elements in this list does not necessarily correspond to the order of entity ids.
[dt.entity.host, dt.entity.service]
entity_tags
A list of entity tags that were assigned to the affected entities at the time of event creation.
[departmentA, department:A]
related_entity_ids
A list of all entities that are related to the affected entities. Each element in the list represents a unique entity.
[HOST-1234567890ABCDEF]
Davis event reports
Davis event reports create, update, refresh, or close events within the Davis system. These reports are uniquely identified and linked through their event.name, event.type, and dt.source_entity. This linkage ensures that identical identifiers contribute to a singular event. A new event is generated in the absence of a matching existing event, while a matching event triggers an update to reflect the latest information.
Davis events are enriched with additional fields, as detailed in the Davis event model documentation.
Event reports can be created through various methods, including direct creation via the REST API. Additionally, features like metric events or log events autonomously generate these reports. These features also offer customization capabilities through event templates.
Event
This section contains general event information that can be set on an event report or an event template.
dt.query
timeseries avg(dt.host.cpu.idle)
dt.source_entity
The ID of the entity considered the source of the event. The string needs to be in the format of any MONITORED_ENTITY type. 1
Tags:
entity-id
HOST-E0D8F94D9065F24F
; PROCESS_GROUP_INSTANCE-E0D8F94D9065F24F
event.description
Human-readable description of an event.
The current response time (11 s) exceeds the auto-detected baseline (767 ms) by 1,336 %
event.end
The event end timestamp in UTC (given in Grail preferred Linux timestamp nano precision format).
16481073970000
event.group_label
Group label of an event.
Availability
event.name
The human readable display name of an event type.
CPU saturation
; User action duration degradation
event.start
The event start timestamp in UTC (given in Grail preferred Linux timestamp nano precision format).
16481073970000
event.type
The unique type identifier of a given event.
Tags:
permission
CUSTOM_ALERT
; AVAILABILITY_EVENT
Value of this attribute will be based on one of dt.entity.<type>
attributes value. That means that both attributes dt.source_entity and corresponding dt.entity.<type>
will be set to the same ID.
Davis config fields
This section contains Davis-specific fields that influence the Davis routine on an event report or an event template.
dt.davis.analysis_time_budget
The time budget (in seconds) that the Davis® engine is granted before it must raise a problem. The analysis time budget can be set per event and controls the balance of sending out alerts early and granting the AI analysis enough time to finish its analysis. The trade-off of a short analysis budget is that the root cause and event merge analysis is limited or even skipped. For example, the time budget of 0 seconds means that the event raises a problem and sends the alert immediately, without any analysis.
dt.davis.analysis_trigger_delay
The time delay (in seconds) before the trigger of Davis® analysis. For example, the delay of 0 seconds triggers a Davis® problem and the root cause analysis immediately. The trigger delay can be used to hold the analysis until all the relevant root cause data has arrived to Dynatrace. For example, it might be beneficial for cloud integrations or log integrations that report data in different schedules - you can delay the analysis until data from all sources is available. Note that while longer delays means more data is available for root cause analysis, it also delays alerts delivery.
dt.davis.is_entity_remapping_allowed
This flag defines whether the remapping of the target entity is enabled (true) or disabled (false). If the remapping is enabled, Dynatrace can map the event to an entity extracted from the event metadata. If the remapping is disabled or the extraction is not possible, Dynatrace maps the event to the entity specified in the event configuration (for example, a specific host) or to the global environment entity.
dt.davis.is_frequent_issue_detection_allowed
The flag controls whether the Davis® engine should consider suppressing frequent events (true) or bypassing frequent issues check (false).
dt.davis.is_merging_allowed
This flag controls whether the Davis® engine is allowed to merge this event into a larger problem (true) or if a new problem must be created (false).
dt.davis.is_problem_suppressed
This flag controls whether the Davis® engine suppresses the problem from showing up in the UI and sending notifications.
dt.davis.is_rootcause_relevant
This flag controls whether the Davis® engine should include this event within the root cause analysis (true) or if it is not (false) relevant.
dt.davis.preferred_entity_type
The preferred entity type for remapping. You can find possible values in Dynatrace UI under Settings > Topology model > Generic types. If the remapping (dt.event.allow_entity_remapping) is enabled, this property defines the entity type to which the event should be mapped. If no entity of the preferred type is extracted, no remapping is applied.
my.custom.entity.type
dt.davis.timeout
The event timeout period (in minutes). Various event sources use this event property to keep an event active by regularly refreshing an initial event. The timeout defines how fast the event source must refresh an event to keep it active. To keep the event active, the event source must send the refresh within the timeout period. If no refresh is sent, the event is automatically closed by Dynatrace after the timeout period. Note that metric sources use their own configurable de-alerting windows to close events. Setting the timeout shorter than the de-alerting window will force events to close and increase the risk of false-positive alerts.
Davis problems
Every Davis problem update is exported to Grail. This includes updates that only refresh the problem, which is guaranteed to be at least 1 update every 6 hours.
Problems in Dynatrace represent anomalies in normal behavior or state. Such anomalies can be, for example, a slow service response or user-login process. Whenever a problem is detected, Dynatrace raises a specific problem event indicating such an anomaly.
Raised problems provide insight into their underlying root causes. To identify the root causes of problems, Dynatrace follows a context-aware approach that detects interdependent events across time, processes, hosts, services, applications, and both vertical and horizontal topological monitoring perspectives. Only through such a context-aware approach is it possible to pinpoint the true root causes of problems. For this reason, newly detected anomalous events in your environment won't necessarily result in the immediate raising of a new problem.
Query
Davis problems can be queried using the following base query.
fetch dt.davis.problems
Event
This section contains general event information.
event.category
Standard categorization based on the significance of an event (similar to the severity level in the previous Dynatrace).
AVAILABILITY
; ERROR
; SLOWDOWN
; RESOURCE_CONTENTION
; CUSTOM_ALERT
; MONITORING_UNAVAILABLE
event.end
The problem end timestamp in UTC (given in Grail preferred Linux timestamp nano precision format).
16481073970000
event.id
Unique identifier string of a problem, is stable across refreshes and updates.
5547782627070661074_1647601320000
event.kind
Gives high-level information about what kind of information the event contains, without being specific to the contents of the event. Helps to determine the record type of a raw event.
Tags:
permission
DAVIS_PROBLEM
event.name
The human readable display name of an event type.
CPU saturation
; Multiple infrastructure problems
event.start
The problem start timestamp in UTC (given in Grail preferred Linux timestamp nano precision format). This is different from the timestamp even for the first record, as event sources require some time to analyze the underlying data, so the time when a problem update is created (timestamp) differs from the time when the event started (event.start)
16481073970000
event.status
Status of an event as being either Active or Closed.
ACTIVE
; CLOSED
event.status_transition
An enum that shows the transition of the above event state.
CREATED
; UPDATED
; REFRESHED
; RESOLVED
; REOPENED
; CLOSED
timestamp
The time (UNIX Epoch time in nanoseconds) when the event originated, typically when it was created by the source. If no original timestamp is available, it will be populated at ingest time. Required for all events. In case of a correlated event (e.g. ITIL events) this time could be different from the event.start time, as this time represents the actual timestamp when the "update" for the event was created.
1649822520123123123
Davis system fields
This section contains fields that the Davis routine sets.
display_id
A pretty, mostly unique id for the problem.
P-2307288
dt.davis.affected_users_count
The estimated count of users affected by the problem for the application with the highest individual impact.
dt.davis.event_ids
A collection of Davis event ids that belong to the problem.
[-2127669892157121805_1688396340000]
dt.davis.is_duplicate
Indicates if the problem has become a duplicate of another problem. Duplicates can be related by looking for event ids that are part of multiple problems.
dt.davis.is_frequent_event
Indicates if the problem was found to frequently happen. Can only be true if dt.davis.is_frequent_issue_detection_allowed is not set to false for any event of the problem.
dt.davis.last_reopen_timestamp
Timestamp in UTC (given in Grail preferred Linux timestamp nano precision format) when the problem has reopened the last time. A reopen can occur when a problem has resolved, but is not yet closed. If Davis causal AI identified a new event that should be part of the problem, the problem reopens. The field is not set if the problem never reopened.
dt.davis.mute.status
Status describing if the problem is muted. It is also set to muted when the problem is manually closed.
MUTED
dt.davis.mute.user
User id of the user who muted the event.
donald_duck@gmail.com
labels.alerting_profile
A list of alerting profiles that match the problem at the current time.
[Production, Team DevOps]
maintenance.is_under_maintenance
Indicates if the problem is within a maintenance window.
resolved_problem_duration
Final duration of the problem in nanoseconds after it was resolved.
Environmental data
This section contains information on entities.
affected_entity_ids
A list of all entities that are directly affected. Each element in the list represents a unique entity.
[HOST-1234567890ABCDEF]
affected_entity_types
A distinct list of entity types corresponding to the entities listed in 'affected_entity_ids'. The order of elements in this list does not necessarily correspond to the order of entity ids.
[dt.entity.host, dt.entity.service]
entity_tags
A combined list of all Davis event entity tags.
[departmentA, department:A]
related_entity_ids
A list of all entities that are related to the affected entities. Each element in the list represents a unique entity.
[HOST-1234567890ABCDEF]
root_cause_entity_id
The problem root cause entity.
HOST-1234567890ABCDEF
root_cause_entity_name
The name of the problem root cause entity at the time when the problem snapshot was created.
Server 1.2.3.4