For full observability into your Kubernetes events, automatic Davis analysis, and custom alerting, you need to enable Kubernetes event monitoring.
To enable event monitoring for specific Kubernetes clusters
When you enable Monitor events, all events are ingested and all important events are considered in the Davis root-cause detection. Alternatively, for maximum flexibility and fine-grained control over the events you want to ingest from Kubernetes, you can filter events.
Even if Monitor events is disabled, the so-called inferred Kubernetes events are still ingested. These inferred events aren't native Kubernetes events but are created by ActiveGate based on the information from the Kubernetes API server.
Examples of inferred events:
cgroup OOM kill events
Workload specification changes (replicas, images, environment variables, resources, probes) for
These events aren't billed and are relevant for Davis root-cause analysis.
After enabling the Kubernetes event monitoring, you can view and analyze events from the Kubernetes cluster.
On your Kubernetes cluster details page, go to Events.
You can filter events by:
For more information about an event, select Details for the event.
Kubernetes events are associated with Kubernetes entities. An event is displayed on the respective entity page and on related entity pages. For example, pod events are displayed on the cluster, namespace, workload, and pod details page.
You can also view events on the Log viewer page (in Dynatrace, go to Logs or Logs & Events (latest Dynatrace)), which allows for advanced search and filtering.
If the environment is platform enabled the events are stored in Grail. The following DQL query can be used as a template to query for specific events in Notebooks or Dashboards.
fetch events| filter event.provider == "KUBERNETES_EVENT"
Inferred Kubernetes events are not shown in Log viewer. They are directly ingested as events.
Filtering is turned off by default, which means that all events are ingested. To set up monitoring only for certain events
Turn on Filter events.
If you don't see the Filter events switch, make sure that Monitor events is turned on first.
Set up multiple field selectors for every Kubernetes environment.
optional Have Davis perform root cause analysis on all important Kubernetes events.
Select Save changes.
Filtering follows the Kubernetes-established syntax of field selectors, so events can be chosen based on event resource fields such as source.component, type, or involvedObject.
A field selector expression must meet the following requirements:
It must conform to the following regular expression: ^[\w\.]{1,1024}((=){1,2}|(!=))[^,!=]{0,256}(,[\w\.]{1,1024}?((=){1,2}|(!=))[^,!=]{0,256}){0,9}$.
It can have up to 10 selectors separated by commas. Events matching all comma-separated selectors will be ingested. The logical operator is AND.
A selector consists of three parts:
= , == or !=.For example,
involvedObject.namespace=hipster-shop,type=Warning, the expression will store all the events related to the namespace hipster-shop that are of type Warning.hipster-shop and all events of type Warning. The logical operator in this case is OR.Example event field selectors:
Node eventsinvolvedObject.kind=NodeWarning eventstype=WarningPod eventsinvolvedObject.kind=PodinvolvedObject.namespace=<your_namespace> (Make sure to replace <your_namespace> with the name of your own namespace)BackOff events for pods across all namespacesreason=BackOfftype!=involvedObject.fieldPath==spec.containers{nginx}To set up event field selectors, select one of the options below:
You can create a maximum of 20 event filter rules per Kubernetes cluster.
When problems with applications, microservices, or infrastructure are detected, Davis performs root cause analysis on all important Kubernetes events for nodes, namespaces, workloads, and pods.
A Kubernetes event is important (relevant for Davis root cause analysis) when at least one of the following two conditions is true.
Warning.BackOff,
DeadlineExceeded,
Killing,
NodeNotSchedulable,
OutOfDisk,
Preempting
By default, all these events are monitored when Monitor events is turned on. If you choose to Filter events, either predefined important events filter or custom events filters can be applied. If multiple filters are set, they are combined using a logical OR. The event is ingested, once a Kubernetes event matches any of the filters.
To enable monitoring of important events, when event filtering is turned on
Go to Kubernetes or Kubernetes Classic (latest Dynatrace).
Find your Kubernetes cluster and, in the Actions column, select More (…) > Settings.
On the Monitoring Settings tab, turn on Include important events.
If you don't see the Include important events switch, make sure that Monitor events and Filter events are turned on first.
Select Save changes.
Kubernetes events are made available in the Kubernetes: Event count (builtin:kubernetes.events) metric. To filter the events count metric for the relevant events, use the k8s.event.reason and k8s.event.type dimensions.
To help you understand the distribution and development of Kubernetes events over time, use Data Explorer to create charts. You can use the charts to compare different timeframes, different entities, event filters, and the use of complex expressions.
To trigger alerts whenever Kubernetes events occur (for example, always alert in case of an Evicted event), define Metric events based on the Kubernetes: Event count metric.
To estimate the number of events that consume DDUs, you can query the dsfm:active_gate.kubernetes.events.processed metric, which provides information about the number of events that are being ingested into Dynatrace per Kubernetes cluster.
Example query for a 24-hour timeframe:
dsfm:active_gate.kubernetes.events.processed:splitBy("dt.entity.kubernetes_cluster"):sum:auto:sort(value(sum,descending)):limit(10)
DDU consumption applies to Kubernetes event monitoring. For details, see DDUs for custom Davis events.