Try it free

Kubernetes alerting

  • Latest Dynatrace
  • Reference
  • 15-min read

Dynatrace provides ready-made alerts for common Kubernetes issues across clusters, nodes, namespaces, workloads, and persistent volume claims. Each alert is either a health alert, which triggers a Problem investigation, or a warning signal, which flags a potential issue without opening a Problem. For details on this severity model, see Health alerts and warning signals.

This page lists every ready-made Kubernetes alert together with its severity, recommended defaults, de-alert timing, and detection logic.

Configure

To review or adjust your Kubernetes alert configuration

  1. Go to Settings Settings.
  2. Select Analyze and alert > Alerts.
  3. Under the Kubernetes section you can find the object types.
  4. Review enabled alerts and adjust thresholds as needed.

You can also open alert configuration directly from a cluster or namespace in Kubernetes (new) Kubernetes. For a full walkthrough of all configuration levels (environment, cluster, namespace), see Alert on common Kubernetes misconfigurations and detect anomalies with Kubernetes metrics.

Severity and recommended defaults

With the latest Dynatrace upgrade, Kubernetes alerts that previously raised a problem have been reclassified as warning signals. This reclassification is automatic and applies to all environments - it doesn't require any configuration change.

Independently, recommended defaults are available for all environments. They define which alerts are enabled and their detection thresholds, but they are not applied automatically — your current configuration isn't affected until you explicitly apply them under Settings Settings > Analyze and alert > Alerts > Kubernetes.

Each alert has two independent dimensions:

  • Enabled/disabled: whether the alert actively monitors your environment.
  • Detection threshold: the condition that triggers the alert.

In the alert reference tables below:

  • ¹ marks alerts whose enablement or threshold changes if you apply the recommended defaults.
  • ² marks alerts reclassified from health alert to warning signal — already in effect, independent of the recommended defaults.

The following alerts are currently disabled and become enabled with Critical severity when you apply the recommended defaults. Review them before applying, since they'll start opening problems.

AlertTrigger condition

Detect stuck deployments

Stops progressing ≥ 3 min; within 5 min

Detect workloads without ready pods

No ready pods ≥ 10 min; within 15 min

Detect pod backoff events

Event-based

Detect pod eviction events

Event-based

Available alerts

Cluster

AlertProblem typeEnabled (recommended)Detection threshold (recommended)SeverityDe-alerts afterCalculation

Detect cluster readiness issues

Availability

Yes

Not ready ≥10 min; within 15 min

Health alert

10 minutes

Cluster readyz metric

Detect cluster CPU-request saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Node CPU requests / Node CPU allocatable

Detect cluster memory-request saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Node memory requests / Node memory allocatable

Detect cluster pod-saturation

Resource

No

>90% pods / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of ready pods / Sum of allocatable pods

Detect monitoring issues

Availability

Yes ¹

Not available ≥20 min; within 30 min

Warning signal ¹ ²

10 minutes

Dynatrace API monitoring availability

Detect monitoring issues has no associated metric or DQL expression — it monitors whether Dynatrace's own API-based monitoring of the cluster is available, not a cluster condition.

Cluster metric and DQL expressions

Detect cluster CPU-request saturation

TypeExpression

Metric expression

builtin:kubernetes.node.requests_cpu:splitBy():sum/builtin:kubernetes.node.cpu_allocatable:splitBy():sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.container.requests_cpu, rollup: avg), o2=sum(dt.kubernetes.node.cpu_allocatable, rollup: avg)}, by: {}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect cluster memory-request saturation

TypeExpression

Metric expression

builtin:kubernetes.node.requests_memory:splitBy():sum/builtin:kubernetes.node.memory_allocatable:splitBy():sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.container.requests_memory, rollup: avg), o2=sum(dt.kubernetes.node.memory_allocatable, rollup: avg)}, by: {}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect cluster pod-saturation

TypeExpression

Metric expression

(builtin:kubernetes.node.pods:filter(and(eq(pod_condition,Ready))):splitBy():sum/builtin:kubernetes.node.pods_allocatable:splitBy():sum):default(0.0)*100.0

DQL

timeseries o1=sum(dt.kubernetes.pods, rollup: avg), nonempty:true, filter: {((pod_condition=="Ready"))}, by: {}| join [timeseries operand=sum(dt.kubernetes.node.pods_allocatable, rollup: avg), nonempty:true, by: {}], on: {interval}, fields: {o2=operand}| fieldsAdd result=if(isNull(o1[]/o2[]), 0.0, else: o1[]/o2[])* 100.0| fieldsRemove {o1,o2}

Detect cluster readiness issues

TypeExpression

Metric expression

builtin:kubernetes.cluster.readyz:splitBy():sum

DQL

timeseries {sum(dt.kubernetes.cluster.readyz, rollup: avg)}, by: {}

Node

AlertProblem typeEnabled (recommended)Detection threshold (recommended)SeverityDe-alerts afterCalculation

Detect node readiness issues

Availability

Yes ¹

Not ready ≥10 min; within 20 min

Warning signal ¹ ²

10 minutes

Node condition metric filtered by 'not ready'

Detect problematic node conditions

Error

Yes ¹

Problematic ≥10 min; within 20 min

Warning signal ¹ ²

10 minutes

Nodes condition metric

Detect node CPU-request saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of node CPU requests / Sum of node CPU allocatable

Detect node memory-request saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of node memory requests / Sum of node memory allocatable

Detect node pod-saturation

Resource

No

>90% pods / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of running pods on node / Node pod limit

Detect problematic node conditions triggers on any of the following: ContainerRuntimeProblem, ContainerRuntimeUnhealthy, CorruptDockerOverlay2, DiskPressure, FilesystemCorruptionProblem, FrequentContainerdRestart, FrequentDockerRestart, FrequentGcfsSnapshotterRestart, FrequentGcfsdRestart, FrequentKubeletRestart, FrequentUnregisterNetDevice, GcfsSnapshotterMissingLayer, GcfsSnapshotterUnhealthy, GcfsdUnhealthy, KernelDeadlock, KubeletProblem, KubeletUnhealthy, MemoryPressure, NetworkUnavailable, OutOfDisk, PIDPressure, ReadonlyFilesystem.

Node metric and DQL expressions

Detect node CPU-request saturation

TypeExpression

Metric expression

builtin:kubernetes.node.requests_cpu:splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum/builtin:kubernetes.node.cpu_allocatable:splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.container.requests_cpu, rollup: avg), o2=sum(dt.kubernetes.node.cpu_allocatable, rollup: avg)}, by: {dt.kubernetes.node.system_uuid,k8s.node.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect node memory-request saturation

TypeExpression

Metric expression

builtin:kubernetes.node.requests_memory:splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum/builtin:kubernetes.node.memory_allocatable:splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.container.requests_memory, rollup: avg), o2=sum(dt.kubernetes.node.memory_allocatable, rollup: avg)}, by: {dt.kubernetes.node.system_uuid,k8s.node.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect node pod-saturation

TypeExpression

Metric expression

builtin:kubernetes.node.pods:filter(and(eq(pod_phase,Running))):splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum/builtin:kubernetes.node.pods_allocatable:splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum*100.0

DQL

timeseries o1=sum(dt.kubernetes.pods, rollup: avg), nonempty:true, filter: {((pod_phase=="Running"))}, by: {dt.kubernetes.node.system_uuid,k8s.node.name}| join [timeseries operand=sum(dt.kubernetes.node.pods_allocatable, rollup: avg), nonempty:true, by: {dt.kubernetes.node.system_uuid,k8s.node.name}], on: {interval}, fields: {o2=operand}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect node readiness issues

TypeExpression

Metric expression

builtin:kubernetes.node.conditions:filter(and(eq(node_condition,Ready),ne(condition_status,True))):splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum

DQL

timeseries {sum(dt.kubernetes.node.conditions, rollup: avg)}, filter: {((node_condition=="Ready")AND(condition_status!=true))}, by: {dt.kubernetes.node.system_uuid,k8s.node.name}

Detect problematic node conditions

TypeExpression

Metric expression

builtin:kubernetes.node.conditions:filter(and(or(eq(node_condition,ContainerRuntimeProblem),eq(node_condition,ContainerRuntimeUnhealthy),eq(node_condition,CorruptDockerOverlay2),eq(node_condition,DiskPressure),eq(node_condition,FilesystemCorruptionProblem),eq(node_condition,FrequentContainerdRestart),eq(node_condition,FrequentDockerRestart),eq(node_condition,FrequentGcfsSnapshotterRestart),eq(node_condition,FrequentGcfsdRestart),eq(node_condition,FrequentKubeletRestart),eq(node_condition,FrequentUnregisterNetDevice),eq(node_condition,GcfsSnapshotterMissingLayer),eq(node_condition,GcfsSnapshotterUnhealthy),eq(node_condition,GcfsdUnhealthy),eq(node_condition,KernelDeadlock),eq(node_condition,KubeletProblem),eq(node_condition,KubeletUnhealthy),eq(node_condition,MemoryPressure),eq(node_condition,NetworkUnavailable),eq(node_condition,OutOfDisk),eq(node_condition,PIDPressure),eq(node_condition,ReadonlyFilesystem)),eq(condition_status,True))):splitBy(dt.kubernetes.node.system_uuid,k8s.node.name):sum

DQL

timeseries {sum(dt.kubernetes.node.conditions, rollup: avg)}, filter: {(((node_condition=="ContainerRuntimeProblem")OR(node_condition=="ContainerRuntimeUnhealthy")OR(node_condition=="CorruptDockerOverlay2")OR(node_condition=="DiskPressure")OR(node_condition=="FilesystemCorruptionProblem")OR(node_condition=="FrequentContainerdRestart")OR(node_condition=="FrequentDockerRestart")OR(node_condition=="FrequentGcfsSnapshotterRestart")OR(node_condition=="FrequentGcfsdRestart")OR(node_condition=="FrequentKubeletRestart")OR(node_condition=="FrequentUnregisterNetDevice")OR(node_condition=="GcfsSnapshotterMissingLayer")OR(node_condition=="GcfsSnapshotterUnhealthy")OR(node_condition=="GcfsdUnhealthy")OR(node_condition=="KernelDeadlock")OR(node_condition=="KubeletProblem")OR(node_condition=="KubeletUnhealthy")OR(node_condition=="MemoryPressure")OR(node_condition=="NetworkUnavailable")OR(node_condition=="OutOfDisk")OR(node_condition=="PIDPressure")OR(node_condition=="ReadonlyFilesystem"))AND(condition_status==true))}, by: {dt.kubernetes.node.system_uuid,k8s.node.name}

Namespace

AlertProblem typeEnabled (recommended)Detection threshold (recommended)SeverityDe-alerts afterCalculation

Detect namespace CPU-request quota saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of resource quota CPU used / Sum of resource quota CPU requests

Detect namespace CPU-limit quota saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of resource quota CPU used / Sum of resource quota CPU limits

Detect namespace memory-request quota saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of resource quota memory used / Sum of resource quota memory requests

Detect namespace memory-limit quota saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of resource quota memory used / Sum of resource quota memory limits

Detect namespace pod quota saturation

Resource

No

>90% / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of resource quota pods used / Sum of resource quota pods limit

Namespace metric and DQL expressions

Detect namespace CPU-limit quota saturation

TypeExpression

Metric expression

builtin:kubernetes.resourcequota.limits_cpu_used:splitBy(k8s.namespace.name):sum/builtin:kubernetes.resourcequota.limits_cpu:splitBy(k8s.namespace.name):sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.resourcequota.limits_cpu_used, rollup: avg), o2=sum(dt.kubernetes.resourcequota.limits_cpu, rollup: avg)}, by: {k8s.namespace.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect namespace CPU-request quota saturation

TypeExpression

Metric expression

builtin:kubernetes.resourcequota.requests_cpu_used:splitBy(k8s.namespace.name):sum/builtin:kubernetes.resourcequota.requests_cpu:splitBy(k8s.namespace.name):sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.resourcequota.requests_cpu_used, rollup: avg), o2=sum(dt.kubernetes.resourcequota.requests_cpu, rollup: avg)}, by: {k8s.namespace.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect namespace memory-limit quota saturation

TypeExpression

Metric expression

builtin:kubernetes.resourcequota.limits_memory_used:splitBy(k8s.namespace.name):sum/builtin:kubernetes.resourcequota.limits_memory:splitBy(k8s.namespace.name):sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.resourcequota.limits_memory_used, rollup: avg), o2=sum(dt.kubernetes.resourcequota.limits_memory, rollup: avg)}, by: {k8s.namespace.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect namespace memory-request quota saturation

TypeExpression

Metric expression

builtin:kubernetes.resourcequota.requests_memory_used:splitBy(k8s.namespace.name):sum/builtin:kubernetes.resourcequota.requests_memory:splitBy(k8s.namespace.name):sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.resourcequota.requests_memory_used, rollup: avg), o2=sum(dt.kubernetes.resourcequota.requests_memory, rollup: avg)}, by: {k8s.namespace.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect namespace pod quota saturation

TypeExpression

Metric expression

builtin:kubernetes.resourcequota.pods_used:splitBy(k8s.namespace.name):sum/builtin:kubernetes.resourcequota.pods:splitBy(k8s.namespace.name):sum*100.0

DQL

timeseries {o1=sum(dt.kubernetes.resourcequota.pods_used, rollup: avg), o2=sum(dt.kubernetes.resourcequota.pods, rollup: avg)}, by: {k8s.namespace.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Workload

AlertProblem typeEnabled (recommended)Detection threshold (recommended)SeverityDe-alerts afterCalculation

Detect container restarts

Error

Yes ¹

≥1 restart per 3 min; within 5 min

Warning signal ¹ ²

15 minutes

Container restarts metric

Detect stuck deployments

Error

Yes ¹

Stops progressing ≥3 min; within 5 min

Health alert ¹

10 minutes

Workload condition metric filtered by 'not progressing'

Detect pods stuck in pending

Resource

Yes ¹

≥1 pod stuck ≥10 min; within 15 min

Warning signal ¹ ²

10 minutes

Pods metric filtered by phase 'Pending'

Detect pods stuck in terminating

Resource

Yes ¹

Termination stopped ≥10 min; within 15 min

Warning signal ¹ ²

10 minutes

Pods metric filtered by status 'Terminating'

Detect workloads without ready pods

Error

Yes ¹

No ready pods ≥10 min; within 15 min

Health alert ¹

10 minutes

Sum of non-failed pods − Sum of non-failed, non-ready pods

Detect workloads with non-ready pods

Error

No

>30% non-ready; within 60 min

Warning signal ²

10 minutes

Sum of non-failed pods − Sum of non-failed, ready pods

Detect memory usage saturation

Resource

No

>90% / ≥10 min; within 15 min

Warning signal ²

10 minutes

Sum of workload working set memory / Sum of workload memory limits

Detect CPU usage saturation

Resource

No

>90% / ≥10 min; within 15 min

Warning signal ²

10 minutes

Sum of workload working set memory / Sum of workload memory limits

Detect high CPU throttling

Resource

No

>100% throttled / ≥20 min; within 30 min

Warning signal ²

10 minutes

Sum of workload CPU throttled / Sum of workload CPU limits

Detect out-of-memory kills

Error

Yes ¹

Event-based

Warning signal ¹ ²

15 minutes

Out-of-memory kills metric

Detect job failure events

Error

Yes ¹

Event-based

Warning signal ¹ ²

60 minutes

Event metric filtered by reason and workload kind

Detect pod backoff events

Error

Yes ¹

Event-based

Health alert ¹

15 minutes

Event metric filtered by reason

Detect pod eviction events

Error

Yes ¹

Event-based

Health alert ¹

60 minutes

Event metric filtered by reason

Detect pod preemption events

Error

Yes ¹

Event-based

Warning signal ¹ ²

60 minutes

Event metric filtered by reason

  • Detect job failure events triggers on events with reason BackoffLimitExceeded, DeadlineExceeded, or PodFailurePolicy, for Jobs and CronJobs only.
  • Detect pod backoff events triggers on events with reason BackOff — check for pods with status ImagePullBackOff or CrashLoopBackOff.
  • Detect pod eviction events triggers on events with reason Evicted.
  • Detect pod preemption events triggers on events with reason Preempted or Preempting.
  • Detect workloads with non-ready pods and Detect workloads without ready pods exclude pods belonging to Jobs and CronJobs.
Workload metric and DQL expressions

Detect CPU usage saturation

TypeExpression

Metric expression

(builtin:kubernetes.workload.cpu_usage:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum/builtin:kubernetes.workload.limits_cpu:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum):default(0.0)*100.0

DQL

timeseries {o1=sum(dt.kubernetes.container.cpu_usage, rollup: avg), o2=sum(dt.kubernetes.container.limits_cpu, rollup: avg)}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}| fieldsAdd result=if(isNull(o1[]/o2[]), 0.0, else: o1[]/o2[])* 100.0| fieldsRemove {o1,o2}

Detect container restarts

TypeExpression

Metric expression

builtin:kubernetes.container.restarts:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries {sum(dt.kubernetes.container.restarts, default:0.0, rollup: avg)}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect high CPU throttling

TypeExpression

Metric expression

(builtin:kubernetes.workload.cpu_throttled:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum/builtin:kubernetes.workload.limits_cpu:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum):default(0.0)*100.0

DQL

timeseries {o1=sum(dt.kubernetes.container.cpu_throttled, rollup: avg), o2=sum(dt.kubernetes.container.limits_cpu, rollup: avg)}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}| fieldsAdd result=if(isNull(o1[]/o2[]), 0.0, else: o1[]/o2[])* 100.0| fieldsRemove {o1,o2}

Detect job failure events

TypeExpression

Metric expression

builtin:kubernetes.events:filter(and(or(eq(k8s.event.reason,BackoffLimitExceeded),eq(k8s.event.reason,DeadlineExceeded),eq(k8s.event.reason,PodFailurePolicy)),or(eq(k8s.workload.kind,job),eq(k8s.workload.kind,cronjob)))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries {sum(dt.kubernetes.events, default:0.0, rollup: avg)}, filter: {(((k8s.event.reason=="BackoffLimitExceeded")OR(k8s.event.reason=="DeadlineExceeded")OR(k8s.event.reason=="PodFailurePolicy"))AND((k8s.workload.kind=="job")OR(k8s.workload.kind=="cronjob")))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect memory usage saturation

TypeExpression

Metric expression

(builtin:kubernetes.workload.memory_working_set:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum/builtin:kubernetes.workload.limits_memory:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum):default(0.0)*100.0

DQL

timeseries {o1=sum(dt.kubernetes.container.memory_working_set, rollup: avg), o2=sum(dt.kubernetes.container.limits_memory, rollup: avg)}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}| fieldsAdd result=if(isNull(o1[]/o2[]), 0.0, else: o1[]/o2[])* 100.0| fieldsRemove {o1,o2}

Detect out-of-memory kills

TypeExpression

Metric expression

builtin:kubernetes.container.oom_kills:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries {sum(dt.kubernetes.container.oom_kills, default:0.0, rollup: avg)}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect pod backoff events

TypeExpression

Metric expression

builtin:kubernetes.events:filter(and(eq(k8s.event.reason,BackOff))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries {sum(dt.kubernetes.events, default:0.0, rollup: avg)}, filter: {((k8s.event.reason=="BackOff"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect pod eviction events

TypeExpression

Metric expression

builtin:kubernetes.events:filter(and(eq(k8s.event.reason,Evicted))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries {sum(dt.kubernetes.events, default:0.0, rollup: avg)}, filter: {((k8s.event.reason=="Evicted"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect pod preemption events

TypeExpression

Metric expression

builtin:kubernetes.events:filter(or(eq(k8s.event.reason,Preempted),eq(k8s.event.reason,Preempting))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries {sum(dt.kubernetes.events, default:0.0, rollup: avg)}, filter: {((k8s.event.reason=="Preempted")OR(k8s.event.reason=="Preempting"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect pods stuck in pending

TypeExpression

Metric expression

builtin:kubernetes.pods:filter(and(eq(pod_phase,Pending))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum

DQL

timeseries {sum(dt.kubernetes.pods, rollup: avg)}, filter: {((pod_phase=="Pending"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect pods stuck in terminating

TypeExpression

Metric expression

builtin:kubernetes.pods:filter(and(eq(pod_status,Terminating))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum

DQL

timeseries {sum(dt.kubernetes.pods, rollup: avg)}, filter: {((pod_status=="Terminating"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect stuck deployments

TypeExpression

Metric expression

builtin:kubernetes.workload.conditions:filter(and(eq(workload_condition,Progressing),eq(condition_status,False))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum

DQL

timeseries {sum(dt.kubernetes.workload.conditions, rollup: avg)}, filter: {((workload_condition=="Progressing")AND(condition_status==false))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}

Detect workloads with non-ready pods

TypeExpression

Metric expression

builtin:kubernetes.pods:filter(and(ne(pod_phase,Failed),ne(pod_phase,Succeeded),ne(k8s.workload.kind,job),ne(k8s.workload.kind,cronjob),ne(pod_status,Terminating))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum-builtin:kubernetes.pods:filter(and(ne(pod_phase,Failed),ne(pod_phase,Succeeded),ne(k8s.workload.kind,job),ne(k8s.workload.kind,cronjob),eq(pod_condition,Ready),ne(pod_status,Terminating))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries o1=sum(dt.kubernetes.pods, rollup: avg), filter: {((pod_phase!="Failed")AND(pod_phase!="Succeeded")AND(k8s.workload.kind!="job")AND(k8s.workload.kind!="cronjob")AND(pod_status!="Terminating"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}| join [timeseries operand=sum(dt.kubernetes.pods, default:0.0, rollup: avg), nonempty:true, filter: {((pod_phase!="Failed")AND(pod_phase!="Succeeded")AND(k8s.workload.kind!="job")AND(k8s.workload.kind!="cronjob")AND(pod_condition=="Ready")AND(pod_status!="Terminating"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}], on: {interval}, fields: {o2=operand}| fieldsAdd result=o1[]-o2[]| fieldsRemove {o1,o2}

Detect workloads without ready pods

TypeExpression

Metric expression

builtin:kubernetes.pods:filter(and(ne(pod_phase,Failed),ne(pod_phase,Succeeded),ne(k8s.workload.kind,job),ne(k8s.workload.kind,cronjob))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum-builtin:kubernetes.pods:filter(and(ne(pod_phase,Failed),ne(pod_phase,Succeeded),ne(k8s.workload.kind,job),ne(k8s.workload.kind,cronjob),ne(pod_condition,Ready))):splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

DQL

timeseries o1=sum(dt.kubernetes.pods, rollup: avg), filter: {((pod_phase!="Failed")AND(pod_phase!="Succeeded")AND(k8s.workload.kind!="job")AND(k8s.workload.kind!="cronjob"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}| join [timeseries operand=sum(dt.kubernetes.pods, default:0.0, rollup: avg), nonempty:true, filter: {((pod_phase!="Failed")AND(pod_phase!="Succeeded")AND(k8s.workload.kind!="job")AND(k8s.workload.kind!="cronjob")AND(pod_condition!="Ready"))}, by: {k8s.namespace.name,k8s.workload.kind,k8s.workload.name}], on: {interval}, fields: {o2=operand}| fieldsAdd result=o1[]-o2[]| fieldsRemove {o1,o2}

Persistent volume claims

AlertProblem typeEnabled (recommended)Detection threshold (recommended)SeverityDe-alerts afterCalculation

Detect low disk space (MiB)

Resource

No

≤100 MiB remaining / ≥3 min; within 5 min

Warning signal ²

10 minutes

Kubelet volume stats available bytes metric

Detect low disk space (%)

Resource

Yes ¹

<3% available / ≥3 min; within 5 min

Warning signal ¹ ²

10 minutes

Volume stats available bytes / Volume stats capacity bytes

Persistent volume claims metric and DQL expressions

Detect low disk space (%)

TypeExpression

Metric expression

builtin:kubernetes.persistentvolumeclaim.available:splitBy(k8s.namespace.name,k8s.persistent_volume_claim.name):avg/builtin:kubernetes.persistentvolumeclaim.capacity:splitBy(k8s.namespace.name,k8s.persistent_volume_claim.name):avg*100.0

DQL

timeseries {o1=avg(dt.kubernetes.persistentvolumeclaim.available), o2=avg(dt.kubernetes.persistentvolumeclaim.capacity)}, by: {k8s.namespace.name,k8s.persistent_volume_claim.name}| fieldsAdd result=o1[]/o2[]* 100.0| fieldsRemove {o1,o2}

Detect low disk space (MiB)

TypeExpression

Metric expression

builtin:kubernetes.persistentvolumeclaim.available:splitBy(k8s.namespace.name,k8s.persistent_volume_claim.name):avg

DQL

timeseries {avg(dt.kubernetes.persistentvolumeclaim.available)}, by: {k8s.namespace.name,k8s.persistent_volume_claim.name}

Related topics

  • Alert on common Kubernetes misconfigurations and detect anomalies with Kubernetes metrics
  • Health alerts and warning signals
  • Alert on common Kubernetes/OpenShift issues — the equivalent alert catalog for Kubernetes Classic
Related tags
Infrastructure ObservabilityKubernetes (new)Kubernetes