Kubernetes metrics migration guide

Latest Dynatrace
5-min read

This guide provides insights into migrating Kubernetes metrics to Grail. Typically, a Grail metric is equivalent to a Metrics Classic metric. In some cases, however, there's no one-to-one relation:

Convergent: a single Grail metric represents multiple Metrics Classic metrics of a similar scope or an increased level of detail.
Replacement: a different Grail metric represents the Metrics Classic metric.
Divergent/Calculated: the Classic metric is not represented 1:1 as Grail metric but can be calculated from other Grail metrics

Identity

Classic Metrics and Grail Metrics have the same level of detail and dimensions available. The only difference is the metric key.

Metric key (Grail)

Metric key (Classic)

dt.kubernetes.cluster.readyz

builtin:kubernetes.cluster.readyz

dt.kubernetes.container.oom_kills

builtin:kubernetes.container.oom_kills

dt.kubernetes.container.restarts

builtin:kubernetes.container.restarts

dt.kubernetes.node.conditions

builtin:kubernetes.node.conditions

dt.kubernetes.node.cpu_allocatable

builtin:kubernetes.node.cpu_allocatable

dt.kubernetes.node.memory_allocatable

builtin:kubernetes.node.memory_allocatable

dt.kubernetes.node.pods_allocatable

builtin:kubernetes.node.pods_allocatable

dt.kubernetes.nodes

builtin:kubernetes.nodes

dt.kubernetes.persistentvolumeclaim.available

builtin:kubernetes.persistentvolumeclaim.available

dt.kubernetes.persistentvolumeclaim.capacity

builtin:kubernetes.persistentvolumeclaim.capacity

dt.kubernetes.persistentvolumeclaim.used

builtin:kubernetes.persistentvolumeclaim.used

dt.kubernetes.resourcequota.limits_cpu

builtin:kubernetes.resourcequota.limits_cpu

dt.kubernetes.resourcequota.limits_cpu_used

builtin:kubernetes.resourcequota.limits_cpu_used

dt.kubernetes.resourcequota.limits_memory

builtin:kubernetes.resourcequota.limits_memory

dt.kubernetes.resourcequota.limits_memory_used

builtin:kubernetes.resourcequota.limits_memory_used

dt.kubernetes.resourcequota.pods

builtin:kubernetes.resourcequota.pods

dt.kubernetes.resourcequota.pods_used

builtin:kubernetes.resourcequota.pods_used

dt.kubernetes.resourcequota.requests_cpu

builtin:kubernetes.resourcequota.requests_cpu

dt.kubernetes.resourcequota.requests_cpu_used

builtin:kubernetes.resourcequota.requests_cpu_used

dt.kubernetes.resourcequota.requests_memory

builtin:kubernetes.resourcequota.requests_memory

dt.kubernetes.resourcequota.requests_memory_used

builtin:kubernetes.resourcequota.requests_memory_used

dt.kubernetes.workload.conditions

builtin:kubernetes.workload.conditions

dt.kubernetes.workload.pods_desired

builtin:kubernetes.workload.pods_desired

dt.kubernetes.workloads

builtin:kubernetes.workloads

Convergent metrics

The following metrics have been consolidated. The Grail metrics that supersede the Classic metrics offer an increased level of detail compared to the Classic metrics.

To achieve this decreased level of detail, the Grail metrics are first aggregated to the granularity of the Classic metric. From there the same set of filters can be applied and the output between Classic metrics and Grail metrics is identical.

The following list of metrics contains the pod and container count metrics and the Kubernetes event count metric that was available at a lower level of detail as Classic metric.

Kubernetes events and container/pod count metrics

Metric key (Grail)

Metric key (Classic)

dt.kubernetes.containers

builtin:kubernetes.containers

dt.kubernetes.pod.containers_desired

builtin:kubernetes.workload.containers_desired

dt.kubernetes.events

builtin:kubernetes.events

dt.kubernetes.pods

builtin:kubernetes.node.pods
builtin:kubernetes.pods

The following table contains the workload and node resource metrics that have been available as separate workload- and node- level Classic metrics. With Grail, there is a single metric at the container level.

Example: The following DQL query returns the amount of memory consumed on the workload level based on aggregated container-level data.

  timeseries memory_working_set = sum(dt.kubernetes.container.memory_working_set)
  by: {
      k8s.cluster.name,
      k8s.namespace.name,
      k8s.workload.name
}

Workload- and node- level resource consumption metrics

Metric key (Grail)

Metric key (Classic)

dt.kubernetes.container.cpu_usage

builtin:kubernetes.node.cpu_usage
builtin:kubernetes.workload.cpu_usage

dt.kubernetes.container.cpu_throttled

builtin:kubernetes.node.cpu_throttled
builtin:kubernetes.workload.cpu_throttled

dt.kubernetes.container.requests_cpu

builtin:kubernetes.node.requests_cpu
builtin:kubernetes.workload.requests_cpu

dt.kubernetes.container.limits_cpu

builtin:kubernetes.node.limits_cpu
builtin:kubernetes.workload.limits_cpu

dt.kubernetes.container.memory_working_set

builtin:kubernetes.node.memory_working_set
builtin:kubernetes.workload.memory_working_set

dt.kubernetes.container.requests_memory

builtin:kubernetes.node.requests_memory
builtin:kubernetes.workload.requests_memory

dt.kubernetes.container.limits_memory

builtin:kubernetes.node.limits_memory
builtin:kubernetes.workload.limits_memory

Replaced metrics

This group of metrics consists of Classic metric keys that have never been made available as Grail metrics. Instead the most similar Classic metric is used to then determine the Grail metric replacement for these deprecated metrics. The reason for the deprecation is a cleanup of duplicate metric keys. In the case of the following metrics, a complete identity of the values between the Classic Metric and Grail Metric is not feasible, but they are closely related and do not deviate very much.

Metric key (Grail)

Metric key (Classic)

Superseding Classic Metric

dt.kubernetes.container.limits_cpu

builtin:containers.cpu.limit

n.a.

dt.kubernetes.container.oom_kills

builtin:kubernetes.container.outOfMemoryKills

builtin:kubernetes.container.oom_kills

Calculated metrics

The following set of Classic container metrics is superseded by Grail container metrics. For most of the CPU metrics in this section the Classic metrics have the unit millicores, while the Grail metrics have the unit nanoseconds/minute. To get to the same values, the Grail metric needs to be divided by the number of nanoseconds in a minute. (The number of nanoseconds per minute is 60 * 1000 * 1000 * 1000)

This is the case for the following Grail metrics.

builtin:containers.cpu.throttledMilliCores

timeseries {
      throttled_time = avg(dt.containers.cpu.throttled_time, rollup: sum, rate: 1m)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      throttled_milli_cores = throttled_time[] * milli_core_per_core / ns_per_min
| summarize {
      throttled_milli_cores = sum(throttled_milli_cores[] )
}, by: { timeframe, interval }

builtin:containers.cpu.usageUserMilliCores

timeseries { usage_user_time = avg(dt.containers.cpu.usage_user_time)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      usage_user_milli_cores = usage_user_time[] * milli_core_per_core / ns_per_min
| summarize { usage_user_milli_cores = sum(usage_user_milli_cores[] )
}, by: { timeframe, interval }

builtin:containers.cpu.usageSystemMilliCores

timeseries {
      usage_system_time = avg(dt.containers.cpu.usage_system_time)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      usage_system_milli_cores = usage_system_time[] * milli_core_per_core / ns_per_min
| summarize {
      usage_system_milli_cores = sum(usage_system_milli_cores[] )
}, by: { timeframe, interval }

builtin:containers.cpu.usageMilliCores

timeseries {
      usage_user_time = avg(dt.containers.cpu.usage_user_time)
    , usage_system_time = avg(dt.containers.cpu.usage_system_time)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      usage_milli_cores = (usage_user_time[] + usage_system_time[] )
    * milli_core_per_core / ns_per_min
| summarize {
      usage_milli_cores = sum(usage_milli_cores[] )
}, by: { timeframe, interval }

builtin:containers.cpu.usagePercent

timeseries {
      // for total usage, user and system cpu usage are added
      userCpuUsage = avg(dt.containers.cpu.usage_user_time)
    , systemCpuUsage = avg(dt.containers.cpu.usage_system_time)
      // cpu logical counts are the fallback, if the throttling ratio doesn't exist
    , cpuLogicalCount = avg(dt.containers.cpu.logical_cores)
}
// filter statement ...
// leftOuter join allows the throttling ratio to be null
| join [
  timeseries {
      throttlingRatio = avg(dt.containers.cpu.throttling_ratio)
      // same filter statement as above ...
}
], on: { interval, timeframe}, fields: { throttlingRatio}, kind:leftOuter
| fieldsAdd
       // sum of system and user cpu usage
       numerator = userCpuUsage[] + systemCpuUsage[]
       // throttling ratio, or as a fallback cpu logical count.
     , denominator = coalesce(throttlingRatio, cpuLogicalCount)
     , nanoseconds_per_minute  = 60 * 1000 * 1000 * 1000
 | fields
       interval, timeframe
     , cpuUsagePercent = 100.0 * numerator[] / ( denominator[] * nanoseconds_per_minute)

builtin:containers.cpu.usageTime

timeseries {
      usageUserTime = avg(dt.containers.cpu.usage_user_time)
    , usageSystemTime = avg(dt.containers.cpu.usage_system_time)
    }
    , by: { dt.entity.container_group_instance},
| fields
      interval, timeframe
    , usageTime = usageSystemTime[] + usageUserTime[]

builtin:containers.memory.limitPercent

timeseries {
      limit_bytes = avg(dt.containers.memory.limit_bytes),
      physical_total_bytes = avg(dt.containers.memory.physical_total_bytes)
}
| fieldsAdd
      limit_percent = (limit_bytes[] / physical_total_bytes[] ) * 100
| summarize {
      limit_percent = sum(limit_percent[] )
}, by: { timeframe, interval }

builtin:containers.memory.usagePercent

timeseries {
      memoryLimits = avg(dt.containers.memory.limit_bytes)
    , totalPhysicalMemory = avg(dt.containers.memory.physical_total_bytes)
    , residentSetBytes = avg(dt.containers.memory.resident_set_bytes)
    }
    , by: { dt.entity.container_group_instance}
| fieldsAdd
      denominator = if (
          arrayFirst(memoryLimits) > 0,
          then: memoryLimits,
          else: totalPhysicalMemory
          )
| fields
     dt.entity.container_group_instance
   , interval, timeframe
   , memoryUsagePercent = 100 * residentSetBytes[] / denominator[]

Kubernetes metrics migration guide

Identity

Convergent metrics

Replaced metrics

Calculated metrics

Related topics