Try it free

Upgrade from Classic Kubernetes metrics to Grail

  • Latest Dynatrace
  • Upgrade guide

This guide provides insights into migrating Kubernetes metrics to Grail, so you can smoothly transition to Latest Dynatrace.

Why upgrade?

  • Full power of Grail: Upgrading your queries lets you use Kubernetes metrics on Grail in Notebooks Notebooks, Dashboards Dashboards, and other Dynatrace apps.
  • Advanced query capabilities: Metrics on Grail use DQL, which lets you filter, aggregate, join, and transform data in a single query.

What will you do?

  • Identify how your Classic Kubernetes metric maps to Grail metrics.
  • Look up your Classic metric key in the tables below and find its Grail equivalent.
  • Convert your metric selector queries to DQL queries using the mapping tables and examples.

Before you begin

Prerequisites

  • Access to Notebooks Notebooks or Dashboards Dashboards to test and validate your converted DQL queries.

Prior knowledge

  • Familiarity with Metrics Classic and metric selector syntax.
  • Basic understanding of DQL concepts.
  • Familiarity with creating DQL queries in Notebooks Notebooks or Dashboards Dashboards.
  • Understanding of the metric mapping types used in this guide: identical, convergent, replacement, and divergent/calculated.

How to upgrade

To convert a Classic Kubernetes metric key:

  1. Find your Classic metric key in the metric key reference below.
  2. Note its Grail equivalent.
  3. Write a DQL query that uses the relevant Grail metric key.

Metric key reference

The following tables list Classic Kubernetes metric keys and their Grail equivalents, organized by mapping type. Use the section that matches how your Classic metric maps to Grail.

For more information about metric mapping, see New concepts.

Identical metrics

This section identifies Classic metrics that have a direct Grail equivalent. These identical metrics have the same level of detail and dimensions available. The only difference is the metric key.

Classic metric keyGrail metric key
builtin:kubernetes.cluster.readyzdt.kubernetes.cluster.readyz
builtin:kubernetes.container.oom_killsdt.kubernetes.container.oom_kills
builtin:kubernetes.container.restartsdt.kubernetes.container.restarts
builtin:kubernetes.node.conditionsdt.kubernetes.node.conditions
builtin:kubernetes.node.cpu_allocatabledt.kubernetes.node.cpu_allocatable
builtin:kubernetes.node.memory_allocatabledt.kubernetes.node.memory_allocatable
builtin:kubernetes.node.pods_allocatabledt.kubernetes.node.pods_allocatable
builtin:kubernetes.nodesdt.kubernetes.nodes
builtin:kubernetes.persistentvolumeclaim.availabledt.kubernetes.persistentvolumeclaim.available
builtin:kubernetes.persistentvolumeclaim.capacitydt.kubernetes.persistentvolumeclaim.capacity
builtin:kubernetes.persistentvolumeclaim.useddt.kubernetes.persistentvolumeclaim.used
builtin:kubernetes.resourcequota.limits_cpudt.kubernetes.resourcequota.limits_cpu
builtin:kubernetes.resourcequota.limits_cpu_useddt.kubernetes.resourcequota.limits_cpu_used
builtin:kubernetes.resourcequota.limits_memorydt.kubernetes.resourcequota.limits_memory
builtin:kubernetes.resourcequota.limits_memory_useddt.kubernetes.resourcequota.limits_memory_used
builtin:kubernetes.resourcequota.podsdt.kubernetes.resourcequota.pods
builtin:kubernetes.resourcequota.pods_useddt.kubernetes.resourcequota.pods_used
builtin:kubernetes.resourcequota.requests_cpudt.kubernetes.resourcequota.requests_cpu
builtin:kubernetes.resourcequota.requests_cpu_useddt.kubernetes.resourcequota.requests_cpu_used
builtin:kubernetes.resourcequota.requests_memorydt.kubernetes.resourcequota.requests_memory
builtin:kubernetes.resourcequota.requests_memory_useddt.kubernetes.resourcequota.requests_memory_used
builtin:kubernetes.workload.conditionsdt.kubernetes.workload.conditions
builtin:kubernetes.workload.pods_desireddt.kubernetes.workload.pods_desired
builtin:kubernetes.workloadsdt.kubernetes.workloads

Convergent metrics

The Grail metrics that supersede the Classic metrics often offer an increased level of detail compared to the Classic metrics.

  • Many Classic metrics, such as metrics related to resource consumption, are available in Classic as separate workload- and node- level metrics. With Grail these have been consolidated to a single metric at the container level.
  • Some metrics, such as builtin:kubernetes.containers and builtin:kubernetes.events, have been extended with additional dimensions.

If a Grail metric now provides more detailed dimensions, such as at the container level, the data can be aggregated back to a higher level (such as workload levels). This aggregation can be useful if you want to re-use existing filters.

Classic metric keysGrail metric key
builtin:kubernetes.containersdt.kubernetes.containers
builtin:kubernetes.workload.containers_desireddt.kubernetes.pod.containers_desired
builtin:kubernetes.eventsdt.kubernetes.events
builtin:kubernetes.node.pods
builtin:kubernetes.pods
dt.kubernetes.pods
builtin:kubernetes.node.cpu_usage
builtin:kubernetes.workload.cpu_usage
dt.kubernetes.container.cpu_usage
builtin:kubernetes.node.cpu_throttled
builtin:kubernetes.workload.cpu_throttled
dt.kubernetes.container.cpu_throttled
builtin:kubernetes.node.requests_cpu
builtin:kubernetes.workload.requests_cpu
dt.kubernetes.container.requests_cpu
builtin:kubernetes.node.limits_cpu
builtin:kubernetes.workload.limits_cpu
dt.kubernetes.container.limits_cpu
builtin:kubernetes.node.memory_working_set
builtin:kubernetes.workload.memory_working_set
dt.kubernetes.container.memory_working_set
builtin:kubernetes.node.requests_memory
builtin:kubernetes.workload.requests_memory
dt.kubernetes.container.requests_memory
builtin:kubernetes.node.limits_memory
builtin:kubernetes.workload.limits_memory
dt.kubernetes.container.limits_memory
Example: Aggregate container-level metrics to workload level

The following DQL query returns the amount of memory consumed at the workload level, based on aggregated container-level data.

timeseries memory_working_set = sum(dt.kubernetes.container.memory_working_set)
by: {
k8s.cluster.name,
k8s.namespace.name,
k8s.workload.name
}

Replaced metrics

The following Classic metric keys are replaced by a similar, but not identical, Grail metric. These metric keys are closely related and any deviations in the output are minor.

Classic metric keyGrail metric key
builtin:containers.cpu.limitdt.kubernetes.container.limits_cpu
builtin:kubernetes.container.outOfMemoryKills
builtin:kubernetes.container.oom_kills
dt.kubernetes.container.oom_kills

Calculated metrics

Dynatrace Classic metrics used millicores as the unit of measure, while the equivalent Grail metrics use nanoseconds per minute. Therefore, to get the same output value for these metrics, you'll need to use DQL to calculate the difference between these units. The sections below list the specific metrics and provide example DQL calculations.

List of metrics requiring calculations

Some Classic metric keys have a direct Grail equivalent, while for others you need to compute the output from multiple Grail metrics.

Classic metric keyGrail metric keys
builtin:containers.cpu.throttledMilliCoresdt.containers.cpu.throttled_time
builtin:containers.cpu.usageUserMilliCoresdt.containers.cpu.usage_user_time
builtin:containers.cpu.usageSystemMilliCoresdt.containers.cpu.usage_system_time
builtin:containers.cpu.usageMilliCoresdt.containers.cpu.usage_user_time
dt.containers.cpu.usage_system_time
builtin:containers.cpu.usagePercentdt.containers.cpu.usage_user_time
dt.containers.cpu.usage_system_time
dt.containers.cpu.logical_cores
dt.containers.cpu.throttling_ratio
builtin:containers.cpu.usageTimedt.containers.cpu.usage_user_time
dt.containers.cpu.usage_system_time
builtin:containers.memory.limitPercentdt.containers.memory.limit_bytes
dt.containers.memory.physical_total_bytes
builtin:containers.memory.usagePercentdt.containers.memory.limit_bytes
dt.containers.memory.physical_total_bytes
dt.containers.memory.resident_set_bytes

Calculation examples

For each Classic metric key, this section provides the equivalent DQL queries, including calculations to convert to nanoseconds per minute.

builtin:containers.cpu.throttledMilliCores
timeseries {
throttled_time = avg(dt.containers.cpu.throttled_time, rollup: sum, rate: 1m)
}
| fieldsAdd
ns_per_min = 60 * 1000 * 1000 * 1000
, milli_core_per_core = 1000
| fieldsAdd
throttled_milli_cores = throttled_time[] * milli_core_per_core / ns_per_min
| summarize {
throttled_milli_cores = sum(throttled_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usageUserMilliCores
timeseries {
usage_user_time = avg(dt.containers.cpu.usage_user_time)
}
| fieldsAdd
ns_per_min = 60 * 1000 * 1000 * 1000
, milli_core_per_core = 1000
| fieldsAdd
usage_user_milli_cores = usage_user_time[] * milli_core_per_core / ns_per_min
| summarize {
usage_user_milli_cores = sum(usage_user_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usageSystemMilliCores
timeseries {
usage_system_time = avg(dt.containers.cpu.usage_system_time)
}
| fieldsAdd
ns_per_min = 60 * 1000 * 1000 * 1000
, milli_core_per_core = 1000
| fieldsAdd
usage_system_milli_cores = usage_system_time[] * milli_core_per_core / ns_per_min
| summarize {
usage_system_milli_cores = sum(usage_system_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usageMilliCores
timeseries {
usage_user_time = avg(dt.containers.cpu.usage_user_time)
, usage_system_time = avg(dt.containers.cpu.usage_system_time)
}
| fieldsAdd
ns_per_min = 60 * 1000 * 1000 * 1000
, milli_core_per_core = 1000
| fieldsAdd
usage_milli_cores = (usage_user_time[] + usage_system_time[] )
* milli_core_per_core / ns_per_min
| summarize {
usage_milli_cores = sum(usage_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usagePercent
timeseries {
// for total usage, user and system cpu usage are added
userCpuUsage = avg(dt.containers.cpu.usage_user_time)
, systemCpuUsage = avg(dt.containers.cpu.usage_system_time)
// cpu logical counts are the fallback, if the throttling ratio doesn't exist
, cpuLogicalCount = avg(dt.containers.cpu.logical_cores)
}
// filter statement ...
// leftOuter join allows the throttling ratio to be null
| join [
timeseries {
throttlingRatio = avg(dt.containers.cpu.throttling_ratio)
// same filter statement as above ...
}
], on: { interval, timeframe}, fields: { throttlingRatio}, kind:leftOuter
| fieldsAdd
// sum of system and user cpu usage
numerator = userCpuUsage[] + systemCpuUsage[]
// throttling ratio, or as a fallback cpu logical count.
, denominator = coalesce(throttlingRatio, cpuLogicalCount)
, nanoseconds_per_minute = 60 * 1000 * 1000 * 1000
| fields
interval, timeframe
, cpuUsagePercent = 100.0 * numerator[] / ( denominator[] * nanoseconds_per_minute)
builtin:containers.cpu.usageTime
timeseries {
usageUserTime = avg(dt.containers.cpu.usage_user_time)
, usageSystemTime = avg(dt.containers.cpu.usage_system_time)
}
, by: { dt.entity.container_group_instance},
| fields
interval, timeframe
, usageTime = usageSystemTime[] + usageUserTime[]
builtin:containers.memory.limitPercent
timeseries {
limit_bytes = avg(dt.containers.memory.limit_bytes),
physical_total_bytes = avg(dt.containers.memory.physical_total_bytes)
}
| fieldsAdd
limit_percent = (limit_bytes[] / physical_total_bytes[] ) * 100
| summarize {
limit_percent = sum(limit_percent[] )
}, by: { timeframe, interval }
builtin:containers.memory.usagePercent
timeseries {
memoryLimits = avg(dt.containers.memory.limit_bytes)
, totalPhysicalMemory = avg(dt.containers.memory.physical_total_bytes)
, residentSetBytes = avg(dt.containers.memory.resident_set_bytes)
}
, by: { dt.entity.container_group_instance}
| fieldsAdd
denominator = if (
arrayFirst(memoryLimits) > 0,
then: memoryLimits,
else: totalPhysicalMemory
)
| fields
dt.entity.container_group_instance
, interval, timeframe
, memoryUsagePercent = 100 * residentSetBytes[] / denominator[]

Related topics

  • Use DQL queries
  • Notebooks
Related tags
Dynatrace Platform