Try it free

Upgrade from Classic Kubernetes metrics to Grail

  • Dynatrace Classic
  • Upgrade guide
  • 5-min read

This guide provides insights into migrating Kubernetes metrics to Grail, so you can smoothly transition to Latest Dynatrace.

Why upgrade?

  • Full power of Grail: Upgrading your queries lets you use Kubernetes metrics on Grail in Notebooks Notebooks, Dashboards Dashboards, and other Dynatrace apps.
  • Advanced query capabilities: Metrics on Grail use DQL, which lets you filter, aggregate, join, and transform data in a single query.

What will you do?

  • Identify how your Classic Kubernetes metric maps to Grail metrics.
  • Look up your Classic metric key in the tables below and find its Grail equivalent.
  • Convert your metric selector queries to DQL queries using the mapping tables and examples.

Before you begin

Prerequisites

  • Access to Notebooks Notebooks or Dashboards Dashboards to test and validate your converted DQL queries.

Prior knowledge

  • Familiarity with Metrics Classic and metric selector syntax.
  • Basic understanding of DQL concepts.
  • Familiarity with creating DQL queries in Notebooks Notebooks or Dashboards Dashboards.
  • Understanding of the metric mapping types used in this guide: identical, convergent, replacement, and divergent/calculated.

How to upgrade

To convert a Classic Kubernetes metric key:

  1. Find your Classic metric key in the metric key reference below.
  2. Note its Grail equivalent.
  3. Write a DQL query that uses the relevant Grail metric key.

Metric key reference

The following tables list Classic Kubernetes metric keys and their Grail equivalents, organized by mapping type. Use the section that matches how your Classic metric maps to Grail.

For more information about metric mapping, see New concepts.

Identical metrics

This section identifies Classic metrics that have a direct Grail equivalent. These identical metrics have the same level of detail and dimensions available. The only difference is the metric key.

Classic metric keyGrail metric key

builtin:kubernetes.cluster.readyz

dt.kubernetes.cluster.readyz

builtin:kubernetes.container.oom_kills

dt.kubernetes.container.oom_kills

builtin:kubernetes.container.restarts

dt.kubernetes.container.restarts

builtin:kubernetes.node.conditions

dt.kubernetes.node.conditions

builtin:kubernetes.node.cpu_allocatable

dt.kubernetes.node.cpu_allocatable

builtin:kubernetes.node.memory_allocatable

dt.kubernetes.node.memory_allocatable

builtin:kubernetes.node.pods_allocatable

dt.kubernetes.node.pods_allocatable

builtin:kubernetes.nodes

dt.kubernetes.nodes

builtin:kubernetes.persistentvolumeclaim.available

dt.kubernetes.persistentvolumeclaim.available

builtin:kubernetes.persistentvolumeclaim.capacity

dt.kubernetes.persistentvolumeclaim.capacity

builtin:kubernetes.persistentvolumeclaim.used

dt.kubernetes.persistentvolumeclaim.used

builtin:kubernetes.resourcequota.limits_cpu

dt.kubernetes.resourcequota.limits_cpu

builtin:kubernetes.resourcequota.limits_cpu_used

dt.kubernetes.resourcequota.limits_cpu_used

builtin:kubernetes.resourcequota.limits_memory

dt.kubernetes.resourcequota.limits_memory

builtin:kubernetes.resourcequota.limits_memory_used

dt.kubernetes.resourcequota.limits_memory_used

builtin:kubernetes.resourcequota.pods

dt.kubernetes.resourcequota.pods

builtin:kubernetes.resourcequota.pods_used

dt.kubernetes.resourcequota.pods_used

builtin:kubernetes.resourcequota.requests_cpu

dt.kubernetes.resourcequota.requests_cpu

builtin:kubernetes.resourcequota.requests_cpu_used

dt.kubernetes.resourcequota.requests_cpu_used

builtin:kubernetes.resourcequota.requests_memory

dt.kubernetes.resourcequota.requests_memory

builtin:kubernetes.resourcequota.requests_memory_used

dt.kubernetes.resourcequota.requests_memory_used

builtin:kubernetes.workload.conditions

dt.kubernetes.workload.conditions

builtin:kubernetes.workload.pods_desired

dt.kubernetes.workload.pods_desired

builtin:kubernetes.workloads

dt.kubernetes.workloads

Convergent metrics

The Grail metrics that supersede the Classic metrics often offer an increased level of detail compared to the Classic metrics.

  • Many Classic metrics, such as metrics related to resource consumption, are available in Classic as separate workload- and node- level metrics. With Grail these have been consolidated to a single metric at the container level.
  • Some metrics, such as builtin:kubernetes.containers and builtin:kubernetes.events, have been extended with additional dimensions.

If a Grail metric now provides more detailed dimensions, such as at the container level, the data can be aggregated back to a higher level (such as workload levels). This aggregation can be useful if you want to re-use existing filters.

Classic metric keysGrail metric key

builtin:kubernetes.containers

dt.kubernetes.containers

builtin:kubernetes.workload.containers_desired

dt.kubernetes.pod.containers_desired

builtin:kubernetes.events

dt.kubernetes.events

  • builtin:kubernetes.node.pods
  • builtin:kubernetes.pods

dt.kubernetes.pods

  • builtin:kubernetes.node.cpu_usage
  • builtin:kubernetes.workload.cpu_usage

dt.kubernetes.container.cpu_usage

  • builtin:kubernetes.node.cpu_throttled
  • builtin:kubernetes.workload.cpu_throttled

dt.kubernetes.container.cpu_throttled

  • builtin:kubernetes.node.requests_cpu
  • builtin:kubernetes.workload.requests_cpu

dt.kubernetes.container.requests_cpu

  • builtin:kubernetes.node.limits_cpu
  • builtin:kubernetes.workload.limits_cpu

dt.kubernetes.container.limits_cpu

  • builtin:kubernetes.node.memory_working_set
  • builtin:kubernetes.workload.memory_working_set

dt.kubernetes.container.memory_working_set

  • builtin:kubernetes.node.requests_memory
  • builtin:kubernetes.workload.requests_memory

dt.kubernetes.container.requests_memory

  • builtin:kubernetes.node.limits_memory
  • builtin:kubernetes.workload.limits_memory

dt.kubernetes.container.limits_memory

Example: Aggregate container-level metrics to workload level

The following DQL query returns the amount of memory consumed at the workload level, based on aggregated container-level data.

timeseries memory_working_set = sum(dt.kubernetes.container.memory_working_set)
by: {
    k8s.cluster.name,
    k8s.namespace.name,
    k8s.workload.name
}

Replaced metrics

The following Classic metric keys are replaced by a similar, but not identical, Grail metric. These metric keys are closely related and any deviations in the output are minor.

Classic metric keyGrail metric key

builtin:containers.cpu.limit

dt.kubernetes.container.limits_cpu

  • builtin:kubernetes.container.outOfMemoryKills
  • builtin:kubernetes.container.oom_kills

dt.kubernetes.container.oom_kills

Calculated metrics

Dynatrace Classic metrics used millicores as the unit of measure, while the equivalent Grail metrics use nanoseconds per minute. Therefore, to get the same output value for these metrics, you'll need to use DQL to calculate the difference between these units. The sections below list the specific metrics and provide example DQL calculations.

List of metrics requiring calculations

Some Classic metric keys have a direct Grail equivalent, while for others you need to compute the output from multiple Grail metrics.

Classic metric keyGrail metric keys

builtin:containers.cpu.throttledMilliCores

dt.containers.cpu.throttled_time

builtin:containers.cpu.usageUserMilliCores

dt.containers.cpu.usage_user_time

builtin:containers.cpu.usageSystemMilliCores

dt.containers.cpu.usage_system_time

builtin:containers.cpu.usageMilliCores

  • dt.containers.cpu.usage_user_time
  • dt.containers.cpu.usage_system_time

builtin:containers.cpu.usagePercent

  • dt.containers.cpu.usage_user_time
  • dt.containers.cpu.usage_system_time
  • dt.containers.cpu.logical_cores
  • dt.containers.cpu.throttling_ratio

builtin:containers.cpu.usageTime

  • dt.containers.cpu.usage_user_time
  • dt.containers.cpu.usage_system_time

builtin:containers.memory.limitPercent

  • dt.containers.memory.limit_bytes
  • dt.containers.memory.physical_total_bytes

builtin:containers.memory.usagePercent

  • dt.containers.memory.limit_bytes
  • dt.containers.memory.physical_total_bytes
  • dt.containers.memory.resident_set_bytes

Calculation examples

For each Classic metric key, this section provides the equivalent DQL queries, including calculations to convert to nanoseconds per minute.

builtin:containers.cpu.throttledMilliCores
timeseries {
      throttled_time = avg(dt.containers.cpu.throttled_time, rollup: sum, rate: 1m)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      throttled_milli_cores = throttled_time[] * milli_core_per_core / ns_per_min
| summarize {
      throttled_milli_cores = sum(throttled_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usageUserMilliCores
timeseries {
      usage_user_time = avg(dt.containers.cpu.usage_user_time)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      usage_user_milli_cores = usage_user_time[] * milli_core_per_core / ns_per_min
| summarize {
      usage_user_milli_cores = sum(usage_user_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usageSystemMilliCores
timeseries {
      usage_system_time = avg(dt.containers.cpu.usage_system_time)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      usage_system_milli_cores = usage_system_time[] * milli_core_per_core / ns_per_min
| summarize {
      usage_system_milli_cores = sum(usage_system_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usageMilliCores
timeseries {
      usage_user_time = avg(dt.containers.cpu.usage_user_time)
    , usage_system_time = avg(dt.containers.cpu.usage_system_time)
}
| fieldsAdd
      ns_per_min = 60 * 1000 * 1000 * 1000
    , milli_core_per_core = 1000
| fieldsAdd
      usage_milli_cores = (usage_user_time[] + usage_system_time[] )
    * milli_core_per_core / ns_per_min
| summarize {
      usage_milli_cores = sum(usage_milli_cores[] )
}, by: { timeframe, interval }
builtin:containers.cpu.usagePercent
timeseries {
      // for total usage, user and system cpu usage are added
      userCpuUsage = avg(dt.containers.cpu.usage_user_time)
    , systemCpuUsage = avg(dt.containers.cpu.usage_system_time)
      // cpu logical counts are the fallback, if the throttling ratio doesn't exist
    , cpuLogicalCount = avg(dt.containers.cpu.logical_cores)
}
// filter statement ...
// leftOuter join allows the throttling ratio to be null
| join [
  timeseries {
      throttlingRatio = avg(dt.containers.cpu.throttling_ratio)
      // same filter statement as above ...
}
], on: { interval, timeframe}, fields: { throttlingRatio}, kind:leftOuter
| fieldsAdd
       // sum of system and user cpu usage
       numerator = userCpuUsage[] + systemCpuUsage[]
       // throttling ratio, or as a fallback cpu logical count.
     , denominator = coalesce(throttlingRatio, cpuLogicalCount)
     , nanoseconds_per_minute  = 60 * 1000 * 1000 * 1000
 | fields
       interval, timeframe
     , cpuUsagePercent = 100.0 * numerator[] / ( denominator[] * nanoseconds_per_minute)
builtin:containers.cpu.usageTime
timeseries {
      usageUserTime = avg(dt.containers.cpu.usage_user_time)
    , usageSystemTime = avg(dt.containers.cpu.usage_system_time)
    }
    , by: { dt.entity.container_group_instance},
| fields
      interval, timeframe
    , usageTime = usageSystemTime[] + usageUserTime[]
builtin:containers.memory.limitPercent
timeseries {
      limit_bytes = avg(dt.containers.memory.limit_bytes),
      physical_total_bytes = avg(dt.containers.memory.physical_total_bytes)
}
| fieldsAdd
      limit_percent = (limit_bytes[] / physical_total_bytes[] ) * 100
| summarize {
      limit_percent = sum(limit_percent[] )
}, by: { timeframe, interval }
builtin:containers.memory.usagePercent
timeseries {
      memoryLimits = avg(dt.containers.memory.limit_bytes)
    , totalPhysicalMemory = avg(dt.containers.memory.physical_total_bytes)
    , residentSetBytes = avg(dt.containers.memory.resident_set_bytes)
    }
    , by: { dt.entity.container_group_instance}
| fieldsAdd
      denominator = if (
          arrayFirst(memoryLimits) > 0,
          then: memoryLimits,
          else: totalPhysicalMemory
          )
| fields
     dt.entity.container_group_instance
   , interval, timeframe
   , memoryUsagePercent = 100 * residentSetBytes[] / denominator[]

Related topics

  • Use DQL queries
  • Notebooks
Related tags
Dynatrace Platform