Send Micrometer metrics to Dynatrace

Micrometer is an open source instrumentation framework for JVM-based application metrics. It's used by Spring Boot to instrument a wide range of metrics. You can ingest Micrometer and Spring Boot metrics and analyze them with Dynatrace Davis® AI end to end in the context of your trace, log, and diagnostics data. With Dynatrace, you get intelligent AI-based observability and automatic root cause analysis for Spring Boot, 15+ pre-instrumented JVM-based frameworks and servers, and custom metrics.

You can use Micrometer in Dynatrace to:

Ingest pre-instrumented metrics from Spring Boot applications
Ingest pre-instrumented metrics from JVM-based frameworks, servers, and cache systems
Define and ingest custom metrics

Metrics ingested from Micrometer consume DDUs for custom metrics.

There are two ways of using Micrometer:

Prerequisites

Micrometer version 1.8.0+
optional Spring Boot version 2.6.0+
The registry dependency must be added to your project:
implementation 'io.micrometer:micrometer-registry-dynatrace:latest.release'
Replace {micrometer.version} with the latest version of Micrometer or a specific version that you want to use. A list of released versions is available on Maven Central. We recommend that you use the latest version.
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-dynatrace</artifactId>
<version>{micrometer.version}</version>
</dependency>
The Spring Boot BOM specifies a Micrometer version that has been tested with the respective version of Spring Boot. It's therefore enough to specify the name of the dependency without specifying the version. This will result in the correct, matching version being pulled by Gradle or Maven.
implementation 'io.micrometer:micrometer-registry-dynatrace'
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-dynatrace</artifactId>
</dependency>

Ingestion channels

You can use one of the following ingestion channels to send your Micrometer metrics:

OneAgent metric API—needs OneAgent installed on the monitored host.
Metrics API v2.

Dynatrace Micrometer registry

Micrometer uses the concept of a registry to export metrics to monitoring systems.

For Micrometer version 1.8.0 or later, Dynatrace Registry v2 is available. It exports metrics via the Metrics API v2. All new integrations of Micrometer and Dynatrace must use this version.
For Micrometer version 1.8.0 or earlier, the legacy Dynatrace Micrometer registry v1 is available. For instructions, see Dynatrace Micrometer registry v1 (legacy) below.

Ingest metrics from Spring Boot apps

Micrometer can be configured via a .properties or .yaml file used for Spring Boot configuration. Spring Boot automatically binds properties with the management.dynatrace.metrics.export prefix to the Dynatrace configuration object.

All configuration should be made through the property files. Manually creating a Micrometer MeterRegistry breaks the auto-configuration.

Property names for binding attributes from Spring Boot have changed in Spring Boot version 3.0.0. If you use a Spring Boot version before 3.0.0, use management.metrics.export.dynatrace instead of management.dynatrace.metrics.export.

OneAgent on Kubernetes nodes does not support the ingestion of Micrometer metrics directly. See Sending Metrics from Kubernetes for more details. If you're using Dynatrace on Kubernetes, we recommend using Dynatrace Operator, which provides autoconfiguration.

For hosts that are monitored by OneAgent, automatic configuration is available. You don't need to specify the API endpoint to ingest the metric—if the uri parameter is not set in the configuration, the metric will be ingested via the OneAgent metric API.

management.dynatrace.metrics.export:
  v2:
    metric-key-prefix: "service.component"
    enrich-with-dynatrace-metadata: true
    default-dimensions:
      stack: "prod"
      region: "us-east-1"

management.dynatrace.metrics.export.v2.metric-key-prefix=service.component
management.dynatrace.metrics.export.v2.enrich-with-dynatrace-metadata=true
management.dynatrace.metrics.export.v2.default-dimensions.stack=prod
management.dynatrace.metrics.export.v2.default-dimensions.region=us-east-1

Automatic injection for endpoint and token via Dynatrace Operator is available only for cloudNativeFullStack and applicationOnly. For details, see Enrich ingested data with Dynatrace-specific dimensions.

For hosts running in an environment managed by Dynatrace Operator, it automatically provides the URI and dataIngestToken to the environment. These credentials are sourced from the secret created to store the required permission tokens, known as the dynakube secret. For more information, see Metadata metric enrichment.

In Micrometer 1.9.0+, the URI and token are automatically picked up by the Dynatrace Micrometer registry, and data is exported to Dynatrace without the need for additional configuration.
Explicitly providing a URI will disable this automatic configuration option and should therefore be avoided if using the Dynatrace Kubernetes Operator autoconfiguration.

HTTP clients connecting to the ActiveGate REST endpoint must trust provided certificates. For details, see Add a custom certificate for ActiveGate.

management.dynatrace.metrics.export:
  v2:
    metric-key-prefix: "service.component"
    enrich-with-dynatrace-metadata: true
    default-dimensions:
      stack: "prod"
      region: "us-east-1"

management.dynatrace.metrics.export.v2.metric-key-prefix=service.component
management.dynatrace.metrics.export.v2.enrich-with-dynatrace-metadata=true
management.dynatrace.metrics.export.v2.default-dimensions.stack=prod
management.dynatrace.metrics.export.v2.default-dimensions.region=us-east-1

To ingest metrics from hosts where OneAgent is not installed, such as serverless deployments (for example, AWS ECS) or other non-Kubernetes environments, you need to use the ingest endpoint of the Metrics API v2. To learn how to use the endpoint, see the POST ingest data points example. The Micrometer Dynatrace registry exports to this API when the URI and token are set.

Ensure that the URI is explicitly configured, as leaving it unset will default to localhost, which does not exist in these environments.

You can use the Spring placeholder notation (for example, api-token: ${YOUR_METRICS_INGEST_API_TOKEN}), which will automatically read the environment variable and supply it to the Micrometer configuration.

management.dynatrace.metrics.export:
  uri: "https://mySampleEnv.live.dynatrace.com/api/v2/metrics/ingest"
  # Read the environment variable YOUR_METRICS_INGEST_API_TOKEN and supply the value of the env var instead.
  api-token: ${YOUR_METRICS_INGEST_API_TOKEN}
  v2:
    metric-key-prefix: "service.component"
    enrich-with-dynatrace-metadata: true
    default-dimensions:
      stack: "prod"
      region: "us-east-1"

management.dynatrace.metrics.export.uri=https://mySampleEnv.live.dynatrace.com/api/v2/metrics/ingest
management.dynatrace.metrics.export.api-token=${YOUR_METRICS_INGEST_API_TOKEN}
management.dynatrace.metrics.export.v2.metric-key-prefix=service.component
management.dynatrace.metrics.export.v2.enrich-with-dynatrace-metadata=true
management.dynatrace.metrics.export.v2.default-dimensions.stack=prod
management.dynatrace.metrics.export.v2.default-dimensions.region=us-east-1

Ingest metrics directly from Micrometer

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
    @Override
    @Nullable
    public String get(String k) {
        // This method can be used for retrieving arbitrary config items;
        // null means accepting the defaults defined in DynatraceConfig
        return null;
    }
};
DynatraceMeterRegistry registry = DynatraceMeterRegistry.builder(config).build();

In Micrometer 1.9.0+, the URI and token are automatically picked up by the Dynatrace Micrometer registry, and data is exported to Dynatrace without the need for additional configuration.
Explicitly providing a URI will disable this automatic configuration option and should therefore be avoided if using the Dynatrace Kubernetes Operator autoconfiguration.

HTTP clients connecting to the ActiveGate REST endpoint must trust provided certificates. For details, see Add a custom certificate for ActiveGate.

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
    @Override
    @Nullable
    public String get(String k) {
        // This method can be used for retrieving arbitrary config items;
        // null means accepting the defaults defined in DynatraceConfig
        return null;
    }
};
DynatraceMeterRegistry registry = DynatraceMeterRegistry.builder(config).build();

Ensure that the URI is explicitly configured, as leaving it unset will default to localhost, which does not exist in these environments.

In this example, the metrics ingest URL and access token are read from environment variables YOUR_METRICS_INGEST_URL and YOUR_METRICS_INGEST_TOKEN. You should never store secrets in the code directly, but read them from a secure source.

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
    @Override
    public DynatraceApiVersion apiVersion() {
        // Not strictly required, but makes the code more clear/explicit
        return DynatraceApiVersion.V2;
    }

    @Override
    public String uri() {
        // The endpoint of the Dynatrace Metrics API v2 including path:
        // "https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest"
        String endpoint = System.getenv("YOUR_METRICS_INGEST_URL");
        return endpoint != null ? endpoint : DynatraceConfig.super.uri();
    }

    @Override
    public String apiToken() {
        // Should be read from a secure source
        String token = System.getenv("YOUR_METRICS_INGEST_TOKEN");
        return token != null ? token : "";
    }

    @Override
    @Nullable
    public String get(String k) {
        // This method can be used for retrieving arbitrary config items;
        // null means accepting the defaults defined in DynatraceConfig
        return null;
    }
};
DynatraceMeterRegistry registry = DynatraceMeterRegistry.builder(dynatraceConfig).build();

Verify the metrics

After you have sent your metrics, verify the data in the Data Explorer.

Configuration properties

To set up the Dynatrace Micrometer registry, you need the Dynatrace configuration object (DynatraceConfig). The object contains the parameters of metric ingestion and is used to construct the Dynatrace registry (DynatraceMeterRegistry), which is registered with Micrometer to ingest metrics to Dynatrace. You can set the following parameters:

Property

Description

Required

api-version

The version of the Dynatrace API to ingest data to Dynatrace:

V2 (default): Metrics API v2.
V1: Timeseries API v1

optional

uri

The ingestion endpoint of the API.

optional

metric-key-prefix

The prefix to be added to all ingested metric keys (for example, a namespace).

optional

enrich-with-dynatrace-metadata

Enrich (true) or do not enrich (false) ingested metrics with additional metadata.

If not set, true is used.

optional

default-dimensions

A list of dimensions to be added to the ingested metrics.

Dimensions are defined as key-value pairs.

optional

use-dynatrace-summary-instruments

Micrometer versions 1.9.x+ Ignore the Dynatrace-specific implementation for summary instruments (Timer and DistributionSummary).

The default (true) uses the new Instruments. Use false to fall back to 1.8.x behavior.

optional

export-meter-metadata

Micrometer versions 1.12.x+ Turn meter metadata export (unit & description) on or off.

The default (true) configures the registry to export meter metadata. Use false to turn off metadata exporting.

optional

Property

Description

Required

apiVersion

The version of the Dynatrace API to ingest data to Dynatrace:

V2 (default): Metrics API v2.
V1: Timeseries API v1

optional

uri

The ingestion endpoint of the API.

optional

metricKeyPrefix

The prefix to be added to all ingested metric keys (for example, a namespace).

optional

enrichWithDynatraceMetadata

Enrich (true) or do not enrich (false) ingested metrics with additional metadata.

If not set, true is used.

optional

defaultDimensions

A list of dimensions to be added to the ingested metrics.

Dimensions are defined as key-value pairs.

optional

useDynatraceSummaryInstruments

Micrometer versions 1.9.x+ Ignore the Dynatrace-specific implementation for summary instruments (Timer and DistributionSummary).

The default (true) uses the new Instruments. Use false to fall back to 1.8.x behavior.

optional

exportMeterMetadata

Micrometer versions 1.12.x+ Turn meter metadata export (unit & description) on or off.

The default (true) configures the registry to export meter metadata. Use false to turn off metadata exporting.

optional

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
    @Override
    public DynatraceApiVersion apiVersion() {
        // Defaults to V2 if not set explicitly.
        return DynatraceApiVersion.V2;
    }

    @Override
    public String uri() {
        // The endpoint of the Dynatrace Metrics API v2 including path. For example:
        // "https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest".
        String endpoint = System.getenv("YOUR_METRICS_INGEST_URL");
        return endpoint != null ? endpoint : DynatraceConfig.super.uri();
    }

    @Override
    public String apiToken() {
        // Should be read from a secure source
        String token = System.getenv("YOUR_METRICS_INGEST_API_TOKEN");
        return token != null ? token : "";
    }

    @Override
    public String metricKeyPrefix() {
        // Will be prepended to all metric keys
        return "service.component";
    }

    @Override
    public boolean enrichWithDynatraceMetadata() {
        // On by default, but can be turned off explicitly.
        return true;
    }

    @Override
    public Map<String, String> defaultDimensions() {
        // Create and return a map containing the desired key-value pairs.
        Map<String, String> dims = new HashMap<>();
        dims.put("dimensionKey", "dimensionValue");
        return dims;
    }

    // Only available in Micrometer 1.9.0 and above.
    @Override
    public boolean useDynatraceSummaryInstruments() {
        return false;
    }

    // Only available in Micrometer 1.12.0 and above.
    @Override
    public boolean exportMeterMetadata() {
        return false;
    }

    @Override
    @Nullable
    public String get(String k) {
        return null; // Accept the rest of the defaults
    }
};

Additional information

Metric types

All metrics are transformed to follow the Metric ingestion protocol types used by Dynatrace.

Micrometer instrument

Dynatrace metric type

Gauge

gauge,X

Counter

count,delta=X

Timer

gauge,min=X,max=Y,sum=Z,count=N

DistributionSummary

gauge,min=X,max=Y,sum=Z,count=N

LongTaskTimer

gauge,min=X,max=Y,sum=Z,count=N

TimeGauge

gauge,X

FunctionCounter

count,delta=X

FunctionTimer

gauge,min=X,max=Y,sum=Z,count=N

Meter metadata

Starting with version 1.12.0 of the Dynatrace Micrometer registry, meter metadata (unit and description) is automatically exported to Dynatrace. No code changes are required to start exporting metadata. To turn off metadata export, use the following configuration:

Meter metadata was introduced with Micrometer 1.12.0, which was first included in Spring Boot 3.2.0. The toggles described below are therefore available from Spring Boot 3.2.0.

management:
  metrics:
    export:
      dynatrace:
        v2:
          export-meter-metadata: false  # default: true

management.dynatrace.metrics.export.v2.export-meter-metadata=false

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
    // Only available in Micrometer 1.12.0 and above.
    @Override
    public boolean exportMeterMetadata() {
        return false;
    }

    @Override
    @Nullable
    public String get(String k) {
        return null; // Accept the rest of the defaults
    }
};

For previous versions of Micrometer, you need to specify the metadata manually using either the Dynatrace web UI or Dynatrace API. For more information, see Custom metric metadata.

Sending metrics from Kubernetes

OneAgent cannot be used for Micrometer metric ingestion on Kubernetes nodes. You can configure your Micrometer setup to push metrics directly to Dynatrace using the Metrics API.

Capture JVM metrics in Micrometer

By default, JVM metrics are turned off when running Micrometer without Spring Boot. To learn how to enable them, see the Micrometer documentation. After they are enabled and registered with the Dynatrace registry (DynatraceMeterRegistry), JVM metrics are created and sent to Dynatrace automatically.

On hosts that are monitored by OneAgent, these metrics might already be captured by OneAgent.

Restrict ingestion of specific metrics

When running Micrometer in Spring Boot, many metrics are automatically created and sent to Dynatrace, including JVM, process, and disk metrics.

To see all metrics created by your Spring Boot application, navigate to the actuator endpoint on your Spring Boot app (/actuator/metrics). Some of these metrics might already be captured by OneAgent.

Disable metrics with Spring properties

Metrics can be disabled based on their prefix in the Spring Boot configuration. To find out which metrics can be excluded, check your Spring Boot applications actuator endpoint. Be sure to exclude the metric key prefix (if any) when using this feature.

management.metrics.enable:
  # disable jvm.memory metrics
  jvm.memory: false
  # disable all jvm metrics:
  jvm: false

# disable jvm.memory metrics
management.metrics.enable.jvm.memory=false
# disable all jvm metrics
management.metrics.enable.jvm=false

It is also possible to disable all metrics first and then only re-enable the desired ones:

management.metrics.enable:
  # disable all metrics
  all: false
  # re-enable only jvm.* metrics
  jvm: true

# disable all metrics
management.metrics.enable.all=false
# re-enable jvm.* metrics
management.metrics.enable.jvm=true

Disable metrics in code

Micrometer provides meter filters to disable metrics based on a variety of reasons. Meter filters can also be configured via Spring Boot with the @Configuration annotation.

@Configuration(proxyBeanMethods = false)
public class MyMeterRegistryConfiguration {

    @Bean
    public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
        return registry -> registry.config()
            .meterFilter(MeterFilter.denyNameStartsWith("jvm.gc"));
    }
}

The metric prefix configured for the Dynatrace registry will be applied after filtering, so properties to enable or disable metrics have to be specified using the original metric key without this prefix.

Add MeterFilters before you turn on additional metrics, as the meter filter will otherwise be overridden.

You can configure the registry to filter out specific metrics by name and/or tags (for example, metrics that are already captured by OneAgent). To achieve that, use Micrometer's meter filters. You need to add meter filters before you enable capture of JVM metrics, otherwise filters will be overridden.

The metric prefix configured for the Dynatrace registry will be applied after filtering, so meterFilters have to be specified using the original metric key without this prefix.

// Disable all metrics with metric names starting with jvm.gc
registry.config()
    .meterFilter(MeterFilter.denyNameStartsWith("jvm.gc"));

Troubleshooting with logs

Spring handles the logging when using Micrometer in the Spring Boot context. The log level for the Micrometer Dynatrace registry can be set via configuration properties.

logging.level.io.micrometer.dynatrace: DEBUG

logging.level.io.micrometer.dynatrace=DEBUG

Micrometer and the Dynatrace Micrometer registry use slf4j internally to log events, such as the lines that are sent to Dynatrace. If you want to get this information, set up your project with the logging framework of your choice (for example, logback) and set the log level to debug.

When using the default logback implementation, debug logging to the console is enabled by default. In order to add logback to your project and log debug information, add the following dependency:

implementation 'ch.qos.logback:logback-classic:latest.release'

Dynatrace summary instruments

Starting with Micrometer version 1.9.x, specialized instruments are used in the Dynatrace meter registry for certain summary instruments (DynatraceTimer and DynatraceDistributionSummary). Their purpose is to get around a peculiarity in how Micrometer records metrics, which, in some cases, led to metrics being rejected by Dynatrace for having an invalid format. The specialized instruments, tailored to Dynatrace metrics ingestion, prevent the creation of invalid metrics.

They are available from version 1.9.0 and are used as a drop-in replacement by default. No action is needed from users upgrading to 1.9.0.
If there's a discrepancy in the observed metrics, it's possible to return to the previous behavior by setting the useDynatraceSummaryInstruments toggle to false.

Dynatrace Micrometer registry v1 (legacy)

If the apiVersion property is set to V1, the registry sends data via the Timeseries API v1. For backward compatibility, it defaults to V1 if a deviceId is specified, because this property is required in V1 and is not used in V2.

Existing setups will continue to work when updating to newer versions of Micrometer. The reported metrics will be assigned to custom devices in Dynatrace.

For the V1 API, you only need to specify the base URL of your environment (for example, https://mySampleEnv.live.dynatrace.com).

management.dynatrace.metrics.export:
    # For v1 export, do not append a path to the endpoint URL. For example:
    # For SaaS: https://{your-environment-id}.live.dynatrace.com
    # For Managed deployments: https://{your-domain}/e/{your-environment-id}
    uri: https://{your-environment-id}.live.dynatrace.com

    # Should be read from a secure source
    api-token: MY_TOKEN

    # When setting the device id, metrics will be exported to the v1 timeseries endpoint
    # Using just device-id (without the v1 prefix) is deprecated, but will work to maintain backwards compatibility.
    v1:
        device-id: sample

    # To disable Dynatrace publishing (for example, in a local development profile), use:
    # enabled: false

    # The interval at which metrics are sent to Dynatrace. The default is 1 minute.
    step: 1m

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
    @Override
    public String uri() {
        // The Dynatrace environment URI without any path. For example:
        // https://{your-environment-id}.live.dynatrace.com
        return MY_DYNATRACE_URI;
    }

    @Override
    public String apiToken() {
        // Should be read from a secure source
        return MY_TOKEN;
    }

    @Override
    public String deviceId() {
        return MY_DEVICE_ID;
    }

    @Override
    @Nullable
    public String get(String k) {
        return null;
    }
};
DynatraceMeterRegistry registry = DynatraceMeterRegistry.builder(config).build();
// Add the Dynatrace registry to Micrometers global registry.
Metrics.addRegistry(registry);