Apache Kafka extension

  • Latest Dynatrace
  • Extension
  • Published Oct 27, 2025

Automatic and intelligent observability with trace and metric insights.

Get an overview of your Kafka monitoring using New Dashboards, which are bundled with the extension upon activation.Easily navigate your Kafka environment and its most important metrics with the out of the box dashboard, included with the activation of the extension.Dedicated screen for your Kafka broker with all its metrics, along with a navigation card for the rest of your Kafka environment.Dedicated screens also for your Kafka producers, consumers and topics.List all your Kafka topics to get a clear overview of your generated entities, with filtering and ordering for the most relevant metrics.Lists also available for the rest of the generated entities.Producer-level metrics can be viewed in Unified-Analysis screens for your Queues.The OneAgent can monitor end-to-end traces that connect producer and consumer services, monitor the performance of topics and queues, and more.
1 of 8Get an overview of your Kafka monitoring using New Dashboards, which are bundled with the extension upon activation.

Get observability for Kafka through automatic monitoring.

Get started

Overview

Seamless end-to-end traces for connected producer and consumer clients allow you to diagnose anomalies and pinpoint the root cause of the broken code before your customers are affected.

kafka

Comprehensive metrics give you insight into your Kafka servers with health and performance metrics for brokers, topics, producers, and consumers. Events point you to critical anomalies, reducing the mean repair time.

kafka

Use cases

  • Capture every message across tiers without blind spots.
  • Improve the performance of your producer and consumer services end-to-end.
  • Troubleshoot asynchronous service problems across your stack proactively.
  • Prevent message processing anomalies to reduce the mean time to repair.
  • Monitor the health and performance of all your brokers and topics.
  • Understand your consumer lag.

Requirements

  • OneAgent version 1.270+
  • Dynatrace version 1.270+
  • OneAgent feature Java Metric Extensions 2.0 (JMX) activated

Activation and setup

Messaging clients (applications)

To get trace insight:

  1. Install OneAgent on the virtual machine or server of your messaging clients (applications).
  2. Set up Dynatrace on Kubernetes or OpenShift for your messaging client (application) workloads.
  3. Activate the following OneAgent features:
    • Java Kafka
    • Java Kafka Streams
    • Spring for Apache Kafka
    • Node.js KafkaJs
    • .NET Messaging Apache Kafka

To get log insight:

Messaging servers (brokers)

Prerequisites:

To get metric insight:

  1. Install OneAgent on the virtual machine or server of your Kafka broker process.
  2. Ensure your Kafka broker, producer and consumer processes are monitored.
  3. Activate the OneAgent feature Java Metric Extensions 2.0 (JMX).
  4. Select in Dynatrace Add to environment to configure the extension.
  5. Open the Apache Kafka Overview dashboard.

Details

This extension package contains:

  • Topology and relationship definitions for Kafka Topics, Kafka Producers, Kafka Consumers, Kafka Brokers, and the relationships between these entities.
  • A dashboard offering a monitoring overview for topics, producers, consumers, and brokers.
  • Alerts covering broker availability, cluster controllers, offline partitions, leader elections, and replicated partitions.
  • A Unified Analysis page for Kafka Topics, Producers, Consumers, and Brokers.

With the Kafka extension, you can get additional insight into your Kafka server with metrics for brokers, topics, producers, consumers, and more. The extension also provides alerts for the most critical metrics. It creates a custom topology and entities for brokers, topics, producers, and consumers. It provides a dashboard for easier access and configuration of the extension and its entities.

The extension will gather different metrics depending on if it's monitoring a Kafka broker, producer or consumer. Make sure to activate the extension on all of them to get all metrics.

The extension has one single variable called Topic name filter, which can be used to filter the topics that will be monitored by this extension. This filter will be applied to every metric with a topic dimension. It uses the ObjectName java notation, meaning it accepts wildcards such as * and ?.

Select Add to environment to get started.

Licensing and cost

This extension queries for data every minute. Once data is produced, this is ingested as metric datapoints and subject to license consumption. Whether you are on a Classic or DPS license model, the measurement unit for metric ingestion is based on metric datapoints.

To estimate the amount of metric datapoints produced by your extension configuration, use the following list which gives you a number of metrics per feature set and a multiplier value depending on which entity the metric is split by. Add all of these up and you will have the number of datapoints produced every minute.

  • broker-metrics 15 x number of Kafka Brokers
  • connect-metrics 4 x number of Kafka Brokers
  • connector-metrics 10 x 1 (metrics are not split by an entity)
  • consumer-metrics 11 x number of Kafka Consumers
  • controller metrics 2 x number of Kafka Brokers
  • disk-metrics 2 x number of Kafka Brokers
  • handler-metrics 1 x number of Kafka Brokers
  • leader-metrics 3 x number of Kafka Brokers
  • log-metrics 2 x number of Kafka Brokers
  • network-metrics 7 x number of Kafka Brokers
  • producer-metrics 2 x number of Kafka Producers
  • purgatory-metrics 2 x number of Kafka Brokers
  • zookeeper-metrics 3 x number of Kafka Brokers

For example, a monitoring configuration where all feature sets are enabled, containing 1 Kafka Broker, 1 Kafka Producer, and 1 Kafka Consumer produces 64 metric datapoints per minute

By extrapolation, it will ingest:

  • 64 x 60 x = 3,840 metric datapoints per hour
  • 64 x 60 x 24 = 92,160 metric datapoints per day
  • 64 x 60 x 24 x 365 = 33,638,400 metric datapoints per year

Included Metrics per host unit Because JMX Extensions run on OneAgent-monitored hosts, are subject to included metrics per host unit.

Please refer to the documentation depending on licensing type:

Feature sets

When activating your extension using monitoring configuration, you can limit monitoring to one of the feature sets. To work properly the extension has to collect at least one metric after the activation.

In highly segmented networks, feature sets can reflect the segments of your environment. Then, when you create a monitoring configuration, you can select a feature set and a corresponding ActiveGate group that can connect to this particular segment.

All metrics that aren't categorized into any feature set are considered to be the default and are always reported.

A metric inherits the feature set of a subgroup, which in turn inherits the feature set of a group. Also, the feature set defined on the metric level overrides the feature set defined on the subgroup level, which in turn overrides the feature set defined on the group level.

Metric nameMetric keyDescription
Kafka Log - Log flush 95th percentilekafka.log.LogFlushStats.LogFlushRateAndTimeMs.Percentile95thLog flush rate and time in milliseconds.
Kafka Log - Log flush mean timekafka.log.LogFlushStats.LogFlushRateAndTimeMs.MeanLog flush rate and time in milliseconds.
Metric nameMetric keyDescription
Kafka Consumer - Requestskafka.consumer.consumer-metrics.request-rateThe average number of requests sent per second for a node.
Kafka Consumer - Request sizekafka.consumer.consumer-metrics.request-size-avgThe average size of all requests in the window.
Kafka Consumer - Incoming byte ratekafka.consumer.consumer-metrics.incoming-byte-rateBytes/second read off all sockets.
Kafka Consumer - Outgoing byte ratekafka.consumer.consumer-metrics.outgoing-byte-rateThe average number of outgoing bytes sent per second to all servers.
Kafka Consumer - Request latencykafka.consumer.consumer-metrics.request-latency-avgThe average request latency in ms for a node.
Kafka Consumer - Messages consumed ratekafka.consumer.consumer-metrics.records-consumed-rateThe average number of records consumed per second.
Kafka Consumer - Bytes consumed ratekafka.consumer.consumer-metrics.bytes-consumed-rateThe average number of bytes consumed per second for a topic.
Kafka Consumer - Fetch latencykafka.consumer.consumer-metrics.fetch-latency-avgThe average time taken for a fetch request.
Kafka Consumer - Consumer lagkafka.consumer.consumer-metrics.records-lagThe latest lag of the partition.
Kafka Consumer - Consumer lag averagekafka.consumer.consumer-metrics.records-lag-avgThe average lag of the partition.
Kafka Consumer - Consumer lag maximumkafka.consumer.consumer-metrics.records-lag-maxThe max lag of the partition.
Metric nameMetric keyDescription
Kafka Server - Purgatory Produce Sizekafka.server.purgatory.produce-delay-sizeRequests waiting in the produce purgatory.
Kafka Server - Purgatory Fetch Sizekafka.server.purgatory.fetch-delay-sizeRequests waiting in the fetch purgatory.
Metric nameMetric keyDescription
Kafka Producer - Incoming byte ratekafka.producer.producer-metrics.incoming-byte-rateThe average number of responses received per second for a node.
Kafka Producer - Outgoing byte ratekafka.producer.producer-metrics.outgoing-byte-rateThe average number of outgoing bytes sent per second to all servers.
Kafka Producer - I/O Wait timekafka.producer.producer-metrics.io-wait-time-ns-avgThe average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.
Kafka Producer - Response ratekafka.producer.producer-metrics.response-rateResponses received sent per second.
Kafka Producer - Request latencykafka.producer.producer-metrics.request-latency-avgThe average request latency in ms.
Kafka Producer - Compression ratekafka.producer.producer-metrics.compression-rate-avgThe average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size.
Kafka Producer - Request sizekafka.producer.producer-metrics.request-size-avgThe average size of all requests in the window.
Kafka Producer - Requestskafka.producer.producer-metrics.request-rateThe average number of requests sent per second.
Kafka Producer - Byte ratekafka.producer.producer-topic-metrics.byte-rateThe average number of bytes sent per second for a topic.
Kafka Producer - Compression ratekafka.producer.producer-topic-metrics.compression-rateThe average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size.
Kafka Producer - Failed Requests Ratekafka.producer.producer-topic-metrics.record-error-rateThe average per-second number of record sends that resulted in errors for a topic.
Kafka Producer - Requests Sent Ratekafka.producer.producer-topic-metrics.record-send-rateThe average number of records sent per second for a topic.
Metric nameMetric keyDescription
Kafka Connector - Statuskafka.connector.statusEquals 1 if the status is running, 0 otherwise.
Kafka Connector - Task statuskafka.connector.task.statusEquals 1 if the status is running, 0 otherwise.
Kafka Connector - Task pause ratiokafka.connector.task.pause-ratioThe fraction of time this task has spent in the pause state.
Kafka Connector - Task running ratiokafka.connector.task.running-ratioThe fraction of time this task has spent in the running state.
Kafka Connector - Task success ratiokafka.connector.task.offset-commit-success-percentageThe average percentage of this task's offset commit attempts that succeeded.
Kafka Connector - Task commit time (max)kafka.connector.task.offset-commit-max-time-msThe maximum time in milliseconds taken by this task to commit offsets.
Kafka Connector - Task failure ratiokafka.connector.task.offset-commit-failure-percentageThe maximum time in milliseconds taken by this task to commit offsets.
Kafka Connector - Task commit time (avg)kafka.connector.task.offset-commit-avg-time-msThe average time in milliseconds taken by this task to commit offsets.
Kafka Connector - Task batch size (max)kafka.connector.task.batch-size-maxThe maximum size of the batches processed by the connector.
Kafka Connector - Task batch size (avg)kafka.connector.task.batch-size-avgThe average size of the batches processed by the connector.
Metric nameMetric keyDescription
Kafka Controller - Offline partitionskafka.controller.KafkaController.OfflinePartitionsCountThe number of partitions that don't have an active leader and are therefore not writable or readable.
Kafka Controller - Active cluster controllerskafka.controller.KafkaController.ActiveControllerCount.ValueIndicates whether the broker is the controller broker.
Metric nameMetric keyDescription
Kafka Server - Handler Pool Idle Percent Ratekafka.server.handler.average-idle-percent.rateThe average fraction of time the request handler threads are idle. Values are between 0 meaning all resources are used and 1 meaning all resources are available.
Metric nameMetric keyDescription
Kafka Server - Disk Read Ratekafka.server.disk.read-bytesThe total number of bytes read by the broker process, including reads from all disks. The total doesn't include reads from page cache. Available only on Linux-based systems.
Kafka Server - Disk Write Ratekafka.server.disk.write-bytesThe total number of bytes written by the broker process, including writes from all disks. Available only on Linux-based systems.
Metric nameMetric keyDescription
Kafka Broker - Incoming byte ratekafka.server.BrokerTopicMetrics.BytesInPerSec.OneMinuteRateThe rate at which data sent from producers is consumed by the broker.
Kafka Broker - Outgoing byte ratekafka.server.BrokerTopicMetrics.BytesOutPerSec.OneMinuteRateThe rate at which data sent from other brokers is consumed by the follower broker.
Kafka Broker - Messages in ratekafka.server.BrokerTopicMetrics.MessagesInPerSec.OneMinuteRateThe rate at which individual messages are consumed by the broker.
Kafka Broker - Follower fetch requests ratekafka.server.BrokerTopicMetrics.TotalFollowerFetchRequestsPerSec.OneMinuteRateThe follower fetch request rate for the broker.
Kafka Broker - Produce message conversions ratekafka.server.BrokerTopicMetrics.ProduceMessageConversionsPerSec.OneMinuteRateThe rate at which produce messages are converted, by topic.
Kafka Broker - Partitionskafka.server.ReplicaManager.PartitionCountThe number of partitions in the broker.
Kafka Broker - Under replicated partitionskafka.server.ReplicaManager.UnderReplicatedPartitionsThe number of partitions that have not been fully replicated in the follower replicas.
Kafka Broker - Produce request ratekafka.server.BrokerTopicMetrics.TotalProduceRequestsPerSec.OneMinuteRateThe produce request rate per second.
Kafka Broker - Fetch request ratekafka.server.BrokerTopicMetrics.TotalFetchRequestsPerSec.OneMinuteRateThe fetch request rate per second.
Kafka Broker - Failed produce requestskafka.server.BrokerTopicMetrics.FailedProduceRequestsPerSec.OneMinuteRateThe produce request rate for requests that failed.
Kafka Broker - Failed fetch requestskafka.server.BrokerTopicMetrics.FailedFetchRequestsPerSec.OneMinuteRateThe fetch request rate for requests that failed.
Kafka Server - Max follower lagkafka.server.ReplicaFetcherManager.MaxLag.Replica.ValueThe maximum lag between the time that messages are received by the leader replica and by the follower replicas.
Kafka Server - Current follower lagkafka.server.FetcherLagMetrics.ConsumerLag.ValueThe lag in number of messages per follower replica.
Kafka Server - Fetch Conversions Ratekafka.server.FetchConversionsRate.OneMinuteRate
Kafka Server - Produce Conversions Ratekafka.server.ProduceConversionsRate.OneMinuteRate
Metric nameMetric keyDescription
Kafka Controller - Leader election ratekafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.OneMinuteRateThe broker leader election rate and latency in milliseconds. This is non-zero when there are broker failures.
Kafka Controller - Unclean election ratekafka.controller.ControllerStats.UncleanLeaderElectionsPerSec.OneMinuteRateThe unclean broker leader election rate. Should be 0.
Kafka Server - Leader countkafka.server.ReplicaManager.LeaderCount.ValueThe number of replicas for which this broker is the leader.
Metric nameMetric keyDescription
Kafka Network - Produce requests per secondkafka.network.RequestMetrics.RequestsPerSec.Produce.OneMinuteRateThe total number of requests made for produce per second
Kafka Network - FetchConsumer requests per secondkafka.network.RequestMetrics.RequestsPerSec.FetchConsumer.OneMinuteRateThe total number of requests made for fetch consumer per second
Kafka Network - FetchFollower requests per secondkafka.network.RequestMetrics.RequestsPerSec.FetchFollower.OneMinuteRateThe total number of requests made for fetch follower per second
Kafka Network - Total time per Produce requestkafka.network.RequestMetrics.TotalTimeMs.Produce.CountTotal time, in milliseconds, spent processing requests, for produce.
Kafka Network - Total time per FetchConsumer requestkafka.network.RequestMetrics.TotalTimeMs.FetchConsumer.CountTotal time, in milliseconds, spent processing requests, for fetch consumer.
Kafka Network - Total time per FetchFollower requestkafka.network.RequestMetrics.TotalTimeMs.FetchFollower.CountTotal time, in milliseconds, spent processing requests, for pfetch follower.
Kafka Network - Request queue sizekafka.network.RequestChannel.RequestQueueSize.ValueSize of the request queue.
Metric nameMetric keyDescription
Kafka Server - ZooKeeper disconnectskafka.server.SessionExpireListener.ZooKeeperDisconnectsPerSec.OneMinuteRateA meter that provides the number of recent ZooKeeper client disconnects.
Kafka Server - ZooKeeper expireskafka.server.SessionExpireListener.ZooKeeperExpiresPerSec.OneMinuteRateThe number of ZooKeeper sessions that have expired.
Kafka Server - Zookeeper Active Connectionskafka.server.active-connectionsThe number of currently open connections to the broker.
Metric nameMetric keyDescription
Kafka Connect - Requestskafka.connect.connect-metrics.request-rateThe average number of requests sent per second.
Kafka Connect - Outgoing byte ratekafka.connect.connect-metrics.outgoing-byte-rateThe average number of outgoing bytes sent per second to all servers.
Kafka Connect - Request sizekafka.connect.connect-metrics.request-size-avgThe average size of all requests in the window.
Kafka Connect - Incoming byte ratekafka.connect.connect-metrics.incoming-byte-rateBytes/second read off all sockets.

Troubleshooting

To troubleshoot this extension, use the guide(s) in the Dynatrace Community.

Related tags
ApplicationJMXStreaming platformApacheApplication Observability