Apache Kafka extension
- Latest Dynatrace
- Concept
- 5-min read
- Published Jun 27, 2025
Automatic and intelligent observability with trace and metric insights.
Compatibility information
Requirements
- OneAgent version 1.270+
- Dynatrace version 1.270+
- OneAgent feature
Java Metric Extensions 2.0 (JMX)activated
Overview
With Dynatrace, you can get observability for Kafka without touching any code, thanks to automatic monitoring. Seamless end-to-end traces for connected producer and consumer clients allow you to diagnose anomalies and pinpoint the root cause of the broken code before your customers are affected. Comprehensive metrics give you insight into your Kafka servers with health and performance metrics for brokers, topics, producers, and consumers. Events point you to critical anomalies, reducing the mean repair time.
1 of 8Get an overview of your Kafka monitoring using New Dashboards, which are bundled with the extension upon activation.
Get started
Messaging clients (applications)
To get trace insight:
- Install OneAgent on the virtual machine or server of your messaging clients (applications).
- Set up Dynatrace on Kubernetes or OpenShift for your messaging client (application) workloads.
- Activate the following OneAgent features:
- Java Kafka
- Java Kafka Streams
- Spring for Apache Kafka
- Node.js KafkaJs
- .NET Messaging Apache Kafka
To get log insight:
Messaging servers (brokers)
Prerequisites:
- Kafka broker, producer and consumer running on a supported Linux or Windows operating system.
- OneAgent version 1.270+
- Dynatrace version 1.270+
To get metric insight:
- Install OneAgent on the virtual machine or server of your Kafka broker process.
- Ensure your Kafka broker, producer and consumer processes are monitored.
- Activate the OneAgent feature
Java Metric Extensions 2.0 (JMX). - Select in Dynatrace Add to environment to configure the extension.
- Open the Apache Kafka Overview dashboard.
Details
This extension package contains:
- Topology and relationship definitions for Kafka Topics, Kafka Producers, Kafka Consumers, Kafka Brokers, and the relationships between these entities.
- A dashboard offering a monitoring overview for topics, producers, consumers, and brokers.
- Alerts covering broker availability, cluster controllers, offline partitions, leader elections, and replicated partitions.
- A Unified Analysis page for Kafka Topics, Producers, Consumers, and Brokers.
With the Kafka extension, you can get additional insight into your Kafka server with metrics for brokers, topics, producers, consumers, and more. The extension also provides alerts for the most critical metrics. It creates a custom topology and entities for brokers, topics, producers, and consumers. It provides a dashboard for easier access and configuration of the extension and its entities.
The extension will gather different metrics depending on if it's monitoring a Kafka broker, producer or consumer. Make sure to activate the extension on all of them to get all metrics.
Topic filtering
The extension has one single variable called Topic name filter, which can be used to filter the topics that will be monitored by this extension. This filter will be applied to every metric with a topic dimension. It uses the ObjectName java notation, meaning it accepts wildcards such as * and ?.
Select Add to environment to get started.
Use cases
- Capture every message across tiers without blind spots.
- Improve the performance of your producer and consumer services end-to-end.
- Troubleshoot asynchronous service problems across your stack proactively.
- Prevent message processing anomalies to reduce the mean time to repair.
- Monitor the health and performance of all your brokers and topics.
- Understand your consumer lag.
Compatibility information
Requirements
- OneAgent version 1.270+
- Dynatrace version 1.270+
- OneAgent feature
Java Metric Extensions 2.0 (JMX)activated
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Broker - Incoming byte rate | kafka.server.BrokerTopicMetrics.BytesInPerSec.OneMinuteRate | The rate at which data sent from producers is consumed by the broker. |
| Kafka Broker - Outgoing byte rate | kafka.server.BrokerTopicMetrics.BytesOutPerSec.OneMinuteRate | The rate at which data sent from other brokers is consumed by the follower broker. |
| Kafka Broker - Messages in rate | kafka.server.BrokerTopicMetrics.MessagesInPerSec.OneMinuteRate | The rate at which individual messages are consumed by the broker. |
| Kafka Broker - Follower fetch requests rate | kafka.server.BrokerTopicMetrics.TotalFollowerFetchRequestsPerSec.OneMinuteRate | The follower fetch request rate for the broker. |
| Kafka Broker - Produce message conversions rate | kafka.server.BrokerTopicMetrics.ProduceMessageConversionsPerSec.OneMinuteRate | The rate at which produce messages are converted, by topic. |
| Kafka Broker - Partitions | kafka.server.ReplicaManager.PartitionCount | The number of partitions in the broker. |
| Kafka Broker - Under replicated partitions | kafka.server.ReplicaManager.UnderReplicatedPartitions | The number of partitions that have not been fully replicated in the follower replicas. |
| Kafka Broker - Produce request rate | kafka.server.BrokerTopicMetrics.TotalProduceRequestsPerSec.OneMinuteRate | The produce request rate per second. |
| Kafka Broker - Fetch request rate | kafka.server.BrokerTopicMetrics.TotalFetchRequestsPerSec.OneMinuteRate | The fetch request rate per second. |
| Kafka Broker - Failed produce requests | kafka.server.BrokerTopicMetrics.FailedProduceRequestsPerSec.OneMinuteRate | The produce request rate for requests that failed. |
| Kafka Broker - Failed fetch requests | kafka.server.BrokerTopicMetrics.FailedFetchRequestsPerSec.OneMinuteRate | The fetch request rate for requests that failed. |
| Kafka Server - Max follower lag | kafka.server.ReplicaFetcherManager.MaxLag.Replica.Value | The maximum lag between the time that messages are received by the leader replica and by the follower replicas. |
| Kafka Server - Current follower lag | kafka.server.FetcherLagMetrics.ConsumerLag.Value | The lag in number of messages per follower replica. |
| Kafka Server - Fetch Conversions Rate | kafka.server.FetchConversionsRate.OneMinuteRate | |
| Kafka Server - Produce Conversions Rate | kafka.server.ProduceConversionsRate.OneMinuteRate |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Server - Disk Read Rate | kafka.server.disk.read-bytes | The total number of bytes read by the broker process, including reads from all disks. The total doesn't include reads from page cache. Available only on Linux-based systems. |
| Kafka Server - Disk Write Rate | kafka.server.disk.write-bytes | The total number of bytes written by the broker process, including writes from all disks. Available only on Linux-based systems. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Server - Handler Pool Idle Percent Rate | kafka.server.handler.average-idle-percent.rate | The average fraction of time the request handler threads are idle. Values are between 0 meaning all resources are used and 1 meaning all resources are available. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Controller - Offline partitions | kafka.controller.KafkaController.OfflinePartitionsCount | The number of partitions that don't have an active leader and are therefore not writable or readable. |
| Kafka Controller - Active cluster controllers | kafka.controller.KafkaController.ActiveControllerCount.Value | Indicates whether the broker is the controller broker. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Connector - Status | kafka.connector.status | Equals 1 if the status is running, 0 otherwise. |
| Kafka Connector - Task status | kafka.connector.task.status | Equals 1 if the status is running, 0 otherwise. |
| Kafka Connector - Task pause ratio | kafka.connector.task.pause-ratio | The fraction of time this task has spent in the pause state. |
| Kafka Connector - Task running ratio | kafka.connector.task.running-ratio | The fraction of time this task has spent in the running state. |
| Kafka Connector - Task success ratio | kafka.connector.task.offset-commit-success-percentage | The average percentage of this task's offset commit attempts that succeeded. |
| Kafka Connector - Task commit time (max) | kafka.connector.task.offset-commit-max-time-ms | The maximum time in milliseconds taken by this task to commit offsets. |
| Kafka Connector - Task failure ratio | kafka.connector.task.offset-commit-failure-percentage | The maximum time in milliseconds taken by this task to commit offsets. |
| Kafka Connector - Task commit time (avg) | kafka.connector.task.offset-commit-avg-time-ms | The average time in milliseconds taken by this task to commit offsets. |
| Kafka Connector - Task batch size (max) | kafka.connector.task.batch-size-max | The maximum size of the batches processed by the connector. |
| Kafka Connector - Task batch size (avg) | kafka.connector.task.batch-size-avg | The average size of the batches processed by the connector. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Producer - Incoming byte rate | kafka.producer.producer-metrics.incoming-byte-rate | The average number of responses received per second for a node. |
| Kafka Producer - Outgoing byte rate | kafka.producer.producer-metrics.outgoing-byte-rate | The average number of outgoing bytes sent per second to all servers. |
| Kafka Producer - I/O Wait time | kafka.producer.producer-metrics.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. |
| Kafka Producer - Response rate | kafka.producer.producer-metrics.response-rate | Responses received sent per second. |
| Kafka Producer - Request latency | kafka.producer.producer-metrics.request-latency-avg | The average request latency in ms. |
| Kafka Producer - Compression rate | kafka.producer.producer-metrics.compression-rate-avg | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. |
| Kafka Producer - Request size | kafka.producer.producer-metrics.request-size-avg | The average size of all requests in the window. |
| Kafka Producer - Requests | kafka.producer.producer-metrics.request-rate | The average number of requests sent per second. |
| Kafka Producer - Byte rate | kafka.producer.producer-topic-metrics.byte-rate | The average number of bytes sent per second for a topic. |
| Kafka Producer - Compression rate | kafka.producer.producer-topic-metrics.compression-rate | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. |
| Kafka Producer - Failed Requests Rate | kafka.producer.producer-topic-metrics.record-error-rate | The average per-second number of record sends that resulted in errors for a topic. |
| Kafka Producer - Requests Sent Rate | kafka.producer.producer-topic-metrics.record-send-rate | The average number of records sent per second for a topic. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Server - Purgatory Produce Size | kafka.server.purgatory.produce-delay-size | Requests waiting in the produce purgatory. |
| Kafka Server - Purgatory Fetch Size | kafka.server.purgatory.fetch-delay-size | Requests waiting in the fetch purgatory. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Consumer - Requests | kafka.consumer.consumer-metrics.request-rate | The average number of requests sent per second for a node. |
| Kafka Consumer - Request size | kafka.consumer.consumer-metrics.request-size-avg | The average size of all requests in the window. |
| Kafka Consumer - Incoming byte rate | kafka.consumer.consumer-metrics.incoming-byte-rate | Bytes/second read off all sockets. |
| Kafka Consumer - Outgoing byte rate | kafka.consumer.consumer-metrics.outgoing-byte-rate | The average number of outgoing bytes sent per second to all servers. |
| Kafka Consumer - Request latency | kafka.consumer.consumer-metrics.request-latency-avg | The average request latency in ms for a node. |
| Kafka Consumer - Messages consumed rate | kafka.consumer.consumer-metrics.records-consumed-rate | The average number of records consumed per second. |
| Kafka Consumer - Bytes consumed rate | kafka.consumer.consumer-metrics.bytes-consumed-rate | The average number of bytes consumed per second for a topic. |
| Kafka Consumer - Fetch latency | kafka.consumer.consumer-metrics.fetch-latency-avg | The average time taken for a fetch request. |
| Kafka Consumer - Consumer lag | kafka.consumer.consumer-metrics.records-lag | The latest lag of the partition. |
| Kafka Consumer - Consumer lag average | kafka.consumer.consumer-metrics.records-lag-avg | The average lag of the partition. |
| Kafka Consumer - Consumer lag maximum | kafka.consumer.consumer-metrics.records-lag-max | The max lag of the partition. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Log - Log flush 95th percentile | kafka.log.LogFlushStats.LogFlushRateAndTimeMs.Percentile95th | Log flush rate and time in milliseconds. |
| Kafka Log - Log flush mean time | kafka.log.LogFlushStats.LogFlushRateAndTimeMs.Mean | Log flush rate and time in milliseconds. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Connect - Requests | kafka.connect.connect-metrics.request-rate | The average number of requests sent per second. |
| Kafka Connect - Outgoing byte rate | kafka.connect.connect-metrics.outgoing-byte-rate | The average number of outgoing bytes sent per second to all servers. |
| Kafka Connect - Request size | kafka.connect.connect-metrics.request-size-avg | The average size of all requests in the window. |
| Kafka Connect - Incoming byte rate | kafka.connect.connect-metrics.incoming-byte-rate | Bytes/second read off all sockets. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Server - ZooKeeper disconnects | kafka.server.SessionExpireListener.ZooKeeperDisconnectsPerSec.OneMinuteRate | A meter that provides the number of recent ZooKeeper client disconnects. |
| Kafka Server - ZooKeeper expires | kafka.server.SessionExpireListener.ZooKeeperExpiresPerSec.OneMinuteRate | The number of ZooKeeper sessions that have expired. |
| Kafka Server - Zookeeper Active Connections | kafka.server.active-connections | The number of currently open connections to the broker. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Network - Produce requests per second | kafka.network.RequestMetrics.RequestsPerSec.Produce.OneMinuteRate | The total number of requests made for produce per second |
| Kafka Network - FetchConsumer requests per second | kafka.network.RequestMetrics.RequestsPerSec.FetchConsumer.OneMinuteRate | The total number of requests made for fetch consumer per second |
| Kafka Network - FetchFollower requests per second | kafka.network.RequestMetrics.RequestsPerSec.FetchFollower.OneMinuteRate | The total number of requests made for fetch follower per second |
| Kafka Network - Total time per Produce request | kafka.network.RequestMetrics.TotalTimeMs.Produce.Count | Total time, in milliseconds, spent processing requests, for produce. |
| Kafka Network - Total time per FetchConsumer request | kafka.network.RequestMetrics.TotalTimeMs.FetchConsumer.Count | Total time, in milliseconds, spent processing requests, for fetch consumer. |
| Kafka Network - Total time per FetchFollower request | kafka.network.RequestMetrics.TotalTimeMs.FetchFollower.Count | Total time, in milliseconds, spent processing requests, for pfetch follower. |
| Kafka Network - Request queue size | kafka.network.RequestChannel.RequestQueueSize.Value | Size of the request queue. |
| Metric name | Metric key | Description |
|---|---|---|
| Kafka Controller - Leader election rate | kafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.OneMinuteRate | The broker leader election rate and latency in milliseconds. This is non-zero when there are broker failures. |
| Kafka Controller - Unclean election rate | kafka.controller.ControllerStats.UncleanLeaderElectionsPerSec.OneMinuteRate | The unclean broker leader election rate. Should be 0. |
| Kafka Server - Leader count | kafka.server.ReplicaManager.LeaderCount.Value | The number of replicas for which this broker is the leader. |
Troubleshooting
- To troubleshoot this extension, use the guide(s) in the Dynatrace Community.
Related topics
Related tags
ExtensionsInfrastructure Observability
ExtensionsInfrastructure Observability