Kafka

  • Explanation
  • 1-min read

When you use Apache Kafka as the transport layer for OTLP, you add durability, scale, and flexibility to your observability pipelines.

Kafka decouples producers (OpenTelemetry Collector agents) from consumers (OpenTelemetry Collector gateways). This helps to:

  • Absorb traffic spikes with persistent buffering.
  • Survive network and backend failures.
  • Enable fan-out to multiple downstream systems.

You can reliably perform the necessary enrichment on high-volume, bursty telemetry—all while keeping your data streams protected.

Overview

You can configure Kafka exporter, Kafka receiver, and Kafka Metrics receiver components, as shown in the figure below.

  • Exporter: Write OTLP data as Kafka messages on a per-signal basis (otlp_logs, otlp_metrics, otlp_spans).
  • Receiver: Consume traces, metrics, and logs from Apache Kafka topics, and then forward these to backends such as Dynatrace.
  • Kafka metrics receiver: Collect Kafka’s own operational metrics (for example, broker count) via OTLP and export them to Dynatrace.

Note that running Kafka may introduce additional end-to-end latencies, operational overhead, and potential bottlenecks.

Communication between OpenTelemetry Collector and Kafka server

Configuration

Configure your OpenTelemetry Collectors to start sending data to/from Kafka and the Dynatrace backend.

Exporter

Configure the OpenTelemetry Collector to export OTLP data to Kafka server.

Receiver

Configure the OpenTelemetry Collector to receive OTLP data from Kafka server.

Kafka Metrics Receiver

Configure the OpenTelemetry Collector to collect Kafka operational metrics and export them to Dynatrace.

Related tags
Application Observability