Buffer data via Kafka with OTel Collector

When you use Apache Kafka as the transport layer for OTLP, you add durability, scale, and flexibility to your observability pipelines.

Kafka decouples producers (OTel Collector agents) from consumers (Collector gateways). This helps to:

You can reliably perform the necessary enrichment on high-volume, bursty telemetry—all while keeping your data streams protected.

Overview

You can configure Kafka exporter, Kafka receiver, and Kafka Metrics receiver components, as shown in the figure below.

Exporter: Write OTLP data as Kafka messages on a per-signal basis (otlp_logs, otlp_metrics, otlp_spans).
Receiver: Consume traces, metrics, and logs from Apache Kafka topics, and then forward these to backends such as Dynatrace.
Kafka metrics receiver: Collect Kafka’s own operational metrics (for example, broker count) via OTLP and export them to Dynatrace.

Note that running Kafka may introduce additional end-to-end latencies, operational overhead, and potential bottlenecks.

Communication between OTel Collector and Kafka server

Configure your Collectors to start sending data to/from Kafka and the Dynatrace backend.

Configure the Collector to export OTLP data to Kafka server.

Configure the Collector to receive OTLP data from Kafka server.

Configure the Collector to collect Kafka operational metrics and export them to Dynatrace.