Try it free

Receive OpenTelemetry data with the Kafka receiver (Dynatrace Classic)

  • Dynatrace Classic
  • How-to guide
  • 3-min read

The following configuration example shows how you configure Kafka to read data from topics and relay this data via OTLP.

Prerequisites

  • A deployed and configured collector distribution, whether
    • Dynatrace Collector
    • OpenTelemetry Contrib
    • custom Builder version
  • The kafkareceiver component.
  • A Kafka server deployed with a reachable BROKER_ADDRESS. For more information, see the Kafka Apache quickstart guide.
  • The Dynatrace API endpoint URL to which the data should be exported.
  • A Classic access token with the relevant access scope. (Only for exports to SaaS and ActiveGate.)

Demo configuration

Here is an example YAML file for a basic Collector configuration that can be used to receive OpenTelemetry traces, metrics, and logs from Kafka.

receivers:
kafka:
tls:
insecure: true # Only necessary if your Kafka server does not provide a certificate that's trusted by the OTel Collector.
traces:
metrics:
logs:
brokers: ["${env:BROKER_ADDRESS}"]
exporters:
otlp_http:
endpoint: ${env:DT_ENDPOINT}
headers:
Authorization: "Api-Token ${env:DT_API_TOKEN}"
service:
pipelines:
traces:
receivers: [kafka]
exporters: [otlp_http]
metrics:
receivers: [kafka]
exporters: [otlp_http]
logs:
receivers: [kafka]
exporters: [otlp_http]

For this configuration to work, you need to set the following environment variables.

  • BROKER_ADDRESS: Specific to your Kafka server.
  • DT_ENDPOINT: The base URL of the Dynatrace API endpoint (for example, https://{your-environment-id}.live.dynatrace.com/api/v2/otlp).
  • DT_API_TOKEN: Your Classic access token.
Configuration validation

Validate your settings to avoid any configuration issues.

Components

For our configuration, we configure certain components as described in the sections below.

Receivers

Under receivers, we specify kafka as the active receiver component for our deployment. This is required to receive data from Kafka server.

Exporters

Under exporters, we specify the otlp_http exporter to forward data into Dynatrace.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.

  • DT_ENDPOINT contains the base URL of the Dynatrace API endpoint (for example, https://{your-environment-id}.live.dynatrace.com/api/v2/otlp).
  • DT_API_TOKEN contains your Classic access token with the Ingest logs (logs.ingest), Ingest metrics (metrics.ingest), and Ingest OpenTelemetry traces (openTelemetryTrace.ingest) scopes.

Service pipeline

Under service, we assemble our receiver, and exporter objects into service pipelines, which will perform these steps:

  1. Receive data from Kafka server.
  2. Export the data to Dynatrace.

Related topics

  • OTel Collector for ingesting telemetry into Dynatrace (Dynatrace Classic)
  • Buffer data via Kafka with OTel Collector (Dynatrace Classic)
  • Forward OpenTelemetry data with the Kafka exporter (Dynatrace Classic)
  • Deploy the Dynatrace OTel Collector (Dynatrace Classic)
  • Configure the OTel Collector (Dynatrace Classic)
Related tags
Application Observability