Try it free

Receive OpenTelemetry data with the Kafka receiver

  • Latest Dynatrace
  • How-to guide
  • 3-min read

The following configuration example shows how you configure Kafka to read data from topics and relay this data via OTLP.

Prerequisites

  • A deployed and configured collector distribution, whether
    • Dynatrace Collector
    • OpenTelemetry Contrib
    • custom Builder version
  • The kafkareceiver component.
  • A Kafka server deployed with a reachable BROKER_ADDRESS. For more information, see the Kafka Apache quickstart guide.
  • The Dynatrace API endpoint URL to which the data should be exported.
  • An API token with the relevant access scope. (Only for exports to SaaS and ActiveGate.)

Demo configuration

Here is an example YAML file for a basic Collector configuration that can be used to receive OpenTelemetry traces, metrics, and logs from Kafka.

receivers:
  kafka:
    tls:
      insecure: true # Only necessary if your Kafka server does not provide a certificate that's trusted by the OTel Collector.
    traces:
    metrics:
    logs:
    brokers: ["${env:BROKER_ADDRESS}"]


exporters:
  otlp_http:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Bearer ${env:DT_PLATFORM_TOKEN}"


service:
  pipelines:
    traces:
      receivers: [kafka]
      exporters: [otlp_http]
    metrics:
      receivers: [kafka]
      exporters: [otlp_http]
    logs:
      receivers: [kafka]
      exporters: [otlp_http]

Assign the scope that matches your signal type: openpipeline:logs:ingest for logs, openpipeline:metrics:ingest for metrics, or openpipeline:traces:ingest for traces.

receivers:
  kafka:
    tls:
      insecure: true # Only necessary if your Kafka server does not provide a certificate that's trusted by the OTel Collector.
    traces:
    metrics:
    logs:
    brokers: ["${env:BROKER_ADDRESS}"]


exporters:
  otlp_http:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"


service:
  pipelines:
    traces:
      receivers: [kafka]
      exporters: [otlp_http]
    metrics:
      receivers: [kafka]
      exporters: [otlp_http]
    logs:
      receivers: [kafka]
      exporters: [otlp_http]

For this configuration to work, you need to set the following environment variables.

  • BROKER_ADDRESS: Specific to your Kafka server.
  • DT_ENDPOINT: The base URL of the Dynatrace API endpoint (for example, https://{your-environment-id}.live.dynatrace.com/api/v2/otlp).
  • DT_PLATFORM_TOKEN or DT_API_TOKEN: Your platform token or Classic access token, depending on the token type you use.
Configuration validation

Validate your settings to avoid any configuration issues.

Components

For our configuration, we configure certain components as described in the sections below.

Receivers

Under receivers, we specify kafka as the active receiver component for our deployment. This is required to receive data from Kafka server.

Exporters

Under exporters, we specify the otlp_http exporter to forward data into Dynatrace.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.

  • DT_ENDPOINT contains the base URL of the Dynatrace API endpoint (for example, https://{your-environment-id}.live.dynatrace.com/api/v2/otlp).

  • One token variable, depending on the token type you use:

    • Platform token: DT_PLATFORM_TOKEN contains your platform token with the openpipeline:logs:ingest, openpipeline:metrics:ingest, and openpipeline:traces:ingest scopes.
    • Classic access token: DT_API_TOKEN contains your Classic access token with the Ingest logs (logs.ingest), Ingest metrics (metrics.ingest), and Ingest OpenTelemetry traces (openTelemetryTrace.ingest) scopes.

Service pipeline

Under service, we assemble our receiver, and exporter objects into service pipelines, which will perform these steps:

  1. Receive data from Kafka server.
  2. Export the data to Dynatrace.

Related topics

  • OTel Collector for ingesting telemetry into Dynatrace
  • Buffer data via Kafka with OTel Collector
  • Forward OpenTelemetry data with the Kafka exporter
  • Deploy the Dynatrace OTel Collector
  • Configure the OTel Collector
Related tags
Application Observability