Transform and filter data with the OpenTelemetry Collector

• How-to guide
• 4-min read
• Published Aug 19, 2024

The following configuration example shows how to configure a Collector instance to transform and manipulate OTLP requests, before forwarding them to Dynatrace.

Using the processors shown in this example (filter and transform), it is possible to streamline requests before sending them to Dynatrace and omit data possibly irrelevant to your use case, and to reduce billing costs.

Prerequisites

See Collector Deployment and Collector Configuration on how to set up your Collector with the configuration below.

Demo configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  transform:
    trace_statements:
      - context: resource
        statements:
          # Only keep a certain set of resource attributes
          - keep_matching_keys(attributes, "^(aaa|bbb|ccc).*")
      - context: span
        statements:
          # Only keep a certain set of span attributes
          - keep_matching_keys(attributes, "(^xyz.pqr$)|(^(aaa|bbb|ccc).*)")
          # Set a static key
          - set(attributes["svc.marker"], "purchasing")
          # Delete a specific key
          - delete_key(attributes, "message")
          # Rewrite a key
          - set(attributes["purchase.id"], ConvertCase(attributes["purchase.id"], "upper"))
          # Apply regex replacement
          - replace_pattern(name, "^.*(DataSubmission-\d+).*$", "$$1")
    metric_statements:
      - context: metric
        statements:
          # Rename all metrics containing '_bad' suffix in their name with `_invalid`
          - replace_pattern(name, "(.*)_bad$", "$${1}_invalid")
  filter:
    error_mode: ignore
    traces:
      span:
        # Filter spans with resource attributes matching the provided regular expression
        - IsMatch(resource.attributes["k8s.pod.name"], "^my-pod-name.*")
    metrics:
      metric:
        # Filter metrics which contain at least one data point with a "bad.metric" attribute
        - 'HasAttrKeyOnDatapoint("bad.metric")'
    logs:
      log_record:
        # Filter logs with resource attributes matching the configured names
        - resource.attributes["service.name"] == "service1"
        - resource.attributes["service.name"] == "service2"

exporters:
  otlp_http:
    endpoint: ${env:DT_ENDPOINT}
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter,transform]
      exporters: [otlp_http]
    metrics:
      receivers: [otlp]
      processors: [filter]
      exporters: [otlp_http]
    logs:
      receivers: [otlp]
      processors: [filter]
      exporters: [otlp_http]

Components

For our configuration, we configure the following components.

Receiver

Under receivers, we specify the standard otlp receiver as active receiver component for our Collector instance.

This is for demonstration purposes. You can specify any other valid receiver here (for example, zipkin).

Processor

Transform

Under processors, we specify the transform processor with a set of different attribute modification statements. context indicates the scope to which the statements should apply (here, resource for resource attributes, span for span attributes, and metric for metrics).

See the OpenTelemetry documentation of the transform processor for more details on the individual configuration options.

The sample configuration above uses the following statements:

Filter

In addition, we also configure an instance of the filter processor, to filter signal based on the following criteria:

See the OpenTelemetry documentation of the filter processor for more details on the individual configuration options.

Exporter

Under exporters, we specify the default otlp_http exporter and configure it with our Dynatrace API URL and the required authentication token.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.

Service pipeline

Under service, we assemble our receiver, processor, and exporter objects into a traces pipeline, which accepts OTLP traces on the configured endpoints and transforms trace attributes according to the configured rules, before forwarding everything to Dynatrace using the exporter.

Limits and limitations

Data is ingested using the OpenTelemetry protocol (OTLP) via the Dynatrace OTLP APIs and is subject to the API's limits and restrictions. For more information see: