Processing stage in OpenPipeline

• Latest Dynatrace
• Explanation
• 6-min read
• Updated on Jun 23, 2026

This article describes the Processing stage in OpenPipeline and the available processors. In the Processing stage, you can prepare data for analysis, extraction, forwarding, and persistence by parsing values into fields, transforming the schema, dropping data records, editing fields, and masking sensitive data.

Processors

The processors in the stage are:

The processors apply to all records matching the condition.

DQL

The DQL processor processes a subset of DQL and formats the result into string, number, bool, duration, timestamp, and respective arrays of those.

This processor uses DQL processing statements, providing high flexibility. By combining different DPL (Dynatrace Pattern Language) and DQL (Dynatrace Query Language) commands, you can use this processor for a variety of use cases, including:

Parameters

The following table describes the parameters available in the DQL processor.

Example: Perform calculations

The following DQL definition extracts the values of two temporary fields, total and failed, out of a JSON file to calculate the failure percentage and store it in a new failed.percentage attribute. Finally, it removes the temporary fields used for the calculation from the JSON file.

parse content, "LD 'total: ' INT:total '; failed: ' INT:failed"
| fieldsAdd failed.percentage = 100.0 * failed / total
| fieldsRemove total, failed

Unprocessed

{
  "content": "Lorem ipsum total: 1000; failed: 255"
}

Processed

{
  "content": "Lorem ipsum total: 1000; failed: 255",
  "failed.percentage": 25.5
}

Example: Mask data

The following example parses out the username of an email address and uses the replaceString function to replace it with a static value.

parse content, "LD 'email: ' LD:user '@'"
| fieldsAdd content = replaceString(content, user, "xxx")
| fieldsRemove user

Unprocessed

{
  "content" : "Lorem ipsum client_ip: 192.168.1.12 email: alex.example@example.com card number: 4012888888881881 server_ip: 215.131.189.194 dolor sit amet"
}

Processed

{
  "content": "Lorem ipsum client_ip: 192.168.1.12 email: xxx@example.com card number: 4012888888881881 server_ip: 215.131.189.194 dolor sit amet"
}

Example: Add fields with dynamic values

The following DQL definition adds two new top-level fields, content.length and content.words, via the fieldsAdd command. The fields store the length and number of words of the content JSON field. The values adapt to the content of the record.

The DQL definition instructs OpenPipeline to count the string length and the array size and to add the corresponding value to the dedicated fields.

fieldsAdd content.length = stringLength(content), content.words = arraySize(splitByPattern(content, "' '"))

Unprocessed

{
  "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis."
}

Processed

{
  "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis.",
  "content.length": "62",
  "content.words": "9"
}

Add fields

The Add fields processor adds fields based on the specified field name and static value. This processor doesn't leverage DQL processing statements and doesn't support dynamic values.

Parameters

The following table describes the parameters available in the Add fields processor.

Example

The following example adds two new top-level fields: company.team stores the team name and company.branch stores the branch location. It also adds their static values: sales-team and New York Sales Office.

The processor filters records that match the following condition:

filter contains(audit.identity, "@sales.example.com") AND geo.city.name == "New York"

The processor configuration specifies the fields to add and the static values.

Unprocessed

{
  "timestamp": "2026-03-18T08:30:00Z",
  "geo.city.name": "New York",
  "audit.identity": "alex.example@sales.example.com",
  "content": "Employee accessed the application from New York.",
  "logLevel": "INFO",
  "application.name": "WebApp"
}

Processed

{
  "timestamp": "2026-03-18T08:30:00Z",
  "geo.city.name": "New York",
  "audit.identity": "alex.example@sales.example.com",
  "content": "Employee accessed the application from New York.",
  "logLevel": "INFO",
  "application.name": "WebApp",
  "company.team": "sales-team",
  "company.branch": "New York Sales Office"
}

Remove fields

The Remove fields processor removes fields from the record. This processor doesn't leverage DQL processing statements.

Parameters

The following table describes the parameters available in the Remove fields processor.

Example

The following example removes the debugging.temp field. The processor defines the name of the field to remove from the record.

The processor filters records that match the following condition:

filter exists(debugging-temp)

Unprocessed

{
  "timestamp": "2023-01-01T10:00:00Z",
  "content": "Request completed",
  "debugging-temp": "Response time: 120ms"
}

Processed

{
  "timestamp": "2023-01-01T10:00:00Z",
  "content": "Request completed"
}

Rename fields

The Rename fields processor changes field names. This processor doesn't leverage DQL processing statements.

Parameters

The following table describes the parameters available in the Rename fields processor.

Example

The following example renames the OpenTelemetry fields trace_id and span_id to Dynatrace Semantic Dictionary syntax. Spans ingested via OpenTelemetry endpoints are routed to the pipeline. The processor then filters records that match the following condition:

status.code = ERROR

The processor defines the name of the field to rename and the new name.

Unprocessed

{
  "trace_id": "a1b2c3d4e5f6",
  "span_id": "1234abcd"
}

Processed

{
  "trace.id": "a1b2c3d4e5f6",
  "span.id": "1234abcd",
}

Drop records

The Drop records processor drops records that match a condition, before they're processed. Records aren't retained. This processor doesn't leverage DQL processing statements.

Parameters

The following table describes the fields available in the Drop records processor.

GeoIP lookup Early Access

The GeoIP lookup processor enriches records with geolocation attributes derived from an IP address field. Use it to map IP addresses to geographic metadata such as country, city, region, and coordinates.

Parameters

The following table describes the parameters available in the GeoIP lookup processor.

Example: Enrich logs with client and server geolocation

The following example enriches log records that contain both a client and a server IP address. Two GeoIP lookup processors run in sequence, each using a different Geo field prefix so the output fields don't overwrite each other.

The processors apply to all records that match the following condition:

isNotNull(client.ip) AND isNotNull(server.ip)

The processors define the source IP field and prefix for each IP address.

Unprocessed

{
  "timestamp": "2026-06-01T10:00:00Z",
  "content": "Request received",
  "client.ip": "192.0.2.xx",
  "server.ip": "198.51.100.xx"
}

Processed

{
  "timestamp": "2026-06-01T10:00:00Z",
  "content": "Request received",
  "client.ip": "192.0.2.xx",
  "client.geo.city.name": "Mountain View",
  "client.geo.country.iso_code": "US",
  "client.geo.country.name": "United States",
  "client.geo.location.latitude": 37.386,
  "client.geo.location.longitude": -122.0838,
  "client.geo.location.accuracy_radius": 1000,
  "server.ip": "198.51.100.xx",
  "server.geo.city.name": "Frankfurt",
  "server.geo.country.iso_code": "DE",
  "server.geo.country.name": "Germany",
  "server.geo.location.latitude": 50.1109,
  "server.geo.location.longitude": 8.6821,
  "server.geo.location.accuracy_radius": 200
}

Technology bundle Logs

The Technology bundle applies pre-built parsers to logs from supported technologies. Use it to automatically enrich logs and improve structure and metadata for downstream analysis.

Availability

The Technology bundle is available only for supported technologies within the OpenPipeline log configuration scope.

Parameters

The following table describes the parameters available in the Technology bundle processor.

Use cases