Dynatrace OpenPipeline can reshape incoming data for better understanding, processing, and analysis. OpenPipeline processing is based on rules that you create and offers a flexible way of extracting value from raw records.
Source of ingestion for a configuration scope, collecting data from the provider into Dynatrace Platform, for example, API endpoints or OneAgent.
Assignation of data to a pipeline, based either on matching conditions (dynamic) or direct assignation (static).
Once data is ingested and routed, OpenPipeline processing occurs in pipelines. Each pipeline contains a set of processing instructions (processors) that are executed in an ordered sequence of stages and define how Dynatrace should structure, separate, and store your data. After a record is processed, it's sent to storage and is available for further analysis.
Processing is based on available records and doesn't take into account record enrichment from external services.
Pipelines can be of a custom or built-in type.
You can create new custom pipelines and modify them to group processing and extraction according to the relevant technology or team. By adding custom pipelines per team, you can manage them via owner-based access control.
Built-in pipelines are provided out of the box. They are essential for OpenPipeline operation and generally cannot be modified within OpenPipeline. Access to these pipelines is intentionally restricted to preserve their configuration.
The default pipeline is a built-in pipeline that processes unassigned incoming data for storage. It's unique to the configuration scope and ensures that records are assigned to the default bucket and not unintentionally dropped. It's available to all users in view-only mode. It's not available for log and business event configuration scopes where the Classic pipeline is available.
Route as much data as feasible to custom pipelines using explicit matching conditions; limit the use of the default pipeline to monitoring unassigned incoming data.
The Classic pipeline is a built-in pipeline specific to the log and business event configuration scopes. It represents and serves as an entry point to the rules you set in Settings Classic for log or business event processing via the classic pipeline.
If you use log processing via the classic pipeline, migrate your rules to OpenPipeline custom pipelines.
A stage is a phase in a pipeline sequence that focuses on a task, such as masking, filtering, processing, or extraction. The sequence of stages is fixed for all pipelines and cannot be modified. Stages contain a predefined list of configurable processors, which define the task of the stage. Each stage can execute one or multiple processors.
The following table is a comprehensive list of stages, ordered in the pipeline sequence of execution, specifying which processors are available and executed for each stage, for the supported configuration scopes.
Specific fields are excluded from matching and processing or restricted. To learn more, see Limits specific to fields.
| Stage | Description | Processors in the stage | Executed processors | Supported data types |
|---|---|---|---|---|
Processing | Prepare data for analysis and storage by parsing values into fields, transforming the schema, and filtering the data records. Fields are edited, and sensitive data is masked. |
| All matches | Logs, Events—Generic, Events—Davis events, Events—Davis, Events—SDLC events, Security events (new) 1, Business events, Spans1 , Metrics, User events, User sessions |
Smartscape node | Extract Smartscape nodes for the records that match the query. |
| All matches | Logs, Events—Generic, Events—SDLC events, Security events (new)1, Business events, System events, Spans1, User events, User sessions, Metrics |
Smartscape edge | Extract Smartscape edges for the records that match the query. |
| All matches | Logs, Events—Generic, Events—SDLC events, Security events (new)1, Business events, System events, Spans1, User events, User sessions, Metrics |
Permission | Apply security context to the records that match the query. |
| First match only | Logs, Events—Generic, Events—Davis events, Events—Davis, Events—SDLC events, Security events (new)1, Business events, Spans1, Metrics, User events, User sessions |
Product allocation | Advanced option to assign product or application usage to specific records that match a query. Make sure to review Cost Allocation documentation when choosing the best approach for your environment. |
| First match only | Logs, Spans1, Metrics |
Cost allocation | Advanced option to assign cost center usage to specific records that match a query. Make sure to review Cost Allocation documentation when choosing the best approach for your environment. |
| First match only | Logs, Spans1, Metrics |
Bucket assignment | Assign records to the best-fit bucket. |
| First match only | Logs, Events—Generic, Events—Davis events, Events—Davis, Events—SDLC events, Security events (new)1, Business events, Spans1 |
Metric extraction | Extract metrics from the records that match the query. |
| All matches | Logs, Events—Generic, Events—SDLC events, Security events (new)1, Business events, System events, User events, User sessions |
Metric extraction | Extract metrics from the records that match the query. |
| All matches | Spans |
Davis | Extract a new record from a pipeline and re-ingest it as a Davis events into another pipeline. |
| All matches | Logs, Events—Generic, Events—SDLC events, Security events (new)1, Business events, System events, Spans1 |
Data extraction | Extract a new record from a pipeline and re-ingest it as a different data type into another pipeline. |
| All matches | Logs, Events—Generic, Events—SDLC events, Security events (new)1, Business events, System events, Spans1, User events, User sessions |
The data remains in its original, structured form. This is important for detailed analysis and troubleshooting, as it ensures that no information is lost or altered.
Extracted metrics are sent to Grail only, except for the security events (new) and span configuration scopes.
A processor is a pre-formatted processing instruction that either modifies the record (for example, by renaming or adding a field) or reads the record to extract data (for example, by creating an event from a log line or extracting metrics). Depending on the stage type, a processor returns either the first matching record or all matching records.
While the processor format is predefined, it contains configurable elements.
The following table lists alphabetically all available processors in a pipeline.
| Processor | Description |
|---|---|
Add fields | Adds fields with name and value. |
Bucket assignment | Assigns a Grail bucket. |
Business event | Extracts fields into a new record and sends it to the business event table. |
Counter metric | Returns the number of occurrences of a metric, from the records that match the query. |
Davis event | Extracts fields into a new record and sends it to an event table. |
DQL | Processes a subset of DQL. The output is formatted to string, number, bool, duration, timestamp, and respective arrays of those. |
DPS Cost Allocation - Cost Center | Assigns cost center usage to a record via |
DPS Cost Allocation - Product | Assigns product or application usage to a record via |
Drop record | Drops a record. The record isn't processed further and isn't stored. |
Early Access GeoIP lookup | Enriches a record with geolocation attributes derived from an IP address field, including country, city, region, and coordinates. |
Histogram metric | Produces histogram metrics that capture a distribution. Histogram metrics can be used to calculate percentiles using the |
Inline lookup | Maps a source field value to a new or updated value using an inline key-value table and writes the result to a target field. |
No storage assignment | Doesn't store the record after processing. The record continues through all configured pipeline stages and isn't stored only at the end of the pipeline. |
Sampling aware counter metric | Sampling might be applied to trace data before it's processed, according to Adaptive Traffic Management for distributed tracing. This span-specific processor supports sampling awareness when returning the number of metric occurrences, from the span records that match the query. Span aggregation and sampling awareness are configurable for all fields available in field extraction, except duration—aggregation of duration is automatically detected and handled. |
Sampling aware histogram metric | Sampling might be applied to trace data before it's processed, according to Adaptive Traffic Management for distributed tracing. This span-specific processor supports sampling awareness when producing histogram metrics that capture a distribution. Span aggregation and sampling awareness are configurable for all fields available in field extraction, except duration—aggregation of duration is automatically detected and handled. Histogram metrics can be used to calculate percentiles using the |
Sampling aware value metric | Sampling might be applied to trace data before it's processed, according to Adaptive Traffic Management for distributed tracing. This span-specific processor supports sampling awareness when returning the aggregated values of a metric, from the span records that match the query. Span aggregation and sampling awareness are configurable for all fields available in field extraction, except duration—aggregation of duration is automatically detected and handled. |
Set dt.security_context | Sets the proper record-level access via |
Smartscape node |
|
Smartscape edge | Extracts Smartscape edges and assigns specified key-value pairs for the fields: source type, source ID, edge type (pre-defined or custom), target type, and target ID. To learn more about Smartscape edges, see Smartscape on Grail. |
Software developement lifecycle event | Extracts fields into a new record and sends it to the SDLC event table. |
Technology bundle | Matches records for the selected technology and processes them according to predefined context-sensitive processing statements. |
Remove fields | Removes fields from the record. |
Rename fields | Changes the name of fields. |
Value metric | Returns the aggregated values of a metric from the records that match the query. |
Prefix a field name with dt.temp. to create a temporary field. Temporary fields are available to the subsequent processors within the same pipeline, starting from the processor that defines them. They're dropped before storage and aren't stored in Grail or forwarded to workflows. They let you pass intermediate values between processors without incurring storage costs.
To use a temporary field as a metric dimension or extraction source, reference it explicitly in the processor configuration.
dt.temp is not a temporary fielddt.temp without a trailing dot and field name (for example, dt.temp used as a field name itself) is not a temporary field and is stored normally.
Temporary fields aren't applicable to Smartscape upsert events. To learn more, see Limits.
Use temporary fields to
Parse two values from content into temporary fields in the Processing stage, calculate a result, and remove the temporary fields. The field containing the result is stored.
The DQL processor definition adds dt.temp.total and dt.temp.failed for the calculation.
parse content, "LD 'total: ' INT:dt.temp.total '; failed: ' INT:dt.temp.failed"| fieldsAdd failed.percentage = 100.0 * dt.temp.failed / dt.temp.total
The temporary fields are automatically removed and won't be stored at the end of the pipeline. Only failed.percentage is stored.
Parse a numeric value directly from log content in the Processing stage and use it as the metric value in the Metric extraction stage.
The DQL processor processing definition is
parse content, "LD 'response_time_ms=' INT:dt.temp.response_time LD"
The following Value metric processor sets dt.temp.response_time field as the metric value. At the end of the pipeline, dt.temp.response_time is removed from the record and is not stored in Grail.
When a Davis event processor uses an array field in a description template, it converts the entire array to a string automatically. If you need a specific format, such as a comma-separated list or a truncated summary, you must transform the array before the Davis stage. By using a temporary field, you can format the array in the Processing stage, reference the formatted string in the Davis event processor, and drop it before it's stored.
The DQL processor processing definition is
fieldsAdd dt.temp.affected_services = arrayJoin(affected.services, ", ")
The Davis event processor references {dt.temp.affected_services}. The formatted string is included in the Davis event. At the end of the pipeline, dt.temp.affected_services is automatically dropped and only the original affected.services array is stored in Grail.