Processing
Dynatrace OpenPipeline can reshape incoming data for better understanding, processing, and analysis. OpenPipeline processing is based on rules that you create and offers a flexible way of extracting value from raw records.
Key terms
- Ingest sources
Source of ingestion for a data type, collecting data from the provider into Dynatrace Platform, for example, API endpoints or OneAgent.
- Routing
Assignation of data to a pipeline, based either on matching conditions (dynamic) or direct assignation (static).
Pipeline
Once data is ingested and routed, OpenPipeline processing occurs in pipelines. Each pipeline contains a set of processing instructions (processors), that are executed in an ordered sequence of stages and define how you want Dynatrace to structure, separate, and store your data. After a record is processed, it's sent to storage and is available for further analysis.
By default data types are processed in dedicated built-in pipelines. You can create new custom pipelines to group processing and extraction by technology or team.
Log and business event processing pipeline conditions are included in the built-in OpenPipeline pipelines.
Processing is based on available records, and doesn't take into account record enrichment from external services.
Use cases
- Prepare, transform, and store data in Grail.
- Grant access to specific records.
Best practices
OpenPipeline processing for logs supports DQL only. If you use already the log processing pipeline, ensure your matchers are converted to DQL.
Stage
A stage is a phase in a pipeline sequence that focuses on a task, such as masking, filtering, processing, or extraction. Stages contain a predefined list of configurable processors, which define the task of the stage. All data types undergo the same stages, which are executed in a specific order.
The following table lists all stages available in a pipeline, ordered in the sequence of execution, specifying which processors are available and executed for each stage.
Stage | Description | Processors in the stage | Executed processors |
---|---|---|---|
Processing | Prepare data for analysis and storage by parsing values into fields, transforming the schema, and filtering the data records. Fields are edited, and sensitive data is masked. |
| All matches |
Metric extraction | Extract metrics from the records that match the query. |
| All matches |
Data extraction Logs | Extract and resend data into another pipeline. |
| All matches |
Permissions | Apply security context to the records that match the query. |
| First match only |
Storage | Assign records to the best-fit bucket. |
| First match only |
Processor
A processor is a pre-formatted processing instruction that focuses either on modifying (for example, by renaming or adding a new field) or extracting data (for example, by creating an event from a log line or extracting metrics).
While the processor format is predefined, it contains a configurable matcher and processing definition.
- The matcher defines the target of a processor via a DQL query. It narrows down the available data to the specific set you want to process.
- The processing definition instructs Dynatrace on how to transform or modify the data filtered by the matcher.
The following table lists alphabetically all available processors in a pipeline.
Processor | Description |
---|---|
Add fields | Adds fields with name and value. |
Business event | Extracts fields into a new record and sends it to the business event table. |
Counter metric | Returns the number of occurrences of a metric, from the records that match the query. |
Davis event | Extracts fields into a new record and sends it to an event table. |
DQL | Processes a subset of DQL. The output is formatted to string, number, bool, duration, timestamp, and respective arrays of those. |
No storage assignment | Skips storage assignment. |
Remove fields | Removes fields from the record. |
Rename fields | Changes the name of fields. |
Bucket assignment | Assigns a Grail bucket. |
Value metric | Returns the aggregated values of a metric from the records that match the query. |
Set dt.security_context | Sets the proper record-level access via |