With OpenPipeline, you can ingest data in the Dynatrace platform from a wide variety of formats and providers, through ingest sources. Data is then routed to pipelines for processing, and stored in Grail buckets.
Collection of processing instructions to structure, separate, and store data.
Configuration scopes, such as logs and events, provide observability insights into the health, performance, and behavior of your system enabling teams to detect, diagnose, and resolve problems. Each configuration scope offers a different perspective because of its unique characteristics.
OpenPipeline provides a unified solution to configure ingestion and processing while ensuring flexibility based on the configuration scope specifics.
The following table lists configuration scopes, summarizing availability in OpenPipeline.
System events supported by OpenPipeline are limited to: App Lifecycle Notifications (event.kind == "AUDIT_EVENT" AND event.provider == "APP_REGISTRY"), Workflow Execution events (event.kind == "WORKFLOW_EVENT" AND event.provider == "AUTOMATION_ENGINE"), and ECC self-monitoring events (event.kind == "EXTENSIONS_EVENT").
Smartscape topology is supported for extensions and cloud providers. Host, processes, and services aren't supported yet in OpenPipeline.
Data reaches the Dynatrace platform via different ingestion sources, such OneAgent, OpenTelemetry, extensions, and other API endpoints, which collect data from data providers. In OpenPipeline, they are defined by a name and a path (dt.openpipeline.source).
Once the records reach your Dynatrace SaaS environment via ingest sources, you can route it to a pipeline.
To learn the ingest sources available in OpenPipeline, see Ingest sources in OpenPipeline.
Ingest sources can be of a built-in, ready-made, or custom type.
| Technical point | Built‑in ingest source | Ready‑made ingest source | Custom ingest source |
|---|---|---|---|
Owner | OpenPipeline | Extension | User or user group |
Permissions |
|
|
|
Access type | View‑only for all users | View‑only for all users | Owner‑based access control |
Configuration scope availability | All | Events, Logs, Metrics | Events (excluding Davis problems and Davis events) |
Routing option | Dynamic | Dynamic, static | Dynamic, static |
Pre‑processing | Not configurable | Not configurable | Configurable |
Built‑in ingest sources represent the primary data entry points provided by Dynatrace. They are owned by OpenPipeline, allowing all users with settings:read permissions to view them, but not modify or delete them. This type is always present in OpenPipeline tables, it's available for all configuration scopes, and only supports dynamic routing.
When an extension is installed, it can automatically create a ready‑made ingest source. In addition to the ingest source, the installation may also generate a ready-made pipeline and an active route. Ready‑made ingest sources are owned by the installing extension and are view‑only for all users with settings:read permissions. They remain unchanged through extension reactivations and are removed only when the extension is uninstalled. This type is available for the event, log, and metric configuration scopes. Typically, a ready-made ingest source routes records to the generated pipeline via static routing, however dynamic routing is supported as well.
Custom ingest sources are created by users, with ownership automatically assigned to them. Owners have view and edit access and can grant view access to other users or user groups, according to the user permissions. Custom ingest sources are available only within event configuration scopes, excluding Davis problems and Davis events. They support pre‑processing and allow both dynamic and static routing.
Optional data processing that transforms raw data into structured formats as soon as it reaches your Dynatrace environment.
Pre-processing runs after primary Grail tag rules and after data is ingested, but before routing.
Dynatrace has built-in pre-processing, and you can also define your own configuration.
Built-in pre-processing
Applied by Dynatrace, on selected ingest sources. Available as read-only as follows:
Pre-processing
Configured by you, on custom ingest sources only. You add processors and they run in the order you list them. Custom ingest sources are available for the generic events, SDLC events, and security events configuration scopes.
Processors in pre-processing run from top to bottom. A later processor can overwrite a field, affecting matching condition results in routes and pipelines.
A custom ingest source receives events from a provider that nests the region inside a payload object, such as payload.region, and sends a customer.id field that's eligible for a primary Grail tag rule on every ingest source. You want to route records by region, but a route can't match on that value while it's nested in the payload.
Because customer.id is eligible on every ingest source, you promote it with a primary Grail tag rule instead of a processor scoped to this one ingest source. For payload.region, which only this provider sends, you add a DQL processor in pre-processing that copies the value to a top-level region field. Routing then matches on region, and the pipelines that the records reach can use region and primary_tags.customer.id in their own processors.
Set up pre-processing to avoid creating complex matching conditions based on provider-specific data formats. This will help you streamline maintenance for routing and processing, for example, when you start ingesting data from a new provider.
OpenPipeline evaluates primary Grail tag rules for every incoming eligible record, before ingest source pre-processing and routing.
These rules promote existing fields to primary_tags.* fields, making them available as routing conditions and for all downstream processing.
After data is ingested (and optionally pre-processed), it's routed to pipelines. The route order is relevant—the position in the list establishes the order of execution. If no route matches the record, the record is routed via the Default route.
Routing is defined according to the following:
Configuration scope: Pipelines are specific to a configuration scope. Different configuration scopes are routed to different pipelines.
Ingest source: You can configure routing for each ingest source. Multiple ingest sources of the same configuration scope can be routed to the same pipeline.
Routing option
Dynamic routing: Data is routed based on a matching condition. The matching condition is a DQL query that defines the data set you want to route.
Static routing: Data is routed to a specific pipeline, which remains fixed unless manually updated. Static routing is available only for custom ingest sources.
If a record matches a different condition but you've already configured static routing for its custom ingest source, the match is skipped and data is routed directly to the pipeline you specified.
OpenPipeline processing occurs in pipelines containing instructions on how to structure, separate, and store your data. To learn more, see Processing in OpenPipeline.
Dynatrace Grail database provides a single unified storage solution for all your configuration scopes. OpenPipeline target storage are Grail buckets. You can leverage built-in buckets and, if available for the configuration scope, create new buckets with custom retention periods. Each bucket is assigned to a DQL database table. Assign permissions to user groups or single users to provide them with access to specific buckets and tables.
By default, OpenPipeline routes data into a built-in pipeline with target storage built-in Grail bucket of the configuration scope. You can configure storage assignment
Storage and retention for system events is not configurable.
Records can optionally be forwarded from OpenPipeline to your cloud object storage. You can forward unprocessed records (right after ingest) or processed records (after pipeline processing).
Forwarding is configured per configuration scope. To learn more, see Forwarding data to cloud storage via OpenPipeline.