Forwarding data to cloud storage via OpenPipeline
- Latest Dynatrace
- Explanation
- 3-min read
You can forward unprocessed or processed data from Dynatrace to your cloud object storage via OpenPipeline.
To set up forwarding, see Configure forwarding via OpenPipeline.
Looking for the other direction? To query data that lives in your cloud object storage directly from Grail, see external datastores.
Overview
Forwarding is an OpenPipeline data-flow step to forward records from OpenPipeline to supported cloud object storage for compliance, auditing, or external system integration.
You can choose to forward records
- Unprocessed—immediately after they're ingested in Dynatrace.
- Processed according to your pipeline configuration.
Forwarding configuration
Forwarding configurations define how Dynatrace should forward a set of records to your cloud object storage. Each configuration is bound to its configuration scope (such as logs, events, or metrics). You can create multiple forwarding configurations and switch on/off, modify, or delete existing ones.
Each forwarding configuration specifies whether to send processed or unprocessed records, the source, and the destination. It consists of the following:
-
Name: A unique custom identifier
-
Source: The existing ingest sources or pipelines whose records you want to forward. Depending on the source, the forwarding configuration sends unprocessed or processed records. If you choose to send from
- Ingest sources: Unprocessed records are forwarded right after ingest, before any processing is applied.
- Pipelines: Processed records are forwarded after processing is applied.
-
Matching condition: The DQL query that defines the data set you want to forward.
-
Destination: The cloud object storage to which you want to forward your records. It's defined by
-
A connection with a Dynatrace AWS or Azure Connector pointing to the cloud object storage.
-
A valid cloud vendor storage identifier (the bucket name in AWS or GCP and the container URL in Azure)
-
Compression
Records are compressed before upload using one of the following codecs:
- ZSTD (default)
- GZIP
The bulk format placeholder (see Segmentation below) must use the file extension matching the selected codec:
.json.gzfor GZIP, or.json.zstfor ZSTD. -
Segmentation
Records to forward are grouped in batches approximately either once a minute or when they reach maximum 16 MB of size, then assigned a bulk pattern and serialized as NDJSON, compressed with the configured codec. Choose one of the following file-naming presets:
-
Dynatrace optimized Recommended: a predefined bulk pattern that partitions the object path by date and includes the batch metadata placeholders, using ZSTD compression:
year=<YYYY>/month=<MM>/day=<DD>/<min-ts>-<max-ts>-<bulk-id>-<raw-size>.json.zst -
Custom: specify your own bulk pattern from the placeholders below, and pick either compression codec.
Forwarded records contain the pattern placeholders in their path in the following order:
-
Optional Date and time placeholders:
-
<DDMMYYYY>/<ddMMyyyy> -
<YYYYMMDD>/<yyyyMMdd> -
<MMDDYYYY>/<MMddyyyy> -
<YYYY>/<yyyy> -
<MM> -
<DD>/<dd> -
<HH> -
<mm> -
<ss> -
<HH:mm> -
<HHmmss.SSSS>Uppercase and lowercase variants of the same placeholder (for example
<YYYY>and<yyyy>) resolve identically.
-
-
Optional Batch metadata placeholders:
<raw-size>: Uncompressed size of the batch, in bytes<min-ts>: Earliest record timestamp in the batch, in epoch milliseconds<max-ts>: Latest record timestamp in the batch, in epoch milliseconds
-
Required Bulk identifier:
<bulk-id> -
Required Bulk format:
.json.gz(GZIP) or.json.zst(ZSTD), matching the codec selected under Compression
-
-
-
Additional optional processing: Apply further processing to records and forward only what the destination system requires. Available processors include DQL, Add fields, Remove fields, Drop, and Rename fields.
API
You can manage forwarding configurations via the Settings API POST /api/v2/settings/objects request and the data-forwarding schema of the configuration scope:
builtin:openpipeline.<configuration.scope>.data-forwarding
Replace <configuration.scope> with the scope you want to forward.
Forwarding schemas
- Business events (
bizevents) - Davis events (
davis.events) - Davis problems (
davis.problems) - Events (
events) - SDLC events (
events.sdlc) - Security events (
events.security) - Logs (
logs) - Metrics (
metrics) - Security events (
security.events) - Smartscape events (
smartscape.events) - Spans (
spans) - System events (
system.events) - User events (
user.events) - User sessions (
usersessions)
Billing and usage
Dynatrace generates billing-relevant usage events when it exports data to your external cloud storage. Use these events to track your consumption and understand charges associated with data movement. To learn more about billing, see Data Egress.
fetch dt.system.events| filter event.type == "Data Egress"| fieldsAdd config_id = concat("[", dt.openpipeline.forwarding.config_id, "](", "/ui/apps/dynatrace.settings/settings/openpipeline-logs/forwarding/", dt.openpipeline.forwarding.config_id, ")")| fieldsAdd datatype = dt.openpipeline.forwarding.datatype| fieldsRemove dt.openpipeline.forwarding.datatype| fieldsRemove dt.openpipeline.forwarding.config_id| fieldsRemove event.version| fieldsRemove event.provider| fieldsRemove dt.security_context| filter matchesValue(datatype, $Configuration)
Use the OpenPipeline usage overview ready-made dashboard in 
Dashboards to track forwarding metrics and usage events.
Use cases
- Forward records to external cloud storage for compliance, auditing, and long‑term retention.
- Maintain flexible storage options across diverse environments.
- Support staged onboarding into Dynatrace by forwarding records before or after Dynatrace processing.
- Choose to send unprocessed records after ingest or processed records before storage.
- Retain records in Grail and apply filtering to forward only what external systems require.
- Secure and control forwarding through hyperscaler credentials and fine‑grained access permissions.
- Use object storage as a drop‑off layer for seamless integration with external systems.