Configure data forwarding via OpenPipeline

Latest Dynatrace
How-to guide
10-min read
Preview

You can forward data from Dynatrace to your cloud object storage via OpenPipeline. You can choose to forward records

Unprocessed—immediately after they're ingested in Dynatrace.
Processed according to your pipeline configuration.

This article explains how to create forwarding configurations, choose to forward unprocessed or processed records, and apply additional filtering to control which data is forwarded.

This article is for application owners integrating Dynatrace OpenPipeline processing and storage with company compliance standards and an established cloud‑storage strategy.

Overview

Forwarding is an OpenPipeline data-flow step to forward unprocessed or processed records from OpenPipeline to supported cloud object storage for compliance, auditing, or external system integration.

Forwarding configuration

Forwarding configurations define how Dynatrace should forward a set of records to your cloud object storage. Each configuration is bound to its configuration scope (such as logs, events, or metrics). You can create multiple forwarding configurations and switch on/off, modify, or delete existing ones.

Each forwarding configuration specifies to send processed or unprocessed records, the source, and the destination. It consists of a

Name: A unique custom identifier
Source: The existing ingest sources or pipelines whose records you want to forward. Depending on the source, the forwarding configuration forwards unprocessed or processed records. If you choose to send from
- Ingest sources: Unprocessed records are forwarded right after ingest, before any processing is applied.
- Pipelines: Processed records are forwarded after processing is applied.
Matching condition: The DQL query that defines the data set you want to forward.
Destination: The cloud object storage to which you want to forward your records. It's defined by
- A connection with a Dynatrace AWS or Azure Connector pointing to a cloud object storage in the same region of your Dynatrace platform environment.
- A valid cloud vendor storage identifier (the bucket name in AWS or GCP and the container URL in Azure)
- Segmentation
  
  Records to forward are grouped in batches approximately either once a minute or when they reach maximum 16 MB of size. Then they are assigned a bulk pattern and are compressed to GZIP NDJSON format. Forwarded records contain the pattern placeholders in their path in the following order:
  - Optional Date and time placeholders:
    
    <DDMMYYYY>
    <YYYYMMDD>
    <HH>
    <HHmmss.SSSS>
  - Required Bulk identifier: <bulk-id>
  - Required Bulk format: .json.gz
    
    If .gz is omitted, GZIP is still executed, and the file has a non-matching file extension.
Additional optional processing: Apply further processing to records and forward only what the destination system requires. Available processors include DQL, Add fields, Remove fields, Drop, and Rename fields.

API

You can manage forwarding configurations via the Settings API POST /api/v2/settings/objects request and the data-forwarding schema of the configuration scope:

builtin:openpipeline.<configuration.scope>.data-forwarding

Replace <configuration.scope> with the scope you want to forward.

Data forwarding schemas

Use cases

Forward records to external cloud storage for compliance, auditing, and long‑term retention.
Maintain flexible storage options across diverse environments.
Support staged onboarding into Dynatrace by forwarding records before or after Dynatrace processing.
Choose to send unprocessed records after ingest or processed records before storage.
Retain records in Grail and apply filtering to forward only what external systems require.
Secure and control forwarding through hyperscaler credentials and fine‑grained access permissions.

Prerequisites

Dynatrace SaaS environment powered by Grail and AppEngine
DPS license capabilities required for the configuration scope you want to forward (for example, Logs powered by Grail (DPS) for logs).
Your configuration uses the Settings API. To learn how to migrate, see Migrate OpenPipeline configurations to Settings API.
Forwarding to Cloud Storage preview enrollement
Environment-level settings:objects:read and settings:objects:write permissions for the connections.aws, connections.azure, or connections.gcp schema

Users with sufficient permissions can:
- View existing configurations.
- View, create, and edit forwarding configurations.
- View hints about pipeline or ingest source data being forwarded.
A valid AWS S3, Azure Blob Storage, or Google Cloud Project in the same region of your Dynatrace platform environment

Depending on your cloud vendor, check the following prerequisites.

Amazon Web Services (AWS)

To write and connect AWS storage you need the following permissions in the AWS Console.
- GetBucketLocation
- PutObject

Microsoft Azure

The user account that operates on the Azure Blob Storage has been assigned the Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write permission.

You can either assign a predefined role, such as Storage Blob Data Contributor, or create a custom one with minimal permissions, as in the following JSON example.

{
    "id": "/subscriptions/e1412bf7-xxxx-xxxx-xxxx-f33ea37e3427/providers/Microsoft.Authorization/roleDefinitions/d93a93fd-xxxx-xxxx-xxxx-3830043e186a",
    "properties": {
        "roleName": "Data Forwarding Role",
        "description": "",
        "assignableScopes": [
            "/subscriptions/e1412bf7-xxxx-xxxx-xxxx-f33ea37e3427"
        ],
        "permissions": [
            {
                "actions": [],
                "notActions": [],
                "dataActions": [
                    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write"
                ],
                "notDataActions": []
            }
        ]
    }
}

Google Cloud Platform (GCP)

Grant the DPSA (Dynatrace Principal Service Account) the Service Account Token Creator role on your target GCP service account.

This allows the DPSA to generate access tokens on behalf of your service account when forwarding data.

You're familiar with the terms Dynatrace Connector and object storage.
You know that cost is associated with object storage, and costs will occur once you start sending data, depending on the retention time and the number of requests.

How-to

Create a connection ID

Access the Set up connection modal.
- Via Settings
Go to Settings > Connections > your cloud vendor ( AWS, Microsoft Azure, or GCP ) > Connection.
- Via OpenPipeline
  
  When you define the forwarding configuration, select Create a new connection.
In Set up connection, enter a new connection name and select Save.

This action generates a connection ID.
Copy the connection ID.
Keep the modal open.

Connect your cloud vendor

Depending on your cloud vendor, do the following.

In the AWS Console, add an IAM role using the AWS account as a trusted entity and the Settings ID as external ID.

This is the role that is assumed when using the AWS connection in Dynatrace.

The resulting trust policy looks as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::314146291599:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<the Connection ID in Dynatrace>"
                }
            }
        }
    ]
}

Use identity or resource-based permissions for resource access.
Once your IAM role is created and its trust policy is configured, copy your AWS Role ARN.

In the Azure portal, create a new service principal.
Create a new federated secret containing
- Issuer: https://token.dynatrace.com
- Type: Explicit subject identifier
- Value: dt:connection-id/<your Dynatrace Connection ID>
- Audience: <dynatrace-tenantid>.live.apps.dynatrace.com/svc-id/com.dynatrace.openpipeline
Assign permissions to the service principal.
Copy the directory and the application IDs.

In Google Cloud Console, select your project and go to IAM & Admin > Service Accounts.
Open your service account and go to Principals with access > Grant access.
Paste the DPSA email in New principals and assign it the Service Account Token Creator role.
Select Save.
Copy your target service account email from Google Cloud Console Service Account list.

Connect Dynatrace

In Dynatrace, paste the value from your clound vendor into the Set up connection modal corresponding field.
Select Save.

Dynatrace immediately verifies that the correct role is being assumed when you save the connection.

Define a forwarding configuration

Go to Settings > Process and contextualize > OpenPipeline and select the configuration scope you want to forward.
Go to Forwarding > Forward.
Define the source.
1. Enter the forwarding configuration name.
2. Choose what you want to forward:
  - The source type: From an ingest source (unprocessed records) or From a pipeline (processed records)
  - One or multiple sources from the available ingest sources or pipelines.
3. Enter the matching condition.
4. Select Next.
Define the destination.
1. Select the cloud vendor.
2. Select a connection. Select Create a new connection to create a new connection.
3. Enter the cloud vendor storage identifier (the bucket name in AWS or GCP and the container URL in Azure).
4. Select Next.
Define the segmentation.
1. Enter a bulk pattern.
2. Select Next.
If you want to further filter records to forward, select Add processor and configure a processor.
Select Finish.

Your forwarding configuration is active by default. Records passing through or entering an ingest source or a pipeline are forwarded to cloud object storage.

Next steps

You learned how to set up a connection with your cloud vendor and how to create a new forwarding configuration in OpenPipeline. You can now start to forward unprocessed or processed records from Dynatrace to your cloud object storage.

Use the following self-monitoring metrics to observe your forwarding configuration performance.

Self-monitoring metric	Description	Dimensions
`dt.sfm.openpipeline.forwarding.successful_records`	The number of records successfully forwarded.	`forwarding.id`, `forwarding.destination`, `forwarding.name`
`dt.sfm.openpipeline.forwarding.failed_records`	The number of records that failed to be forwarded.	`forwarding.id`, `forwarding.destination`, `forwarding.name`, `reason`¹

The reason dimension holds predefined values indicating possible errors, such as unauthorized, bucket_not_found, resource_unavailable, target_configuration_missing, and other.

You can query them in Notebooks and Dashboards.

// success
timeseries { sum=sum(dt.sfm.openpipeline.forwarding.successful_records) },
by: { forwarding.destination, forwarding.name, forwarding.id }

// failure
timeseries { sum=sum(dt.sfm.openpipeline.forwarding.failed_records) },
by: { forwarding.destination, forwarding.name, reason, forwarding.id }

Limits

Supported cloud vendors are limited to AWS S3 and Azure Blob Storage.
Preview Records can be forwarded only to cloud vendors in the same region of your Dynatrace platform environment.
The maximum number of configurations is 1,000 per data type.
If the cloud storage is not reachable, Dynatrace automatically drops the record.

Records are forwarded in bulks.
- The bulk pattern must contain the bulk identifier (<bulk-id>) and end with the bulk format (.json.gz).
Dynatrace forwards unencrypted records in NDJSON-GZIP file format.