Try it free

Forward data to cloud storage via OpenPipeline

  • Latest Dynatrace
  • How-to guide
  • 8-min read

Configure data forwarding via OpenPipeline to send records to your cloud object storage. For limits, see Forwarding.

This article explains how to create forwarding configurations, choose to forward unprocessed or processed records, and apply additional filtering to control which data is forwarded.

This article is for application owners integrating Dynatrace OpenPipeline processing and storage with company compliance standards and an established cloud‑storage strategy.

Looking for the other direction? To query data that lives in your cloud object storage directly from Grail, see external datastores.

Prerequisites

  • A valid AWS S3, Azure Blob Storage, or Google Cloud Project

  • Dynatrace SaaS environment powered by Grail and AppEngine

  • DPS license capabilities required for the configuration scope you want to forward (for example, Logs powered by Grail (DPS) for logs).

  • Environment-level settings:objects:read and settings:objects:write permissions for the connections.aws, connections.azure, or connections.gcp schema

    Users with sufficient permissions can:

    • View existing configurations.
    • View, create, and edit forwarding configurations.
    • View hints about pipeline or ingest source data being forwarded.

  • Depending on your cloud vendor, check the following prerequisites.

    To write and connect the AWS storage you need the following permissions in the AWS Console.

    • GetBucketLocation
    • PutObject

    If your S3 bucket is encrypted with a customer-managed KMS key (SSE-KMS), you also need KMS permissions. See Connect your cloud vendor.

    The user account that operates on the Azure Blob Storage has been assigned the Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write permission.

    You can either assign a predefined role, such as Storage Blob Data Contributor, or create a custom one with minimal permissions, as in the following JSON example.

    {
        "id": "/subscriptions/e1412bf7-xxxx-xxxx-xxxx-f33ea37e3427/providers/Microsoft.Authorization/roleDefinitions/d93a93fd-xxxx-xxxx-xxxx-3830043e186a",
        "properties": {
            "roleName": "Data Forwarding Role",
            "description": "",
            "assignableScopes": [
                "/subscriptions/e1412bf7-xxxx-xxxx-xxxx-f33ea37e3427"
            ],
            "permissions": [
                {
                    "actions": [],
                    "notActions": [],
                    "dataActions": [
                        "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write"
                    ],
                    "notDataActions": []
                }
            ]
        }
    }

    Grant the DPSA (Dynatrace Principal Service Account) the Service Account Token Creator role on your target GCP service account.

    This allows the DPSA to generate access tokens on behalf of your service account when forwarding data.

  • You're familiar with the terms Dynatrace Connector and object storage.

  • You know that cost is associated with object storage, and costs will occur once you start sending data, depending on the retention time and the number of requests.

Configure forwarding

Create a connection ID

  1. Access the Set up connection modal.

    • Via Settings Settings

    Go to Settings Settings > Connections > your cloud vendor ( AWS Connector AWS, Microsoft Azure Connector Microsoft Azure, or GCP ) > Connection.

    • Via OpenPipeline

      When you define the forwarding configuration, select Create a new connection.

  2. In Set up connection, enter a new connection name and select Save.

    This action generates a connection ID.

  3. Copy the connection ID.

  4. Keep the modal open.

Connect your cloud vendor

Depending on your cloud vendor, do the following.

  1. In the AWS Console, add an IAM role using the AWS account as a trusted entity and the Settings ID as external ID.

    This is the role that is assumed when using the AWS connection in Dynatrace.

    The resulting trust policy looks as follows:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::314146291599:root"
                },
                "Action": "sts:AssumeRole",
                "Condition": {
                    "StringEquals": {
                        "sts:ExternalId": "<the Connection ID in Dynatrace>"
                    }
                }
            }
        ]
    }

  2. Use identity or resource-based permissions for resource access.

  3. KMS permissions Grant KMS access to the role.

    This step is required if your S3 bucket is encrypted with a customer-managed KMS key (SSE-KMS). If your bucket doesn't use SSE-KMS, you don't need it.

    The role needs the kms:Decrypt and kms:GenerateDataKey permissions for your KMS key, and the key policy must allow the role to use the key. Without them, uploads fail with an AccessDeniedException. You configure these permissions in AWS, not in Dynatrace.

    Use one of these options:

    • Add a statement to your KMS key policy. This works in every case. Choose it if you're unsure which option applies to you.

      In the AWS Console, go to KMS > Customer managed keys, select your key, and add the following statement to the key policy.

      {
          "Sid": "Allow use of the key",
          "Effect": "Allow",
          "Principal": {
              "AWS": "<your-role-arn>"
          },
          "Action": [
              "kms:Decrypt",
              "kms:GenerateDataKey"
          ],
          "Resource": "*"
      }

    • Add KMS permissions to the IAM role policy. This only works if your key policy includes a root delegation statement ("Principal": {"AWS": "arn:aws:iam::<account-id>:root"}, "Action": "kms:*"). Without it, IAM policies have no effect on the key.

      In the AWS Console, add the following statement to the IAM policy attached to your role.

      {
          "Effect": "Allow",
          "Action": [
              "kms:Decrypt",
              "kms:GenerateDataKey"
          ],
          "Resource": "<your-kms-key-arn>"
      }

  4. Once your IAM role is created and its trust policy is configured, copy your AWS Role ARN.

  1. In the Azure portal, create a new service principal.

  2. Create a new federated credential containing

    • Scenario: Other issuer
    • Issuer: https://token.dynatrace.com
    • Value: dt:connection-id/<your Dynatrace Connection ID>
    • Audience: <dynatrace-tenantid>.live.apps.dynatrace.com/svc-id/com.dynatrace.openpipeline

  3. Copy the directory and the Application IDs.

  4. Assign permissions to the service principal.

    1. Go to Storage Account > Access control (IAM).
    2. Select Add > Add role assignment.
    3. Assign the Storage Blob Data Contributor role to the app registration. Note that the minimum required permission is Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write.

  5. Configure your storage account access settings.

    • Security + networking > Networking > Public network access: Enabled from all networks

    • Settings > Configuration > Allow Blob anonymous access: Disabled

      This setup prevents unauthenticated reads while the service principal can still write.

  6. Allow outbound traffic.

    1. Go to Settings Settings > General > External requests.

    2. Add the following host patterns:

      • management.azure.com
      • *.blob.core.windows.net

Dynatrace doesn't provide static egress IP addresses for forwarding, so restricting your storage account firewall to Dynatrace IP addresses isn't supported. The federated credential restricts access by identity instead of by network origin.

  1. In Google Cloud Console, select your project and go to IAM & Admin > Service Accounts.

  2. Open your service account and go to Principals with access > Grant access.

  3. Paste the DPSA email in New principals and assign it the Service Account Token Creator role.

  4. Select Save.

  5. Copy your target service account email from Google Cloud Console Service Account list.

Connect Dynatrace

  1. In Dynatrace, paste the value from your clound vendor into the Set up connection modal corresponding field.
  2. Select Save.

Dynatrace immediately verifies that the correct role is being assumed when you save the connection.

Define a forwarding configuration

  1. Go to Settings Settings > Process and contextualize > OpenPipeline and select the configuration scope you want to forward.
  2. Go to Forwarding > Forward.
  3. Define the source.
    1. Enter the forwarding configuration name.
    2. Choose what you want to forward:
      • The source type: From an ingest source (unprocessed records) or From a pipeline (processed records)
      • One or multiple sources from the available ingest sources or pipelines.
    3. Enter the matching condition.
    4. Select Next.
  4. Define the destination.
    1. Select the cloud vendor.
    2. Select a connection. Select Create a new connection to create a new connection.
    3. Enter the cloud vendor storage identifier (the bucket name in AWS or GCP and the container URL in Azure).
    4. Select Next.
  5. Define the segmentation.
    1. Enter a bulk pattern.
    2. Select Next.
  6. If you want to further filter records to forward, select Add processor and configure a processor.
  7. Select Finish.

Your forwarding configuration is active by default. Records passing through or entering an ingest source or a pipeline are forwarded to cloud object storage.

Next steps

You learned how to set up a connection with your cloud vendor and how to create a new forwarding configuration in OpenPipeline. You can now start to forward unprocessed or processed records from Dynatrace to your cloud object storage.

Use the self-monitoring metrics to observe

  • Successfully forwarded records (dt.sfm.openpipeline.forwarding.successful_records)
  • Failed forwarding records (dt.sfm.openpipeline.forwarding.failed_records)

You can query them via Notebooks Notebooks and Dashboards Dashboards.

// success
timeseries { sum=sum(dt.sfm.openpipeline.forwarding.successful_records) },
by: { forwarding.destination, forwarding.name, forwarding.id }
// failure
timeseries { sum=sum(dt.sfm.openpipeline.forwarding.failed_records) },
by: { forwarding.destination, forwarding.name, reason, forwarding.id }

For the full reference, including dimensions and values, see OpenPipeline self-monitoring metrics.

Use the OpenPipeline usage overview ready-made dashboard in Dashboards Dashboards to track forwarding metrics and usage events.

Related topics

  • Forwarding data to cloud storage via OpenPipeline
  • Set up external datastores
Related tags
Dynatrace Platform