Try it free

Forward data to cloud storage via OpenPipeline

  • Latest Dynatrace
  • How-to guide
  • 8-min read

Configure data forwarding via OpenPipeline to send records to your cloud object storage. For limits, see Forwarding.

This article explains how to create forwarding configurations, choose to forward unprocessed or processed records, and apply additional filtering to control which data is forwarded.

This article is for application owners integrating Dynatrace OpenPipeline processing and storage with company compliance standards and an established cloud‑storage strategy.

Prerequisites

  • A valid AWS S3, Azure Blob Storage, or Google Cloud Project

  • Dynatrace SaaS environment powered by Grail and AppEngine

  • DPS license capabilities required for the configuration scope you want to forward (for example, Logs powered by Grail (DPS) for logs).

  • Your configuration uses the Settings API. To learn how to migrate, see Migrate OpenPipeline configurations to Settings API.

  • Environment-level settings:objects:read and settings:objects:write permissions for the connections.aws, connections.azure, or connections.gcp schema

    Users with sufficient permissions can:

    • View existing configurations.
    • View, create, and edit forwarding configurations.
    • View hints about pipeline or ingest source data being forwarded.

  • Depending on your cloud vendor, check the following prerequisites.

    To write and connect the AWS storage you need the following permissions in the AWS Console.

    • GetBucketLocation
    • PutObject

    The user account that operates on the Azure Blob Storage has been assigned the Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write permission.

    You can either assign a predefined role, such as Storage Blob Data Contributor, or create a custom one with minimal permissions, as in the following JSON example.

    {
        "id": "/subscriptions/e1412bf7-xxxx-xxxx-xxxx-f33ea37e3427/providers/Microsoft.Authorization/roleDefinitions/d93a93fd-xxxx-xxxx-xxxx-3830043e186a",
        "properties": {
            "roleName": "Data Forwarding Role",
            "description": "",
            "assignableScopes": [
                "/subscriptions/e1412bf7-xxxx-xxxx-xxxx-f33ea37e3427"
            ],
            "permissions": [
                {
                    "actions": [],
                    "notActions": [],
                    "dataActions": [
                        "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write"
                    ],
                    "notDataActions": []
                }
            ]
        }
    }

    Grant the DPSA (Dynatrace Principal Service Account) the Service Account Token Creator role on your target GCP service account.

    This allows the DPSA to generate access tokens on behalf of your service account when forwarding data.

  • You're familiar with the terms Dynatrace Connector and object storage.

  • You know that cost is associated with object storage, and costs will occur once you start sending data, depending on the retention time and the number of requests.

Configure forwarding

Create a connection ID

  1. Access the Set up connection modal.

    • Via Settings Settings

    Go to Settings Settings > Connections > your cloud vendor ( AWS Connector AWS, Microsoft Azure Connector Microsoft Azure, or GCP ) > Connection.

    • Via OpenPipeline

      When you define the forwarding configuration, select Create a new connection.

  2. In Set up connection, enter a new connection name and select Save.

    This action generates a connection ID.

  3. Copy the connection ID.

  4. Keep the modal open.

Connect your cloud vendor

Depending on your cloud vendor, do the following.

  1. In the AWS Console, add an IAM role using the AWS account as a trusted entity and the Settings ID as external ID.

    This is the role that is assumed when using the AWS connection in Dynatrace.

    The resulting trust policy looks as follows:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::314146291599:root"
                },
                "Action": "sts:AssumeRole",
                "Condition": {
                    "StringEquals": {
                        "sts:ExternalId": "<the Connection ID in Dynatrace>"
                    }
                }
            }
        ]
    }

  2. Use identity or resource-based permissions for resource access.

  3. Once your IAM role is created and its trust policy is configured, copy your AWS Role ARN.

  1. In the Azure portal, create a new service principal.

  2. Create a new federated secret containing

    • Issuer: https://token.dynatrace.com
    • Type: Explicit subject identifier
    • Value: dt:connection-id/<your Dynatrace Connection ID>
    • Audience: <dynatrace-tenantid>.live.apps.dynatrace.com/svc-id/com.dynatrace.openpipeline

  3. Assign permissions to the service principal.

  4. Copy the directory and the application IDs.

  1. In Google Cloud Console, select your project and go to IAM & Admin > Service Accounts.

  2. Open your service account and go to Principals with access > Grant access.

  3. Paste the DPSA email in New principals and assign it the Service Account Token Creator role.

  4. Select Save.

  5. Copy your target service account email from Google Cloud Console Service Account list.

Connect Dynatrace

  1. In Dynatrace, paste the value from your clound vendor into the Set up connection modal corresponding field.
  2. Select Save.

Dynatrace immediately verifies that the correct role is being assumed when you save the connection.

Define a forwarding configuration

  1. Go to Settings Settings > Process and contextualize > OpenPipeline and select the configuration scope you want to forward.
  2. Go to Forwarding > Forward.
  3. Define the source.
    1. Enter the forwarding configuration name.
    2. Choose what you want to forward:
      • The source type: From an ingest source (unprocessed records) or From a pipeline (processed records)
      • One or multiple sources from the available ingest sources or pipelines.
    3. Enter the matching condition.
    4. Select Next.
  4. Define the destination.
    1. Select the cloud vendor.
    2. Select a connection. Select Create a new connection to create a new connection.
    3. Enter the cloud vendor storage identifier (the bucket name in AWS or GCP and the container URL in Azure).
    4. Select Next.
  5. Define the segmentation.
    1. Enter a bulk pattern.
    2. Select Next.
  6. If you want to further filter records to forward, select Add processor and configure a processor.
  7. Select Finish.

Your forwarding configuration is active by default. Records passing through or entering an ingest source or a pipeline are forwarded to cloud object storage.

Next steps

You learned how to set up a connection with your cloud vendor and how to create a new forwarding configuration in OpenPipeline. You can now start to forward unprocessed or processed records from Dynatrace to your cloud object storage.

Use the following self-monitoring metrics to observe your forwarding configuration performance.

Self-monitoring metricDescriptionDimensions

dt.sfm.openpipeline.forwarding.successful_records

The number of records successfully forwarded.

forwarding.id, forwarding.destination, forwarding.name

dt.sfm.openpipeline.forwarding.failed_records

The number of records that failed to be forwarded.

forwarding.id, forwarding.destination, forwarding.name, reason1

1

The reason dimension holds predefined values indicating possible errors, such as unauthorized, bucket_not_found, resource_unavailable, target_configuration_missing, and other.

You can query them in Notebooks Notebooks and Dashboards Dashboards.

// success
timeseries { sum=sum(dt.sfm.openpipeline.forwarding.successful_records) },
by: { forwarding.destination, forwarding.name, forwarding.id }
// failure
timeseries { sum=sum(dt.sfm.openpipeline.forwarding.failed_records) },
by: { forwarding.destination, forwarding.name, reason, forwarding.id }

Related topics

  • Forwarding data to cloud storage via OpenPipeline
Related tags
Dynatrace Platform