Try it free

Set up external datastores

  • Latest Dynatrace
  • How-to guide
  • 8-min read

Preview

Preview availability

External datastores are available to customers enrolled in the Query and import archived data with DQL preview. For eligibility, conditions, and enrollment instructions, see the Preview program.

Preview features provide early access to new capabilities and help us refine them based on real-world use—they're not intended for production environments. Behavior, configuration options, and supported integrations may change before general availability.

An external datastore lets you query data stored in your own cloud object storage directly from Grail using DQL, without ingesting or uploading it first. Grail reads the data on demand at query time, so it stays in your cloud account under your control.

This is particularly useful when:

  • You have large historical archives that are impractical to ingest in full.
  • You want to join or enrich ingested Grail data with reference data in your cloud store.
  • Compliance or data-residency requirements prevent you from copying data outside your cloud account.
  • You want to query raw data from existing pipelines or exports without redesigning them.

Setting up an external datastore is a one-time administrative task. Once it's configured and access is granted, users can query the data using DQL external data commands just like any other Grail data source.

This page walks you through:

  • Meeting the prerequisites on the Dynatrace and cloud vendor side.
  • Setting up a connection to your cloud account.
  • Creating an external datastore in Grail.
  • Granting users access to the external datastore.

Currently, only AWS S3 buckets are supported as external datastores. Support for other hyperscalers and storage types is planned.

Prerequisites

To create and manage external datastores and connections, the following Dynatrace policies are required:

ALLOW storage:external-datastore-definitions:read;
ALLOW storage:external-datastore-definitions:write;
ALLOW storage:external-datastore-definitions:delete;

Depending on your cloud vendor, check the following additional prerequisites.

Dynatrace permissions to configure connections to AWS

To set up a connection in Dynatrace from Grail to AWS, the following Dynatrace policies are required:

ALLOW app-engine:apps:install;
ALLOW settings:objects:read, settings:objects:write, settings:schemas:read
      WHERE settings:schemaId = "builtin:hyperscaler-authentication.connections.aws";
ALLOW settings:objects:read, settings:objects:write, settings:schemas:read
      WHERE settings:schemaId = "builtin:dt-javascript-runtime.allowed-outbound-connections";
ALLOW hyperscaler-authentication:aws:authenticate;

For more information, see Access control for Connectors.

Allow outbound connections to AWS

For Dynatrace to reach your S3 bucket, you need to add the AWS domain to the external request allow-list.

  1. Go to Settings > General > External requests.
  2. Select New host pattern.
  3. Enter *.amazonaws.com and select Add.

For more information, see Add a host to the allowlist.

Permissions in AWS

You need access to your AWS account with permissions to create an IAM role and attach permission policies to it.

Create an external datastore

Step 1 Open the external datastore form

  1. Go to Settings and select Storage Management > External data storage management.

  2. Select New Data Store. A dialog opens.

  3. Enter the datastore details:

    FieldDescription

    External data store name

    Unique identifier for the datastore. Must be 3–100 characters, start with a letter, and contain only lowercase alphanumeric characters, underscores, or hyphens. Can't be changed after creation.

    Display name

    An optional descriptive name for the external data store.

    Data store type

    AWS S3 is preselected (the only supported type at this time).

    Connection

    Select Create a new connection to set up a connection inline, or reuse an existing connection.

    Bucket name

    Name of your S3 bucket (example: amzn-s3-demo-bucket1). Enter only the bucket name, not the full ARN. For access points use the access point alias.

    Region

    AWS Region your bucket resides in (example: us-east-2).

Step 2 Initiate the connection setup

  1. Within the datastore form, select Create a new connection in the Connection field. This opens up a connection panel.

  2. Enter a Name for the connection. The name must be unique across all connection types in your environment.

    Connections are specific to their use case. Grail requires its own dedicated connection — separate from connections configured for Workflows or OpenPipeline. For more on how connections are shared and administered, see Access control for Connectors.

  3. Select Create. A Connection ID is generated and displayed.

  4. Copy the Connection ID—you need it in the next step—and keep the modal open.

Step 3 Connect to your cloud vendor

Depending on your cloud vendor, do the following.

Switch to the AWS Console. You need to create an IAM role that trusts Dynatrace and attach an S3 permission policy, then copy the Role ARN back to Dynatrace.

Create an IAM role

To create a new IAM role with the Dynatrace AWS account as a trusted entity:

  1. In the AWS Console, go to IAM > Roles. Select Create role and, under Trusted entity type, select Custom trust policy.

  2. Use the following trust policy, replacing <your-connection-id> with the Connection ID you copied earlier:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::314146291599:root"
                },
                "Action": "sts:AssumeRole",
                "Condition": {
                    "StringEquals": {
                        "sts:ExternalId": "<your-connection-id>"
                    }
                }
            }
        ]
    }

  3. Select Next. You can now attach permissions to your new role. Attach a custom policy that grants only the minimum S3 permissions required for the specific bucket and prefixes that this datastore reads. Do not grant AmazonS3ReadOnlyAccess unless its broad scope is intentional.

    The following example policy allows the role to list and read objects in a single bucket. Replace <your-bucket> with your bucket name.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "s3:ListBucket",
                "Resource": "arn:aws:s3:::<your-bucket>"
            },
            {
                "Effect": "Allow",
                "Action": "s3:GetObject",
                "Resource": "arn:aws:s3:::<your-bucket>/*"
            }
        ]
    }

  4. Select Next and enter a Role name.

  5. Select Create role.

  6. On the role overview, search for and select the role you just created. Look for the ARN of the role and copy it.

Using S3 access points

If you access data through an S3 access point, with the bucket, the access point, and the IAM role all in the same AWS account, one way to set this up is:

  • In the IAM role's policy, allow s3:ListBucket on arn:aws:s3:<region>:<account-id>:accesspoint/<access-point-name> and s3:GetObject on arn:aws:s3:<region>:<account-id>:accesspoint/<access-point-name>/object/*. If you limited the policy to the bucket ARNs, as in the example above, replace them with these access point ARNs.
  • In the bucket policy, allow the IAM role to access the bucket through the access point. Allow s3:ListBucket on arn:aws:s3:::<bucket-name> and s3:GetObject on arn:aws:s3:::<bucket-name>/*, with an s3:DataAccessPointArn condition set to the access point ARN.
  • If the access point has a policy, check that it doesn't deny these actions for the IAM role.
Additional permissions for server-side encryption with AWS KMS keys (SSE-KMS)

If objects in your bucket are encrypted with SSE-KMS, the IAM role also needs permission to decrypt them:

  • In the IAM role's policy, allow kms:Decrypt on the KMS key ARN.
  • If the key is in a different account than the IAM role, or the key policy doesn't allow IAM policies to control access to the key, also allow the IAM role to perform kms:Decrypt in the key policy. The default key policy allows IAM policies to control access, so for a key in the same account, you usually don't need to change it.

For cross-account access, use a customer managed key. The AWS managed key aws/s3 can't be used across accounts. For details, see Allowing users in other accounts to use a KMS key.

You don't need to grant the Dynatrace AWS account access to the key. Only the IAM role in your account uses it.

Step 4 Complete the connection and create the datastore

  1. Return to Dynatrace. In the connection panel, paste the Role ARN into the corresponding field and select Create.

    The connection is now created, and you're brought back to the external datastore creation dialog.

  2. Select the created connection in the Connection dropdown and select Save to finish creating the external datastore.

Grant users access to an external datastore

By default, no user can access external data. You must explicitly grant access via Dynatrace IAM policies. You can reference one or more datastores by name and use wildcards:

// Listing objects in the datastore
ALLOW storage:external-data:list
      WHERE storage:store-name MATCH ("your-datastore-name", "aws-external-eu-logs-*", ...);
// Access to content
ALLOW storage:external-data:load
      WHERE storage:store-name MATCH ("your-datastore-name", "aws-external-eu-logs-*", ...);

Query the list of external datastores

You can query the metadata of your external datastores with DQL:

fetch dt.system.external_datastores

In case you don't see an external datastore in this list, make sure you have the storage:external-data:list or storage:external-data:load permissions granted.

Best practices for organizing data in your cloud store

How you lay out data in your cloud store has a direct impact on query performance and cost. Because Grail reads the data on demand at query time, a good layout lets it skip irrelevant data and read only what a query actually needs.

When you organize data in your cloud store, keep the following in mind.

File size — Aim for files that are neither too small nor too large:

  • Avoid many tiny files (roughly below 1 MB). Per-file overhead dominates and query performance suffers when a query has to open thousands of small objects.
  • A good target is single-digit MB up to about 1 GB per file. Files of about 8–16 MB work well; where the number of files becomes large, consolidating into bigger files (for example, around 200 MB) reduces that overhead.
  • Avoid very large files (for example, 100 GB). A single file that covers a wide time range hurts time selectivity, because the whole file has to be read even when a query targets a narrow time window.

Partitioning and paths — Segment data into a time-based folder or prefix structure so queries can prune data by path:

  • Include the time component in the path and order it from coarse to fine — year, then month, then day, then hour (for example, .../year=2026/month=07/day=24/...).
  • Keeping the most significant time unit first lets queries scan only the prefixes that fall inside their time range and skip the rest.
  • Choose the path time so that it reliably indicates there are no records newer than that boundary, so queries can safely skip more recent paths.

Compression — Compress files to reduce storage footprint and the volume of data transferred at query time, balancing the savings against decompression cost.

Next steps

You're now set up to load and parse external data with DQL. See DQL commands for external data for the previewExternalList and previewExternalParse commands, file listing patterns, and examples.

Related topics

  • How to organize your data stored in Grail
  • Lookup data in Grail
  • Permissions in Grail
  • DQL external data commands
  • Access control for Connectors
Related tags
Dynatrace Platform