Assess log and span traffic, then execute a bucket strategy if needed. Default buckets work for most environments, but high-volume environments (above 5 TB/day) or those with compliance-driven retention requirements need custom buckets to keep query performance and costs predictable.
A bucket strategy transforms Grail from a flat data store into a partitioned, cost-efficient storage layer, improving query performance, enforcing retention by compliance requirement, and making DPS consumption predictable as volume grows.
Without a bucket strategy, all signals land in the default bucket and get the same retention policy. This becomes a problem as volume grows: queries scan the full data lake instead of targeted partitions, response times degrade, and DPS consumption increases unnecessarily.
Start by assessing volume before committing to any bucket design. The assessment determines whether custom buckets are needed at all.
Enrich your observability signals: signals must be enriched with relevant dimensions (such as dt.security_context or application identifiers) before routing rules can be configured.
| Role | Involvement | Responsibility |
|---|---|---|
Dynatrace Admin | Required | Designs bucket routing and configures OpenPipeline storage processors |
Application Team Leads | Recommended | Provide input on data volumes, retention needs, and compliance requirements per application |
Security / Compliance | Optional | Defines retention mandates (for example, PCI DSS 12-month requirements) |
Calculate daily signal volume using the Data Partitioning Helper notebook to identify whether custom buckets are needed and which applications drive the most traffic.
Verify: Volume report complete with per-application breakdown; decision made on whether default buckets are sufficient or custom buckets are required.
Complete the volume assessment first: deploying with default buckets in high-volume environments (above 5 TB/day) leads to slow queries and unnecessarily high DPS consumption. Complete the assessment before committing to a bucket strategy.
Define which signals need their own bucket based on volume, retention, and compliance requirements from the assessment.
dt.security_context value or application identifier routes to which bucket.Verify: Bucket strategy document complete with retention periods, routing criteria, and compliance alignment confirmed with the Security/Compliance stakeholder where required.
Avoid over-partitioning: creating too many buckets adds operational overhead without meaningful performance or cost benefit. Focus on applications with high volume or distinct retention requirements.
Create buckets in Dynatrace and configure OpenPipeline storage processors to route traffic according to the strategy.
dt.security_context value or application identifier.Verify: All buckets created with correct retention periods; OpenPipeline storage processors active and routing test traffic to the expected buckets.
Test routing rules before enabling in production: misconfigured OpenPipeline storage processors send data to wrong buckets, breaking retention policies and access boundaries. Test routing rules before enabling them on production traffic.
Resolve enrichment gaps before configuring routing: if dt.security_context or application identifiers are missing from signals, storage processors cannot route them correctly. Resolve enrichment gaps before configuring bucket routing.
Confirm that signals are landing in the correct buckets and that query performance has improved against the pre-partitioning baseline.
Verify: Target percentage of signal traffic routed to non-default buckets; all retention policies applied correctly; query performance at or above the pre-partitioning baseline.
Stage complete when:
With data organized and retention policies in place, your environment is ready for the configuration upgrade phase. The next stage is Upgrade configurations and settings: migrate the foundational classic configurations (service naming, metrics, SLOs, and processing rules) that dashboards and alerting depend on. Complete stage 6 before starting stages 7 or 8.
The following resources support your work in this stage. Documentation covers platform concepts, configuration reference, and related guides; best practice cards provide implementation guidance from Dynatrace experts.
Configure data storage and retention for logs
Configure log bucket assignment and routing rules to direct log traffic to the correct storage buckets.
How to organize your data stored in Grail
Overview of Grail data organization, buckets, and retention management.
Best practices for data partitioning
Design a bucket strategy, calculate per-application ingest volumes, and configure OpenPipeline routing for optimal query performance.
Best practices for a logs bucket strategy
Define a logs bucket strategy by line of business, set retention policies, and optimize query performance and DPS costs.
When signal routing and retention policies are confirmed, you're ready to migrate the foundational classic configurations that dashboards and alerting both depend on. Complete this stage before starting the parallel dashboard and alerting stages; the service detection, SLO, and calculated metric changes made here are prerequisites for the stages that follow.
Continue to Upgrade configurations & settings.