How to organize your data stored in Grail

Latest Dynatrace
Overview
4-min read

Grail data model

Dynatrace Grail organizes data in buckets, tables, and views to ensure efficient storage, flexible access, and scalable querying.

Buckets are the logical storage units where records are stored. Buckets are always associated with a specific record type, such as logs, events, or spans. Each record type has a predefined built-in bucket. Administrators can create custom buckets to optimize performance, apply different retention times, or meet specific compliance requirements.

Tables group records by type. Fetching a table retrieves records from all corresponding buckets. For example, the logs table includes all log records, regardless of whether they're stored in the default logs bucket or a custom one. This abstraction allows you to access data uniformly, independent of the underlying storage structure.

System tables such as dt.system.buckets, dt.system.data_objects, and dt.system.files represent information that is not stored in buckets.

Views are virtual tables defined by queries on existing tables. They provide a filtered or transformed perspective of the underlying records. For example, you can use dt.entity.* views to query classic entities.

Built-in Grail buckets

There is a set of predefined built-in buckets that cannot be modified, including:

Default buckets, whose name starts with default_
System buckets, whose name starts with dt_

Built-in buckets with corresponding retention periods

This section has been updated to align with the new Grail security events table. For the complete list of updates and actions needed to accomplish the migration, follow the steps in the Grail security table migration guide.

These are examples of built-in buckets with corresponding retention periods. For a full list of available built-in buckets, run this DQL query:

fetch dt.system.buckets
| filter startsWith(name, "default_") or startsWith(name, "dt_")

Name	Table	Retention
`default_events`	events	35 days
`default_user_events`	user.events	35 days
`default_user_sessions`	user.sessions	35 days
`default_mobile_user_replays`	mobile.user.replays	35 days
`default_web_user_replays`	web.user.replays	35 days
`default_synthetic_user_events`	synthetic.user.events	35 days
`default_synthetic_user_sessions`	synthetic.user.sessions	35 days
`default_synthetic_detailed_events`	synthetic.detailed.events	35 days
`default_synthetic_events`	synthetic.events	35 days
`default_securityevents_builtin`	security.events	3 years
`default_securityevents`	security.events	1 year
`default_bizevents`	bizevents	35 days
`default_logs`	logs	35 days
`default_metrics`	metrics	15 months
`default_spans`	spans	10 days
`dt_system_events`	dt.system.events	1 year
`default_application_snapshots`	application.snapshots	10 days

You can extend the bucket retention period after you join the preview program. Go to Preview program and request access.

Custom Grail buckets

You can create a bucket tailored to your needs. Grail buckets behave like folders in a file system and are designed for records that should be handled together. For example, you might need to store together:

Data with the same retention period
Data that needs to be queried/analyzed together
Data that needs to be deleted at the same time

Defining buckets can improve query performance by reducing query execution time and the scope of data read. Finally, having your data stored in a bucket streamlines your permission management because you can easily provide a user group or single users with access to needed data.

The default limit per environment is 80 custom buckets. The default buckets such as default_logs are not taken into account. This typically satisfies ingestion volumes up to 5TB/day per table (e.g. logs). For larger ingestion volumes, more buckets can be requested in coordination with your Dynatrace account team.

For custom buckets, the possible retention periods range from 1 day to 10 years, with an additional week.

Shortening the retention period on update requests will delete the data that is over the new period.
Any operation that deletes data is a long-running process. Deleting data can take up to a few days, depending on the amount of data you've deleted.

Manage custom Grail buckets

To manage your buckets, ensure that you have configured the following permissions:

storage:bucket-definitions:read
storage:bucket-definitions:write
storage:bucket-definitions:delete
storage:bucket-definitions:truncate

With Storage Management you can:

Create custom buckets for events, security events, bizevents, logs, and spans.
Edit custom buckets.
Delete custom buckets.

Creating new buckets with Storage Management

To create a new custom Grail bucket with Storage Management you need to specify:

Unique bucket name. It has to be between 3-100 characters long and has to start with a letter. The bucket name can only contain lowercase alphanumeric characters, underscores and hyphens. The bucket name can't be edited or changed at a later time.
Display name. You can use this field to describe your bucket.
Retention period between 1-3657 days.

Manage custom Grail buckets via REST API

To manage your custom Grail buckets via REST API

Search for and select Dynatrace API.
In the Select a definition field, select Grail Storage Management.
Authenticate with your API token.

For details, see Authentication.

Perform one of the following actions.

To do this	Go to Bucket Definitions and select this	Description
List my buckets	GET /bucket-definitions	Retrieves a list of all Grail buckets available in the environment, including default, internal, and user-defined buckets. The response contains metadata such as bucket name, associated table, retention period, and status.
Create buckets	POST /bucket-definitions	Creates a new user-defined bucket with a specified name, table type (for example, logs, events, bizevents, or spans), and retention period. This enables you to separate data streams based on retention requirements or access control needs, such as isolating audit logs for long-term compliance storage.
Update buckets	PATCH /bucket-definitions/{bucketName} or PUT /bucket-definitions/{bucketName}	Modifies the configuration of an existing user-defined bucket, such as adjusting the retention period. Note that shortening the retention period will permanently delete all data exceeding the new limit.
Truncate buckets	POST /bucket-definitions/{bucketName}:truncate	Permanently deletes all records stored in a specified bucket without removing the bucket itself. This endpoint enables you to delete your stored data—for example, from completed POCs—while allowing immediate ingestion of new data. All truncation operations are recorded in the audit log for compliance purposes. This operation is irreversible.
Delete buckets	DELETE /bucket-definitions/{bucketName}	Permanently deletes a user-defined bucket, including all data stored within it. Before deletion, the system verifies that the bucket is not actively referenced by any OpenPipeline processors. If the bucket is still in use, the request returns a `409` conflict status along with a list of the affected processors; no deletion is executed in this case. Only user-defined buckets can be deleted; default and internal system buckets prefixed with `dt_` can't be deleted. This operation is irreversible.

The delete buckets operation is irreversible. This operation will remove the content of a given bucket and then delete the bucket itself. Delete is an asynchronous task. Runtime will depend on the amount of data that has to be removed. The status of this operation can be tracked via the status field within GET bucket definitions. Status will show deleting as long as data will be drained, and finally the bucket will be deleted. Afterwards, the bucket will cease to exist. This operation can be executed on all types of buckets, except buckets where bucketName starts with dt_ or default_. Before a bucket is deleted, checks are performed to verify that the bucket is not in use. To delete a bucket, you need the storage:bucket-definitions:delete permission.

See when to create custom buckets and how to allow access to them.

Audit events for bucket management

Audit events for bucket management provide visibility into all changes to your Grail buckets. These events help you track who made changes, what was changed, and when the changes occurred.

What gets tracked?

An audit event is recorded whenever one of the following actions is performed:

Create a new bucket
Update a bucket
Truncate a bucket
Delete a bucket

Each audit event consists of the following:

The user who performed the action
Status of the action (success or failure)
A detailed description of the changes. If the bucket was updated, the event includes both the previous and the new configuration.

Access audit events

To access audit events, you can use the following DQL query. The query retrieves audit events related to bucket management and sorts them in descending order (newest events on top).

fetch dt.system.events
| filter event.kind == "AUDIT_EVENT" and event.category == "BUCKET_MANAGEMENT"
| sort timestamp desc

Use buckets to partition data

You can use buckets to partition data, which allows your teams to work efficiently. As teams have different data perspectives and requirements, the right partitioning strategy is vital for accurate ingestion, efficient queries, cost-effective retention, and secure access.

For more information, see Use Grail buckets to partition data.

What is Dynatrace Grail?