Configure data storage and retention for logs

Latest Dynatrace
Tutorial
4-min read

Logs are stored in Grail buckets with a retention period from 10 days to 10 years. By default, log data is stored for 35 days in the default_logs bucket.

In this tutorial, you'll learn how to store your log data in retention buckets based on specific retention periods. We'll examine three different examples:

Retain logs for three years
Skip storage for selected logs
Manage query billing per bucket

Who is this for?

This tutorial is intended for Site Reliability Engineers (SREs) and architects who want to configure storage and retention settings for access control, optimization, or compliance purposes.

What will you learn?

In this tutorial, you'll learn how to:

Store log data in a custom bucket for a specific user group or with a longer retention period up to 10 years.
Skip storage of log data from a specific ingest source or based on matching conditions.
Manage how queries are billed for a log bucket.

Prerequisites

Permissions to manage custom Grail buckets.
You have sufficient access level to create a new pipeline or edit an existing pipeline. To learn more, see Owner-based access control in OpenPipeline.

Retain logs for three years

Using buckets can improve query performance by reducing query execution time and the scope of data read. With this procedure, you create a new bucket with a custom retention period for your log data. Log records that match the route and the pipeline conditions are stored according to the chosen bucket retention period and are readable to users based on permissions.

Using a custom log bucket, you can:

Store log data with the same retention period.
Store log data that needs to be queried and analyzed together.
Store log data that needs to be deleted at the same time.

For more information, see Best practices for Log Management and Analytics.

1. Create a custom bucket

Go to Settings > Storage management > Bucket storage management > Bucket.
Define the new bucket.
1. Enter a bucket name and the custom retention period in days.
2. From the Bucket table type dropdown list, choose logs.
Select Create.
Select (Refresh) to update the bucket list.

2. Assign log data to a bucket

Go to Settings > Process and contextualize > OpenPipeline > Logs > Pipelines.
Choose an existing pipeline or create a new one.
In the Storage stage, select Processor > Bucket assignment and define the new processor.
1. Enter the processor name and the matching condition.
2. In the Storage dropdown list, choose the bucket that you've created in the previous step
Select Save.
Ensure records are routed to your pipeline.

3. Assign bucket permissions to users

Go to Account Management, and select one of your accounts.
Set up permissions and assign bucket permissions to the right users.

Skip storage for selected logs

With this example, you skip the storage of logs that match the route and pipeline conditions. Log records are not retained.

This can be useful when you parse log lines and extract metrics, and access to original records is not needed.

To skip storage for selected log records

Go to Settings > Process and contextualize > OpenPipeline > Logs >Pipelines; choose an existing pipeline or create a new one.
In the Storage stage, select Processor > No storage assignment.
Enter the processor name and matching condition.
Select Save.
Ensure records are routed to your pipeline.

Manage query billing per bucket

There are two retention models that you can configure on a per-bucket basis:

Usage-based: Each query execution is charged separately.
Retain with Included Queries: Log data for the defined timeframe is included in the retention cost. Querying this data does not incur additional costs.

For more information, see Take control of log query costs using Retain with Included Queries.

Congratulations!

You've completed this tutorial.

Buckets are the foundation of log management—set them up right to avoid data silos and optimize retention. A few best practices can make a big difference in performance and cost. For more information, see Best practices for Log Management and Analytics.