Configure data storage and retention for logs
- Latest Dynatrace
- Tutorial
- 4-min read
Logs are stored in Grail buckets with a retention period from 10 days to 10 years. By default, log data is stored for 35 days in the default_logs bucket.
In this tutorial, you'll learn how to store your log data in retention buckets based on specific retention periods. We'll examine three different examples:
Who is this for?
This tutorial is intended for Site Reliability Engineers (SREs) and architects who want to configure storage and retention settings for access control, optimization, or compliance purposes.
What will you learn?
In this tutorial, you'll learn how to:
- Store log data in a custom bucket for a specific user group or with a longer retention period up to 10 years.
- Skip storage of log data from a specific ingest source or based on matching conditions.
- Manage how queries are billed for a log bucket.
Prerequisites
- Permissions to manage custom Grail buckets.
- You have sufficient access level to create a new pipeline or edit an existing pipeline. To learn more, see Owner-based access control in OpenPipeline.
Retain logs for three years
Using buckets can improve query performance by reducing query execution time and the scope of data read. With this procedure, you create a new bucket with a custom retention period for your log data. Log records that match the route and the pipeline conditions are stored according to the chosen bucket retention period and are readable to users based on permissions.
Using a custom log bucket, you can:
- Store log data with the same retention period.
- Store log data that needs to be queried and analyzed together.
- Store log data that needs to be deleted at the same time.
For more information, see Best practices for Log Management and Analytics.
1. Create a custom bucket
- Go to Settings > Storage management > Bucket storage management > Bucket.
- Define the new bucket.
- Enter a bucket name and the custom retention period in days.
- From the Bucket table type dropdown list, choose logs.
- Select Create.
- Select (Refresh) to update the bucket list.
2. Assign log data to a bucket
- Go to
Settings > Process and contextualize > OpenPipeline > Logs > Pipelines. - Choose an existing pipeline or create a new one.
- In the Bucket assignment stage, select > Bucket assignment and define the new processor.
- Enter the processor name and the matching condition.
- In the Storage dropdown list, choose the bucket that you've created in the previous step
- Select Save.
- Ensure records are routed to your pipeline.
3. Assign bucket permissions to users
- Go to Account Management, and select one of your accounts.
- Set up permissions and assign bucket permissions to the right users.
Don't store selected logs
With this example, you skip the storage of logs that match the route and pipeline conditions. Log records aren't stored.
This can be useful when you parse log lines and extract metrics, and access to original records isn't needed.
To skip storage for selected log records
- Go to Settings > Process and contextualize > OpenPipeline > Logs >Pipelines; choose an existing pipeline or create a new one.
- In the Bucket assignment stage, select > No storage assignment.
- Enter the processor name and matching condition.
- Select Save.
- Ensure records are routed to your pipeline.
When you configure No storage assignment, the record continues through all configured pipeline stages and is not stored only at the end of the pipeline. This means you can extract metrics and generate alerts from records that you won't store.
Manage query billing per bucket
There are two retention models that you can configure on a per-bucket basis:
- Usage-based: Each query execution is charged separately.
- Retain with Included Queries: Log data for the defined timeframe is included in the retention cost. Querying this data does not incur additional costs.
For more information, see Take control of log query costs using Retain with Included Queries.
Congratulations!
You've completed this tutorial.
Buckets are the foundation of log management—set them up right to avoid data silos and optimize retention. A few best practices can make a big difference in performance and cost. For more information, see Best practices for Log Management and Analytics.
OpenPipeline