Bucket assignment

Logs powered by Grail can be stored in different buckets. Buckets can improve query performance by reducing query execution time and the scope of data read.

Default log buckets

Grail contains built-in Grail buckets with default retention periods. The default built-in bucket intended for log data is default_logs with a retention period of 35 days.

Create and manage custom log buckets

To manage your buckets, ensure that you have configured the following permissions:

  • storage:bucket-definitions:read
  • storage:bucket-definitions:write
  • storage:bucket-definitions:delete
  • storage:bucket-definitions:truncate

With Storage Management you can:

  • Create custom buckets for Events, BizEvents, Logs and Spans. Learn more about creating new buckets.
  • Edit custom buckets.
  • Delete custom buckets.

With the Dynatrace API, you can create a custom log bucket tailored to your needs. Using a custom log bucket, you can:

  • Store log data with the same retention period.
  • Store log data that needs to be queried/analyzed together.
  • Store log data that needs to be deleted at the same time.

Defining log buckets can improve query performance by reducing query execution time and the scope of data read. Also, you can use buckets to set unique access rules to different logs or log areas.

Permissions can be assigned on the bucket, table, and entity levels. You can set up permissions and choose from six predefined global policies. For details, see:

When configuring a custom log storage retention period, this applies only to the logs that were ingested after you have created the bucket and the rule.

In scenarios where two buckets with varying retention periods can potentially store overlapping logs due to their matchers, the system ensures no duplication by prioritizing the rule that matches first. This guarantees that only the first matching rule governs the storage, preventing redundancy.

To identify the logs directed to specific buckets, use the DQL filter | filter dt.system.bucket == 'bucketName'. This filter allows for a targeted view of logs assigned to a particular bucket. However, note that the field dt.system.bucket isn't included in the default result set. To include it, employ the fields command: | fields dt.system.bucket, .

Assign log data to a bucket

To choose a log bucket for specific log data, create a log bucket rule.

  1. Go to Settings > Log monitoring > Bucket assignment.
  2. Select Add rule.
  3. Define the rule.
    • Set Rule name to the name you want to display for your rule.
    • Set Bucket to the bucket where you want to store the log data.
      • If you need buckets with specific retention times, you can create custom Grail buckets. For details, see Custom Grail buckets.
      • If you need to exclude certain logs from storage, select Don't store in a bucket. Logs excluded from storage are still processed by log events and metrics.
    • Set Matcher (DQL) to a Dynatrace Query Language (DQL) expression that describes the log data for this rule.
      • See DQL matcher in logs for details on writing a matcher that will match the ingested log data for log processing, log metrics, log events, and log buckets.
      • If you set Bucket to Don't store in a bucket, the expression needs to match the logs that you do not want to store.
  4. Select Save changes.

Once data is ingested, the bucket it is assigned to can't be changed.

Example 1: Store logs for 3 years

In this example, we need to have a 3-year retention period for the logs whose content contains the word error.

  1. Create a custom log bucket with a 3-year retention period and name it Logs 3-year retention.
    For details, see Manage custom Grail buckets.

  2. Go to Settings > Log monitoring > Bucket assignment and create a rule.

    • Set Rule name to Error log 3-year retention.
    • Set Bucket to Logs 3-year retention (the bucket created in the previous step).
    • Set Matcher (DQL) to the matcher for the ingested log data. 
      matchesPhrase(content, "error")

  3. Select Save changes.

This rule will place all ingested log data that matches the phrase error in the content in the log bucket with the 3-year retention period.

Example 2: Exclude logs

In this example, we need to exclude all logs that have the drop value in their log.source attribute.

  1. Go to Settings > Log monitoring > Bucket assignment and create a rule.
    • Set Rule name to Exclude drop logs.
    • Set Bucket to Don't store in a bucket.
    • Set Matcher (DQL) to the matcher for the ingested log data. 
      matchesValue(log.source, "drop")
  2. Select Save changes.

This rule will exclude log data that has the drop value in their log.source attribute.

Related topics