DDUs for Log Management and Analytics
The DDU consumption model outlined here for Log Management and Analytics only affects Dynatrace SaaS environments that are activated for Dynatrace Grail for log data management. DDU consumption for all other DDU capability types, including Log Monitoring for Dynatrace SaaS & Managed, remains unchanged. For details about activating Log Management and Analytics for your Dynatrace environment, contact a Dynatrace product expert via live chat within your Dynatrace environment.
This page explains how DDUs are consumed by Dynatrace Log Management and Analytics and how you can estimate and track your environment’s DDU consumption.
DDU consumption model for Log Management and Analytics
The DDU consumption model for Log Management and Analytics is based on three dimensions of data usage (Ingest & Process, Retain, and Query). The unit of measure for consumed data volume is gigabytes (GB). Each of the three data usage dimensions consumes DDUs based on a different weight (see the DDU consumption row in the table below for details).
Total DDU consumption for Log Management and Analytics is calculated by multiplying the DDU weight of each of the three data-usage dimensions with the data volume in GB.
Unit of measure | Ingest & Process | Retain | Query |
---|---|---|---|
Definition | Ingested data is the amount of raw data in bytes (logs and events) sent to Dynatrace after decompression and before enrichment and transformation. | Retained data is the amount of data saved to storage after data parsing, enrichment, transformation, and filtering but before compression. | Queried data is the data read during the execution of a DQL query, including sampled data. |
DDU consumption | 100.00 DDUs per GB | 0.30 DDUs per GB retained per day | 1.70 DDUs per GB read |
Ingest & Process
What‘s included with the Ingest & Process data-usage dimension?
Concept | Explanation |
---|---|
Data delivery |
|
Topology enrichment |
|
Data transformation |
|
Data-retention control |
|
Conversion to time series |
|
Apply the following calculation to determine your DDU consumption for the Ingest & Process data-usage dimension:
(number of GBs ingested) × (100.00 DDU weight) = DDUs consumed
Be aware that data enrichment and processing can increase your data volume significantly. Depending on the source of the data, the technology, the attributes, and metadata added during processing, the total data volume after processing can increase by a factor of 2 or more.
Retain
What’s included with the Retain data-usage dimension?
Concept | Explanation |
---|---|
Data availability |
|
Retention periods |
|
Apply the following calculation to determine your DDU consumption for the Retain data-usage dimension:
(number of GB of processed data ingested) × (retention period in days) × (0.30 DDU weight) × (number of days that data is stored) = DDUs consumed
Query
Query data usage occurs when:
Accessing the Logs & Events viewer in simple mode with filters.
Submitting custom DQL queries in the Logs & Events viewer in advanced mode.
Unified analysis pages show the log data of a specific entity.
Dashboard tiles that are based on log data trigger the execution of DQL queries on refresh and include sampled data.
Executing DQL queries via API.
Dynatrace Query Language (DQL) queries consume Davis data units (DDUs) while they are active, even when the results of such queries are not returned. To avoid unnecessary consumption of DDUs, cancel ongoing queries for logs and aggregations that are no longer necessary. Otherwise, you might be billed for incomplete queries. For details on canceling such queries in API calls, see Logs and Events viewer.
What’s included with the Query data-usage dimension?
Concept | Explanation |
---|---|
On-read parsing |
|
Aggregation |
|
Reporting |
|
Context |
|
Apply the following calculation to determine your DDU consumption for the Query data-usage dimension:
(number of GB of uncompressed data read during query execution) × (1.70 DDU weight) = DDUs consumed
Calculate DDU consumption across data-usage dimensions
Following are example DDU calculations which show how each data-usage dimension contributes to overall DDU consumption.
Step 1 – Ingest & Process
For example, say that you produce 500 GB of log data per day which you ingest into Log Management and Analytics for processing. The monthly DDU consumption for Ingest & Process in this case would be 1,500,000 DDUs:
Ingest volume per day | 500 GB | |
Ingest volume per month | 15,000 GB | 500 (GB data per day) × 30 (days) |
DDU consumption per month | 1,500,000 DDUs | 15,000 (GB per month) × 100.00 (DDUs per GB) |
Step 2 - Retain
Following the Ingest & Process step, your data is retained and enriched on an on-going basis. If you ingested 500 GB of raw data in step 1, 900 GB of enriched data (500 GB × 1.8 for enrichment) is added to your storage daily. In this example, your enriched data is retained for 35 days. The monthly DDU consumption (after a ramp-up period of 35 days) for Retain in this case is 283,500 DDUs:
Retained volume for 1 day | 900 GB | 500 (GB data per day) × 1.8 (enrichment) |
Retained volume for 35 days | 31,500 GB | 900 (GB data per day) × 35 (days) |
DDU consumption per day | 9,450 DDUs | 31,500 (GB) × 0.3 (DDUs per GB per day) |
DDU consumption per month | 283,500 DDUs | 9,450 (DDUs) × 30 (days) |
If the same amount of processed data is to be retained for a year, the monthly DDU consumption (after a ramp-up of 365 days in this case) for Retain is 2,956,500 DDUs.
Retained volume for 1 day | 900 GB | 500 (GB data per day) × 1.8 (enrichment) |
Retained volume for 365 days | 328,500 GB | 900 (GB data per day) × 365 (days) |
DDU consumption per day | 98,550 DDUs | 328,500 (GB) × 0.3 (DDUs per GB per day) |
DDU consumption per month | 2,956,500 DDUs | 98,550 (DDUs) × 30 (days) |
Step 3 - Query
Let’s assume that to resolve incidents and analyze performance issues your team executes DQL queries with a total of 25 TB of data read per day. The monthly DDU consumption for Query in this case is 1,275,000 DDUs.
Data volume read per day | 25,000 GB | |
Data volume read per month | 750,000 GB | 25,000 (GB data per day) × 30 (days) |
DDU consumption per month | 1,275,000 DDUs | 750,000 (GB per month) × 1.70 (DDUs per GB) |
Step 4 – Total DDU consumption
The total monthly DDU consumption for this example scenario of 35 days of data retention is 3,058,500 DDUs.
Ingest – DDU consumption per month | 1,500,000 DDUs |
Retain – DDU consumption per month | 283,500 DDUs |
Query – DDU consumption per month | 1,275,000 DDUs |
Total DDU consumption per month | 3,058,500 DDUs |
Frequently asked questions
Yes. The following retention periods apply to log buckets:
10 days (10 days)
2 weeks (15 days)
- 1 month (35 days) default
3 months (95 days)
1 year (372 days)
15 months (462 days)
3 years (1,102 days)
5 years (1,832 days)
7 years (2,562 days)
10 years (3,657 days)
- In the Dynatrace menu, go to Manage > Consumption. You’ll need the change monitoring settings permission or an admin account to access this page.
- On the Davis data units (DDU) page:
a. In the Consumption by DDU pool table, see the Log Monitoring row.
b. In the DDU consumption details section, go to the Log Monitoring tab.
For Log Management and Analytics, volume of data in gigabytes (GB) is the unit of measure. Consumption is based on three data-usage dimensions. The weight of DDU consumption per dimension is as follows:
Ingest & Process: 100.00 DDUs per GB of data ingested and processed
Retain: 0.30 DDUs per GB per day for data stored
Query: 1.70 DDUs per GB for data read from storage during query execution
The number and size of individual log records isn’t relevant for DDU consumption. For Log Management and Analytics, DDU consumption for a given period is calculated based on volume of data ingested and processed, volume of data stored per day, and volume of data read from storage during query execution.
No. Log Management and Analytics always consumes DDUs for Ingest & Process, Retain, and Query. Note that you can choose to only ingest and process data without storing it or querying it. Each Dynatrace environment includes a free tier of 200,000 DDUs per year, which can be used for Ingest & Process, Retain, and Query.
No. The DDU consumption model outlined here for Log Management and Analytics only affects Dynatrace SaaS environments that are activated for and connected to a Dynatrace GrailTM cluster for log data management. DDU consumption for all other DDU capability types, including Log Monitoring for Dynatrace SaaS & Managed, remains unchanged.
If Log Management and Analytics is activated for your environment, the DDU consumption model that uses gigabytes as the unit of measure will replace the event-based consumption model that’s used for Log Monitoring for Dynatrace SaaS & Managed. Consumption of other DDU-based capability types, including Log Monitoring for Dynatrace SaaS & Managed (if you continue to use it in parallel) will remain unchanged.
Yes. Every ingested GB (or fraction thereof, before enrichment and processing) is added together and then multiplied by the DDU weight of 100.00 DDUs. For example, DDU consumption for Ingest & Process of 10.50 GB equates to 1,050 DDUs. 10.50 (GB log data) × 100.00 (DDU weight) = 1,050 DDUs
While writing ingested logs to time series is a cost-effective way of visualizing log-based metrics, this approach consumes additional DDUs for creating custom metrics in addition to the DDUs consumed for Ingest & Process.
No, when internal failures occur while executing a query (for example, a time-out) no DDUs are consumed.
If you cancel a query execution, all data read before the cancellation will be factored into your DDU consumption.