What is Dynatrace Grail data lakehouse?

Grail is the Dynatrace database designed explicitly for observability and security data. It acts as single unified storage solution for logs, metrics, traces, events, and more. All data stored in Grail is interconnected within a real-time model that reflects the topology and dependencies within a monitored environment.

Grail availability

Grail is generally available for Dynatrace SaaS customers deployed on AWS or Azure.

Grail is schema-on-read and indexless, built with scaling in mind. There's no need to think about schema and indexes, re-hydration, or hot/cold storage. This architecture also means you are not required to determine your log data use cases beforehand (for example, at ingest). On-read parsing enables you to solve all analytics tasks on historical data stored in Grail.

Benefits of Grail:

  • Unified storage for all observability and security data
  • Single interface to query a variety of data
  • No need to define schemas and indexes
  • No re-hydration
  • No hot/cold storage

Grail is built and optimized for Dynatrace Davis® AI, which processes billions of dependencies in a time-efficient manner to provide precise anomaly detection, root cause analysis, or business impact analysis. The Dynatrace Query Language (DQL) serves as the single interface to explore, query, combine, and process all data persisted in Grail. Queries are executed in parallel and retrieve results in real time without undermining execution performance.

The cost-effective licensing model supports your storage and analytics needs. You can filter, enrich, aggregate, and preserve your data without limiting analytic capabilities.

The Grail-based data management process

The Grail-based data management and analytics process has four stages.

  1. Ingest

    Any data ingested into Dynatrace eventually ends up in Grail.

    Dynatrace forwards logs and business events into the Grail data lakehouse. The ingest endpoints receive the data and channel it as records into the processing pipeline .

  2. Process

    The processing pipeline transforms and enriches records with additional fields and determines target buckets.

  3. Store

    Records are stored in buckets. The retention period and the record type are set separately for each bucket. Each bucket is assigned to a DQL database table. You need to assign permissions to user groups or single users to provide them with access to buckets and tables.

  4. Query

    Dynatrace provides a single interface to query all kinds of data using Dynatrace Query Language (DQL), which offers a set of commands that you can use to query logs, events, and more.

    The DQL fetch command loads records from a table. Tables serve to form sets of stored records into logical groups. The predefined tables for Grail are logs and business events.

Log Management and Analytics powered by Grail technology

Log Management and Analytics for Dynatrace SaaS leverages the unique architecture of the Grail™ storage engine. Grail allows you to benefit from your log data with these innovations:

Scalability
Your log data—no matter the volume, rate of growth, or configured retention period—is always accessible for detailed analysis. Storage is managed for you automatically, without the need to manage indexes or archives.

Adjustable consumption

A three-tier log data-usage model tracks your DDU consumption across three dimensions: Ingest & Process, Retain, and Query. This approach ensures that your environment's DDU consumption is closely aligned with your usage and the value you receive. For details, see DDUs for Log Management and Analytics.

Advanced analytics
Advanced analytics gives you direct access to the log content of all your system's critical processes. You can filter your log data using the log viewer, as well as retrieve and organize your data using DQL.

Flexible retention
Choose the period of retention to match specific use-cases of log data.

  • Bucket assignment
    Logs on Grail can be stored in different buckets, where you can also choose retention periods.

Business Analytics powered by Grail technology

Dynatrace Business Analytics is a powerful tool for ingesting, exploring, and analyzing your organization's critical business data.

  • Business Analytics ensures 100% accuracy of your analyses. Events are captured without sampling, on a real-time basis, from multiple sources.
  • Business Analytics allows you to collect business events and transform them according to your needs using pipeline and DQL capabilities so that you can create the visualizations, charts, dashboards, and metrics your teams need to make informed business decisions.

Business event processing uses the capabilities of Grail and Dynatrace Query Language (DQL) to refine your data. Within the business event ingest pipeline, you can define how your data will be further processed.

Related topics