definity extension
- Latest Dynatrace
- Extension
Monitor cloud and on-premises lakehouse platforms — including Databricks, AWS EMR, GCP Dataproc, Spark, and dbt — by ingesting pipeline metrics, traces, and events into Dynatrace.
Get started
Overview
definity is an agentic data engineering platform for the lakehouse and Spark ecosystem. It provides runtime intelligence that helps data engineering teams optimize platform costs, prevent data and job incidents, and resolve issues faster.
The definity extension brings pipeline and data observability into Dynatrace, so you can observe your data ecosystem alongside business, application, infrastructure, and AI observability.
Use cases
- Optimize platform cost and pipeline performance with actionable job-specific recommendations
- Proactively prevent job and data incidents in real time
- Resolve issues up to 10x faster with unified run-time execution context, deep lineage, and AI-powered insights
Requirements
You must have an existing definity environment with monitored data observability platforms.
The following API endpoints are monitored:
- {definity_url}/api/envs
- {definity_url}/api/envs/{env_name}/pipelines
- {definity_url}/api/envs/{env_name}/apps/{pipeline_name}/runs/{app_pit}
- {definity_url}/api/envs/{env_name}/apps/{pipeline_name}/task-runs
Activation and setup
-
Install Dynatrace Environment ActiveGate.
-
Ensure connectivity between this ActiveGate and your definity environment URL.
-
Create a definity API token with access to query all of the API endpoints listed in the Requirements.
-
Create a Dynatrace API token with
openTelemetryTrace.ingestscope, required to ingest pipeline runs as traces. -
Create a new monitoring configuration in Dynatrace. Provide your definity URL, the definity API token, the Dynatrace API token, and the definity data observability environments you want to monitor.
Details
- Unified data observability: Ingest pipeline metadata and run telemetry from Databricks, Spark, and dbt into Dynatrace for a single, centralized view.
- Unified event ingestion: Ingest data quality, pipeline health, and performance degradation incidents from definity into Dynatrace for centralized visibility and correlation.
- Pipeline health: Visualize failure counts, run duration, and cost trends with ready-to-use dashboards.
- Deep-link drill-through: Jump from Dynatrace events directly to the corresponding definity pipeline run for detailed investigation and remediation.
Licensing and cost
If all the feature sets are enabled, the number of metric data points is:
1 * # of pipelines
The number of spans is:
# of pipeline runs * (1 + tasks per pipeline)
Log ingestion varies, with log lines reported per:
- Overall pipeline run
- Each task run within a pipeline
Feature sets
When activating your extension using a monitoring configuration, you can limit monitoring to one of the feature sets. To work properly, the extension has to collect at least one metric after the activation.
In highly segmented networks, feature sets can reflect the segments of your environment. Then, when you create a monitoring configuration, you can select a feature set and a corresponding ActiveGate group that can connect to this particular segment.
All metrics that aren't categorized into any feature set are considered to be the default and are always reported.
A metric inherits the feature set of a subgroup, which in turn inherits the feature set of a group. Also, the feature set defined on the metric level overrides the feature set defined on the subgroup level, which in turn overrides the feature set defined on the group level.
Pipelines
| Metric name | Metric key | Description |
|---|---|---|
| definity pipeline properties | definity.pipeline.properties | — |
