definity extension
- Latest Dynatrace
- Extension
Monitor cloud and on-premises lakehouse platforms — including Databricks, AWS EMR, GCP Dataproc, Spark, and dbt — by ingesting pipeline metrics, traces, and events into Dynatrace.
Get started
Overview
definity is an agentic data engineering platform for the lakehouse and Spark ecosystem. It provides runtime intelligence that helps data engineering teams optimize platform costs, prevent data and job incidents, and resolve issues faster.
The definity extension brings pipeline and data observability into Dynatrace, so you can observe your data ecosystem alongside business, application, infrastructure, and AI observability.
Use cases
- Optimize platform cost and pipeline performance with actionable job-specific recommendations
- Proactively prevent job and data incidents in real time
- Resolve issues up to 10x faster with unified run-time execution context, deep lineage, and AI-powered insights
Requirements
You must have an existing definity environment with monitored data observability platforms.
The following API endpoints are monitored:
- {definity_url}/api/envs
- {definity_url}/api/envs/{env_name}/pipelines
- {definity_url}/api/envs/{env_name}/apps/{pipeline_name}/runs/{app_pit}
- {definity_url}/api/envs/{env_name}/apps/{pipeline_name}/task-runs
- {definity_url}/api/envs/{env_name}/apps/{pipeline_name}/app-runs
- {definity_url}/api/envs/{env_name}/incidents
- {definity_url}/api/envs/{env_name}/lineage
- {definity_url}/api/envs/{env_name}/apps/{pipeline_name}/lineage
- {definity_url}/api/envs/{env_name}/health/runs
- {definity_url}/api/envs/{env_name}/health/incidents
- {definity_url}/api/envs/{env_name}/insights
- {definity_url}/api/envs/{env_name}/apps/{pipeline_name}/resources
- {definity_url}/api/envs/{env_name}/pipeline-events
Activation and setup
-
Install Dynatrace Environment ActiveGate.
-
Ensure connectivity between this ActiveGate and your definity environment URL.
-
Create a definity API token with access to query all of the API endpoints listed in the Requirements.
-
Create a Dynatrace API token with
openTelemetryTrace.ingestscope, required to ingest pipeline runs as traces. -
Create a new monitoring configuration in Dynatrace. Provide your definity URL, the definity API token, the Dynatrace API token, and the definity data observability environments you want to monitor.
Details
- Unified data observability: Ingest pipeline metadata and run telemetry from Databricks, Spark, and dbt into Dynatrace for a single, centralized view.
- Unified event ingestion: Ingest data quality, pipeline health, and performance degradation incidents from definity into Dynatrace for centralized visibility and correlation.
- Pipeline health: Visualize failure counts, run duration, and cost trends with ready-to-use dashboards.
- Deep-link drill-through: Jump from Dynatrace events directly to the corresponding definity pipeline run for detailed investigation and remediation.
- Automated Problem lifecycle management: Report definity pipeline failure and degradation events as Davis Problems, enabling proactive alerting and root-cause analysis directly within Dynatrace.
Licensing and costs
If all the feature sets are enabled, the number of metric data points (per definity-environment) is:
Every 5 mins: 1 * # of pipelines
Every 1 min: 12 + (1 * Unique pipeline event type seen each minute - e.g. PIPELINE_RUN_RESOLVED, TASK_RUN_FAILED, etc.) - Up to 21 total
The number of spans is:
# of pipeline runs * (1 + tasks per pipeline)
Log ingestion varies, with log lines reported per:
- Overall pipeline run
- Task run within a pipeline
- Pipeline and environment incident (logs reported at open and close times)
- Lineage relationship (pipelines datasets and input output datasets for tasks). Total dataset entries + total task input×output pairs reported every hour
- Health insight (logs reported on change of remediation status)
- Health run
Davis events are reported per:
- Pipeline event (failure and degradation incidents)
Feature sets
When activating your extension using a monitoring configuration, you can limit monitoring to one of the feature sets. To work properly, the extension has to collect at least one metric after the activation.
In highly segmented networks, feature sets can reflect the segments of your environment. Then, when you create a monitoring configuration, you can select a feature set and a corresponding ActiveGate group that can connect to this particular segment.
All metrics that aren't categorized into any feature set are considered to be the default and are always reported.
A metric inherits the feature set of a subgroup, which in turn inherits the feature set of a group. Also, the feature set defined on the metric level overrides the feature set defined on the subgroup level, which in turn overrides the feature set defined on the group level.
Pipelines
| Metric name | Metric key | Description |
|---|---|---|
| definity pipeline properties | definity.pipeline.properties | — |
| Pipeline events received per poll | definity.pipeline.events.events_received | — |
| Pipeline events received by type | definity.pipeline.events.events_by_type | — |
| Pipeline Problems opened per poll | definity.pipeline.events.problems_opened | — |
| Pipeline Problems refreshed per poll | definity.pipeline.events.problems_refreshed | — |
| Pipeline Problems closed per poll | definity.pipeline.events.problems_closed | — |
| Pipeline open episodes (gauge) | definity.pipeline.events.open_episodes | — |
| Pipeline event reporting failures per poll | definity.pipeline.events.conversion_failures | — |
| Pipeline events API errors per poll | definity.pipeline.events.api_errors | — |
| Pipeline events poll duration | definity.pipeline.events.polling_duration | — |
| Pipeline events polling lag (now minus latest event timestamp) | definity.pipeline.events.polling_lag | — |
