Try it free

Root cause analysis for OpenTelemetry services, frontends, and databases

  • Latest Dynatrace
  • Explanation
  • 11-min read

Causal AI, used in the latest Dynatrace root cause analysis (RCA), also covers OpenTelemetry services, frontends that call them, and databases used by them.

When directly connected components degrade because of the same issue, Dynatrace merges their alerts into one problem with the same detected root cause, instead of opening one problem per component.

This page explains how RCA covers OpenTelemetry components in new and existing environments, prerequisites needed to run it, and what you need to do to upgrade the root cause engine.

Overview

Causal AI follows the Smartscape topology on Grail data and merges alerts along the topology for components that call each other directly:

  • Frontend to service
  • Service to service
  • Service to database
  • Service to the host or process it runs on

Causal AI can merge alerts when their start timestamps are within about 15 minutes of each other. The model ranks the candidates by timing and position in the topology to identify a common root cause.

Causal AI only connects components with existing alerts. If any component in the topology doesn't raise an alert, the chain splits on that component.

Availability

Causal AI for OpenTelemetry services, frontends, and databases is available starting with SaaS version 1.349+.

For all environments in Group 3 (Enhanced), the causal AI support for OpenTelemetry is available by default.

For environments in Groups 1 and 2, causal AI works as before, but causal AI support for OpenTelemetry isn't enabled by default and there's no manual setting for it. If your environment is in Group 1 or Group 2 and runs only or mostly on OpenTelemetry, you can request causal AI support for OpenTelemetry. To do so, open a support ticket or contact your Dynatrace account team.

After you switch to the latest version of the root cause engine on environments in Groups 1 or 2, Problems app - new Problems starts showing only problems detected by the latest version of the root cause engine, while Problems Problems Classic shows only problems detected by the classic version of the root cause engine. As a result, Problems app - new Problems and Problems Problems Classic might show different problems for the same situation after the switch.

Requirements

Latest Dynatrace uses dt.smartscape.* instead of dt.entity.*, so you need to ensure that each OpenTelemetry component uses Smartscape properties.

To ensure that causal AI can identify and merge alerts raised by your OpenTelemetry components, check if your components fulfill the requirements:

  • Services: services get traced from OpenTelemetry or OneAgent, and each service raises its own alert.
  • Web and mobile frontends: Dynatrace RUM is active, with frontend-backend linking and frontend health alerts turned on. Proxies, API gateways, and OpenTelemetry Collectors forward both W3C headers, traceparent and tracestate, without changing traceparent or the Dynatrace dtr entry.
  • Self-managed databases, such as PostgreSQL, MySQL, Oracle, SQL Server, SAP HANA, MariaDB, and Db2: a Dynatrace database extension points at the same address that the application uses, and the availability health alert is active. If the extension uses a private IP address and the application uses a public one (or the reverse), causal AI can't connect the database to its health alerts.
  • Cloud databases, such as Amazon RDS, Amazon ElastiCache, Azure SQL Database, Azure Cache for Redis, and Google Cloud Firestore: you monitor the database through the new cloud connections, and health alerts are active on it.

Alert merging rules

Causal AI in latest Dynatrace operates based on a set rule list when merging alerts. These rules largely depend on the edge type, which defines causal direction.

Edge typeAlert typeMerged alert typeCausal direction
SERVICE calls SERVICESLOWDOWNSLOWDOWNDownstream (called SERVICE causes the slowdown of the calling SERVICE)
SERVICE calls SERVICEERRORERRORDownstream (called SERVICE causes the failure of the calling SERVICE)
FRONTEND calls SERVICESLOWDOWNSLOWDOWNDownstream (called SERVICE causes the slowdown of the calling FRONTEND)
FRONTEND calls SERVICEERRORERRORDownstream (called SERVICE causes the failure of the calling FRONTEND)
FRONTEND calls SERVICEAVAILABILITYERRORDownstream (called SERVICE causes the failure of the calling FRONTEND)
SERVICE runs_on HOST*1*1Downstream (HOST causes the failure of the SERVICE that runs on it)
SERVICE runs_on PROCESS*1*1Downstream (PROCESS causes the failure of the SERVICE that runs on it)
FRONTEND calls SERVICERESOURCE_CONTENTIONRESOURCE_CONTENTIONUpstream (calling FRONTEND causes the resource contention for the called SERVICE)
SERVICE calls SERVICERESOURCE_CONTENTIONRESOURCE_CONTENTIONUpstream (calling SERVICE causes the resource contention for the called SERVICE)
SERVICE runs_on HOSTRESOURCE_CONTENTIONRESOURCE_CONTENTIONUpstream (SERVICE causes the resource contention for the HOST it runs on)
SERVICE runs_on PROCESSRESOURCE_CONTENTIONRESOURCE_CONTENTIONUpstream (SERVICE causes the resource contention for the PROCESS it runs on)
SERVICE runs_on OTEL_HOSTRESOURCE_CONTENTIONRESOURCE_CONTENTIONUpstream (SERVICE causes the resource contention for the OTEL_HOST it runs on)
SERVICE runs_on OTEL_PROCESSRESOURCE_CONTENTIONRESOURCE_CONTENTIONUpstream (SERVICE causes the resource contention for the OTEL_PROCESS it runs on)
SERVICE runs_on OTEL_HOST or SERVICE runs_on OTEL_PROCESS*1*1Downstream (OTEL_HOST or OTEL_PROCESS causes the failure of the SERVICE that runs on it)
SERVICE calls DB_INSTANCE_{POSTGRES,MSSQL,MYSQL,ORACLE,HANA,MARIADB,DB2}ERRORAVAILABILITYDownstream (called DB_INSTANCE_* causes the failure of the calling SERVICE)
SERVICE calls AWS_RDS_DBINSTANCESLOWDOWNRESOURCE_CONTENTIONDownstream (called AWS_RDS_DBINSTANCE causes the resource contention for the calling SERVICE)
SERVICE calls AWS_RDS_DBCLUSTERSLOWDOWNRESOURCE_CONTENTIONDownstream (called AWS_RDS_DBCLUSTER causes the resource contention for the calling SERVICE)
SERVICE calls AWS_ELASTICACHE_CACHECLUSTERSLOWDOWNSLOWDOWNDownstream (called AWS_ELASTICACHE_CACHECLUSTER causes the slowdown of the calling SERVICE)
SERVICE calls AZURE_MICROSOFT_SQL_SERVERS_DATABASESSLOWDOWNRESOURCE_CONTENTIONDownstream (called AZURE_MICROSOFT_SQL_SERVERS_DATABASES causes the resource contention for the calling SERVICE)
SERVICE calls AZURE_MICROSOFT_CACHE_REDISERRORERRORDownstream (called AZURE_MICROSOFT_CACHE_REDIS causes the failure of the calling SERVICE)
SERVICE calls GCP_FIRESTORE_GOOGLEAPIS_COM_DATABASEERRORERRORDownstream (called GCP_FIRESTORE_GOOGLEAPIS_COM_DATABASE causes the failure of the calling SERVICE)
1

Causal AI correlates service problems with infrastructure events on which it runs. For example, suppose there's a new deployment event on the infrastructure and the service failure rate goes up. Then latest RCA diagnoses the problem (ERROR) and concludes that the service fails because of a new deployment.

CUSTOM_ALERT behavior

A custom alert can trigger the root cause analysis only if it's activated on a service (dt.smartscape_source.id should be the ID of that service) and its event type is ERROR or SLOWDOWN. If the event type is CUSTOM_ALERT, no root cause search happens.

Keep in mind that classic custom alerting events use CUSTOM_ALERT by default. If your alert is set to activate on service failures or slowdowns, we recommend setting the event type to ERROR or SLOWDOWN.

A custom alert on a host or, for example, a Kubernetes deployment doesn't start the root cause search itself and can only be merged with an event on a service.

Known limitations

Causal AI in latest Dynatrace is currently subject to the following limitations:

Not identified as a root cause

  • External services and third-party APIs. A failing provider shows up as a problem on the calling services.
  • Databases without an extension or cloud monitoring.
  • Database slowdown from warning-level signals on self-managed databases.
  • Load balancer and ingress outages.

Not merged into a single problem

  • Kubernetes containers and cloud deployments aren't merged into the service problem.
  • If a component in a topology chain raises no alert, the chain breaks on that component.

Use cases

  • Get one problem with a common root cause when a frontend, the services it calls, and their database degrade together.
  • Identify a database instance as the root cause for services that depend on it.
  • Reduce alert noise from OpenTelemetry-instrumented environments by merging related alerts.

Upgrade to the latest Dynatrace root cause engine

If your environment is in Group 1 or Group 2 and runs only or mostly on OpenTelemetry, you can request causal AI support for OpenTelemetry. To request it, open a support ticket or contact your Dynatrace account team. To make the switch smoother, prepare your environment first.

Before the latest root cause engine is active in your environment, ensure that workflow filters are replaced by their latest Dynatrace equivalent. To do so, follow the Upgrade from Classic problem notification to simple workflows guide.

We recommend using root_cause.smartscape_entity fields as a part of notification content instead of the trigger filters and using tags or smartscape.affected_entities as trigger filters instead.

For more information, see Upgrade from Classic problem notification to simple workflows.

For further upgrade details, see:

  • Problems upgrade guide
  • Upgrade from Classic problem notification to simple workflows
  • External providers upgrade guide
Related tags
Dynatrace Platform