Frequently asked questions about AI Observability and Dynatrace

• Latest Dynatrace
• Overview
• 8-min read
• Updated on Jul 27, 2026

This FAQ page provides answers to your most common questions about how AI Observability works within Dynatrace.

How do I instrument my services and send data?

Use the AI Observability onboarding page to configure OpenTelemetry/OpenLLMetry, define permissions, sampling strategies, and tokens. It includes scenario‑based guidance (for example, "data not in", "no access") and validation tools to confirm successful ingestion.

Which metrics are available?

Within AI Observability, use the Explorer tab to see AI performance metrics such as volume (LLM request counts), errors and success/failure rates, latency and traffic by model, tokens (input, completion, and total), cost (per model, input and output totals), and guardrails (invocation counts and provider-specific dimensions).

Select any service or model in the Explorer table to open the service detail view, which also shows service-level metrics (open problems, endpoints, vulnerabilities, running processes, pods, and clusters) and debug information (logs, prompts, traces, and additional collected telemetry).

Views are customizable, and you can add DQL-based metrics.

Behind these dashboards, the underlying data comes from three distinct sources. Knowing the source helps you troubleshoot when a metric is missing or empty:

For the full list of standard metrics, their instrument types, and which tile each one feeds, see AI Observability metrics reference.

How do guardrail metrics work? Who provides guardrail runtime protection?

Dynatrace does not execute or enforce guardrail runtime protection. Guardrail enforcement happens at the model/provider level during inference (for example, Amazon Bedrock Guardrails). These then exposes results (such as whether a guardrail intervened) via response payloads and/or provider metrics.

AI Observability instruments these provider signals and surfaces them for monitoring and analysis. This includes pre-filled alert creation flows and real‑time monitoring with customizable views.

You need to configure guardrails with your model provider. Once configured, Dynatrace ingests and displays the resulting guardrail outcomes and metrics so you can observe behavior and trends centrally.

You can build on top of these guardrail metrics in Dynatrace just like with other AI observability signals: create custom alerts and notifications, add tiles to dashboards, and trigger workflows.

How do I automate LLM evaluations in Dynatrace?

Use dt-evals, an open-source CLI, to run continuous LLM evaluations against your production AI traffic.

It reads the GenAI spans already in your Dynatrace environment, samples spans in the configured timeframe, scores them with an LLM-as-a-judge, and writes the results back to Dynatrace as bizevents. Results are queryable in DQL and visible in AI Observability linked back to their source traces.

dt-evals supports:

You can run evaluations locally on demand, schedule recurring runs, or use CI mode to gate a deployment on quality—the run fails if scores fall below your configured thresholds.

To set up dt-evals end to end, see Run LLM-as-a-judge evaluations with dt-evals.

How do I create custom LLM-as-a-judge evaluators?

You can create custom evaluators in two ways:

If your spans don't follow OTel GenAI semconv, you can also map custom span fields to the evaluator's input slots using scope.spanFields in your .dt-eval.yaml config.

How do I configure the required token scopes for dt-evals?

dt-evals requires a Dynatrace platform token with the following scopes. To create or update a token, see Platform tokens. Store the token as an environment variable (DT_API_TOKEN), or run dt-evals configure to set it up interactively.

How does dt-evals handle sensitive data in prompts?

dt-evals masks personally identifiable information (PII) in memory before sending content to the judge provider. Email addresses, phone numbers, credit card numbers, and social security numbers are redacted from prompts and responses before the judge call—the original values are never sent externally.

Evaluated prompt and response content is also excluded from the bizevents written to Dynatrace by default. You can opt in to storing it by setting storeEvaluatedPrompt: true in your .dt-eval.yaml config.

Does Dynatrace support agent frameworks and protocols?

Yes. OpenTelemetry and OpenLLMetry integrations cover:

For examples in GitHub, see Dynatrace AI Agent instrumentation examples.

Can I configure proactive alerts?

Yes. You can create custom alerts directly in AI Observability. These include context with pre‑filled fields, embed notifications (Slack/email), and link back to investigate. You also get a centralized view to review all related alerts that you and your teams have created.

What attributes do I need to add to my GenAI spans?

For dashboards to render data properly, add all GenAI Required, Conditionally required, and Recommended attributes to your spans.

For more information, see Semantic Conventions for GenAI agent and framework spans

What if I am already using ready-made dashboards?

You can continue using your existing ready‑made dashboards, which remain available until the AI Observability is generally available. The dashboards are tagged for discoverability and include navigation that redirects into AI Observability for deeper, contextual analysis. For a more integrated and centralized workflow (instrumentation guidance, insights, proactive alerts, and in‑context prompt/log/trace debugging), we recommend using AI Observability as your primary entry point, because it provides a dedicated end‑to‑end experience purpose‑built for AI workloads. Dashboards alone can be limited or inconsistent for GenAI‑specific workflows.

Does AI observability generate additional query cost?

Some out-of-the-box AI Observability dashboards use span queries, which consume Traces powered by Grail - Query. This is true even if AI Observability isn’t fully configured yet, or the dashboards show no data.

To control your trace consumption, you can:

Note that we're currently working on reducing costs for both AI Observability and Dashboards, by moving away from span queries.