Terms and concepts about AI Observability and GenAI in Dynatrace

• Latest Dynatrace
• Explanation
• 3-min read
• Updated on Jul 27, 2026

Dynatrace supports GenAI Observability by troubleshooting conversation issues, performance, costs, and correlating data throughout the product for effectively resolving issues in Large Language Models (LLMs) or traditional Machine Learning (ML).

This page provides definitions and explanations of the key terms used in our documentation.

Retrieval-Augmented Generation (RAG)

RAG is an AI technique that enhances the performance of LLMs by retrieving relevant documents or information from external sources. The retrieved data is used to enhance the input to the LLM during text generation.

Unlike traditional LLMs that rely solely on internal training data, RAG leverages real-time information to deliver more accurate, up-to-date, and contextually relevant responses.

Agents

In the context of AI and language models, an agent is an autonomous or semi-autonomous entity designed to perform specific tasks or solve problems.

Agents perform actions on behalf of users, professionals, or other systems, often based on received inputs or objectives. These agents can operate with varying levels of independence and intelligence, making them suitable for complex decision-making tasks, often powered by a language model.

Agentic

An agentic system utilizes intelligent agents that manage or request specific RAG tasks in real-time, enhancing control over the retrieval process.

In an agentic system, several agents work together to address complex queries. These agents assess the relevance of information dynamically, prioritize it, and modify the generation process based on evolving contexts.

Guardrails

Guardrails are runtime controls that help keep AI and Agentic applications safe, compliant, and predictable by detecting and handling unsafe or disallowed behavior (for example, policy-violating content, sensitive data/PII exposure, or abuse patterns). For AI observability, guardrails are critical because they explain why a model interaction was blocked, modified (redacted/truncated), or allowed. This turns a “the model failed” error into an actionable, policy-driven root cause that you can investigate and tune.

We recommend enabling provider-native guardrails as close to the model as possible (for example, Amazon Bedrock guardrails, Azure OpenAI safety filters, and OpenAI safety mechanisms) and emitting their outcomes via OpenTelemetry. OpenLLMetry captures guardrail data on the relevant GenAI spans using the OpenTelemetry GenAI semantic conventions (for example, outcome, category/reason, action taken, and provider/system), and Dynatrace consumes these exposed attributes/metrics through OpenTelemetry ingestion so you can filter and correlate guardrail-triggered requests with latency, token usage, and cost, and build dashboards/alerts on trends like rising blocks or redactions.

Evaluations (evals)

An evaluation (eval) is a systematic process of testing a model or AI-powered system to determine whether it meets a defined expectation: a reference answer, a policy, a functional check, or a quality rubric.

Unlike performance signals such as latency or token count, which describe how a request ran, an evaluation assesses the quality of the response itself. Evaluations let you catch quality regressions before they reach users, compare model or prompt candidates objectively, and build a structured feedback loop that speeds up iteration.

LLM as a judge

LLM-as-a-judge is a scoring method in which one language model rates the output of another LLM or agent, aiming to approximate a human reviewer at a scale manual review can't reach. It is one of several graders an evaluation can draw on, alongside deterministic code checks and human review.

Use it when quality is nuanced and no deterministic or one-size-fits-all test applies, for open-ended responses where "good" depends on context, tone, and intent. You give the judge the request, the response, and a rubric, and it returns a score with a short rationale. This makes it practical to measure qualities that resist automation, such as faithfulness, relevance, tone, and bias. Because the judge is a model too, its verdicts can carry bias, so scores are typically cross-checked against periodic human review.

Code evals

Code evals score a response with fixed rules instead of another model. The checks are deterministic, so the same input always returns the same verdict.

A check can be as simple as an exact match, a keyword lookup, or a regex pattern, or any function you write, such as validating JSON, running generated code against tests, or confirming an agent called the right tool. Fast and inexpensive, code evals complement LLM as a judge. Use them for anything with a definite answer, and bring in a judge for the nuanced, open-ended qualities they cannot capture.

Online evals

Online evals run against live production traffic in real time or near real time, scoring real user interactions as they happen. The data is unlabeled, unpredictable, and representative of actual usage, surfacing issues that never appear in pre-release testing, such as new question patterns, edge cases, drift, and anomalies.

Because there are no pre-defined labels, scoring relies on heuristics, implicit signals, or async LLM-as-a-judge pipelines. Traffic is typically sampled rather than scored in full, keeping costs and latency in check. Results feed continuous quality assurance, A/B testing, and production monitoring without slowing down user-facing requests.

Offline evals

Offline evals run against a fixed, pre-collected dataset during development, before deploying a new model, prompt, retrieval change, or release. The dataset is curated, labeled, or synthetically generated, and is often versioned alongside the code it tests. Over time, teams typically enrich it with real examples sampled from live production traffic, keeping it grounded in actual usage patterns.

Full control over the test distribution makes them well-suited for regression testing, model comparison, prompt optimization, safety red-teaming, and fine-tune evaluation. Because they run in batch without user-facing latency pressure, scoring can be thorough and unhurried. The tradeoff is coverage: a curated dataset may not capture the full range of real user behavior or the long tail of production inputs, which is where online evals pick up.

Instrumentation

Instrumentation is the process of adding observability code to an application.

Dynatrace uses OpenLLMetry, an instrumentation library based on OpenTelemetry. OpenLLMetry automatically registers an OpenTelemetry SDK and a list of instruments for popular GenAI frameworks, models, and vector databases.

To learn how to configure OpenLLMetry in your application, see the Get started page.

Traces

A trace describes a user request and all the operations performed to satisfy it. We can analyze the steps relevant to observing AI/ML workloads.

Traces help bring visibility into complex workflows, providing information about costs, performance, and insights into the quality of the generated output in the context of AI/ML workloads.

It's common to have LLM applications with complex and autonomous logic in which a model makes the decision. We can leverage traces to understand how requests propagate across RAG or agentic pipelines and see the details of each step that was executed.

Each action performed in a trace is stored as a span. The span attributes contain information relevant to AI/ML workloads (such as token costs for the operation, and input and output prompts). OpenLLMetry follows the OpenTelemetry Semantic Conventions for GenAI, so it's easy to find the relevant attribute keys.

The Distributed Tracing page explains the trace concept in more detail.

Agentic trace screenshot

trace-agentic-pipeline

RAG trace screenshot

trace-rag-pipeline

Traceloop span kind

Traceloop marks spans that belong to an LLM framework with a particular attribute, traceloop.span.kind. This attribute helps to organize and understand the structure of your application's traces, making it easier to analyze and debug complex LLM-based systems.

The traceloop.span.kind attribute can have one of four possible values:

OpenTelemetry GenAI semantic conventions

The OpenTelemetry GenAI semantic conventions standardize the attributes and metrics captured for generative AI operations, so data from different providers stays comparable.

For the full, current list of GenAI metrics, span attributes, agent, and provider-specific attributes, and which of them AI Observability visualizes today, see the AI Observability metrics reference.