Run LLM-as-a-judge evaluations with dt-evals

Latest Dynatrace
Tutorial
10-min read
Published Jul 28, 2026

In this tutorial, you run continuous LLM-as-a-judge evaluations against the production GenAI traffic already flowing into Dynatrace, using the open-source dt-evals CLI. You score real interactions, detect quality drift over time, and pivot from any low score straight to the trace that produced it.

Who is this for?

DevOps engineers and AI engineers who want to continuously score the quality of a GenAI service already sending traces to Dynatrace.

What will you learn?

How to configure dt-evals against your Dynatrace environment and a judge provider
How to run evaluations on sampled production GenAI spans
How to add a custom LLM-as-a-judge evaluator alongside the built-in catalog
How to automate recurring runs and gate a CI pipeline on quality
How to visualize scores, drift, and threshold breaches in Dynatrace and pivot to the source trace

Before you begin

Prerequisites

Node.js 20 or later
A Dynatrace environment already receiving GenAI spans
A Dynatrace platform token with the following scopes. For how to create or update a token, see Platform tokens. You can let dtevals help you set up this token, see Set up credentials.
- storage:spans:read
- storage:buckets:read
- storage:bizevents:read
- storage:events:write
- storage:metrics:write
- storage:logs:read
Credentials for one judge provider: OpenAI, Anthropic, Azure OpenAI, Google Gemini Enterprise Platform (Vertex AI), or Amazon Bedrock

Sample application

The tutorial uses GenAI spans that already exist in your environment, so any service that sends them to Dynatrace will work.

If you don't have one yet, you can start with the pydantic-ai/opentelemetry sample application from dynatrace-oss/dynatrace-ai-agent-instrumentation-examples. It is a music history agent that uses pydantic-ai's native OpenTelemetry support to emit GenAI spans to Dynatrace.

Ensure your application is already running and its spans are visible in AI Observability before continuing.

Evaluate your production traces

Follow these steps to run evaluations with dt-evals.

This example uses OpenAI as the judge provider. To configure a different provider, such as Anthropic, Azure OpenAI, Google Gemini Enterprise Platform (Vertex AI), or Amazon Bedrock, see Supported providers.

1. Install the CLI

Install dt-evals globally, or run it on demand with npx.

npm install -g @dynatrace-oss/dt-evals

Verify the installation:

dt-evals --version

2. Set up credentials

dt-evals needs a Dynatrace platform token (to read spans and write results) and a key for your judge provider. The CLI reads both from the environment and never stores them in the config file; it picks them up automatically, regardless of how you supply them.

The easiest way is to let the CLI walk you through it.

dt-evals doctor

When you do this, dt-evals doctor:

Opens the platform tokens page so you can create a scoped token.
Saves it to a gitignored .env file.
Verifies your token scopes and provider connectivity.

The dt-evals configure wizard in the next step also prompts for these credentials.

If you prefer to set the variables yourself, export them directly and the CLI will use them. For information about how to create a token, see Platform tokens; grant the scopes listed in Prerequisites.

export DT_API_TOKEN=dt0s16.<your-token>
export OPENAI_API_KEY=<your-key>

For CI, set the same variables as pipeline secrets instead of exporting them locally.

3. Configure the CLI

Run the interactive wizard. The wizard connects to your Dynatrace environment, selects your judge provider, and writes a .dt-eval.yaml file to the current directory.

dt-evals configure

The wizard produces a config similar to what's shown in the following code block. Set scope.service to the service that emits your GenAI spans, and choose which evaluators to run under metrics.enabled.

schemaVersion: 2
name: music-agent-evals

dynatrace:
  environmentUrl: https://<your-env-id>.live.dynatrace.com
  # apiToken loaded from the DT_API_TOKEN environment variable

judge:
  provider: openai
  model: gpt-4.1

scope:
  service: pydantic-ai-music-agent
  since: 1h
  sampling:
    strategy: random
    percent: 10

metrics:
  enabled:
    - faithfulness
    - relevance
    - hallucination
    - toxicity
    - drift

alerts:
  thresholds:
    faithfulness: 0.7
    relevance: 0.7

storeEvaluatedPrompt: false

By default, dt-evals reads OpenTelemetry GenAI semantic convention fields. If your spans expose LLM input and output under different attribute names, map them to the canonical fields with scope.spanFields. For more information, see the dt-evals README.

The wizard writes .dt-eval.yaml to the current directory. Continue to the next step to validate your config and connectivity.

4. Validate connectivity

Before the first run, confirm that your config, Dynatrace connectivity, and judge provider are all reachable.

dt-evals validate

5. Run evaluations

Run all enabled evaluators against a sample of recent traces. The CLI writes results back to Dynatrace as bizevents with event.type == "gen_ai.evaluation.result", linked to their source traces.

Here are some example runs that you could do:

To evaluate only a certain percentage of traces, use --sample.
```
dt-evals run --since 1h --sample 10
```
- --since sets the trace lookback window, for example 1h or 24h.
- --sample is the percentage of traces to evaluate.

To run a single evaluator, use --metric.

dt-evals run --since 6h --metric faithfulness

To preview the work without calling the judge or writing results, use --dry-run.
```
dt-evals run --since 1h --sample 5 --dry-run
```

6. Add a custom evaluator

Built-in evaluators cover common safety, grounding, and quality checks. When you need a project-specific rubric, create your own LLM-as-a-judge evaluator with the interactive wizard.

For example, suppose you want to score whether the chatbot answers in a concise, on-brand style.

Run the wizard and define an answer-style evaluator with your own judge prompt, scoring range, and pass threshold.
```
dt-evals evaluators add
```
The wizard stores the evaluator locally. Enable it in .dt-eval.yaml like any built-in evaluator.
```
metrics:
  enabled:
    - faithfulness
    - answer-style
```

Inspect and test it before a full run.

dt-evals evaluators show answer-style
dt-evals evaluators test answer-style

7. Automate the runs

To gate a deployment on quality, add the --ci flag to your run. The command exits with a non-zero code when a score falls below its configured threshold, which fails the pipeline step.

dt-evals run --ci

To repeat a run on a schedule, invoke it from your own scheduler, such as a cron job or a CI pipeline trigger.

After the scheduled run completes, open the Evaluations tab in AI Observability AI Observability to confirm new scores appear.

What happens during a run

When you run dt-evals, it samples the GenAI spans in your configured timeframe, masks personally identifiable information in memory, scores each interaction with your judge model, and writes the results back to Dynatrace as bizevents.

Sample: The CLI fetches GenAI spans for the configured service and timeframe, then samples them by random percentage, latest count, or errors only.
Mask: Email addresses, phone numbers, credit card numbers, and social security numbers are redacted before any content leaves the process. The original values are never sent to the judge.
Score: Your judge model rates each interaction against the enabled evaluators and returns a score with a short explanation.
Store: Each score is written as a bizevents entry linked to its source trace, so a low score is never a dead end.

For gradual regressions rather than single-run breaches, enable the drift evaluator. The drift evaluator compares recent scores against a seven-day baseline of prior evaluation results.

Congratulations!

With your evaluations in place, you can explore scores, drift, and threshold breaches in Dynatrace.

AI Observability

In AI Observability, open the Evaluations tab to see pass and fail rates, average score per evaluator, score distribution, and drift over time. Select any failing score to open the originating trace and inspect the prompt, retrieval context, model call, and tool usage behind it.

For more information about the tabs, see AI Observability app.

The Evaluations tab is currently part of our Preview program and is governed by our preview terms. The features offered in the preview are not complete and may undergo significant changes prior to general availability.

Query with DQL

Evaluation results are queryable as bizevents. To see the average score per evaluator, run the following query:

fetch bizevents
| filter event.type == "gen_ai.evaluation.result"
| summarize avg_score = avg(gen_ai.evaluation.score.value), by: { gen_ai.evaluation.name }
| sort avg_score asc

Each result carries the score (gen_ai.evaluation.score.value), the pass or fail label (gen_ai.evaluation.score.label), the judge's explanation, and the source trace_id.

Alert on quality

Because results are bizevents, you can alert on quality as you would any other signal. Create a custom alert on a rising fail rate or a dropping average score, and route notifications to your team.

Run LLM-as-a-judge evaluations with dt-evals

Latest Dynatrace
Tutorial
10-min read
Published Jul 28, 2026

Who is this for?

DevOps engineers and AI engineers who want to continuously score the quality of a GenAI service already sending traces to Dynatrace.

What will you learn?

How to configure dt-evals against your Dynatrace environment and a judge provider
How to run evaluations on sampled production GenAI spans
How to add a custom LLM-as-a-judge evaluator alongside the built-in catalog
How to automate recurring runs and gate a CI pipeline on quality
How to visualize scores, drift, and threshold breaches in Dynatrace and pivot to the source trace

Before you begin

Prerequisites

Node.js 20 or later
A Dynatrace environment already receiving GenAI spans
A Dynatrace platform token with the following scopes. For how to create or update a token, see Platform tokens. You can let dtevals help you set up this token, see Set up credentials.
- storage:spans:read
- storage:buckets:read
- storage:bizevents:read
- storage:events:write
- storage:metrics:write
- storage:logs:read
Credentials for one judge provider: OpenAI, Anthropic, Azure OpenAI, Google Gemini Enterprise Platform (Vertex AI), or Amazon Bedrock

Sample application

The tutorial uses GenAI spans that already exist in your environment, so any service that sends them to Dynatrace will work.

Ensure your application is already running and its spans are visible in AI Observability before continuing.

Evaluate your production traces

Follow these steps to run evaluations with dt-evals.

1. Install the CLI

Install dt-evals globally, or run it on demand with npx.

npm install -g @dynatrace-oss/dt-evals

Verify the installation:

dt-evals --version

2. Set up credentials

The easiest way is to let the CLI walk you through it.

dt-evals doctor

When you do this, dt-evals doctor:

Opens the platform tokens page so you can create a scoped token.
Saves it to a gitignored .env file.
Verifies your token scopes and provider connectivity.

The dt-evals configure wizard in the next step also prompts for these credentials.

export DT_API_TOKEN=dt0s16.<your-token>
export OPENAI_API_KEY=<your-key>

For CI, set the same variables as pipeline secrets instead of exporting them locally.

3. Configure the CLI

Run the interactive wizard. The wizard connects to your Dynatrace environment, selects your judge provider, and writes a .dt-eval.yaml file to the current directory.

dt-evals configure

schemaVersion: 2
name: music-agent-evals

dynatrace:
  environmentUrl: https://<your-env-id>.live.dynatrace.com
  # apiToken loaded from the DT_API_TOKEN environment variable

judge:
  provider: openai
  model: gpt-4.1

scope:
  service: pydantic-ai-music-agent
  since: 1h
  sampling:
    strategy: random
    percent: 10

metrics:
  enabled:
    - faithfulness
    - relevance
    - hallucination
    - toxicity
    - drift

alerts:
  thresholds:
    faithfulness: 0.7
    relevance: 0.7

storeEvaluatedPrompt: false

The wizard writes .dt-eval.yaml to the current directory. Continue to the next step to validate your config and connectivity.

4. Validate connectivity

Before the first run, confirm that your config, Dynatrace connectivity, and judge provider are all reachable.

dt-evals validate

5. Run evaluations

Run all enabled evaluators against a sample of recent traces. The CLI writes results back to Dynatrace as bizevents with event.type == "gen_ai.evaluation.result", linked to their source traces.

Here are some example runs that you could do:

To evaluate only a certain percentage of traces, use --sample.
```
dt-evals run --since 1h --sample 10
```
- --since sets the trace lookback window, for example 1h or 24h.
- --sample is the percentage of traces to evaluate.

To run a single evaluator, use --metric.

dt-evals run --since 6h --metric faithfulness

To preview the work without calling the judge or writing results, use --dry-run.
```
dt-evals run --since 1h --sample 5 --dry-run
```

6. Add a custom evaluator

Built-in evaluators cover common safety, grounding, and quality checks. When you need a project-specific rubric, create your own LLM-as-a-judge evaluator with the interactive wizard.

For example, suppose you want to score whether the chatbot answers in a concise, on-brand style.

Run the wizard and define an answer-style evaluator with your own judge prompt, scoring range, and pass threshold.
```
dt-evals evaluators add
```
The wizard stores the evaluator locally. Enable it in .dt-eval.yaml like any built-in evaluator.
```
metrics:
  enabled:
    - faithfulness
    - answer-style
```

Inspect and test it before a full run.

dt-evals evaluators show answer-style
dt-evals evaluators test answer-style

7. Automate the runs

To gate a deployment on quality, add the --ci flag to your run. The command exits with a non-zero code when a score falls below its configured threshold, which fails the pipeline step.

dt-evals run --ci

To repeat a run on a schedule, invoke it from your own scheduler, such as a cron job or a CI pipeline trigger.

After the scheduled run completes, open the Evaluations tab in AI Observability AI Observability to confirm new scores appear.

What happens during a run

Sample: The CLI fetches GenAI spans for the configured service and timeframe, then samples them by random percentage, latest count, or errors only.
Mask: Email addresses, phone numbers, credit card numbers, and social security numbers are redacted before any content leaves the process. The original values are never sent to the judge.
Score: Your judge model rates each interaction against the enabled evaluators and returns a score with a short explanation.
Store: Each score is written as a bizevents entry linked to its source trace, so a low score is never a dead end.

For gradual regressions rather than single-run breaches, enable the drift evaluator. The drift evaluator compares recent scores against a seven-day baseline of prior evaluation results.

Congratulations!

With your evaluations in place, you can explore scores, drift, and threshold breaches in Dynatrace.

AI Observability

For more information about the tabs, see AI Observability app.

Query with DQL

Evaluation results are queryable as bizevents. To see the average score per evaluator, run the following query:

fetch bizevents
| filter event.type == "gen_ai.evaluation.result"
| summarize avg_score = avg(gen_ai.evaluation.score.value), by: { gen_ai.evaluation.name }
| sort avg_score asc

Each result carries the score (gen_ai.evaluation.score.value), the pass or fail label (gen_ai.evaluation.score.label), the judge's explanation, and the source trace_id.

Alert on quality

Because results are bizevents, you can alert on quality as you would any other signal. Create a custom alert on a rising fail rate or a dropping average score, and route notifications to your team.

Run LLM-as-a-judge evaluations with dt-evals

Who is this for?

What will you learn?

Before you begin

Prerequisites

Sample application

Evaluate your production traces

1. Install the CLI

2. Set up credentials

3. Configure the CLI

4. Validate connectivity

5. Run evaluations

6. Add a custom evaluator

7. Automate the runs

What happens during a run

Congratulations!

AI Observability

Query with DQL

Alert on quality

Related topics

Run LLM-as-a-judge evaluations with dt-evals

Who is this for?

What will you learn?

Before you begin

Prerequisites

Sample application

Evaluate your production traces

1. Install the CLI

2. Set up credentials

3. Configure the CLI

4. Validate connectivity

5. Run evaluations

6. Add a custom evaluator

7. Automate the runs

What happens during a run

Congratulations!

AI Observability

Query with DQL

Alert on quality

Related topics