Run agent evaluations in Dynatrace
- Latest Dynatrace
- How-to guide
- 8-min read
- Published Aug 05, 2026
Use agent and LLM evaluations to assess the quality, safety, and reliability of your AI application responses. Configure the required settings, run evaluations in the UI or with dt-evals, and review the resulting runs and results.
Evaluations are currently part of the Dynatrace Preview program and are governed by the preview terms. The features offered in the preview are not complete and might change significantly before general availability.
Before you begin
Make sure that:
- Your application sends GenAI spans to Dynatrace.
- To configure LLM providers and evaluation methods, you need
app-settings:objects:readandapp-settings:objects:write. - To run manual evaluations in the UI, you need
app-settings:objects:read,storage:bizevents:read, andstorage:events:writepermissions. - An LLM provider and at least one evaluation method are configured.
Configure evaluation settings
In 
AI Observability, you can configure evaluation settings in:
- The settings page. Select the Settings icon in the upper-right corner, and then configure an LLM provider including its credentials, and create custom evaluation methods.
- The Prompts tab. Review and manage prompts before you run an evaluation.
Configure an LLM provider
Dynatrace evaluations use a judge model that you provide. Add a connection for Anthropic, OpenAI, Microsoft Foundry, Azure OpenAI, Google Gemini, Google Gemini Enterprise Platform (Vertex AI), AWS Bedrock, or a custom OpenAI-compatible endpoint.
Before you add an LLM provider, allow its API host in AI Observability > Settings > General > External requests.
- For OpenAI, add
api.openai.com. - For Anthropic, add
api.anthropic.com. - For Azure OpenAI, Google, AWS Bedrock, or a custom provider, add the host for the endpoint that you use.
To add API hosts, select New host pattern, enter the host without a protocol or path, and select Add. For details, see Allow external requests.
If the provider endpoint restricts inbound network traffic, confirm with your network administrator that it accepts requests from your Dynatrace environment.
Select the model that you want to use for evaluation, provide the required connection details and credentials, then save the configuration. After you configure a provider, you can select its model as the judge when you run a manual evaluation.
Your LLM provider manages API rate limits, quotas, and context-window size. Make sure that your provider configuration supports the expected evaluation volume and prompt size.
Protect LLM provider credentials
Dynatrace stores the credentials for an LLM provider connection as secret fields in app settings.
Use a dedicated provider project or account for evaluation workloads, and grant it only the permissions required to run evaluations. Rotate provider credentials according to your organization's credential-rotation policy, and update the LLM provider connection when you rotate a credential.
When you run automated evaluations with the open-source dt-evals CLI in a CI/CD workflow, store provider credentials as CI/CD secrets. Do not add credentials to the dt-evals configuration file.
Create a custom evaluation method
Alongside LLM providers, the settings menu includes the Evaluation method library. Select Evaluation method to create a custom LLM-as-a-judge method when the built-in methods don't meet your requirements.
Enter a method name and description, then define the LLM-as-a-judge system prompt. When the evaluation runs, Dynatrace replaces the input and output variables with data from the selected GenAI span.
input: Prompt input fromgen_ai.input.messagesor user-prompt fields undergen_ai.prompt.*.output: Span output.
If available, Dynatrace places a system prompt from gen_ai.system_instructions or a system-role gen_ai.prompt.* field before the prompt input.
Tell the judge which criterion to assess, how to score it, and which score, label, and concise explanation to return. Save the method when you finish. The new method is then available when you run a manual evaluation.
For evaluations that require fields beyond the input and output, use Run LLM-as-a-judge evaluations with dt-evals. The CLI supports additional fields such as context and an expected output.
Review and manage prompts
Evaluations run against the GenAI spans that your AI services already send to Dynatrace. Use the Prompts tab to confirm that your service sends prompt data and to explore the prompts associated with it.
Select a prompt to inspect its related prompt runs, traces, and evaluation results. You can then select a prompt run for a manual evaluation.
Run manual evaluations from the UI
Use manual evaluations to assess prompt runs captured from your GenAI spans. To evaluate prompts manually:
- Go to the Prompts tab.
- Select the prompt runs that you want to evaluate.
- Select Run manual evaluations.
- Select an LLM provider and its configured model.
- Select one or more evaluation methods.
- Select Run evaluation.
Dynatrace writes the result for each selected prompt run and method as a business event associated with the source span. You can review the result with the related trace and prompt context.
Understand manual evaluation limits
Use manual evaluations to validate built-in and custom LLM-as-a-judge methods against captured prompt runs. When you are ready to evaluate at scale, configure the method in dt-evals for larger batches, recurring evaluations, or CI/CD workflows.
For the complete setup, configuration, and execution procedure, see Run LLM-as-a-judge evaluations with dt-evals.
If a manual run includes more than 20 evaluations, some evaluations might not complete. Use the open-source dt-evals CLI for larger batches, recurring evaluations of production traffic, or evaluation runs in a CI/CD workflow.
The number of evaluations in a manual run is equal to selected prompt runs × Selected evaluation methods. Currently, you can't run more than 30 evaluations at once, however Dynatrace plans to lift this temporary limit in a future update.
Review evaluation runs and results
Manual and dt-evals evaluations both write results to the Evaluations tab. Use Runs to review the results from one execution, then open Evals to inspect individual prompt-run results.
Review evaluation runs
Select Evaluations > Runs to review the evaluation runs available in your environment. The table shows the run ID, pass rate, evaluation model, AI app, evaluation methods, and last-run time.
Filter runs by pass rate, evaluation method, provider, model, AI app, or run ID. Select View evals for a run to open the individual evaluation results filtered to that run.
Review individual evaluation results
In Evals, select a result to review its score, verdict, and explanation. The details show:
- The evaluation method, type, version, and scoring format.
- The judge provider and model.
- The run ID, trace ID, and span ID.
Investigate a result in the prompt trace
Use the evaluation result as a starting point. The following investigation ideas can help you understand why it passed or failed.
- Review the evaluated prompt and response to confirm what the judge assessed. Compare the score and explanation with the input, output, and system prompt.
- Review the evaluation configuration to confirm the selected method, scoring format, provider, and model. This helps determine whether the result reflects the expected evaluation criteria.
- Select Open prompts to open the evaluated span in the Prompts tab. Review the span fields and related prompt context, then select View trace to examine the surrounding trace.
- If the result might relate to agent behavior, open Agents topology for the same timeframe. Select the relevant agent, model, provider, or service to review LLM requests, token usage, and response time. Select View Prompts to investigate the filtered prompts for that topology node.
- Use the
dtctlCLI and its agent skills to query evaluation business events and spans. Provide the structured results to your coding assistant or SRE agent to automate investigations and find the root cause.
