Try it free

OpenAI Observability

  • Latest Dynatrace
  • Tutorial
  • 6-min read

Dynatrace enables enterprises to automatically collect, visualize and alert on OpenAI Agent, API request consumption, latency, and stability information in combination with all other services used to build their AI application. This includes OpenAI and Azure OpenAI services, such as GPT-5.2, Codex, DALL-E, or ChatGPT.

The example dashboard below visualizes OpenAI token consumption, showing critical SLOs for latency and availability as well as the most important OpenAI generative AI service metrics, such as response time, error count, and overall number of requests.

OpenAI dashboard
OpenAI dashboard

Dynatrace with OpenTelemetry or OpenInference helps to:

  • Reveal the full context of used technologies.
  • Shows the service interaction topology.
  • Analyzes security vulnerabilities.
  • Observes metrics, traces, logs, and business events in real time.

The sections below show how to:

  • Trace OpenAI/GPT model requests.
  • Chart OpenAI golden signals within a dashboard.
  • Use Dynatrace Intelligence to automatically detect abnormal service behavior such as slowdowns in OpenAI/GPT requests as the root cause of large-scale situations.

OpenAI Agents SDK sample app

Use the OpenAI Agents SDK sample to see AI Observability in action. The sample is a Customer Service Agent interface built on the OpenAI Agents SDK and based on the openai-cs-agents-demo. The backend uses OpenInference to emit traces with gen_ai.* attributes to Dynatrace.

In the sample, api.py configures the OpenInference OpenAIAgentsInstrumentor and exports spans over OTLP. An included OpenTelemetry Collector forwards the spans to Dynatrace and derives GenAI duration and token usage metrics from them, because OpenInference emits spans only.

For more configuration options, see Get started with OpenInference and AI Observability.

Run the sample

Prerequisites:

  • A Dynatrace access token with the openTelemetryTrace.ingest and metrics.ingest scopes.
  • Python 3.10 or later and uv.
  • Docker, to run the OpenTelemetry Collector.
  • An Azure OpenAI endpoint, key, and deployment.
  1. Clone the sample repository and go to the openai-agents/openinference directory.

  2. Copy .env.sample to .env and set DT_ENDPOINT, DT_API_TOKEN, and your Azure OpenAI values.

    cp .env.sample .env
  3. Install dependencies.

    make install
  4. Start the collector and the app on port 8000.

    make run
  5. In a second terminal, send two chat turns that share one conversation.

    make request

Observe OpenAI request costs using logs

Using logs to collect and observe OpenAI request costs is both simple and powerful. It allows the owner of the service to write a log line using a standard Node.js logging framework such as Winston and to collect and analyze those logs within Dynatrace.

The following code snippet writes a Winston log line containing the OpenAI request token counts.

logger.log('info', `OpenAI response promt_tokens:${response.data.usage.prompt_tokens} completion_tokens:${response.data.usage.completion_tokens} total_tokens:${response.data.usage.total_tokens}`);

The OpenAI request log lines can be viewed in the Dynatrace log viewer.

We recommend that you also enable trace context enrichment for Node.js logs (go to Settings Settings > Preferences > OneAgent features and turn on Node.js Trace/span context enrichment for logs) to automatically map all log lines to their captured request traces.

Trace context enrichment setting
Trace context enrichment setting

With Node.js Trace/span context enrichment for logs enabled, you even get a convenient mapping of all OpenAI traces with their log lines. You can view them at AI Observability AI Observability > Explorer > Logs.

Logs explorer
Logs explorer

Observe OpenAI request cost by sending custom metrics

Each request to an OpenAI model, such as GPT-5.2 or GPT-5.1, reports back how many tokens were used for the request prompt (the length of your text question) and how many tokens the model generated as a response.

By extracting token measurements from the returning payload and reporting them through OneAgent, users can observe the token consumption across all OpenAI-enhanced services in their monitoring environment.

To extract the token count from the OpenAI response and report those measurements to the local OneAgent, add the following instrumentation to your Node.js service.

function report_metric(openai_response) {
var post_data = "openai.promt_token_count,model=" + openai_response.model + " " + openai_response.usage.prompt_tokens + "\n";
post_data += "openai.completion_token_count,model=" + openai_response.model + " " + openai_response.usage.completion_tokens + "\n";
post_data += "openai.total_token_count,model=" + openai_response.model + " " + openai_response.usage.total_tokens + "\n";
console.log(post_data);
var post_options = {
host: 'localhost',
port: '14499',
path: '/metrics/ingest',
method: 'POST',
headers: {
'Content-Type': 'text/plain',
'Content-Length': Buffer.byteLength(post_data)
}
};
var metric_req = http.request(post_options, (resp) => {}).on("error", (err) => { console.log(err); });
metric_req.write(post_data);
metric_req.end();
}

The three new OpenAI token consumption metrics are available in Dynatrace.

OpenAI token consumption metrics collected in Dynatrace
OpenAI token consumption metrics collected in Dynatrace

Dynatrace Intelligence automatically detects GPT as root cause

Dynatrace Intelligence automatically learns the typical behavior of monitored services.

When an abnormal slowdown or increase of errors is detected, Dynatrace Intelligence triggers a root cause analysis to identify the cause of complex situations.

Our simple example of a Node.js service entirely depends on the ChatGPT model response. Whenever the latency of the model response degrades or the model request returns with an error, Dynatrace Intelligence automatically detects it.

Related tags
AI Observability