Try it free

AI data governance with Amazon Bedrock

  • Latest Dynatrace
  • Tutorial
  • 5-min read

Emerging AI regulations such as the European Union Artificial Intelligence Act provide the means to deploy a comprehensive strategy combining organizational and AI model oversight, covering everything from model training to AI/user interactions.

When running your AI models through Amazon Bedrock, Dynatrace helps you to comply with regulatory record-keeping requirements.

What you will learn

In this tutorial, we first configure your model training and deployment observability. Afterward, we configure your application to observe user inference requests.

Steps

The general steps are as follows:

  1. Configure Dynatrace
  2. Configure your AWS account to send data to your Dynatrace tenant
  3. Configure your application

Step 1 Configure Dynatrace

In this step, we create a Dynatrace token and we configure OpenPipeline to retain the data for 5+ years.

Create Dynatrace token

To authenticate, create one of the following:

  • Platform token:
    1. Go to My platform tokens.
    2. Create a platform token with the required scope (listed below). See Platform tokens for details.
  • Classic access token:
    1. Go to Access tokens Access Tokens.
    2. Select Generate new token.
    3. Enter a name for your token.
    4. Select the required scopes listed below.
    5. Select Generate token.
    6. Copy the generated token. Store it securely — you can only access it once.

Required scopes:

  • Platform token: openpipeline:bizevents:ingest, openpipeline:traces:ingest
  • Classic access token: bizevents.ingest, openTelemetryTrace.ingest

Configure OpenPipeline

The default retention period for BizEvents is 35 days. Depending on the regulations, this might not be enough.

To change the retention period, you can create a custom Grail bucket.

  1. Go to Settings > Storage management > Bucket storage management.
  2. In Bucket Storage Management, select Bucket.
  3. On New bucket:
    • Set Bucket name (for example, gen_ai_events)
    • Set Retention period (in days) (for example, 1,825, which is about 5 years)
    • Set Bucket table type to bizevents
  4. Select Save.
Grail Bucket Creation
Grail Bucket Creation

When the bucket is available, we can configure OpenPipeline to redirect AI-relevant events to storage there.

  1. Go to Settings Settings > Process and contextualize > OpenPipeline > Business events > Pipelines.
  2. On the Pipelines tab, select Pipeline and name your pipeline (for example, AI Data Governance).
  3. Go to Bucket assignment > > Bucket assignment.
  4. Configure the processor:
    1. Enter a Name for the processor
    2. Set Matching condition to matchesValue(event.type, "gen_ai.auditing")
    3. Set Storage to the bucket you created in the previous procedure
  5. Select Save.
OpenPipeline Bucket Assignment
OpenPipeline Bucket Assignment

Finally, we route the ingestion of AI events to the pipeline.

Configure the dynamic routes in Settings Settings, even if you use dtctl for the bucket and pipelines. All routes of a configuration scope are stored in one settings object, so applying a routing file with dtctl would replace your existing routes.

  1. Still in OpenPipeline > Business events, select the Dynamic routing tab.
  2. On the Dynamic routing tab, select Dynamic route to Add a new dynamic route.
    • Enter a Name for the new route (for example, AI Event Routing)
    • Set Matching condition to matchesValue(event.type,"gen_ai.auditing")
    • Set Pipeline to the pipeline you created in the previous procedure.
  3. Select Add.
  4. Select Save.

Finally, to mark it as the first pipeline to trigger, drag it up to be the first row in the table.

OpenPipeline Routing
OpenPipeline Routing

Extract business events from LLM spans

Your application sends traces, not business events. To keep a record of each user/AI interaction, configure OpenPipeline to create a gen_ai.auditing business event from every LLM span. The dynamic route from the previous procedure then sends these events to your bucket.

  1. Go to Settings Settings > Process and contextualize > OpenPipeline > Spans > Pipelines.
  2. On the Pipelines tab, select Pipeline and name your pipeline (for example, AI Data Governance - Spans).
  3. Go to Data extraction > > Business event.
  4. Configure the processor:
    1. Enter a Name for the processor (for example, gen_ai.auditing extraction).
    2. Set Matching condition to isNotNull(gen_ai.input.messages) OR isNotNull(gen_ai.output.messages).
    3. Set Event type to Static string with the value gen_ai.auditing.
    4. Set Event provider to Field name with the value service.name.
    5. In Field extraction, select Fields and add the fields you need to retain, for example:
      • gen_ai.input.messages
      • gen_ai.output.messages
      • gen_ai.request.model
      • gen_ai.response.model
      • gen_ai.provider.name
      • gen_ai.operation.name
      • gen_ai.usage.input_tokens
      • gen_ai.usage.output_tokens
      • service.name
      • trace.id
      • span.id
  5. Select Save.

Then route the spans of your AI application to this pipeline.

  1. Still in OpenPipeline > Spans, select the Dynamic routing tab.
  2. Select Dynamic route.
    • Enter a Name for the new route (for example, AI Span Routing).
    • Set Matching condition to match your application, for example, service.name == "<your-service>".
    • Set Pipeline to the pipeline you created in the previous procedure.
  3. Select Add.
  4. Select Save.

The extracted events go through the business events pipeline, so the bucket must exist before you send data. If the bucket doesn't exist, the events are dropped without an error.

Create the bucket and pipelines with dtctl

Instead of using Settings Settings, you can create the bucket and both pipelines with dtctl. Configure the two dynamic routes as described in Configure OpenPipeline.

  1. Create the bucket. This requires the storage:buckets:write permission.

    dtctl create bucket --name gen_ai_events --table bizevents --retention 1825 --display-name "GenAI auditing events"
  2. Save the business events pipeline as bizevents-pipeline.yaml. Set bucketName to your bucket.

    schemaid: builtin:openpipeline.bizevents.pipelines
    scope: environment
    value:
    customId: ai-data-governance
    displayName: AI Data Governance
    storage:
    processors:
    - id: gen-ai-events-bucket
    type: bucketAssignment
    enabled: true
    description: Store gen_ai.auditing events in gen_ai_events
    matcher: matchesValue(event.type, "gen_ai.auditing")
    bucketAssignment:
    bucketName: gen_ai_events
  3. Save the spans pipeline as spans-pipeline.yaml. Adjust the list of extracted fields to what you need to retain.

    schemaid: builtin:openpipeline.spans.pipelines
    scope: environment
    value:
    customId: ai-data-governance-spans
    displayName: AI Data Governance - Spans
    dataExtraction:
    processors:
    - id: gen-ai-auditing-bizevent
    type: bizevent
    enabled: true
    description: Extract gen_ai.auditing business event from LLM spans
    matcher: isNotNull(gen_ai.input.messages) OR isNotNull(gen_ai.output.messages)
    bizevent:
    eventType:
    type: constant
    constant: gen_ai.auditing
    eventProvider:
    type: field
    field:
    sourceFieldName: service.name
    fieldExtraction:
    type: include
    include:
    - { extractionType: field, sourceFieldName: gen_ai.input.messages }
    - { extractionType: field, sourceFieldName: gen_ai.output.messages }
    - { extractionType: field, sourceFieldName: gen_ai.request.model }
    - { extractionType: field, sourceFieldName: gen_ai.response.model }
    - { extractionType: field, sourceFieldName: gen_ai.provider.name }
    - { extractionType: field, sourceFieldName: gen_ai.operation.name }
    - { extractionType: field, sourceFieldName: gen_ai.usage.input_tokens }
    - { extractionType: field, sourceFieldName: gen_ai.usage.output_tokens }
    - { extractionType: field, sourceFieldName: service.name }
    - { extractionType: field, sourceFieldName: trace.id }
    - { extractionType: field, sourceFieldName: span.id }
  4. Apply both files. This requires the settings:objects:write permission.

    dtctl apply -f bizevents-pipeline.yaml
    dtctl apply -f spans-pipeline.yaml

Step 2 Configure your AWS account

Amazon Bedrock emits events for every configuration action executed, such as when you deploy a new model or when the fine-tuning of your model finishes.

We can set up a rule to forward these events to Dynatrace. Please refer to our integration with Amazon EventBridge using BizEvent to configure the rule.

The only change is in the InputTemplate field, where the property "type" should be set to gen_ai.auditing. This change is required to match the values that OpenPipeline uses to redirect the events to our Grail bucket.

Example AWS configuration

The following screenshots show the connection, transformer, and target configuration in AWS.

AWS <-> Dynatrace connection
AWS <-> Dynatrace connection
AWS CloudTrail Transformer configuration
AWS CloudTrail Transformer configuration
AWS CloudTrail Target configuration
AWS CloudTrail Target configuration

Step 3 Configure your application

Instrument your application with OpenInference. The BedrockInstrumentor traces your Amazon Bedrock calls, including the prompts and responses.

The business event extraction reads gen_ai.* attributes. To emit them alongside the OpenInference attributes, set the following environment variable:

OPENINFERENCE_ENABLE_GENAI_SEMCONV=true

For details, see Use OpenTelemetry GenAI semantic conventions when available.

To install, use the following command:

pip install openinference-instrumentation-bedrock opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Afterward, add the following code at the beginning of your main file:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from openinference.instrumentation.bedrock import BedrockInstrumentor
exporter = OTLPSpanExporter(
endpoint="https://<YOUR_ENV>.live.dynatrace.com/api/v2/otlp/v1/traces",
headers={"Authorization": "Bearer <your-platform-token>"},
)
provider = TracerProvider(resource=Resource.create({SERVICE_NAME: "<your-service>"}))
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
BedrockInstrumentor().instrument(tracer_provider=provider)

With a Classic access token, set the header to Api-Token <YOUR_DT_API_TOKEN> instead. The required scope is openpipeline:traces:ingest for a platform token, or openTelemetryTrace.ingest for a Classic access token.

Use the same <your-service> value as in the span routing condition.

For a complete runnable example, see the Amazon Bedrock OpenInference sample.

And that's it! Progressive delivery

Now you can:

  • Fetch all the user/AI interactions, training status, and more on demand.
  • Use NotebooksNotebooks or DashboardsDashboards to create data-driven documents for custom analytics on it.
GenAI Compliance Auditing
GenAI Compliance Auditing
Related tags
AI ObservabilityOpenPipelineOpenPipeline