Deploy Prometheus scraping with Dynatrace Operator

Latest Dynatrace
How-to guide
15-min read
Published Sep 24, 2026
Preview

Dynatrace Operator version 1.11+

Dynatrace Operator can run a complete Prometheus scraping pipeline inside your Kubernetes cluster. You declare a single PrometheusMonitoring custom resource, and Dynatrace Operator deploys and configures a target allocator, a scraper pool, and a gateway pool that ingests the collected metrics into Dynatrace.

Scrape targets are declared with the standard Prometheus Operator custom resources (ServiceMonitor, PodMonitor, ScrapeConfig, and Probe), so existing Prometheus configuration can be reused as-is.

Experimental feature

This integration is experimental and turned off by default. It isn't generally available, and the PrometheusMonitoring API can change in a breaking way between Dynatrace Operator releases, including field renames and removals that require you to rewrite your manifests. Don't use it for production monitoring.

To try it out, you need to explicitly opt in through a Helm value. See Step 1.

How it works

PrometheusMonitoring deploys the same three-tier architecture described in Scrape Prometheus metrics with the OTel Collector (standard), but Dynatrace Operator owns the rollout and the collector configuration instead of you managing Helm values. For a diagram of the data flow, see Architecture overview.

Target allocator: a Deployment that watches ServiceMonitor, PodMonitor, ScrapeConfig, and Probe resources, resolves them into scrape targets, and distributes those targets across the scraper pool using consistent hashing. Because the hashing is deterministic, every replica computes the same assignment without coordination, so additional replicas add failure tolerance only.
Scraper pool: a Deployment of OpenTelemetry Collectors. Each scraper polls the target allocator for its own share of targets, scrapes them, and forwards the metrics to the gateway pool through the load_balancing exporter. The exporter routes by resource hash, so all metrics from a given source pod consistently reach the same gateway.
Gateway pool: a StatefulSet of OpenTelemetry Collectors. It receives OTLP from the scrapers, runs the stateful metric_start_time and cumulative_to_delta processors, enriches the metrics with Kubernetes context through the k8sattributes and transform processors, and exports them to Dynatrace over OTLP/HTTP.

These three components are everything Dynatrace Operator deploys.

Self-monitoring for the pipeline is planned but isn't part of this first experimental release. Until it ships, diagnose problems from the component logs and the target allocator API.

Connection settings

All connection settings (API URL, tokens, proxy, and trusted CA certificates) come from the DynaKube that the PrometheusMonitoring references. There's no separate endpoint or token configuration. The gateway exports to <apiUrl>/v2/otlp on your Dynatrace environment.

With a proxy configured on the DynaKube, the gateway receives HTTP_PROXY, HTTPS_PROXY, and a NO_PROXY covering the Kubernetes API and (if an ActiveGate is enabled) the ActiveGate service. Your Dynatrace environment host is deliberately not in NO_PROXY, so the OTLP export goes through the proxy.
With spec.trustedCAs set on the DynaKube, the CA bundle is mounted into the gateway and used for the export connection.
The scraper receives neither proxy nor trusted-CA settings. Configure TLS and proxy behavior for individual scrape targets in the Prometheus custom resource that defines them.

A future release may route the exported metrics through an ActiveGate rather than sending them to your Dynatrace environment directly. That path isn't implemented yet.

Prerequisites

Dynatrace Operator version 1.11+, installed with the Prometheus integration turned on. See Step 1.
The Prometheus Operator custom resource definitions (ServiceMonitor, PodMonitor, ScrapeConfig, and Probe) present in the cluster. They're mandatory. Install them yourself or let the Dynatrace Operator Helm chart install them. See Step 1.
A DynaKube in the same namespace as the PrometheusMonitoring.
The DynaKube token secret needs to contain a dataIngestToken with the Ingest metrics (metrics.ingest) scope. The gateway uses it to export the scraped metrics. See Tokens and permissions.
Dynatrace OpenTelemetry Collector version 0.55.0 or later for the scraper and the gateway. Earlier versions don't support the configuration Dynatrace Operator generates.
Image pull access to the public Amazon ECR registry or a mirror of it.

Step 1: Install Dynatrace Operator with the Prometheus integration

The Prometheus integration is gated behind the experimental.enablePrometheus Helm value. Without it, Dynatrace Operator doesn't install the PrometheusMonitoring custom resource definition, doesn't create the required RBAC, and doesn't start the PrometheusMonitoring controller.

The target allocator discovers scrape targets exclusively through Prometheus Operator custom resources, so the ServiceMonitor, PodMonitor, ScrapeConfig, and Probe custom resource definitions are mandatory. Without them, the integration has no way to learn about a single target. Pick one of the two options below to get them into the cluster.

Set prometheus.installCRDs=true alongside experimental.enablePrometheus=true, and the chart deploys the four Prometheus Operator definitions with the integration. Use this option on a cluster that doesn't have them yet.

New installation

helm install dynatrace-operator oci://public.ecr.aws/dynatrace/dynatrace-operator \
  --create-namespace \
  --namespace dynatrace \
  --set experimental.enablePrometheus=true \
  --set prometheus.installCRDs=true \
  --atomic

Existing installation: Add the same values to the helm upgrade command you normally use for Dynatrace Operator.

helm upgrade dynatrace-operator oci://public.ecr.aws/dynatrace/dynatrace-operator \
  --namespace dynatrace \
  --reset-then-reuse-values \
  --set experimental.enablePrometheus=true \
  --set prometheus.installCRDs=true \
  --atomic

Both values default to false and are independent of each other. experimental.enablePrometheus turns the integration on, and prometheus.installCRDs governs only whether the chart brings the Prometheus Operator definitions along.

Verify that the custom resource definition and the ServiceAccounts are present.

kubectl get crd prometheusmonitorings.dynatrace.com
kubectl get serviceaccount -n dynatrace \
  dynatrace-target-allocator dynatrace-prometheus-scraper dynatrace-prometheus-gateway

Don't switch prometheus.installCRDs back to false on a cluster that relies on the Prometheus Operator definitions. The chart manages them as ordinary release resources, so turning the value off deletes them. Deleting a definition deletes every ServiceMonitor, PodMonitor, ScrapeConfig, and Probe in the cluster, including resources that have nothing to do with Dynatrace.

Both the target allocator and any existing Prometheus Operator read the same custom resources. Use distinct labels for the resources each side should pick up, so the same target isn't scraped twice. See Control which resources are discovered.

Step 2: Create a DynaKube

The PrometheusMonitoring doesn't hold any connection settings of its own. It references a DynaKube by name, and Dynatrace Operator reads the API URL, the tokens, the proxy configuration, and the trusted CA certificates from it.

A minimal DynaKube is sufficient. No OneAgent, ActiveGate, or other capability needs to be enabled.

apiVersion: dynatrace.com/v1beta6
kind: DynaKube
metadata:
  name: dynakube
  namespace: dynatrace
spec:
  apiUrl: https://<environment-id>.live.dynatrace.com/api
  tokens: dynakube-tokens

The PrometheusMonitoring is only reconciled while the DynaKube reports Running. Check the phase before you continue.

kubectl get dynakube dynakube -n dynatrace -o jsonpath='{.status.phase}'

If you pull component images from a private registry, configure customPullSecret on the DynaKube. Dynatrace Operator propagates it as imagePullSecrets to the target allocator, scraper, and gateway pods. See DynaKube parameters and Component images.

Step 3: Create the PrometheusMonitoring resource

Create a PrometheusMonitoring in the same namespace as the DynaKube. The example names it app-metrics; Dynatrace Operator derives the names of all workloads it creates from that name, so pick something you'll recognize in kubectl get pods. dynaKubeName is the only field the schema requires; every other field has a default.

Component images are resolved automatically if your Dynatrace environment supports it. If not, you need to set image for all three components explicitly.

apiVersion: dynatrace.com/v1alpha1
kind: PrometheusMonitoring
metadata:
  name: app-metrics
  namespace: dynatrace
spec:
  dynaKubeName: dynakube

  targetAllocator: {}

  scraper: {}

  gateway: {}

Apply it and wait for the pipeline to become available.

kubectl apply -f app-metrics.yaml
kubectl get prometheusmonitoring -n dynatrace

NAME          DYNAKUBE   STATUS    AGE
app-metrics   dynakube   Running   2m

The resource kind has the short names pm and pms, so kubectl get pms -n dynatrace is equivalent.

A validating webhook checks every PrometheusMonitoring on create and update. The webhook never rejects a resource, but it warns when a Prometheus Operator custom resource definition isn't installed:

Warning: Required Prometheus Operator CRDs are not installed on this cluster: ServiceMonitor.monitoring.coreos.com. Related functionality will not work until they are installed.

The resource is still accepted, so nothing in .status reports the problem afterward. Install the missing definitions before you declare any scrape targets. See Step 1.

With this configuration, the target allocator discovers only the Prometheus custom resources labeled prometheus.dynatrace.com: "true", in any namespace. The default scrape interval is 60 seconds, and a resource that sets its own interval overrides it. To change the label or restrict the namespaces, see Control which resources are discovered.

Setting replicas on a component pins the count and conflicts with a HorizontalPodAutoscaler targeting the same workload; Dynatrace Operator and the HPA will fight over the value. Use one or the other: set replicas in the PrometheusMonitoring for static scaling, or leave it unset and manage the count with an HPA. See Autoscaling.

Step 4: Declare scrape targets with a PodMonitor

A PodMonitor selects pods directly by label, without requiring a Service in front of them. Use it when your application pods expose a metrics port but aren't fronted by a Service, or when you want to scrape every pod individually.

The following example scrapes all pods labeled k8s-app: kube-dns in the kube-system namespace on their metrics port.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kube-dns
  labels:
    prometheus.dynatrace.com: "true"
spec:
  podMetricsEndpoints:
    - path: /metrics
      port: metrics
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: kube-dns

Note the following:

The prometheus.dynatrace.com: "true" label on the PodMonitor itself is what makes the target allocator consider the resource. Without it, the resource is ignored.

This label is the target allocator's default discovery selector, so the example works without configuring anything on the PrometheusMonitoring. If you'd rather use your own label, or restrict discovery to specific namespaces, set customResourceSelector and customResourceNamespaceSelector on spec.targetAllocator. See Control which resources are discovered.
spec.selector matches pod labels. By default, a PodMonitor selects pods in its own namespace only. To select pods in other namespaces, add spec.namespaceSelector.
port refers to the name of a container port, not a port number. Your pod template needs to name the port.
```
ports:
  - name: metrics
    containerPort: 9100
```
If your pods expose an unnamed port, use portNumber instead of port.
interval and scrapeTimeout override the target allocator's default interval for these targets. scrapeTimeout needs to be smaller than interval.

Apply the PodMonitor and confirm that the target allocator picked it up.

kubectl apply -f podmonitor.yaml
kubectl get podmonitor -A -l prometheus.dynatrace.com=true

To scrape through a Service instead of individual pods, use a ServiceMonitor with spec.endpoints in place of spec.podMetricsEndpoints. Both resource kinds are discovered by the same selector.

Step 5: Verify the rollout

Check the PrometheusMonitoring status

All three conditions need to report True.

kubectl get prometheusmonitoring app-metrics -n dynatrace \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.message}{"\n"}{end}'

GatewayAvailable=True gateway is ready
TargetAllocatorAvailable=True target allocator is ready
ScraperAvailable=True scraper is ready

Check the workloads

kubectl get deployment,statefulset -n dynatrace \
  -l app.kubernetes.io/managed-by=dynatrace-operator

You should see the app-metrics-allocator and app-metrics-scraper Deployments and the app-metrics-gateway StatefulSet.

Inspect the target assignments

Query the target allocator API to confirm that it discovered your targets and distributed them across the scraper pool.

Port-forward to the target allocator Service.

kubectl port-forward -n dynatrace svc/app-metrics-allocator 8080:80

List the discovered scrape jobs.

curl -s http://localhost:8080/jobs | jq .

List the targets of a job and the scraper each one is assigned to. Replace <job-name> with a job name from the previous step.
```
curl -s http://localhost:8080/jobs/<job-name>/targets | jq .
```
Verify that all expected targets appear, that they're spread across more than one collector ID, and that none are unassigned.

Confirm the metrics in Dynatrace

After at least two scrape intervals, open Notebooks and query the scraped metrics. The following DQL example lists all metrics that arrived from a given namespace.

timeseries value = avg(`<metric-key>`), by: { k8s.namespace.name, k8s.workload.name }

Scraped metrics carry the Kubernetes resource attributes added by the gateway, such as k8s.namespace.name, k8s.pod.name, k8s.workload.name, k8s.workload.kind, and k8s.cluster.name.

Step 6: Uninstall the Prometheus integration

Turning off experimental.enablePrometheus removes the integration's custom resource definition and RBAC. Cleaning up your PrometheusMonitoring resources and the workloads they own then depends on Kubernetes garbage collection resolving in the right order. Delete them yourself first, and confirm the workloads are gone before you change the flag.

Delete the PrometheusMonitoring resources and wait for their workloads to disappear.

kubectl delete prometheusmonitoring --all -n dynatrace
kubectl get deployment,statefulset,configmap,service -n dynatrace \
  -l app.kubernetes.io/managed-by=dynatrace-operator

Turn the flag off.

helm upgrade dynatrace-operator oci://public.ecr.aws/dynatrace/dynatrace-operator \
  --namespace dynatrace \
  --reset-then-reuse-values \
  --set experimental.enablePrometheus=false \
  --set prometheus.installCRDs=true \
  --atomic

Dynatrace Operator pod restarts without the PrometheusMonitoring controller, so nothing reconciles these resources afterward.

Reversing the order can leave orphaned workloads. If you turn off the flag while PrometheusMonitoring resources still exist, the RBAC disappears immediately while the pods may keep running, now without API permissions and with no controller watching them. If you installed with prometheus.installCRDs=false, the Prometheus Operator custom resource definitions aren't managed by the chart at all, so turning off the flag removes only the integration RBAC and leaves the resources and their workloads in place.

Control which resources are discovered

Two selectors on spec.targetAllocator decide which Prometheus custom resources become scrape targets. Both apply identically to ServiceMonitor, PodMonitor, ScrapeConfig, and Probe.

customResourceSelector filters by the labels on the custom resource. If you omit it, it defaults to prometheus.dynatrace.com: "true".

customResourceNamespaceSelector filters by the labels on the namespace that holds the custom resource. If you omit it, no namespace filter is applied.

The target allocator watches the matching resources, so a new or edited resource takes effect as soon as you apply it.

Configuration	Result
Both omitted	Resources labeled `prometheus.dynatrace.com: "true"` in any namespace.
`customResourceSelector: {}`	All resources, regardless of their labels. Use with care: this also picks up resources managed by another Prometheus instance.
`customResourceSelector` with `matchLabels` or `matchExpressions`	Only resources matching the selector.
`customResourceNamespaceSelector: {}`	All namespaces. Equivalent to omitting the field.
`customResourceNamespaceSelector` with `matchLabels`	Only resources in matching namespaces.

The following example picks up resources labeled metrics-pipeline: dynatrace in namespaces labeled monitoring: enabled, and scrapes their targets every 30 seconds unless a target sets its own interval.

spec:
  targetAllocator:
    scrapeInterval: 30s
    customResourceSelector:
      matchLabels:
        metrics-pipeline: dynatrace
    customResourceNamespaceSelector:
      matchLabels:
        monitoring: enabled

Both selectors are atomic. A server-side apply replaces the whole selector rather than merging individual keys into it.

Discovery is label-based only

Discovery works exclusively through the label on the Prometheus custom resource. There's no built-in preset that discovers workloads from Dynatrace pod annotations such as metrics.dynatrace.com/scrape, and Dynatrace Operator never creates a ScrapeConfig on your behalf.

If you're moving off the annotation-based discovery of the ActiveGate Kubernetes module, reproduce it with your own ScrapeConfig that keeps pods carrying those annotations and reads their port, path, and secure values. For a complete relabeling example you can copy, see ScrapeConfig. Remember to add the discovery label to the ScrapeConfig itself so the target allocator picks it up.

Don't confuse the two annotation families. metrics.dynatrace.com/* annotations describe how to scrape a pod and are meaningful only inside a ScrapeConfig you write. metadata.dynatrace.com/* annotations enrich the resulting metrics with resource attributes and are read automatically. See Resource attributes.

Sizing and scaling

Each tier scales on a different dimension, so size them independently.

The target allocator scales with the number of scrape targets, not with metric volume. It holds all Prometheus service-discovery metadata in memory and doesn't process data itself, so it stays cheap even under heavy ingest.

Scale it vertically, through resources. Adding replicas doesn't lower its memory footprint: every replica has to hold the complete set of discovered targets and the complete list of scrapers, because that's precisely what lets each replica compute the same assignment as the others without coordinating. A new replica therefore arrives at the same working set as the existing ones, and doubling the replica count doubles the memory the tier consumes. Add replicas for failure tolerance only.
The scraper pool scales with metric volume and is memory-bound. The Prometheus receiver buffers an entire scrape response in memory before processing, which produces a sharp spike at every scrape cycle.
The gateway pool scales with metric volume and series cardinality. The cumulative_to_delta processor keeps per-series state, so its memory grows with the number of tracked series.

Resource limits and replica counts

The PrometheusMonitoring integration uses the same three-tier architecture as Scrape Prometheus metrics with the OTel Collector (standard). Per-pod resource limits and replica counts at load levels from 1 million to 90 million data points per minute are documented in Resource sizing reference and apply here without change.

Set resources on each component to match those values, and run at least two replicas of each component. The target allocator is the single source of truth for target assignments, and a gateway that restarts loses the in-memory delta state of every series routed through it.

The following example shows a configuration for approximately 30 million data points per minute (~334 scrape targets at a 60-second interval), using the resource limits from the reference table.

apiVersion: dynatrace.com/v1alpha1
kind: PrometheusMonitoring
metadata:
  name: app-metrics
  namespace: dynatrace
spec:
  dynaKubeName: dynakube

  targetAllocator:
    image: public.ecr.aws/dynatrace/otel-target-allocator:0.5.0
    replicas: 2
    resources:
      requests:
        cpu: "1"
        memory: 1Gi
      limits:
        cpu: "1"
        memory: 1Gi

  scraper:
    image: public.ecr.aws/dynatrace/dynatrace-otel-collector:0.55.0
    replicas: 38
    resources:
      requests:
        cpu: "1"
        memory: 6Gi
      limits:
        cpu: "1"
        memory: 6Gi

  gateway:
    image: public.ecr.aws/dynatrace/dynatrace-otel-collector:0.55.0
    replicas: 8
    resources:
      requests:
        cpu: "2"
        memory: 8Gi
      limits:
        cpu: "2"
        memory: 8Gi

Autoscaling

Dynatrace Operator doesn't create HorizontalPodAutoscalers for any of the three components. If you want autoscaling, create the HPAs yourself against the operator-managed workloads.

To let an HPA own the replica count, leave replicas unset on that component. Dynatrace Operator then doesn't manage .spec.replicas, so it won't fight the autoscaler. Setting replicas explicitly pins the count and overrides whatever the HPA decides.

Because both the target allocator and the load_balancing exporter distribute work by hash rather than by measured load, adding replicas doesn't rebalance existing assignments, and a single scrape target can never be split across scrapers. For the resulting hotspot behavior and how to mitigate it, see Limitations of hash-based load distribution.

Component images

Each component runs an image you either pin yourself or let Dynatrace Operator resolve.

How resolution is meant to work

When image is empty on a component, Dynatrace Operator asks your Dynatrace environment which image to run. Updates are automatically picked up during regular reconciliations.

Set every image explicitly

Automatic resolution isn't usable in the experimental phase. The gateway and the scraper have no component identifier to look up yet, so their lookup can't succeed, and because the gateway reconciles first, a missing gateway image aborts the reconcile before the target allocator is attempted. Set image on all three components, as shown in Step 3.

The kubectl explain description for this field still claims Dynatrace Operator falls back to the latest public image. Disregard it for now.

Failed resolution surfaces as a component condition with reason Error and the message image discovery failed, and the workload isn't created. See Troubleshooting.

Pinning images for air-gapped clusters

Setting image skips the lookup entirely, which is what you want when the cluster can't reach the Dynatrace API or a public registry. Mirror the three images into your registry and reference them by digest so the tag can't drift:

spec:
  dynaKubeName: dynakube
  targetAllocator:
    image: registry.example.com/dynatrace/otel-target-allocator@sha256:<digest>
  scraper:
    image: registry.example.com/dynatrace/dynatrace-otel-collector@sha256:<digest>
  gateway:
    image: registry.example.com/dynatrace/dynatrace-otel-collector@sha256:<digest>

For a registry that needs credentials, configure customPullSecret on the DynaKube. Dynatrace Operator propagates it as imagePullSecrets to these pods. See Step 2.

To route resolution through a mirror of the public registry instead of pinning, set spec.publicRegistryOverride. Reconciliation fails if the image the endpoint returns doesn't come from the registry you asked for.

Security

What the chart creates

With experimental.enablePrometheus=true, the chart adds the PrometheusMonitoring custom resource definition, three ServiceAccounts, four ClusterRoles with matching bindings, and one namespaced Role and binding for Dynatrace Operator itself. Every component grant is read-only.

ServiceAccount	Read access
`dynatrace-target-allocator`	`monitoring.coreos.com` `servicemonitors`, `podmonitors`, `scrapeconfigs`, `probes`; core `namespaces`, `pods`, `services`, `endpoints`, `nodes`, `nodes/metrics`, `secrets`; `endpointslices`; `ingresses`; the `/metrics` endpoint
`dynatrace-prometheus-scraper`	core `pods`, `endpoints`, `services`, `namespaces`, `nodes`; `endpointslices`; `apps` workloads; `batch` `jobs` and `cronjobs`
`dynatrace-prometheus-gateway`	core `pods`, `namespaces`, `nodes`; `endpointslices`; `replicasets`; `jobs` and `cronjobs`: the metadata the `k8sattributes` processor needs for enrichment

Dynatrace Operator additionally gets get, list, update, and watch on prometheusmonitorings, update on their finalizers, and patch on their status, all scoped to its own namespace. On OpenShift, each component is granted use on the nonroot-v2 security context constraint.

All three containers run with a restricted security context: read-only root filesystem, no privilege escalation, all capabilities dropped, and the RuntimeDefault seccomp profile. The target allocator and the scraper run as non-root user and group 65532. The gateway runs as non-root user and group 10001, with a matching pod fsGroup so it can read the data ingest token without the token file being world-readable.

Because the feature is off by default, a default installation is unaffected. These objects only exist once you opt in.

Permissions the installer needs

Enabling the integration makes helm install and helm upgrade create cluster-scoped objects, so the identity that runs Helm needs to be able to create CustomResourceDefinitions, ClusterRoles and ClusterRoleBindings, a namespaced Role and RoleBinding, and ServiceAccounts. Adding prometheus.installCRDs=true puts four more CustomResourceDefinitions under the same requirement.

If your installer doesn't hold the escalate verb, Kubernetes additionally requires it to already hold every permission it hands out (escalation prevention). Because the Prometheus integration creates ClusterRoles for the three component ServiceAccounts, the installer needs to pre-hold all of those permissions. A restricted installer is rejected with attempt to grant extra privileges otherwise.

Apply the following ClusterRole and binding, and replace the subject with the identity that runs Helm. This is a standalone grant that doesn't depend on or extend the deployer samples used for a regular Dynatrace Operator installation.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: dynatrace-prometheus-installer
rules:
  # Object types the chart creates for the integration.
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["get", "list", "create", "update", "patch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterroles", "clusterrolebindings"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["roles", "rolebindings"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # Passed on to the target allocator, which discovers scrape targets from these resources.
  - apiGroups: ["monitoring.coreos.com"]
    resources: ["servicemonitors", "podmonitors", "scrapeconfigs", "probes"]
    verbs: ["get", "list", "watch"]
  # Passed on to the target allocator (Kubernetes service discovery).
  - apiGroups: [""]
    resources: ["namespaces", "pods", "services", "endpoints", "nodes", "nodes/metrics", "secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]
  # Passed on to the scraper (Kubernetes service discovery and workload metadata).
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "watch"]
  # Passed on to the operator (PrometheusMonitoring reconciliation).
  - apiGroups: ["dynatrace.com"]
    resources: ["prometheusmonitorings"]
    verbs: ["get", "list", "update", "watch"]
  - apiGroups: ["dynatrace.com"]
    resources: ["prometheusmonitorings/finalizers"]
    verbs: ["update"]
  - apiGroups: ["dynatrace.com"]
    resources: ["prometheusmonitorings/status"]
    verbs: ["patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dynatrace-prometheus-installer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dynatrace-prometheus-installer
subjects:
  - kind: ServiceAccount
    name: <installer-service-account>
    namespace: <installer-namespace>

On OpenShift, each component is additionally granted use on the nonroot-v2 security context constraint. If your installer is restricted, add that grant to the ClusterRole as well.

Installers that hold escalate, and the default cluster-admin path, need none of this.

Next steps

See PrometheusMonitoring parameters for a complete list of fields, status conditions, and managed Kubernetes objects.
To compare this setup with a manually configured pipeline, see Scrape Prometheus metrics with the OTel Collector (standard).

Known limitations

The following gaps apply to the experimental phase and are expected to close before general availability.

Configuration

spec.targetAllocator.image, spec.scraper.image, and spec.gateway.image need to be set explicitly. See Component images.
spec.targetAllocator.updateStrategy and spec.gateway.updateStrategy are ignored, so the Kubernetes defaults apply. In particular, the rollingUpdate.partition canary workflow for the gateway has no effect. spec.scraper.updateStrategy works, but only when you set its type. Supplying rollingUpdate alone is dropped.

Isolation and validation

Only one PrometheusMonitoring per namespace is usable. Nothing prevents you from creating more, but each target allocator claims every scraper pool in its namespace, so two resources would double-assign targets and duplicate metrics. Use separate namespaces if you need more than one pipeline.
The validating webhook doesn't reject any configuration. Beyond the schema checks in the custom resource definition, misconfiguration surfaces as a failed condition in .status rather than a rejected kubectl apply. A dynaKubeName pointing at a nonexistent DynaKube is accepted, and reports Deploying before any condition exists and Error afterward. The one thing the webhook does report is a warning about missing Prometheus Operator custom resource definitions. See Step 3.
Restricted Helm installers need two additional RBAC grants that aren't in the deployer samples. See Permissions the installer needs.

Transport and observability

The target allocator's HTTPS endpoint isn't served. The Service exposes port 443 and the container exposes port 8443, but nothing listens there, and scraper-to-target-allocator traffic is plain HTTP.
The scraper-to-gateway hop is unencrypted by design, on the assumption that the cluster-internal network is trusted.

Troubleshooting

Common failure modes

Symptom	Cause	Resolution
`phase: Deploying` and no conditions	The referenced DynaKube doesn't exist, or isn't in the `Running` phase	Check `spec.dynaKubeName` and the DynaKube's own status. The `PrometheusMonitoring` is only reconciled while the DynaKube reports `Running`.
Error log `dynakube not found`	No DynaKube with that name exists in the `PrometheusMonitoring` namespace	Both resources need to be in the same namespace.
Error log `image discovery failed`	`image` isn't set for a component, and automatic resolution isn't available yet	Set `image` on `spec.targetAllocator`, `spec.scraper`, and `spec.gateway` explicitly.
Error log `image registry "…" does not match requested registry "…"`	`publicRegistryOverride` doesn't match the registry of the resolved image	Remove the override, or set it to the registry that your environment returns.
`kubectl apply` fails with `no matches for kind "PrometheusMonitoring"`	Dynatrace Operator was installed without `experimental.enablePrometheus=true`	Repeat the Helm installation or upgrade with the flag set. See Step 1.
Target allocator API returns no jobs	No Prometheus custom resource matches `customResourceSelector`, or the resources are in a namespace excluded by `customResourceNamespaceSelector`	Verify the labels on the `PodMonitor` or `ServiceMonitor` itself, not on the target workload. Check the target allocator logs for permission errors.
A job is listed, but has no targets	The resource's `spec.selector` matches no pods or Services, or the named port doesn't exist	Compare `spec.selector` against the actual workload labels. For a `ServiceMonitor`, confirm the Service has active endpoints. For a `PodMonitor`, confirm the container port name matches `port`.
Targets are listed but no metrics arrive	The scraper can't reach the target endpoint, or the gateway can't reach Dynatrace	Check for a `NetworkPolicy` between the `dynatrace` namespace and the target namespace. Check the gateway logs for export errors, and confirm the DynaKube `dataIngestToken` has the `metrics.ingest` scope.
Duplicate metric values	The same target is also scraped by another Prometheus instance that reads the same custom resources	Use distinct labels per pipeline and narrow `customResourceSelector` accordingly.
Scraper or gateway pods are `OOMKilled`	The Prometheus receiver buffers a whole scrape response in memory; the `cumulative_to_delta` processor on the gateway keeps per-series state	Raise `resources.limits.memory` for the affected component, or split heavy targets into separate scrape targets. For sizing guidance, see Resource sizing reference.

Collect logs

Each component writes to standard output. Select its pods by label and name the container explicitly.

# Target allocator  -  discovery and target assignment
kubectl logs -n dynatrace -c targetallocator --tail=100 \
  -l app.kubernetes.io/instance=otel-allocator

# Scraper  -  target polling and scraping
kubectl logs -n dynatrace -c scraper --tail=100 \
  -l app.kubernetes.io/instance=otel-scraper

# Gateway  -  enrichment and export to Dynatrace
kubectl logs -n dynatrace -c gateway --tail=100 \
  -l app.kubernetes.io/instance=otel-gateway

The gateway is a StatefulSet, so its pods have stable ordinals. Address one directly when you're debugging a single replica:

kubectl logs -n dynatrace app-metrics-gateway-0 -c gateway

To raise verbosity on any of the three components, use args. See Container arguments.

Collect a support archive

The support archive includes the PrometheusMonitoring custom resource definition and all PrometheusMonitoring resources, alongside Dynatrace Operator logs. To collect it for a support case, see Generate a support archive.