Try it free

Troubleshoot GCP connections

  • Latest Dynatrace
  • How-to guide

Use this checklist when a GCP connection is set up but no topology, metrics, or logs appear in Dynatrace.

Before you begin

This checklist uses dtctl, the Dynatrace command-line tool. Install it and authenticate it against your Dynatrace environment before running the commands — see Create a GCP connection via CLI for setup.

Each command takes your monitoring configuration ID (<mconfig_id>). To find it, run dtctl get gcp monitoring to list your monitoring configurations, or open the connection in the Settings app and copy the Configuration ID from its properties.

Step 1: Check connection and monitoring config status

dtctl describe gcp monitoring <mconfig_id>

Look for:

  • Enabled: true — if false, the monitoring config is disabled. Enable it via the Settings app or with dtctl enable gcp monitoring <mconfig_id>.
  • Service account — the output shows both the Dynatrace principal and your monitoring service account. Verify your service account email (<sa-id>@<project>.iam.gserviceaccount.com) is present and not empty. If only the Dynatrace principal appears and the service account is blank, the connection was never linked to your service account — set it with dtctl update gcp connection --serviceAccountId <sa-email> or enter the email in the Settings app.
  • Extension — with manual setup (gcloud CLI and dtctl), the GCP extension must be activated once per Dynatrace environment. The Settings app wizard does this automatically. Check it with dtctl get extension com.dynatrace.extension.da-gcp. If it is missing, activate it with dtctl create extension --hub-extension com.dynatrace.extension.da-gcp.
  • Metrics ingest and Smartscape updates — no data is expected for the first few minutes after deployment. If still no data after 15 minutes, continue the checklist.
  • Events — no recent data acquisition events means no errors yet. If error messages appear here, they indicate authentication or permission failures.

Step 2: Verify the service account exists

gcloud iam service-accounts describe <sa-id>@<project>.iam.gserviceaccount.com \
  --project=<project>

If not found, the service account was not created. Complete the Google Cloud setup steps for your deployment method and verify the service account exists before proceeding.

Step 3: Verify IAM roles on the monitored project

gcloud projects get-iam-policy <project> \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:<sa-id>@<project>.iam.gserviceaccount.com" \
  --format="value(bindings.role)"

Required roles:

  • roles/monitoring.viewer
  • roles/cloudasset.viewer
  • roles/browser

Missing roles mean Dynatrace cannot read metrics or topology. Re-run terraform apply or grant them manually.

roles/browser at folder scope is required for folder filtering. Granting it only at project scope does not enable folder listing.

Step 4: Verify Dynatrace principal has impersonation right

Find the Dynatrace principal email under Dynatrace > Principal in the output of:

dtctl describe gcp monitoring <mconfig_id>
Dynatrace:
  Principal ID:  <internal-id>
  Principal:     dynatrace-<env-id>-<hash>@dtp-prod-gcp-auth.iam.gserviceaccount.com

Then check it has roles/iam.serviceAccountTokenCreator on your monitoring service account:

gcloud iam service-accounts get-iam-policy <sa-id>@<project>.iam.gserviceaccount.com \
  --project=<project>

The Dynatrace principal must appear as a member of roles/iam.serviceAccountTokenCreator. If missing, Dynatrace cannot authenticate and no data flows.

Confirm authentication in the audit logs

Every time Dynatrace polls for metrics, it first exchanges a short-lived token by calling iamcredentials.googleapis.com / GenerateAccessToken on your monitoring service account. This call is visible in Cloud Audit Logs on the service account project and is a reliable signal that the integration is authenticating.

gcloud logging read \
  'logName="projects/<sa-project>/logs/cloudaudit.googleapis.com%2Fdata_access"
   AND protoPayload.serviceName="iamcredentials.googleapis.com"
   AND protoPayload.methodName="GenerateAccessToken"' \
  --project=<sa-project> \
  --limit=10 \
  --format="table(timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.request.name)"

A successful log entry shows:

  • principalEmail — the Dynatrace principal.
  • request.name — your monitoring service account being impersonated.
  • authorizationInfo[].granted: true — impersonation succeeded.

If no entries appear, either Data Access audit logging is not enabled for iamcredentials.googleapis.com on the project, or Dynatrace has not yet attempted authentication.

Step 5a: Verify required APIs on the service account project

gcloud services list --project=<sa-project> --enabled \
  --format="value(config.name)" \
  --filter="config.name=(monitoring.googleapis.com OR cloudasset.googleapis.com OR cloudresourcemanager.googleapis.com OR pubsub.googleapis.com OR iam.googleapis.com OR logging.googleapis.com OR dataflow.googleapis.com OR storage.googleapis.com OR secretmanager.googleapis.com)"

The five base APIs must always be listed: monitoring, cloudasset, cloudresourcemanager, pubsub, and iam. If you enabled log forwarding, four more must also appear: logging, dataflow, storage, and secretmanager. Missing APIs cause silent failures. Enable any that are missing:

gcloud services enable <api-name> --project=<sa-project>

Step 5b: Verify monitoring API on each monitored project

This is separate from the service account project. monitoring.googleapis.com must be enabled on every project whose metrics you want to collect. Topology can appear without it — metrics will be silently missing.

gcloud services list --project=<monitored-project> --enabled \
  --format="value(config.name)" \
  --filter="config.name=monitoring.googleapis.com"

If the output is empty, the API is disabled on that project. Enable it:

gcloud services enable monitoring.googleapis.com --project=<monitored-project>

Repeat for each project in your monitoring scope.

Step 6: Check that the required tokens were set before terraform apply

Neither token is stored in terraform.tfvars; both are passed as environment variables.

  • Settings token (TF_VAR_dynatrace_settings_token) — always required. It creates the connection and enables the monitoring configuration.
  • Ingest token (TF_VAR_dynatrace_api_token) — required only when log forwarding is enabled (logs_enabled = true). If it is missing while logs_enabled = true, terraform plan fails with the error dynatrace_api_token must be non-empty when the logs module is enabled. If you are not forwarding logs, an empty ingest token is expected and harmless.

The asset inventory feed does not use a Dynatrace token. Its Pub/Sub push subscription authenticates to Dynatrace with a Google-issued OIDC token, so the feed works without the ingest token.

[ -n "$TF_VAR_dynatrace_settings_token" ] && echo "settings token is set" || echo "settings token is NOT set"
[ -n "$TF_VAR_dynatrace_api_token" ] && echo "ingest token is set" || echo "ingest token is NOT set"   # only relevant when logs_enabled = true

If a required token is empty in your current shell, this was likely the cause. Re-export it and re-run terraform apply:

export TF_VAR_dynatrace_settings_token="<your-settings-token>"
export TF_VAR_dynatrace_api_token="<your-ingest-token>"   # only when forwarding logs
terraform apply

Step 7: Wait for the first data cycle

SignalFirst data expected

Metrics

Within one 5-minute polling cycle after a successful terraform apply, plus Google's metric ingest delay

Topology (asset inventory)

Within about 15 minutes — Dynatrace scrapes the full inventory when the connection is established

If dtctl describe shows no data but it has been less than 15 minutes, wait before continuing. After the initial scrape, Dynatrace works from asset feed events (if you enabled the asset feed) and runs a full scrape every hour.

Metrics depend on topology: Dynatrace collects Cloud Monitoring metrics only for resources that already exist as Smartscape entities. Until a resource is discovered in topology, its metrics are not collected — even after the five-minute metric cycle. If metrics are missing while topology is still populating, wait for the initial topology scrape (about 15 minutes) before troubleshooting metrics further. Metrics ingested via Ingest any Google Cloud Monitor metrics are exempt from this requirement only for resource types that are not in the supported services list. For supported resource types, the topology requirement still applies.

Step 8 (logs only): Verify Dataflow job is running

gcloud dataflow jobs list --project=<project> --region=<region> \
  --filter="name~<mconfig_id>" \
  --format="table(id,name,state,location)"

Use the region that you configured for log forwarding (region in terraform.tfvars).

Look for the job whose name contains your mconfig_id. Its state must be Running. Previous failed or cancelled attempts may also appear — only the current job matters.

If the current job state is FAILED or CANCELLED:

  1. Check Secret Manager — the Ingest token secret must hold a valid token. It can be empty or invalid if you created it manually or the token expired (see Step 6).
  2. Check that roles/dataflow.admin and roles/logging.admin were granted to the account that ran the setup.
  3. Re-run the setup after fixing the root cause.

Step 9 (logs only): Verify log sink and Pub/Sub are wired

Terraform doesn't create log sinks. You create them in each project whose logs you want to forward, so check the sinks in those projects, and the topics in the service account project:

gcloud logging sinks list --project=<source-project> \
  --filter="destination~dt-logs-"


gcloud pubsub topics list --project=<service-account-project> \
  --filter="labels.dt_created_by=dynatrace"

Expected: in every project whose logs you want to forward, one log sink with a destination that points to the Pub/Sub topic dt-logs-<mconfig_id> in the service account project. If a project has no such sink, its logs aren't forwarded — see Create log sinks. If a sink's FILTER column shows (empty filter), all logs of that project are forwarded.

Expected Pub/Sub topics:

  • dt-logs-<mconfig_id> — main log ingestion topic
  • dt-logs-dlq-<mconfig_id> — dead-letter queue for failed messages

Check the subscriptions in the service account project as well:

gcloud pubsub subscriptions list --project=<service-account-project> \
  --filter="name~dt-logs"

Expected Pub/Sub subscriptions:

  • dt-logs-sub-<mconfig_id> — main subscription that the Dataflow job reads
  • dt-logs-dlq-sub-<mconfig_id> — dead-letter subscription for inspecting failed messages
  • dt-logs-dlq-replay-sub-<mconfig_id> — replay subscription that the Dataflow job reads to retry failed messages

The topics and subscriptions that the Terraform bundle creates carry the label dt_created_by=dynatrace. The asset feed topic dt-asset-feed-<mconfig_id> and its subscription dt-asset-feed-push-<mconfig_id> exist only if you enabled the asset inventory feed. See Asset feed events not arriving.

Missing resources indicate an incomplete setup. Rerun the setup steps for your deployment method.

Step 10: Audit all deployed resources via labels

This label-based audit — and the labels.dt_created_by=dynatrace filter in Step 9 — applies only to resources deployed by the Terraform template, which labels every resource it creates. If you deployed manually with the gcloud CLI, the resources are not labelled; filter by the dt-logs- and dt-asset-feed- name prefixes instead.

All resources created by the Terraform template are labelled with dt_created_by=dynatrace and dt_connection_id=<mconfig_id>. Filter by the connection ID so that the resources of other connections in the same project don't appear. Use this to confirm the deployment is complete:

gcloud asset search-all-resources \
  --project=<project> \
  --query="labels.dt_connection_id=<mconfig_id>" \
  --format="table(assetType,displayName)"

A complete deployment includes: Dataflow job, Compute instance (Dataflow worker), Pub/Sub topics and subscriptions, and GCS buckets (log staging).

Cloud Asset feeds, the Secret Manager secret, the service account, and IAM bindings don't carry labels and don't appear in this list.

If the list is empty or incomplete, Terraform did not apply cleanly. Check terraform.tfstate for errors and re-run terraform apply.

If you enabled the asset inventory feed (asset_feed_enabled = true), verify the feeds separately. List the feeds of each scope you configured; the command lists only the feeds directly under the scope you pass:

gcloud asset feeds list --project=<project>
gcloud asset feeds list --folder=<folder-id>
gcloud asset feeds list --organization=<organization-id>

Expected: one feed named dt-asset-feed-<mconfig_id> for each project, folder, and organization in the asset feed scope.

Connection status: Pending

A connection shows Pending after creation while Google Cloud setup completes. Status changes to Healthy once Dynatrace successfully authenticates.

While Pending, the Manage and Enable/Disable actions in the menu are disabled. If the connection stays Pending, verify:

  1. Google Cloud deployment complete: Terraform has been applied or manual setup is done. See Via Terraform or Via gcloud CLI and dtctl.
  2. IAM configuration correct: Service account exists, required APIs are enabled, and the Dynatrace principal has impersonation rights. See prerequisites.

Connection status: Unhealthy

SymptomPossible causeSolution

Authentication failed

Service account doesn't exist

Verify service account was created in Google Cloud

Authentication failed

Dynatrace service account lacks impersonation rights

Grant roles/iam.serviceAccountTokenCreator to the Dynatrace service account on your monitoring service account

Authentication failed

Domain-restricted sharing blocks Dynatrace

Add the Dynatrace customer identity to allowed domains

No metrics data

Required APIs not enabled

Enable monitoring.googleapis.com

No topology data

Required APIs not enabled

Enable cloudasset.googleapis.com and cloudresourcemanager.googleapis.com

Partial data

Service account lacks permissions on some projects

Grant roles/monitoring.viewer, roles/cloudasset.viewer, and roles/browser on all projects to monitor

Healthy connection but no data appears

Filters may be excluding all projects. Check folder and project filters in connection settings.

No metrics — scoping projects mode with no scoping projects

If the monitoring configuration is set to collect metrics from scoping projects only, Dynatrace polls only GCP projects whose metrics scope includes other projects. If no such scoping project exists in your environment, zero metrics are collected with no error — the connection stays Healthy. Dynatrace caches scoping project detection, so recent changes to your Google Cloud metrics scopes can take some time to appear.

Check which monitoring configurations use scoping projects mode:

dtctl get gcp monitoring -o yaml | grep -B1 "observabilityscopesenabled: true"

If the output is empty, no monitoring configuration has scoping projects mode enabled. If one or more IDs appear, those configurations collect metrics only from scoping projects.

Check whether any scoping projects exist in your GCP environment:

A scoping project is a GCP project whose Cloud Monitoring metrics scope includes other projects. Check each project's metrics scope via the Cloud Monitoring API:

TOKEN=$(gcloud auth print-access-token)
for PROJECT_ID in $(gcloud projects list --format="value(projectId)"); do
  RESPONSE=$(curl -s \
    -H "Authorization: Bearer $TOKEN" \
    "https://monitoring.googleapis.com/v1/projects/${PROJECT_ID}/metricsScopes")
  COUNT=$(echo "$RESPONSE" | grep -c '"name":')
  if [ "$COUNT" -gt 1 ]; then
    echo "Scoping project: $PROJECT_ID (scope includes $COUNT projects)"
    echo "$RESPONSE" | grep '"name":' | grep -v "metricsScopes/${PROJECT_ID}\"" | sed 's/.*projects\//  - /' | sed 's/".*//'
  fi
done

If the loop finds no scoping projects, switch the affected monitoring configuration to all visible projects:

  • Settings app: Edit the monitoring configuration and change the metrics collection scope.

  • dtctl: Export the configuration, disable scoping projects mode, and apply:

    dtctl get gcp monitoring <mconfig_id> -o yaml | \
      sed 's/observabilityscopesenabled: true/observabilityscopesenabled: false/' | \
      dtctl apply -f -

Metrics missing or delayed at scale

If metrics are missing or arrive late in large environments, the Cloud Monitoring API quota of the project that holds the monitoring service account may be exhausted. Check the ListTimeSeries quota usage in that project and request an increase if needed. See GCP performance and quotas.

Missing metrics for a specific service

The service may be disabled in the metrics configuration. Go to Settings Settings > Collect and capture > Cloud and virtualization > Google Cloud, open the connection, select Edit, and enable the service under Google Cloud Monitor metrics.

Missing metrics for specific resources (service is enabled)

Dynatrace only collects metrics for resources already discovered in GCP topology. If a resource was recently created or not yet discovered, its metrics are not collected until the entity appears in Smartscape on Grail.

  • Check whether the resource appears in Clouds Clouds or via a DQL query in Notebooks Notebooks.
  • If not yet visible: wait for the next topology poll cycle or verify the asset type is included in your asset feed configuration.
  • Metrics ingested via Ingest any Google Cloud Monitor metrics are not subject to this restriction only for resource types that are not in the supported services list. For supported resource types, such as gce_instance, the restriction still applies.

Labels and tags not appearing on entities or telemetry

Google Cloud label and tag enrichment is best effort. Due to the eventual consistency of Cloud Asset Inventory, values on Smartscape entities may be briefly out of date after a resource change.

  • Labels consistently missing: Verify label enrichment is configured in the connection settings under Label and tag enrichment.
  • Tags consistently missing: Confirm the tag keys are listed under Google Cloud tags in the connection settings.
  • For time-sensitive queries, consult the resource directly in the Google Cloud Console.

Asset feed events not arriving

If the asset inventory feed is configured but topology is not updating in near-real time:

  1. Verify the Pub/Sub subscription is in ACTIVE state:

    gcloud pubsub subscriptions describe dt-asset-feed-push-${MCONFIG_ID} --project=${PROJECT_ID}

  2. Check for delivery errors in the subscription's error statistics in the Google Cloud Console (Pub/Sub > Subscriptions > your subscription > Monitoring tab).

  3. Verify IAM: The Pub/Sub service agent must have roles/iam.serviceAccountTokenCreator on your monitoring service account. The Cloud Asset service agent must have roles/pubsub.publisher on the Pub/Sub topic: the agent of each project that has a project-scope feed, and the agent of the service account project for folder and organization feeds.

  4. Verify the asset feed exists for each monitored scope:

    gcloud asset feeds list --project=${PROJECT_ID}

  5. Check asset types: If a resource type is missing from topology, verify it is in the --asset-types list of the feed. See Update asset feed asset types.

Log forwarding not delivering logs to Dynatrace

  1. Check the Dataflow job is in Running state (see also Step 8):

    gcloud dataflow jobs list --region=${REGION} --project=${PROJECT_ID} \
        --filter="name~${MCONFIG_ID}" --format='value(state)'

  2. Check the dead-letter queue for permanently failed messages:

    gcloud pubsub subscriptions pull dt-logs-dlq-sub-${MCONFIG_ID} \
        --project=${PROJECT_ID} --limit=5

    Without --auto-ack, the messages stay in the subscription. Adding --auto-ack would remove them.

  3. Verify the log sink in each source project routes to the correct Pub/Sub topic and its writer identity has roles/pubsub.publisher on that topic. See Create log sinks.

  4. Token expired: If the Log Ingest token has expired, rotate it — see Rotate Log Ingest token.

Duplicate data from overlapping connections

If the same Google Cloud project is monitored by more than one connection, its resources are ingested more than once. This produces duplicate metrics, topology, and logs, inflates ingest and cost, and makes the data harder to interpret. Dynatrace detects overlapping visibility and flags it on the connection's Projects tab.

Find projects monitored by more than one connection:

smartscapeEdges "*"
| filter source_type == "DA_MONITORING_CONFIGURATION" and type == "monitors" and startsWith(target_type, "GCP")
| lookup [smartscapeNodes "GCP*" | fieldsAdd project = gcp.project.id | fields id, project],
         sourceField:target_id, lookupField:id, prefix:"p."
| summarize connections = countDistinct(source_id), by:{ project = p.project }
| filter connections > 1

This situation must be resolved so that each project is monitored by exactly one connection. Use one of the following:

  • Adjust IAM permissions — grant the monitoring roles (roles/monitoring.viewer, roles/cloudasset.viewer, roles/browser) for each project to only one connection's service account.
  • Remove service account visibility — revoke the overlapping service account's access to the project, for example by removing a roles/browser binding granted at the folder or organization level that exposes the project.
  • Use Dynatrace project filters — scope each connection to a distinct set of projects with folder or project include filters in the connection settings.
Related tags
Infrastructure Observability