GCP performance and quotas
- Latest Dynatrace
- Explanation
Dynatrace reads metrics from the Cloud Monitoring API, discovers resources through Cloud Asset Inventory, and can forward logs through Pub/Sub and Dataflow. Google Cloud limits how many requests and how much capacity each of these services provides. If your environment needs more than a quota allows, data arrives late or not at all.
Increase the Cloud Monitoring API quota in the Google Cloud project that holds the monitoring service account. The number of requests Dynatrace sends depends on how many metrics you configure, so Dynatrace can't tell you in advance how much quota you need. Pay particular attention to the Time series queries per minute quota (ListTimeSeries). See Cloud Monitoring API quota.
Google Cloud can change quota values at any time. This page names the quotas that matter and links to the Google Cloud pages that list the current values. Always check the quota in your own project.
Cloud Monitoring API quota
Which quota to raise
Dynatrace polls the timeSeries.list method of the Cloud Monitoring API every five minutes. These calls count against the Time series queries per minute quota of the monitoring.googleapis.com service. Google Cloud applies quotas per project. See Cloud Monitoring quotas and limits.
Google Cloud doesn't publish a fixed default for this quota. To see the value in your project, open IAM & Admin > Quotas & System Limits in the Google Cloud console and filter for the Cloud Monitoring API, or run:
gcloud quotas info list --service=monitoring.googleapis.com --project=<service-account-project-id>
The default can be enough for many environments, but it can be too low when Dynatrace collects many metrics. The more projects, services, and metric types you collect, the more requests Dynatrace sends in each polling cycle. Because the demand depends on your configuration, check the quota usage after you onboard, and raise the limit before usage reaches it. See Watch your quota usage.
Where the quota applies
Dynatrace calls the Cloud Monitoring API with your monitoring service account. Google Cloud counts all of these calls against the quota of the project that holds that service account, no matter how many projects you monitor. The monitored and scoping projects aren't charged. Raise the quota in the service account project.
Request an increase
- In the Google Cloud console, select the project that holds your monitoring service account.
- Open IAM & Admin > Quotas & System Limits.
- Filter for the Cloud Monitoring API and the Time series queries per minute quota.
- Select the quota, choose Edit, and enter the new limit. For larger values, choose Apply for higher quota and describe your use case.
You need a role such as Quota Administrator (roles/servicemanagement.quotaAdmin) in that project. Google Cloud evaluates most requests automatically and escalates some to a review. Google Cloud doesn't publish a lead time, so plan ahead for large rollouts. For details, see View and manage quotas.
Recognize an exhausted quota
When the quota is exhausted, Dynatrace can't read all metrics in a polling cycle, so metrics arrive late or are missing. Dynatrace doesn't show the Google Cloud error itself. To find out whether the quota is the cause:
- Open the connection in
Settings and check the connection logs on the Health tab. - In the Google Cloud console, open IAM & Admin > Quotas & System Limits in the service account project and check the usage of the Time series queries per minute quota.
If the usage is at the limit, request an increase.
Watch your quota usage
Use the Google Cloud quota dashboard and quota alerts to find out before you reach the limit. See Monitor quota usage.
Reduce the polling load
If you can't raise the quota, or you want to stay well below it, reduce the number of requests.
- Collect from scoping projects only. With the Scoping metric collection scope, Dynatrace polls only your scoping projects instead of every visible project, which reduces Cloud Monitoring API traffic. See Metric collection scope.
- Use Recommended metric sets. Auto-discovery collects every metric of a service and can generate a large number of requests. Use it only to explore. See Metric collection sets.
- Split large environments across connections. Each connection uses its own service account. If you place each service account in a different project, each connection has its own quota. Group projects with the same monitoring requirements under one service account.
- Keep a metrics scope at 375 projects or fewer. Cloud Monitoring officially supports up to 375 projects per metrics scope. You can add more, but Google Cloud warns of performance issues. See Cloud Monitoring quotas and limits.
Limits of other Google Cloud services
Topology discovery and log forwarding use other Google Cloud services. Each has its own quotas, which Google Cloud applies per project or per organization.
| Service | What to know | Details |
|---|---|---|
Cloud Asset Inventory | Dynatrace lists assets once an hour. Google Cloud applies request quotas per project and per organization, and all tools in your organization share the organization quota. A parent (project, folder, or organization) can have up to 200 asset feeds. A new or changed feed can take up to 10 minutes to take effect. | |
Resource Manager | Listing and searching projects and folders has per-project request limits. Very large resource hierarchies can reach them. | |
Pub/Sub | If the receiver of a push subscription responds slowly or with an error other than a success status, Pub/Sub backs off delivery. Subscriptions keep unacknowledged messages for 7 days by default. | |
Dataflow | The log forwarding job needs Compute Engine capacity: CPUs, in-use IP addresses, and persistent disk in the job's region. A project can run 25 concurrent Dataflow jobs by default. Dataflow doesn't raise Compute Engine quotas for you. | |
Cloud Logging | A project can have 200 sinks by default, and a sink can have 50 exclusion filters. Exclusion filters don't reduce the write quota. A narrow inclusion filter does reduce log volume. |
Organization policies that block a setup
Organization policies can make a deployment fail in a way that looks like a quota or permission problem. Check them when a resource can't be created or a role can't be granted.
- Domain restriction (
iam.allowedPolicyMemberDomains). This policy can stop Google Cloud from granting roles to service accounts and service agents outside your allowed domains. Add your organization to the allowed principals. See Restricting identities by domain. - External IP access (
compute.vmExternalIpAccess). Dataflow workers are Compute Engine VMs. If your organization denies external IP addresses, the log forwarding job can fail to start. Check this policy when the Dataflow job doesn't start. - Resource locations (
gcp.resourceLocations). This policy blocks resources in locations you didn't allow. Create the resources in an allowed region. See Restricting resource locations.
For other causes of failed deployments, see Troubleshoot GCP connections.