Set up the Dynatrace Google Cloud log integration in a Kubernetes container (GKE)

Dynatrace version 1.230+

As an alternative to the main deployment, that provides Google Cloud monitoring for both metrics and logs, you can choose to set up monitoring for logs only. In this scenario, you'll run the deployment script in Google Cloud Shell. Instructions will depend on the location where you want the deployment script to run:

  • On a new GKE Autopilot cluster created automatically recommended

  • On an existing GKE standard or GKE Autopilot cluster

During setup, a new Pub/Sub subscription will be created. GKE will run a log forwarder container. After installation, you'll get logs, dashboards, and alerts for your configured services in Dynatrace.

For other deployment options, see Alternative deployment scenarios.

This page describes how to install version 1.0 of the Google Cloud integration on a GKE cluster.

Limitations

Dynatrace Google Cloud log integration supports up to 8 GB of data processing per hour (with base resources—without scaling). With bigger loads, messages will start to be retained in the PubSub Subscription. To measure latency, look for these metrics: Oldest unacked message age and Unacked messages. For scaling recommendations, see the scaling guide below.

Prerequisites

To deploy the integration, you need to make sure the following requirements are met on the machine where you are running the installation.

  • Linux OS only

  • Internet access

  • GKE Cluster access

  • Dynatrace environment access

    You need to configure the Dynatrace endpoint (environment, cluster or ActiveGate URL) to which the GKE cluster should send metrics and logs. Make sure that you have direct network access or, if there is a proxy or any other component present in between, that communication is not affected.

Tools

You can deploy the Dynatrace GCP integration in Google Cloud Shell or in bash.

If you use bash, you need to install:

Google Cloud permissions

Running the deployment script requires a list of permissions. You need to create a custom role (see below) and use it to deploy dynatrace-gcp-monitor.

  1. Create a YAML file named dynatrace-gcp-monitor-helm-deployment-role.yaml with the following content:
title: Dynatrace GCP Monitor helm deployment role
description: Role for Dynatrace GCP Monitor helm and pubsub deployment
stage: GA
includedPermissions:
  - container.clusters.get
  - container.configMaps.create
  - container.configMaps.delete
  - container.configMaps.get
  - container.configMaps.update
  - container.deployments.create
  - container.deployments.delete
  - container.deployments.get
  - container.deployments.update
  - container.namespaces.create
  - container.namespaces.get
  - container.pods.get
  - container.pods.list
  - container.secrets.create
  - container.secrets.delete
  - container.secrets.get
  - container.secrets.list
  - container.secrets.update
  - container.serviceAccounts.create
  - container.serviceAccounts.delete
  - container.serviceAccounts.get
  - iam.roles.create
  - iam.roles.list
  - iam.roles.update
  - iam.serviceAccounts.actAs
  - iam.serviceAccounts.create
  - iam.serviceAccounts.getIamPolicy
  - iam.serviceAccounts.list
  - iam.serviceAccounts.setIamPolicy
  - pubsub.subscriptions.create
  - pubsub.subscriptions.get
  - pubsub.subscriptions.list
  - pubsub.topics.attachSubscription
  - pubsub.topics.create
  - pubsub.topics.getIamPolicy
  - pubsub.topics.list
  - pubsub.topics.setIamPolicy
  - pubsub.topics.update
  - resourcemanager.projects.get
  - resourcemanager.projects.getIamPolicy
  - resourcemanager.projects.setIamPolicy
  - serviceusage.services.enable
  - serviceusage.services.get
  - serviceusage.services.list
  - serviceusage.services.use

Each group of permissions is used to handle the different resources included in the integration. Creation and access are for new resources, update is for reusing existing resources, and deletion is for uninstalling.

  • container.configMaps: for mapping secrets and other variables used by the containers.
  • container.deployments: for the K8s' deployment within the cluster (which includes the pods, containers, etc.).
  • container.namespaces: for the K8s namespace in which we are deploying the resources.
  • container.pods: for the K8s pods.
  • container.secrets: for the K8s secrets in which to store the data-sensitive variables.
  • container.serviceAccounts: for the SA to be taken in the K8s context.
  • iam.roles: for the necessary permissions for data collection.
  • iam.serviceAccounts: for the general context SA.
  • resourcemanager.projects: for handling the project in which we are deploying our integration.
  • serviceusage.services: for handling the services' APIs.
  • pubsub.subscriptions: for the PubSub subscription we are using to collect and ingest logs.
  • pubsub.topics: for the PubSub topic we are using to collect and ingest logs.
  1. Run the command below, replacing <your_project_ID> with the project ID where you want to deploy the Dynatrace integration.
gcloud iam roles create dynatrace_monitor.helm_deployment --project=<your_project_ID> --file=dynatrace-gcp-monitor-helm-deployment-role.yaml

Be sure to add this role to your Google Cloud user. For details, see Grant or revoke a single role.

Configure log export

  1. Run the following shell script in the Google Cloud project you've selected for deployment.

Be sure to replace <your-subscription-name> and <your-topic-name> with your own values.

wget https://raw.githubusercontent.com/dynatrace-oss/dynatrace-gcp-monitor/master/scripts/deploy-pubsub.sh
chmod +x deploy-pubsub.sh
./deploy-pubsub.sh --topic-name <your-topic-name> --subscription-name <your-subscription-name>
  1. Configure log export to send the desired logs to the Google Cloud Pub/Sub topic created above.

To monitor logs from multiple projects, you need to create Log Routing Sinks in each source project selecting as a destination for your main project (in which you also deployed the integration and the PubSub Topic and Subscription). For more information, see Route logs to supported destinations.

Dynatrace permissions

Log ingestion

  • If you are using Log Monitoring v1, enable the latest version of Dynatrace log monitoring.

  • Determine where log ingestion will be performed, according to your deployment. This distinction is important when configuring the parameters for this integration.

    • For SaaS deployments: SaaS log ingest, where log ingestion is performed directly through the Cluster API. recommended

    • For Managed deployments: You can use an existing ActiveGate for log ingestion. For information on how to deploy it, see ActiveGate installation.

Because of GCP's implementation of Cloud Function 2nd gen, logs from those resources will be linked to the underlying Cloud Run instances. Both extensions will have to be enabled.

To learn more, visit Google Cloud Functions version comparison.

Install

Complete the steps below to finish your setup.

Step 1

Download the Helm deployment package in Google Cloud Shell

Step 2

Configure parameter values

Step 3

Connect your Kubernetes cluster

Step 4

Run the deployment script

Step 1 Download the Helm deployment package in Google Cloud Shell 

wget -q "https://github.com/dynatrace-oss/dynatrace-gcp-monitor/releases/latest/download/helm-deployment-package.tar"; tar -xvf helm-deployment-package.tar; chmod +x helm-deployment-package/deploy-helm.sh

Step 2 Configure parameter values

The Helm deployment package contains a values.yaml file with the necessary configuration for this deployment. Go to helm-deployment-package/dynatrace-gcp-monitorand edit the values.yaml file, setting the required and optional parameter values as follows.

You might want to store this file somewhere for future updates, since it will be needed in case of redeployments. Also, keep in mind that its schema can change. In such case, you should use the new file and only copy over the parameter values.

Parameter nameDescriptionDefault value

gcpProjectId

required The ID of the Google Cloud project you've selected for deployment.

Your current project ID

deploymentType

required Set to logs.

all

dynatraceAccessKey

required Your Dynatrace API token with the required permissions.

dynatraceAccessKeySecretName

optional You can specify the key to fetch the endpoint from Google Cloud Secret Manager, instead of using dynatraceAccessKey.

dynatraceUrlSecretName

optional You can specify the key to fetch the endpoint from Google Cloud Secret Manager, instead of using dynatraceUrl.

dtSecurityContext

optional Assign the attribute value used for data segmentation, analysis, and permission mapping within the Dynatrace platform. Refer to Grant access to entities with security context for more information. If left empty, the value of gcpProjectId will be assigned automatically.

Value of gcpProjectId

dynatraceUrl

required For SaaS log ingestion, it's your environment URL (https://<your-environment-id>.live.dynatrace.com).

logsSubscriptionId

required The ID of your log Sink Pub/Sub subscription. For details, see Configure log export.

requireValidCertificate

optional If set to true, Dynatrace requires the SSL certificate of your Dynatrace environment.
For SaaS log ingestion, we recommend leaving the default value.

true

selfMonitoringEnabled

optional Send custom metrics to Google Cloud to quickly diagnose if dynatrace-gcp-monitor processes and sends logs to Dynatrace properly.For details, see Self-monitoring metrics for the Dynatrace Google Cloud integration.

false

serviceAccount

optional Name of the service account to be created.

dockerImage

optionalDynatrace Google Cloud Monitor Docker image. We recommend using the default value, but you can adapt it if needed.

dynatrace/dynatrace-gcp-monitor:v1-latest

logIngestContentMaxLength

optional The maximum content length of a log event. Should be the same as or lower than the setting on your Dynatrace environment.

8192

logIngestAttributeValueMaxLength

optional The maximum length of the log event attribute value. If it exceeds the server limit, content will be truncated.

250

logIngestRequestMaxEvents

optional The maximum number of log events in a single payload to the logs ingestion endpoint. If it exceeds the server limit, payload will be rejected with code 413.

5000

logIngestRequestMaxSize

optional The maximum size in bytes of a single payload to the logs ingestion endpoint. If it exceeds the server limit, payload will be rejected with code 413.

1048576

logIngestEventMaxAgeSeconds

optional Determines the maximum age of a forwarded log event. Should be the same as or lower than the setting on your Dynatrace environment.

86400

clusterIpv4Cidr

optional Set the IP address range for the pods in this cluster in CIDR notation, if you want to use a custom range.

servicesIpv4Cidr

optional Set the IP range for the services IPs. It can be specified as a netmask size or as in the CIDR notion.

useCustomMasterCidr

optional If set to true, you can specify the IPv4 CIDR range to use for the master network.

false

masterIpv4Cidr

optional IPv4 CIDR range to use for the master network. Requires the useCustomMasterCidr value to be true.

For DDU consumptiom information, see Monitoring consumption.

Step 3 Connect your Kubernetes cluster

  • If you want to have a new GKE Autopilot cluster created by the deployment script, add --create-autopilot-cluster to the script. Setting up the connection to the cluster will happen automatically in this case and you can proceed to step 4.
  • If you run the deployment script on an existing standard GKE or GKE Autopilot cluster, you can connect to your cluster from the Google Cloud console or via terminal. Follow the instructions below.

Step 4 Run the deployment script

  • If you run the deployment script on an existing standard GKE or GKE Autopilot cluster, the deployment script will create an IAM service account with the necessary roles and deploy dynatrace-gcp-monitor to your Kubernetes cluster.
  • If you run the deployment script with the --create-autopilot-cluster option, the deployment script will automatically create the new GKE Autopilot cluster and deploy dynatrace-gcp-monitor to it.

To run the deployment script, follow the instructions below.

Verify installation

To check whether installation was successful

  1. Check if the container is running.

    After the installation, it may take couple of minutes until the container is up and running.

    kubectl -n dynatrace get pods

  2. Check the container logs for errors or exceptions. You have two options:

  1. Check if dashboards are imported.

    In Dynatrace, go to Dashboards or Dashboards Classic (latest Dynatrace) and filter by Tag for Google Cloud. A number of dashboards for Google Cloud Services should be available.

Enable alerting

To activate alerting, you need to enable metric events for alerting in Dynatrace.

To enable metric events

  1. Go to Settings.
  2. In Anomaly detection, select Metric events.
  3. Filter for Google Cloud alerts and turn on On/Off for the alerts you want to activate.

View logs

After deploying the integration, you can view and analyze Google Cloud logs in Dynatrace if you go to Logs or Logs & Events (latest Dynatrace) and filter by cloud.provider: gcp.

Change deployment settings

Change parameters from values.yaml

To load a new values.yaml file, you need to upgrade your Helm release.

To update your Helm release

  1. Find out what Helm release version you're using.

    helm ls -n dynatrace

  2. Run the command below, making sure to replace <your-helm-release> with the value from the previous step.

    helm upgrade <your-helm-release> dynatrace-gcp-monitor -n dynatrace

For details, see Helm upgrade.

Change deployment type

To change the deployment type (all, metrics, or logs)

  1. Find out what Helm release version you're using.

    helm ls -n dynatrace

  2. Uninstall the release.

    Be sure to replace <your-helm-release> with the release name from the previous output.

    helm uninstall <your-helm-release> -n dynatrace

  3. Edit deploymentType in values.yaml with the new value and save the file.

  4. Run the deployment command again. For details, see Run the deployment script.

Verification

To investigate potential deployment and connectivity issues

  1. Verify installation
  2. Enable self-monitoring optional
  3. Check the dynatrace_gcp_<date_time>.log log file created during the installation process.
  • This file will be created each time the installation script runs.
  • The debug information won't contain sensitive data such as the Dynatrace access key.
  • If you are contacting a Dynatrace product expert via live chat:
    • Make sure to provide the dynatrace_gcp_<date_time>.log log file described in the previous step.
    • Provide version information.
      • For issues during installation, check the version.txt file.
      • For issues during runtime, check container logs.

Scaling guide for logs

The default container with 1.25vCPU and 1Gi (with default configuration) can handle 8 GB of log throughput per hour. Achieving more throughput requires allocating more resources to the container (scale up), increasing the number of container replicas (scale out), and changing configuration numbers to use allocated resources efficiently. All config variables can be found and changed in dynatrace-gcp-monitor-config.

The following table presents tested configuration and achieved throughput with scaled up&out containers:

Achieved throughput

Machine resources

Replica sets

Config variable values

~8MB/s => ~480MB/min

4vCPU 4Gi RAM

1

PARALLEL_PROCESSES=4,
NUMBER_OF_CONCURRENT_MESSAGE_PULL_COROUTINES = 30,
NUMBER_OF_CONCURRENT_PUSH_COROUTINES=20

~25MB/s => ~1.5GB/min => ~2TB/day

4vCPU 4Gi RAM

4

PARALLEL_PROCESSES=4,
NUMBER_OF_CONCURRENT_MESSAGE_PULL_COROUTINES = 30,
NUMBER_OF_CONCURRENT_PUSH_COROUTINES=20

~46MB/s => ~2.7GB/min => ~4TB/day

4vCPU 4Gi RAM

6

PARALLEL_PROCESSES=4,
NUMBER_OF_CONCURRENT_MESSAGE_PULL_COROUTINES = 30,
NUMBER_OF_CONCURRENT_PUSH_COROUTINES=20

Autoscaling guide for logs

Autoscaling works only for logs type of deployment, not all.

We recommend manually scaling the container to have a 4vCPU 4Gi machine and then enabling autoscaling.

GCP provides autoscaling of containers in both directions: horizontal and vertical. However, Dynatrace recommends only horizontal scaling.

If you have a 4vCPU 4Gi machine, you can enable autoscaling horizontally. However, we don't recommend scaling horizontally with the base resources of the container (1.25vCPU, 1Gi). It hasn't been proven to be efficient during testing. One 4vCPU machine does better than four 1vCPU machines. To enable autoscaling horizontally, use the horizontal autoscaling command:

kubectl autoscale deployment dynatrace-gcp-monitor --namespace dynatrace --cpu-percent=90 --min=1 --max=6

Autoscaling is recommended only when you have a minimum of 450 MB/min throughput and can provide a 4vCPU 4Gi RAM machine. Autoscaling is only scaling out, not scaling the machine up.

We don't recommend scaling vertically because every time a machine is scaled up, an environment variable needs to be changed to create more processes corresponding to machine cores.

Uninstall

  1. Find out what Helm release version you're using.
helm ls -n dynatrace
  1. Uninstall the release.

Be sure to replace <your-helm-release> with the release name from the previous output.

helm uninstall <your-helm-release> -n dynatrace

Alternatively, you can delete the namespace.

kubectl delete namespace dynatrace
  1. To remove all monitoring assets (such as dashboards and alerts) from Dynatrace, you need to remove all Google Cloud extensions.

You can find and delete relevant extensions via Dynatrace Hub.

Make sure to uninstall the following resources manually:
  • The initial Role created and attached to the Service Account that you used to deploy the integration.
  • The PubSub Topic.
  • The PubSub Subscription.
  • The LogRoute Sink.

Monitoring consumption

DDU consumption applies to cloud Log Monitoring. See DDUs for Log Monitoring for details.

Related topics