General troubleshooting

This guide provides general troubleshooting steps and guidance for common issues encountered when using Dynatrace with Kubernetes. It covers how to access debug logs, use the troubleshoot subcommand, or generate a support archive.

Troubleshoot common Dynatrace Operator setup issues using the `troubleshoot` subcommand

Dynatrace Operator version 0.9.0+

Run the command below to retrieve a basic output on DynaKube status, such as:

Namespace: If the dynatrace namespace exists (name can be overwritten via parameter)
DynaKube:
- If CustomResourceDefinition exists
- If CustomResource with the given name exists (name can be overwritten via parameter)
- If the API URL ends with /api
- If the secret name is the same as DynaKube (or .spec.tokens if used)
- If the secret has Dynatrace Operator and Data Ingest tokens set
- If the secret for customPullSecret is defined
Environment: If your environment is reachable from the Dynatrace Operator pod using the same parameters as the Dynatrace Operator binary (such as proxy and certificate).
OneAgent and ActiveGate image: If the registry is accessible; if the image is accessible from the Dynatrace Operator pod using the registry from the environment with (custom) pull secret.

1kubectl exec deploy/dynatrace-operator -n dynatrace -- dynatrace-operator troubleshoot

If you use a different DynaKube name, add the --dynakube <your_dynakube_name> argument to the command.

Example output if there are no errors for the above-mentioned fields:

1{"level":"info","ts":"2022-09-12T08:45:21.437Z","logger":"dynatrace-operator-version","msg":"dynatrace-operator","version":"<operator version>","gitCommit":"<commithash>","buildDate":"<release date>","goVersion":"<go version>","platform":"<platform>"}
2[namespace ]     --- checking if namespace 'dynatrace' exists ...
3[namespace ]      √  using namespace 'dynatrace'
4[dynakube  ]     --- checking if 'dynatrace:dynakube' Dynakube is configured correctly
5[dynakube  ]         CRD for Dynakube exists
6[dynakube  ]         using 'dynatrace:dynakube' Dynakube
7[dynakube  ]         checking if api url is valid
8[dynakube  ]         api url is valid
9[dynakube  ]         checking if secret is valid
10[dynakube  ]         'dynatrace:dynakube' secret exists
11[dynakube  ]         secret token 'apiToken' exists
12[dynakube  ]         customPullSecret not used
13[dynakube  ]         pull secret 'dynatrace:dynakube-pull-secret' exists
14[dynakube  ]         secret token '.dockerconfigjson' exists
15[dynakube  ]         proxy secret not used
16[dynakube  ]      √  'dynatrace:dynakube' Dynakube is valid
17[dtcluster ]     --- checking if tenant is accessible ...
18[dtcluster ]      √  tenant is accessible

Debug logs

By default, OneAgent logs are located in /var/log/dynatrace/oneagent.

To debug Dynatrace Operator issues, run

You might also want to check the logs from OneAgent pods deployed through Dynatrace Operator.

Generate a support archive using the `support-archive` subcommand

Dynatrace Operator version 0.11.0+

Use support-archive to generate a support archive containing all the files that can be potentially useful for the RFA analysis:

operator-version.txt—a file containing the current Operator version information
logs—logs from all containers of the Dynatrace Operator pods in the Dynatrace Operator namespace (usually dynatrace); this also includes logs of previous containers, if available:
- dynatrace-operator
- dynatrace-webhook
- dynatrace-oneagent-csi-driver
manifests—the Kubernetes manifests for Dynatrace Operator components and deployed DynaKubes in the Dynatrace Operator namespace
troubleshoot.txt—output of a troubleshooting command that is automatically executed by the support-archive subcommand
supportarchive_console.log—complete output of the support-archive subcommand

Usage

To create a support archive, execute the following command.

1kubectl exec -n dynatrace deployment/dynatrace-operator -- dynatrace-operator support-archive

The collected files are stored in a zipped tarball and can be downloaded from the pod using the kubectl cp command.

1kubectl -n dynatrace cp <operator pod name>:/tmp/dynatrace-operator/operator-support-archive.tgz ./tmp/dynatrace-operator/operator-support-archive.tgz

The recommended approach is to use the --stdout parameter line switch to stream the tarball directly to your disk.

1kubectl exec -n dynatrace deployment/dynatrace-operator -- dynatrace-operator support-archive --stdout > operator-support-archive.tgz

If you use the --stdout parameter, all support archive command output is written to stderr so as not to corrupt the support archive tar file.

Sample output

The following is sample output from running support-archive with the --stdout parameter.

1kubectl exec -n dynatrace deployment/dynatrace-operator -- dynatrace-operator support-archive --stdout > operator-support-archive.tgz

1[support-archive]       dynatrace-operator      {"version": "v0.11.0", "gitCommit": "...", "buildDate": "...", "goVersion": "...", "platform": "linux/amd64"}
2[support-archive]       Storing operator version into operator-version.txt
3[support-archive]       Starting log collection
4[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/server.log
5[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/provisioner.log
6[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/registrar.log
7[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/liveness-probe.log
8[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/server.log
9[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/provisioner.log
10[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/registrar.log
11[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/liveness-probe.log
12[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/server.log
13[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/provisioner.log
14[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/registrar.log
15[support-archive]       Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/liveness-probe.log
16[support-archive]       Successfully collected logs logs/dynatrace-operator-6d9fd9b9fc-sw5ll/dynatrace-operator.log
17[support-archive]       Successfully collected logs logs/dynatrace-webhook-7d84599455-bfkmp/webhook.log
18[support-archive]       Successfully collected logs logs/dynatrace-webhook-7d84599455-vhkrh/webhook.log
19[support-archive]       Starting K8S object collection
20[support-archive]       Collected manifest for manifests/injected_namespaces/Namespace-default.yaml
21[support-archive]       Collected manifest for manifests/dynatrace/Namespace-dynatrace.yaml
22[support-archive]       Collected manifest for manifests/dynatrace/Deployment-dynatrace-operator.yaml
23[support-archive]       Collected manifest for manifests/dynatrace/Deployment-dynatrace-webhook.yaml
24[support-archive]       Collected manifest for manifests/dynatrace/StatefulSet-dynakube-activegate.yaml
25[support-archive]       Collected manifest for manifests/dynatrace/DaemonSet-dynakube-oneagent.yaml
26[support-archive]       Collected manifest for manifests/dynatrace/DaemonSet-dynatrace-oneagent-csi-driver.yaml
27[support-archive]       Collected manifest for manifests/dynatrace/DynaKube-dynakube.yaml

Debug configuration and monitoring issues using the Kubernetes Monitoring Statistics extension

The Kubernetes Monitoring Statistics extension can help you:

Troubleshoot your Kubernetes monitoring setup
Troubleshoot your Prometheus integration setup
Get detailed insights into queries from Dynatrace to the Kubernetes API
Receive alerts when your Kubernetes monitoring setup experiences issues
Get alerted on slow response times of your Kubernetes API

Potential issues when changing the monitoring mode

Changing the monitoring mode from classicFullStackto cloudNativeFullStack affects the host ID calculations for monitored hosts, leading to new IDs being assigned and no connection between old and new entities.
If you want to change the monitoring method from applicationMonitoring or cloudNativeFullstack to classicFullstack or hostMonitoring, you need to restart all the pods that were previously instrumented with applicationMonitoring or cloudNativeFullstack.