Migrate from classic full stack to application monitoring mode

Dynatrace Operator version 1.0.0+

This guide describes the steps required to migrate your Dynatrace deployment from classic full stack monitoring to application monitoring mode.

Advantages

To only monitor selected applications on Kubernetes, application monitoring offers a flexible approach with the following benefits:

The application monitoring mode, similar to the cloud native full stack mode, prevents race conditions that can occur when OneAgent DaemonSet pods and monitored application pods start simultaneously.
By leveraging Kubernetes concepts such as admission webhooks and CSI driver for Code Module injection, application monitoring mode reduces the required privileges for OneAgent.

Considerations and implications

When switching to application monitoring, previously deployed OneAgents will get deactivated and deep monitoring of applications will stop. Consequently, restarting all application pods that require deep monitoring becomes mandatory. Restarting these pods ensures that applications are reinjected, allowing deep monitoring to resume.
In application monitoring mode, container monitoring rules are ignored. Instead, label selectors should be employed to precisely manage OneAgent injection.
Log monitoring requires additional setup.

Migrate to application monitoring mode

This section provides all the information needed to migrate from classic to application monitoring mode.

Using CRI-O container runtime

The standard migration procedure described below requires OneAgent version 1.281 or higher for Kubernetes clusters using CRI-O as their container runtime, so you need to upgrade OneAgents accordingly before continuing with the steps below.

If that upgrade cannot be performed, follow the Running CRI-O with OneAgent versions 1.279 or earlier procedure for an alternative migration flow, and then return to step 1 of this procedure.

recommended

Add CSI Driver to existing installation:

helm upgrade dynatrace-operator oci://docker.io/dynatrace/dynatrace-operator \
--atomic \
--csidriver.enabled="true" \ # By default CSI driver is enabled
--namespace dynatrace

Kubernetes

kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v1.3.2/kubernetes-csi.yaml

OpenShift

kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v1.3.2/openshift-csi.yaml

Reconfigure (existing) DynaKube for application monitoring mode:

The following side-by-side comparison outlines how to reconfigure a DynaKube CR from classic full stack to application monitoring:

apiVersion: dynatrace.com/v1beta2
kind: DynaKube
metadata:
  name: dynakube
  namespace: dynatrace
spec:
  apiUrl: https://<environment-id>.live.dynatrace.com/api

  networkZone: <network-zone>

  oneAgent:
    classicFullStack:
      args:
        - "--set-host-group=<host-group>"

activeGate:
    capabilities:
      - routing
      - kubernetes-monitoring
      - dynatrace-api

apiVersion: dynatrace.com/v1beta2
kind: DynaKube
metadata:
  name: dynakube
  namespace: dynatrace
spec:
  apiUrl: https://<environment-id>.live.dynatrace.com/api

  networkZone: <network-zone>

  oneAgent:
    hostGroup: <host-group>
    applicationMonitoring:
      useCSIDriver: true # recommended; defaults to false

activeGate:
    capabilities:
      - routing
      - kubernetes-monitoring
      - dynatrace-api

For further information on how to configure DynaKube for application monitoring mode, visit the deployment guide or DynaKube parameters. Alternatively, download the DynaKube custom resource sample for application monitoring from GitHub and adapt the DynaKube custom resource according to your requirements.

Apply the DynaKube custom resource:

Run the command below to apply the DynaKube custom resource. A validation webhook will provide helpful error messages if there's a problem.
```
kubectl apply -f dynakube.yaml
```
This action will lead to the removal of OneAgents in classic full stack mode and subsequently result in the termination of deep monitoring for application pods shortly thereafter.
Wait for code modules to become ready:

Dynatrace Operator picks up the changes in the DynaKube custom resource and ensures code modules are available on each node.

The CSI driver emits Kubernetes events attached to the DynaKube custom resource when the code modules are ready and available on each node. Wait until an event has been logged for each node before proceeding with the next step.
Restart application workloads:

Restart all application workloads promptly to trigger code module injection and enable deep monitoring minimizing monitoring outages.

Running CRI-O with OneAgent versions 1.279 or earlier

This section outlines the migration procedure for Kubernetes clusters utilizing a CRI-O container runtime and running OneAgent version 279 or earlier.

It is necessary to remove CRI-O hooks installed and utilized for OneAgent injection in classic full-stack mode. For additional details on CRI-O hooks, refer to this Red Hat blog post.

Follow these instructions to successfully migrate from classic full-stack mode:

Delete DynaKube custom resource:

Delete the DynaKube configured in classic full-stack mode by running the following command:
```
kubectl delete dynakube -n dynatrace <dynakube-name>
```
This action will lead to the removal of OneAgents in classic full-stack mode and subsequently result in the termination of deep monitoring for application pods shortly thereafter. Additionally, if Kubernetes monitoring is configured in the DynaKube custom resource, Kubernetes monitoring will stop instantly with the removal of the ActiveGate.
Wait for the OneAgent pods to terminate.
Run uninstall script:

Execute the uninstall.sh script on each node previously monitored by OneAgent to remove mentioned CRI-O hooks:
```
/opt/dynatrace/oneagent/agent/uninstall.sh
```
Continue with step 1 of the standard migration procedure.

Changes in Kubernetes resources

The migration impacts several Kubernetes resources, altering their functions or introducing new components to support the application monitoring mode. Key changes include:

Component

Classic full stack

Application monitoring

Dynatrace Oneagent

Deployed as a DaemonSet
Collect host metrics on nodes
Inject code modules into application pods

Not present

Dynatrace Webhook Server

Validates DynaKube definitions

Validates DynaKube definitions
Inject code modules into application pods by modifying pod definitions

Dynatrace CSI Driver

optional

Not present

Deployed as a DaemonSet
Optimizes the download of code modules to speed up pod injection and reduce storage consumption