Containerized, auto-scalable private Synthetic locations on Kubernetes

Dynatrace version 1.264+

Containerized, auto-scalable private Synthetic locations on Kubernetes and its commercial distribution OpenShift are an alternative to deploying Synthetic-enabled ActiveGates on separate hosts or virtual machines and then assigning them to private locations for the execution of synthetic monitors.

Unlike individual Synthetic-enabled ActiveGates that are deployed and assigned to private locations (and then tracked via utilization metrics) containerized locations are deployed as a whole, with a minimum and maximum number of ActiveGates as the necessary input parameters.

Kubernetes and OpenShift aren't just additional supported ActiveGate platforms along with Windows and Linux; with this offering, containerized private Synthetic locations:

You can execute scheduled as well as on-demand executions of all types of synthetic monitors on containerized locations.

You can manage Kubernetes/OpenShift locations via the Dynatrace web UI and the existing Synthetic - Locations, nodes, and configuration API v2. Additional Early Adopter endpoints in this API facilitate the deployment of Kubernetes locations; the new endpoints help you generate the commands that need to be executed on the Kubernetes cluster.

Architecture

Containerized private Synthetic locations are deployed as a whole.

Locations are scaled automatically by adjusting the number of ActiveGates per location by the following additional parts of the containerized location architecture.

Requirements

Containerized private Synthetic locations are supported with Dynatrace version 1.264+ on Kubernetes 1.22–1.25 with persistent volume and kubectl support.

Internet connectivity is required to access the public repositories where Docker images for the Synthetic-enabled ActiveGate and Synthetic metric adapter are available. These image locations are referenced in the respective template files—see Install a containerized location and Update a containerized location below.

The ActiveGate hardware requirements below are listed by size.

Best practices and caveats

ActiveGates

Locations

Synthetic metric adapter

Auto-scaling specifics

For auto-scaling purposes, the Synthetic metric adapter needs access to and extends the Kubernetes API by specifying a new API service—v1beta1.external.metrics.k8s.io.

This API service is defined in the Synthetic metric adapter template—see #install below.

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  service:
    name: dynatrace-metrics-apiserver
    namespace: {{adapterNamespace}}
  group: external.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

Note that any other metric adapter, such as a Prometheus adapter, in the Kubernetes cluster might also use this service.

The Synthetic metric adapter also modifies an existing resource in its template—the horizontal-pod-autoscaler ServiceAccount in kube-system namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hpa-controller-dynatrace-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dynatrace-metrics-server-resources
subjects:
  - kind: ServiceAccount
    name: horizontal-pod-autoscaler
    namespace: kube-system

Install a containerized location

Set up and manage a containerized location in the Dynatrace web UI at Settings > Web and Mobile monitoring > Private Synthetic locations.

Initial setup for a Kubernetes/OpenShift location

Deploy the location

Deploy the Synthetic metric adapter

Initial setup for a Kubernetes/OpenShift location

Your named location is displayed on the Private Synthetic locations page with the Kubernetes or OpenShift logo. Note that no ActiveGates are assigned to the location at this point.

Deploy the location

Select your location in Private Synthetic locations to download the location template and generate the commands that need to be executed on the Kubernetes cluster.

Deploy the Synthetic metric adapter

This procedure generates a separate template for the Synthetic metric adapter. You then execute generated commands on your Kubernetes cluster to deploy the metric adapter.

Update a containerized location or its ActiveGates

Any updates to a location require that you download the location template file again and apply the changes via kubectl.

To update ActiveGate versions

Any update redeploys ActiveGates in the reverse order of their deployment. For example, if your location contains the ActiveGates activegate-name-0 and activegate-name-1, activegate-name-1 is stopped and redeployed first.

The redeployed ActiveGate pod uses the same persistent volume deployed for log continuity.

Delete a location or metric adapter on Kubernetes

The commands generated when deploying a location and the Synthetic metric adapter also include code snippets for deleting them on Kubernetes. You may copy and store these commands for future reference.

At any point, you can regenerate the commands for the respective namespaces.

Location

Note that this procedure only deletes Kubernetes resources; it doesn't delete the location you initially set up in Dynatrace.

Synthetic metric adapter

Multi-AZ PVC access on cloud clusters

Using a multi-availability zone (multi-AZ) cluster with deployments utilizing PVC can result in pods being stuck in a pending state upon recreation. This happens because the storage volumes like EBS are not replicated between zones.

PVC is only shared between nodes located in the same availability zone. When you use a multi-AZ cluster and a node tries to access PVC from a different availability zone, it will become stuck in pending state and display an error message.

Currently, there are two possible solutions for multi-AZ Kubernetes deployments:

Use node affinity to limit pods to a specific zone

You can configure node affinities to use only specific zones for a deployment.

To set node affinity

Nodes with the same zone label will be deployed in the same availability zone and you'll be able to share PVC between them without causing an error.

Use shared storage systems

Each cloud service provides its own shared storage systems options. To explain how to use shared storage systems, we will use AWS EFS as the example. For information about storage systems used by other cloud storage providers, see:

We assume that you already have EFS that you can use. If you don't, see Getting started with Amazon EFS to learn how to set up EFS.

To use storage class with EFS

Now, if the pod is redeployed on a node in a different zone, the PVC should be automatically bound to the new deployment zone.

NAM monitors on containerized locations

Network availability monitors are supported on containerized Synthetic-enabled ActiveGate deployments, but additional permissions are required for ICMP tests.

To enable ICMP request type for NAM execution

The entire securityContext for the synthetic-vuc container with enabled network availability monitors should look as follows.

securityContext:
  readOnlyRootFilesystem: true
  privileged: false
  allowPrivilegeEscalation: true
  runAsNonRoot: true
  capabilities:
    drop: ["all"]
    add: ["NET_RAW"]

OpenShift

OpenShift uses Security Context Constraint for limiting capabilities used by the pods. By default, deployed pods will use the restricted-v2 SCC, which does not allow any additional capabilities. The recommended solution is to prepare a custom Security Context Constraint.

Azure RedHat OpenShift (ARO)

If the OpenShift cluster is deployed as an Azure Red Hat OpenShift (ARO) resource, by default, the Network Security Group won't allow ICMP traffic outside the cluster.

The AROs Network Security Group is not modifiable, but a custom NSG can be created and imported during the ARO cluster creation. To learn more about it, see Bring your own Network Security Group (NSG) to an Azure Red Hat OpenShift (ARO) cluster.

Running the cluster with default settings will only allow for using ICMP NAM monitors for resources inside the OpenShift cluster. Any requests going outside the cluster will fail.

Proxy configuration

Add the following code at the top of your location template file to insert a ConfigMap resource containing your proxy server information.

In the code sample below:

kind: ConfigMap
apiVersion: v1
data:
  custom.properties: |-
    [http.client]
    proxy-server = 10.102.43.210
    proxy-port = 3128
    proxy-user = proxyuser
    proxy-password = proxypass
metadata:
  name: ag-custom-configmap
  namespace: dynatrace
---

Add the following code at template.volumes.

- name: ag-custom-volume
  configMap:
    name: ag-custom-configmap
    items:
    - key: custom.properties
      path: custom.properties

Add the following code to the ActiveGate container configuration under volumeMounts:.

- name: ag-custom-volume
  mountPath: /var/lib/dynatrace/gateway/config_template/custom.properties
  subPath: custom.properties

Synthetic metric adapter

Add the following code to Synthetic metric adapter template under env:

- name: HTTPS_PROXY
  value: "http://proxyuser:proxypass@10.102.43.210:3128"
- name: NO_PROXY
  value: "172.20.0.0/16"  # do not proxy internal calls to Kubernetes cluster

For more details about these environment variables see Go httpproxy package documentation.

The way of obtaining Service CIDR depends on Kubernetes distribution, for example for AWS EKS the following command can be used:

aws eks describe-cluster --name my-cluster --query 'cluster.kubernetesNetworkConfig'

Non-scalable containerized locations

Auto-scalable locations become non-scalable for any of the following reasons.

API: Synthetic - Locations, nodes, and configuration API v2

You can automate the deployment of and manage containerized locations via the existing Synthetic - Locations, nodes, and configuration API v2. Early Adopter endpoints added to this API to facilitate the deployment of Kubernetes locations. The new endpoints help generate the commands you need to execute on the Kubernetes cluster.