Get deep insights into your OpenShift control plane using etcd metrics.
This is an extension for monitoring etcd as part of the OpenShift control plane. It comes with an out-of-the-box dashboard and pre-configured alerts for easiness of use. It requires metrics already ingested into Dynatrace via Prometheus integration.
Prerequisites
Enable Prometheus monitoring in Dynatrace In your Dynatrace environment, go to Kubernetes, edit your Kubernetes cluster settings, and turn on Enable monitoring and Monitor annotated Prometheus exporters. For more information see our official documentation.
Start ingesting etcd metrics To ingest metrics exposed by etcd, the related service needs to be annotated with specific Dynatrace annotations. Based on the used OpenShift version, use the corresponding instructions below. Before executing any of the instructions below, ensure your kubectl is configured to use the Kubernetes cluster you want to monitor.
Check if secret and configmap referenced below exist in OpenShift cluster (see documentation on syntax). The resource and namespace name can differ between different versions without any announcement from RedHat. E.g., note that namespace with relevant resources changed to openshift-etcd-operator for Openshift 4.18.
OpenShift 4
kubectl annotate --overwrite service etcd -n openshift-etcd \metrics.dynatrace.com/port='9979' metrics.dynatrace.com/scrape='true' metrics.dynatrace.com/secure='true' \metrics.dynatrace.com/tls.ca.crt='configmap:openshift-config:etcd-metric-serving-ca:ca-bundle.crt' \metrics.dynatrace.com/tls.crt='secret:openshift-config:etcd-metric-client:tls.crt' \metrics.dynatrace.com/tls.key='secret:openshift-config:etcd-metric-client:tls.key'
OpenShift 4.18
kubectl annotate --overwrite service etcd -n openshift-etcd \metrics.dynatrace.com/port='9979' metrics.dynatrace.com/scrape='true' metrics.dynatrace.com/secure='true' \metrics.dynatrace.com/tls.ca.crt='configmap:openshift-etcd-operator:etcd-metric-serving-ca:ca-bundle.crt' \metrics.dynatrace.com/tls.crt='secret:openshift-etcd-operator:etcd-metric-client:tls.crt' \metrics.dynatrace.com/tls.key='secret:openshift-etcd-operator:etcd-metric-client:tls.key'
OpenShift 3
For OpenShift 3, the required TLS files need to be created and stored in secrets - creating the same situation that is the default for OpenShift 4. To achieve this easily, we provide a script that follows the official documentation of OpenShift. For executing the script, download the script itself and the dynatrace-monitoring-etcd-ocp3-service.yaml file. Store both files in a single folder and execute the activate_etcd_scraping_ocp3.sh script.
After some minutes, you can verify if everything works as expected by finding etcd related metrics using Dynatrace metrics browser. etcd metrics are prefixed with etcd_ string, for example etcd_server_has_leader.
For troubleshooting and further annotation methods, see Monitor Prometheus metrics in Dynatrace documentation.
Add extension to environment To add this extension to your environment click Add to environment on the page of this extension in the Dynatrace Hub. After activating the extension, select Dashboards in Dynatrace. You should now find an out-of-the-box etcd for OpenShift dashboard.
Activate metric events for alerting Additionally, the extension comes with the two pre-configured metric events for alerting. To activate them:
From the Dynatrace navigation menu, select Settings > Anomaly detection > Custom events for alerting.
Find the following events
If necessary, select the Edit button to customize the event conditions.
Move the switch next an event to the On position to activate it.
Stop metric ingest As metric ingest might be subject to additional costs (see licensing), it can be stopped by removing the annotations from the etcd service. If you followed the instructions described in the "Start ingesting etcd metrics" section, you can use kubectl to remove the annotations.
OpenShift 4
kubectl annotate service etcd -n openshift-etcd \metrics.dynatrace.com/port- metrics.dynatrace.com/scrape- metrics.dynatrace.com/secure- \metrics.dynatrace.com/tls.ca.crt- metrics.dynatrace.com/tls.crt- metrics.dynatrace.com/tls.key-
OpenShift 3
kubectl delete configmap etcd-data -n dynatracekubectl delete secret etcd-data -n dynatracekubectl delete service dynatrace-monitoring-etcd-ocp3 -n kube-system
Red Hat OpenShift (self-managed).
When activating your extension using monitoring configuration, you can limit monitoring to one of the feature sets. To work properly the extension has to collect at least one metric after the activation.
In highly segmented networks, feature sets can reflect the segments of your environment. Then, when you create a monitoring configuration, you can select a feature set and a corresponding ActiveGate group that can connect to this particular segment.
All metrics that aren't categorized into any feature set are considered to be the default and are always reported.
A metric inherits the feature set of a subgroup, which in turn inherits the feature set of a group. Also, the feature set defined on the metric level overrides the feature set defined on the subgroup level, which in turn overrides the feature set defined on the group level.