Rack-aware deployment
- Explanation
- 3-min read
In a standard high-availability Dynatrace Managed deployment, you are protected against data loss if one node fails in either small or large deployment.
Dynatrace Managed rack-aware deployment allows you to group Managed Cluster nodes into three fault domains (racks). A rack-aware deployment is resilient to an outage of all nodes in a rack.
Prerequisites
Use rack awareness only if:
- The final number of racks is three, corresponding to the replication factor of Dynatrace data storage.
- Racks reflect the underlying physical deployment location of nodes.
- Racks are in the same low-latency network. Typically, this means the same local area network. If racks are in separate sites connected over a wide area network, network latency between sites need to stay below 10 ms.
Otherwise, you may lose data and have issues with Managed Cluster availability.
Rack-aware deployment
Rack-aware deployment ensures that no replica is stored redundantly inside a single rack, so replicas are spread across all racks. In case one rack goes down, the other two full replicas are available, ensuring data consistency and availability. For example, in the deployment below, the Managed Cluster can handle up to three node failures in a rack before data loss.
In a standard Dynatrace Managed high availability deployment, you need at least three Managed Cluster nodes to prevent data loss. Similarly, in rack-aware deployments, you must have three racks (fault domains) to prevent data loss. In an event where the rack fails, the surviving two racks maintain the data. Given that the rack contains at least three nodes, in rack-aware deployments, you can afford a failure of the entire rack and still maintain data integrity.
The same concept applies to Premium High Availability (PHA) Managed deployments. Using rack-aware Managed Cluster deployments in separate data centers increases your resilience to data loss.
PHA Managed deployment.
PHA rack-aware Managed deployment.
For the ultimate high availability and redundancy, use the PHA deployment that is rack-aware.
To create a rack-aware deployment during the initial Managed deployment, use the installation parameters to indicate the data center and the rack to which to add the node. See Install a Managed Cluster and Customize installation for Dynatrace Managed for example:
dynatrace-managed.sh --rack-name az-1 --rack-dc datacenter1
Convert to rack-aware
Use either the Cluster expansion or Cluster restore method to convert the existing Managed deployment.
Cluster expansion (no Managed Cluster downtime)
You can scale nodes vertically in two locations, so they can handle additional load when you terminate the third location, and reinstall with rack-aware settings. See Rack-aware conversion using replication.
If your current metric storage (Cassandra database) per node is more than 1 TB, use the cluster restore method. While the cluster expansion method will work, the Cassandra bootstrapping required in this method takes an unreasonably long time.
Cluster restore (Managed Cluster downtime during the restore)
You can backup and restore with rack-aware settings. See Rack-aware conversion using restore.