To extend your Managed Cluster into a second data center for a Premium High Availability (PHA) deployment, follow the steps below.
A PHA deployment is globally distributed. You add a redundant set of nodes that mirrors your original Managed Cluster, and the two node sets typically sit in different data centers.
This procedure uses these terms:
The procedure migrates and replicates each Dynatrace Managed component individually to prepare for cross-data center data replication. See Overview of Dynatrace Managed components.
The DC-1 Managed Cluster must have backup turned off before you start the migration procedure. Create a fresh Managed Cluster backup, then turn off backup shortly before you deploy the additional data center.
The DC-1 Managed Cluster must have automatic update turned off before you start the migration procedure. Do not upgrade the Managed Cluster during migration. See Automatic update. Contact a Dynatrace product expert via live chat if the automatic update option is turned off.
Make sure the machines for the DC-2 Managed Cluster are ready.
DC-2 replicates the data of DC-1, so designate the same number of nodes with the same hardware, including disk storage. Set up Network Time Protocol on every node in DC-1 and DC-2, so all nodes stay time-synchronized.
PHA deployment requires at least three nodes in DC-1 and three corresponding nodes in DC-2.
All nodes in both data centers must be able to communicate with each other.
To check whether a node in DC-1 is reachable from DC-2, run this health-check REST call from a host in DC-2:
curl -k https://<DC-1-node-IP>/rest/health
Replace <DC-1-node-IP> with the IP address of any node in DC-1. The response contains "RUNNING" when the connection succeeds.
Ensure that your system meets the specified hardware and operating system requirements.
The REST API calls in this procedure use variables. Collect this information before you start:
<seed-node-ip>: IP address of the seed node in DC-1
<nodes-ips>: List of IPv4 addresses of the new nodes in DC-2.
Example: "176.16.0.5", "176.16.0.6", "176.16.0.7"
<api-token>: Valid Cluster API token, which requires the ServiceProviderAPI scope.
Generate it in the Cluster Management Console. See Cluster API - Authentication.
<dynatrace-directory>: Directory where Dynatrace Managed is installed on the seed node.
The default installation directory is /opt/dynatrace-managed.
<datacenter-1>: DC-1 name, which must match the Cassandra DC name.
The default Cassandra DC name is datacenter1.
To get the DC name, run this command on the seed node before you start migration:
sudo <dynatrace-directory>/utils/cassandra-nodetool.sh status
The response includes the DC-1 name. The example shows a DC named datacenter1:
Datacenter: datacenter1=======================Status=Up/Down|/ State=Normal/Leaving/Joining/Moving-- Address Load Tokens Owns (effective) Host ID RackUN 10.176.42.20 65.54 GB 256 100.0% f053dd8d-ecf3-7834-b099-68542439817b rack1UN 10.176.42.244 65.47 GB 256 100.0% 2aa7e790-a423-9273-88f9-45bcd158dd6e rack1UN 10.176.42.168 65.47 GB 256 100.0% 48543bca-41f5-26d3-b2fd-6cfdf5c0f3b2 rack1
<datacenter-2>: DC-2 name, which can be any string that begins and ends with an alphanumeric character and is no longer than 80 characters. Underscores and dashes are allowed within the name. Example: dc-us-east-2.To streamline the numerous REST API calls during the deployment, set environment variables on every node in DC-1 and DC-2.
SEED_IP=<seed-node-ip>DT_DIR=<dynatrace-directory>NODES_IPS=$(echo '[<nodes-ips>]')API_TOKEN=<api-token>DC1_NAME=<datacenter-1>DC2_NAME=<datacenter-2>
For example:
SEED_IP=10.176.37.201DT_DIR=/opt/dynatrace-managedNODES_IPS=$(echo '["10.176.37.218", "10.176.37.227", "10.176.37.120"]')API_TOKEN=R_SZOpV4RTOmjr9fFmK4xDC1_NAME=datacenter1DC2_NAME=dc-us-east-2
Before you migrate to PHA, review your network zone setup for OneAgents and Environment ActiveGates that use the default network zone.
When you migrate and restart a Cluster node, Dynatrace changes the Embedded ActiveGate network zone from default to the node's data center name. The network zone reassignment can reroute OneAgent and Environment ActiveGate traffic that currently uses the default network zone.
Although PHA optimizes traffic between the data centers, ActiveGates should send data to both data centers for redundancy. Configure OneAgents and ActiveGates to prefer specific network zones without losing their ability to fail over to another part of the Managed Cluster during a data center outage. You can also use load balancers for this purpose.
For active-passive application deployments, keep ActiveGates active in the passive portions of the deployment so the Dynatrace infrastructure fails over without reconfiguration or rediscovery.
Complete these steps in order.
Update Elasticsearch license
Distribute the installer
Create the data center topology
Open firewall rules
Install second data center nodes
Replicate Cassandra
Replicate Elasticsearch
Migrate the server
Migrate the Nodekeeper
Enable the new data center
Each REST API call in this procedure returns an HTTP code. Go to the next step only when the call returns 200.
| Return code | What to do |
|---|---|
| The step succeeded. Go to the next step. |
| The request is in progress. Repeat the step after a few minutes if there is no response. |
| Revise your request path and arguments, then repeat the request. |
| Contact a Dynatrace product expert via live chat. |
Fetch the PHA license. Run the following command on each existing DC-1 node successively:
sudo nohup $DT_DIR/installer/reconfigure.sh --only els --premium-ha on &
In this step, you copy the node installer to every node in DC-2.
Sign in to the Cluster Management Console.
Go to Home for the Dynatrace Managed deployment status page.
Select Install cluster node.
The Run this installer script with root rights text field contains a command for the installation script. Ignore this command, and do not run the provided script.
Copy the wget command line from the Run this command on the target host text field.
Paste and run only the wget command line into your terminal window.
In this step, you create a configuration that defines which node belongs to which data center.
Run the following Cluster API call only on the seed node:
curl -ikS -X POST -d "{\"newDatacenterName\" : \"$DC2_NAME\", \"nodesIp\" :$NODES_IPS}" https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/datacenterTopology?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
In this step, you add firewall rules that open ports for traffic to the new DC-2 nodes.
To open ports to traffic from the new DC-2 nodes, run the following Cluster API call only on the seed node:
curl --noproxy '*' -ikS -X POST -d "$NODES_IPS" https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID you need to check the firewall rules status.
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
Set the request ID environment variable on seed node only. The request ID is from the response in the previous API call.
REQ_ID=<topology-configuration-request-id>
To check the firewall rules status, run the following Cluster API call only on the seed node:
curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code from this call isn't 200, try again after a few minutes.
In this step, you install Cluster nodes on all hosts within DC-2 and, once completed, you check for the presence of a Nodekeeper service. A running Nodekeeper service indicates that all nodes were successfully installed in DC-2.
Run this command on every node in DC-2, then follow the installer prompts. The installation follows the standard Cluster node procedure.
sudo /bin/sh ./managed-installer.sh --install-new-dc --premium-ha on --datacenter $DC2_NAME --seed-auth $API_TOKEN
For rack-aware deployments, also add the --rack-dc <data-center> and --rack-name <rack> parameters. For the full procedure, go to Combine Premium High Availability with rack awareness.
The installation takes three to five minutes. The expected result is similar to the following:
Installation in new data center completed successfully after 2 minutes 51 seconds.
Run the following Cluster API call only on the seed node when all nodes in DC-2 finish installing:
curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/nodekeeper/healthCheck?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
In this step, you reconfigure Cassandra in DC-1 and DC-2 for cross-data center replication, trigger data synchronization, rebuild Cassandra data, and verify the Cassandra state.
Cassandra replication can take minutes to hours, depending on your metric storage size.
Reconfigure Cassandra in DC-1 for cross-data center replication. Run this Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/currentDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID that you need to check replication status. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.
REQ_ID=<replication-old-datacenter-request-id>
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
To check replication status, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/currentDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
Reconfigure Cassandra in DC-2 for cross-data center replication and trigger data synchronization. Run this Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID that you need to check replication status. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.
REQ_ID=<replication-new-datacenter-request-id>
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
To check the replication status, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
In this step, you rebuild Cassandra and verify the progress by checking the status. Depending on the size of your Cassandra database, this can take several hours.
To rebuild Cassandra data in the DC-2 data center, run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200. If the status code isn't 200 and the response doesn't suggest the following steps, contact a Dynatrace product expert via live chat within your Dynatrace environment.
To check the rebuild data status, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after approximately 15 minutes. Remember that the rebuilding data process can be time-consuming.
If the response has an error flag set to true, contact a Dynatrace product expert via live chat within your environment.
In this step, you replicate Elasticsearch to the DC-2 data center and verify the configuration and data replication. Elasticsearch replication can take minutes or hours, depending on your storage size.
To start replication of Elasticsearch to the DC-2 data center, run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/elasticsearch?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID that you need to check replication status. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.
REQ_ID=<replication-elasticsearch-request-id>
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
To check the replication status of Elasticsearch, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/elasticsearch/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
To verify Elasticsearch data replication, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/elasticsearch/indexMigrationStatus?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
In this step, you migrate the server, refresh authorization tokens that enable OneAgent connectivity, and start NGINX in the DC-2 data center. Also, you refresh installers in DC-1 that are used to add nodes.
Start the Managed Cluster in DC-2. Run this Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/server?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID that you need to check cluster readiness. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.
REQ_ID=<replication-server-request-id>
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
To check if the Managed Cluster is ready, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/server/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
In this step, you migrate the Nodekeeper service in DC-1.
Run the script below manually on each node in DC-1 before you start migration.
/opt/dynatrace-managed/installer/reconfigure.sh --only ndk
To start the Nodekeeper migration in DC-1, run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/nodekeeper/currentDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
To check if the Managed Cluster is migrated, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/nodekeeper/currentDc/status?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200, and the response contains the Migration completed successfully message.
Go to the Deployment status page in the Cluster Management Console and confirm that it lists both data centers with all nodes healthy.