Recover from a backup
- How-to guide
- 8-min read
To restore a lost data center (DC) from backup in a Premium High Availability deployment, follow these steps.
The procedure uses the following terms:
- Source-DC: Surviving data center that contains the Managed Cluster
- Target-DC: Lost data center designated for recovery
- seed node: Node in Source-DC that performs installation tasks and distributes configuration
The procedure migrates and replicates Dynatrace Managed components individually to prepare them for data replication across two DCs. See Managed components.
Uninstall Dynatrace Managed on all running nodes
Restore data center from backup
Remove lost data center from configuration
Distribute the installer
Prepare Managed Cluster data for replication
Create the data center topology
Open firewall rules
Install second data center nodes
Migrate Cassandra
Migrate Elasticsearch
Migrate the server
Enable the new data center
Gather information
Collect the following information before running the API calls:
-
<seed-node-ip>: The IP address of the seed node from Source-DC.
Use any node that runs in the existing data center and can perform installation tasks and distribute configuration. -
<nodes-ips>: The list of IPv4 addresses of new nodes in Target-DC.
Example:"176.16.0.5", "176.16.0.6", "176.16.0.7" -
<api-token>: A valid Cluster API token. The token requires the ServiceProviderAPI scope.
You can generate it in the Cluster Management Console (CMC). See Cluster API - Authentication. -
<dynatrace-directory>: The directory where Dynatrace Managed is installed on the seed node.
The default Dynatrace Managed installation directory is/opt/dynatrace-managed. -
<datacenter-1>: The Source-DC name must be the same as the Cassandra DC name.
The default Cassandra DC name isdatacenter1.
Get the DC name
To get the DC name, run this command on the seed node before you start migration:
sudo <dynatrace-directory>/utils/cassandra-nodetool.sh status
The response includes the Source-DC name. The example shows a DC named datacenter1:
Datacenter: datacenter1=======================Status=Up/Down|/ State=Normal/Leaving/Joining/Moving-- Address Load Tokens Owns (effective) Host ID RackUN 10.176.42.20 65.54 GB 256 100.0% f053dd8d-ecf3-7834-b099-68542439817b rack1UN 10.176.42.244 65.47 GB 256 100.0% 2aa7e790-a423-9273-88f9-45bcd158dd6e rack1UN 10.176.42.168 65.47 GB 256 100.0% 48543bca-41f5-26d3-b2fd-6cfdf5c0f3b2 rack1
<datacenter-2>: The Target-DC name must remain unchanged. Example:dc-us-east-2.
You must use the same name of the lost DC during the recovery to Target-DC.
Set variables
Set the following environment variables on the seed node in Source-DC and on every node in Target-DC:
SEED_IP=<seed-node-ip>DT_DIR=<dynatrace-directory>NODES_IPS=$(echo '[<nodes-ips>]')API_TOKEN=<api-token>SDC_NAME=<datacenter-1>TDC_NAME=<datacenter-2>
For example:
SEED_IP=10.176.37.201DT_DIR=/opt/dynatrace-managedNODES_IPS=$(echo '["10.176.37.218", "10.176.37.227", "10.176.37.120"]')API_TOKEN=R_SZOpV4RTOmjr9fFmK4xSDC_NAME=datacenter1TDC_NAME=dc-us-east-2
Each REST API call in this procedure returns an HTTP code. Go to the next step only when the call returns 200.
| Return code | What to do |
|---|---|
| The step succeeded. Go to the next step. |
| The request is in progress. Retry after a few minutes. |
| Revise your request path and arguments, then repeat the request. |
| Contact support. |
Uninstall Dynatrace Managed on all running nodes
Follow the official procedure to remove a Cluster node using either the command prompt or the CMC. See Remove a cluster node.
Restore data center from backup
Follow the official procedure to restore the DC from the backup. See Back up and restore a cluster.
Remove lost data center from configuration
Run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/lostDatacenterCleanUp?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
Distribute the installer
-
Sign in to the CMC.
-
Go to Home for the Dynatrace Managed deployment status page.
-
Select Add new cluster node.
The Run this installer script with root rights text field contains a command for the installation script. Ignore this command. Do not run the provided script.
-
Copy the
wgetcommand line from the Run this command on the target host text field. -
Paste and run only the
wgetcommand line on every node in the Target-DC terminal window.
Prepare Managed Cluster data for replication
Start Managed Cluster data preparation
Run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterReplicationPreparation?Api-Token=$API_TOKEN
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
Create the data center topology
Run the following Cluster API call only on the seed node:
curl -ikS -X POST -d "{\"newDatacenterName\" : \"$TDC_NAME\", \"nodesIp\" :$NODES_IPS}" https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/datacenterTopology?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
Open firewall rules
Open ports
To open ports to traffic from the new Target-DC nodes, run the following Cluster API call only on the seed node:
curl --noproxy '*' -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID you need to check the firewall rules status.
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
Verify firewall rules
Set the request ID environment variable on seed node only. The request ID is from the response in the previous API call.
REQ_ID=<topology-configuration-request-id>
To check the firewall rules status, run the following Cluster API call only on the seed node:
curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code from this call isn't 200, try again after a few minutes.
Install second data center nodes
Install nodes in Target-DC
Run the following command on every node in Target-DC. Follow the installation prompts as this will be a typical node installation.
sudo /bin/sh ./managed-installer.sh --install-new-dc --premium-ha on --datacenter $TDC_NAME --seed-auth $API_TOKEN
For rack-aware deployments, also add the --rack-dc <data-center> and --rack-name <rack> parameters, which assign the node to a data center and a rack. For the parameter descriptions, go to Customize Managed Cluster installation.
The installation takes from three through five minutes. The expected result is similar to this:
Installation in new data center completed successfully after 2 minutes 51 seconds.
Check Nodekeeper in Target-DC
Run the following Cluster API call only on the seed node when all nodes in Target-DC finish installing:
curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/nodekeeper/healthCheck?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
Migrate Cassandra
Cassandra migration can take minutes to hours, depending on your metric storage size.
Migrate Cassandra
To start migration of Cassandra in Target-DC, run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID which you need to check migration status. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.
REQ_ID=<migration-new-datacenter-request-id>
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
Check migration status
To check the migration status, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
Rebuild data
To rebuild Cassandra data in Target-DC, run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200. If the status code isn't 200 and the response doesn't suggest the following steps, contact a Dynatrace product expert via live chat within your Dynatrace environment.
Check the rebuild data status
To check the rebuild data status, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after approximately 15 minutes. Remember that the rebuilding data process can be time-consuming.
If the response has an error flag set to true, contact a Dynatrace product expert via live chat within your environment.
Migrate Elasticsearch
Elasticsearch migration can take minutes or hours, depending on your Elasticsearch storage.
Migrate Elasticsearch to Target-DC
Start Elasticsearch. Run the following command successively on every node in Target-DC only:
sudo $DT_DIR/launcher/elasticsearch.sh start
To start migration of Elasticsearch to Target-DC, run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/elasticsearch/recover?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200.
Verify progress and status
To check the migration status of Elasticsearch, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/elasticsearch/recover?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
Verify data migration
To verify Elasticsearch data migration, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/elasticsearch/indexMigrationStatus?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
Migrate the server
Migrate server
To start the Managed Cluster in Target-DC, run the following Cluster API call only on the seed node:
curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/server/recovery?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If successful, the status code is 200 and the response body contains a request ID which you need to check Managed Cluster readiness. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.
REQ_ID=<migration-server-request-id>
If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.
Check Managed Cluster readiness
To check if the Managed Cluster is ready, run the following Cluster API call only on the seed node:
curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/server/recovery/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"
If the status code isn't 200, try again after a few minutes.
Enable the new data center
- Turn on OneAgent traffic.
For details, see Cluster node capabilities. - Turn on backup in one of the DCs. Migration turns off backup.
For details, see Back up and restore a cluster.
Go to the Deployment status page in the Cluster Management Console and confirm that it lists both data centers with all nodes healthy.