Try it free

Recover from a backup

  • How-to guide
  • 8-min read

To restore a lost data center (DC) from backup in a Premium High Availability deployment, follow these steps.

Step 1

Gather information and set variables

Step 2

Restore the Managed Cluster from backup

Step 3

Remove lost data center from configuration

Step 4

Prepare the Managed Cluster for replication

Step 5

Install second data center nodes

Step 6

Migrate Cassandra

Step 7

Migrate Elasticsearch

Step 8

Migrate the server

Step 9

Enable the new data center

Step 1 Gather information and set variables

Collect the values that the commands need, and then set them as environment variables.

The procedure uses the following terms:

  • Source-DC: Surviving data center that contains the Managed Cluster
  • Target-DC: Lost data center designated for recovery
  • seed node: Node in Source-DC that performs installation tasks and distributes configuration

The procedure migrates and replicates Dynatrace Managed components individually to prepare them for data replication across two DCs. See Managed components.

Gather information

Collect the following information before running the API calls:

  • <seed-node-ip>: The IP address of the seed node from Source-DC.
    Use any node that runs in the existing data center and can perform installation tasks and distribute configuration.

  • <nodes-ips>: The list of IPv4 addresses of new nodes in Target-DC.
    Example: "176.16.0.5", "176.16.0.6", "176.16.0.7"

  • <api-token>: A valid Cluster API token. The token requires the ServiceProviderAPI scope.
    You can generate it in the Cluster Management Console (CMC). See Cluster API - Authentication.

  • <dynatrace-directory>: The directory where Dynatrace Managed is installed on the seed node.
    The default Dynatrace Managed installation directory is /opt/dynatrace-managed.

  • <datacenter-1>: The Source-DC name must be the same as the Cassandra DC name.
    The default Cassandra DC name is datacenter1.

Get the DC name

To get the DC name, run this command on the seed node before you start migration:

sudo <dynatrace-directory>/utils/cassandra-nodetool.sh status

The response includes the Source-DC name. The example shows a DC named datacenter1:

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.176.42.20 65.54 GB 256 100.0% f053dd8d-ecf3-7834-b099-68542439817b rack1
UN 10.176.42.244 65.47 GB 256 100.0% 2aa7e790-a423-9273-88f9-45bcd158dd6e rack1
UN 10.176.42.168 65.47 GB 256 100.0% 48543bca-41f5-26d3-b2fd-6cfdf5c0f3b2 rack1
  • <datacenter-2>: The Target-DC name must remain unchanged. Example: dc-us-east-2.
Lost data center name

You must use the same name of the lost DC during the recovery to Target-DC.

Set variables

Set the following environment variables on the seed node in Source-DC and on every node in Target-DC:

SEED_IP=<seed-node-ip>
DT_DIR=<dynatrace-directory>
NODES_IPS=$(echo '[<nodes-ips>]')
API_TOKEN=<api-token>
SDC_NAME=<datacenter-1>
TDC_NAME=<datacenter-2>

For example:

SEED_IP=10.176.37.201
DT_DIR=/opt/dynatrace-managed
NODES_IPS=$(echo '["10.176.37.218", "10.176.37.227", "10.176.37.120"]')
API_TOKEN=R_SZOpV4RTOmjr9fFmK4x
SDC_NAME=datacenter1
TDC_NAME=dc-us-east-2
API return codes

Each REST API call in this procedure returns an HTTP code. Go to the next step only when the call returns 200.

Return codeWhat to do

200

The step succeeded. Go to the next step.

207

The request is in progress. Retry after a few minutes.

40x

Revise your request path and arguments, then repeat the request.

5xx

Contact support.

Step 2 Restore the Managed Cluster from backup

Uninstall Dynatrace Managed on the nodes that still run, and then restore the data center from the backup.

Uninstall Dynatrace Managed on all running nodes

Follow the official procedure to remove a Cluster node using either the command prompt or the CMC. See Remove a Cluster node.

Restore the data center

Follow the official procedure to restore the DC from the backup. See Back up and restore a Managed Cluster.

Step 3 Remove lost data center from configuration

Run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/lostDatacenterCleanUp?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Step 4 Prepare the Managed Cluster for replication

Copy the installer to Target-DC, and then configure the Managed Cluster so it can replicate data to Target-DC.

Distribute the installer

  1. Sign in to the CMC.

  2. Go to Home for the Dynatrace Managed deployment status page.

  3. Select Add new cluster node.

Do not run the installer script

The Run this installer script with root rights text field contains a command for the installation script. Ignore this command and do not run it.

  1. Copy the wget command line from the Run this command on the target host text field.

  2. Paste and run only the wget command line on every node in the Target-DC terminal window.

Prepare data for replication

Run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterReplicationPreparation?Api-Token=$API_TOKEN

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Create the data center topology

Run the following Cluster API call only on the seed node:

curl -ikS -X POST -d "{\"newDatacenterName\" : \"$TDC_NAME\", \"nodesIp\" :$NODES_IPS}" https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/datacenterTopology?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Open ports

To open ports to traffic from the new Target-DC nodes, run the following Cluster API call only on the seed node:

curl --noproxy '*' -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200 and the response body contains a request ID you need to check the firewall rules status.

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Verify firewall rules

Set the request ID environment variable on seed node only. The request ID is from the response in the previous API call.

REQ_ID=<topology-configuration-request-id>

To check the firewall rules status, run the following Cluster API call only on the seed node:

curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code from this call isn't 200, try again after a few minutes.

Step 5 Install second data center nodes

Install nodes in the target data center

Run the following command on every node in Target-DC. Follow the installation prompts as this will be a typical node installation.

sudo /bin/sh ./managed-installer.sh --install-new-dc --premium-ha on --datacenter $TDC_NAME --seed-auth $API_TOKEN

For rack-aware deployments, also add the --rack-dc <data-center> and --rack-name <rack> parameters, which assign the node to a data center and a rack. For the parameter descriptions, go to Managed Cluster installation parameters.

The installation takes from three to five minutes. The expected result is similar to this:

Installation in new data center completed successfully after 2 minutes 51 seconds.

Check Nodekeeper in the target data center

Run the following Cluster API call only on the seed node when all nodes in Target-DC finish installing:

curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/nodekeeper/healthCheck?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Step 6 Migrate Cassandra

Cassandra migration can take minutes to hours, depending on your metric storage size.

Trigger migration

To start migration of Cassandra in Target-DC, run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200 and the response body contains a request ID which you need to check migration status. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.

REQ_ID=<migration-new-datacenter-request-id>

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Check migration status

To check the migration status, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Rebuild data

To rebuild Cassandra data in Target-DC, run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200. If the status code isn't 200 and the response doesn't suggest the following steps, contact a Dynatrace product expert via live chat within your Dynatrace environment.

Check the rebuild data status

To check the rebuild data status, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after approximately 15 minutes. Remember that the rebuilding data process can be time-consuming.

If the response has an error flag set to true, contact a Dynatrace product expert via live chat within your environment.

Step 7 Migrate Elasticsearch

Elasticsearch migration can take minutes or hours, depending on your Elasticsearch storage.

Migrate Elasticsearch to the target data center

Start Elasticsearch. Run the following command successively on every node in Target-DC only:

sudo $DT_DIR/launcher/elasticsearch.sh start

To start migration of Elasticsearch to Target-DC, run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/elasticsearch/recover?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200.

Verify progress and status

To check the migration status of Elasticsearch, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/elasticsearch/recover?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Verify data migration

To verify Elasticsearch data migration, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/elasticsearch/indexMigrationStatus?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Step 8 Migrate the server

Start the Managed Cluster in the target data center

To start the Managed Cluster in Target-DC, run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/server/recovery?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200 and the response body contains a request ID which you need to check Managed Cluster readiness. Set the request ID environment variable only on the seed node. The request ID is from the response in the previous API call.

REQ_ID=<migration-server-request-id>

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Check Managed Cluster readiness

To check if the Managed Cluster is ready, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/server/recovery/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Step 9 Enable the new data center

  1. Turn on OneAgent traffic.
    For details, see Cluster node capabilities.
  2. Turn on backup in one of the DCs. Migration turns off backup.
    For details, see Back up and restore a Managed Cluster.

Go to the Deployment status page in the Cluster Management Console and confirm that it lists both data centers with all nodes healthy.

Related topics

  • Multi-data center high availability
  • Multi-data center failover