Try it free

Rebuild a data center

  • How-to guide
  • 8-min read

To rebuild a lost data center (DC) in a Premium High Availability deployment, follow the steps below.

Step 1

Gather information and set variables

Step 2

Remove the lost data center

Step 3

Prepare the Managed Cluster for replication

Step 4

Install second data center nodes

Step 5

Migrate Cassandra

Step 6

Migrate Elasticsearch

Step 7

Migrate the server

Step 8

Restore traffic and backup

Step 1 Gather information and set variables

Collect the values that the commands need, and then set them as environment variables.

The procedure uses the following terms:

  • Source-DC: Surviving data center that contains the Managed Cluster
  • Target-DC: Lost data center designated for recovery
  • seed node: Node in Source-DC that performs installation tasks and distributes configuration

Gather information

The commands use these variables in Cluster API calls. Gather the following information:

  • <seed-node-ip>: The IP address of the seed node from Source-DC.
    Use any node that runs in the existing data center and can perform installation tasks and distribute configuration.

  • <nodes-ips>: The list of IPv4 addresses of new nodes in Target-DC.
    Example: "176.16.0.5", "176.16.0.6", "176.16.0.7"

  • <api-token>: A valid Cluster API token. The token requires the ServiceProviderAPI scope.
    You can generate it in the Cluster Management Console (CMC). See Cluster API authentication.

  • <dynatrace-directory>: The directory where Dynatrace Managed runs on the seed node.
    The default Dynatrace Managed installation directory is /opt/dynatrace-managed.

  • <datacenter-1>: The Source-DC name must be the same as the Cassandra DC name.
    The default Cassandra DC name is datacenter1.

    Get the DC name

    To get the DC name, run this command on the seed node before you start migration:

    sudo <dynatrace-directory>/utils/cassandra-nodetool.sh status

    The response includes the Source-DC name. The example shows a DC named datacenter1:

    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    -- Address Load Tokens Owns (effective) Host ID Rack
    UN 10.176.42.20 65.54 GB 256 100.0% f053dd8d-ecf3-7834-b099-68542439817b rack1
    UN 10.176.42.244 65.47 GB 256 100.0% 2aa7e790-a423-9273-88f9-45bcd158dd6e rack1
    UN 10.176.42.168 65.47 GB 256 100.0% 48543bca-41f5-26d3-b2fd-6cfdf5c0f3b2 rack1
  • <datacenter-2>: The Target-DC name must remain unchanged. Example: dc-us-east-2.

Lost data center name

During recovery to Target-DC, use the same name as the lost data center.

Set variables

API return codes

Each Cluster API call in this procedure returns an HTTP code. Go to the next step only when the call returns 200.

Return codeWhat to do

200

Go to the next step. The current step succeeded.

207

The request is still processing. Repeat the step after a few minutes.

40x

Revise your request path and arguments, and repeat the request.

5xx

Contact support.

Set the following environment variables on the seed node in Source-DC and on every node in Target-DC:

SEED_IP=<seed-node-ip>
DT_DIR=<dynatrace-directory>
NODES_IPS=$(echo '[<nodes-ips>]')
API_TOKEN=<api-token>
SDC_NAME=<datacenter-1>
TDC_NAME=<datacenter-2>

For example:

SEED_IP=10.176.37.201
DT_DIR=/opt/dynatrace-managed
NODES_IPS=$(echo '["10.176.37.218", "10.176.37.227", "10.176.37.120"]')
API_TOKEN=R_SZOpV4RTOmjr9fFmK4x
SDC_NAME=datacenter1
TDC_NAME=dc-us-east-2

Step 2 Remove the lost data center

Stop the lost data center, remove its nodes, and then remove it from the Managed Cluster configuration.

Stop the lost data center

Stop all required Dynatrace Managed services in the recommended order. See Start/stop/restart a Cluster node.

Remove nodes

  1. Go to the CMC.
  2. For each node in the unavailable data center, go to the Node details page and remove the node.

For more information on other ways to remove a node, see Remove a Cluster node.

Remove the data center from the configuration

Run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/lostDatacenterCleanUp?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Step 3 Prepare the Managed Cluster for replication

Copy the installer to Target-DC, and then configure the Managed Cluster so it can replicate data to Target-DC.

Distribute the installer

  1. Sign in to the CMC.
  2. Go to Home for the Dynatrace Managed deployment status page.
  3. Open Add new cluster node.
  4. Copy only the wget command line from Run this command on the target host. Ignore the command in Run this installer script with root rights and don't run the installer script.
  5. Paste and run only the wget command line in your terminal window.

Prepare data for replication

Run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterReplicationPreparation?Api-Token=$API_TOKEN

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Create the data center topology

Run the following Cluster API call only on the seed node:

curl -ikS -X POST -d "{\"newDatacenterName\" : \"$TDC_NAME\", \"nodesIp\" :$NODES_IPS}" https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/datacenterTopology?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Open ports

To open ports to traffic from the new Target-DC nodes, run the following Cluster API call only on the seed node:

curl --noproxy '*' -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200 and the response body contains a request ID. Use this ID to check the firewall rules status.

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Verify firewall rules

Set the request ID environment variable only on the seed node. Use the request ID from the response in the previous API call.

REQ_ID=<topology-configuration-request-id>

To check the firewall rules status, run the following Cluster API call only on the seed node:

curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/clusterNodes/currentDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code from this call isn't 200, try again after a few minutes.

Step 4 Install second data center nodes

Install Cluster nodes on all hosts in Target-DC. Then check for the Nodekeeper service to verify that all nodes installed successfully in Target-DC.

Install nodes in the target data center

Run the following command on every node in Target-DC. Follow the prompts for a typical node installation.

sudo /bin/sh ./managed-installer.sh --install-new-dc --premium-ha on --datacenter $TDC_NAME --seed-auth $API_TOKEN

For rack-aware deployments, also add the --rack-dc <data-center> and --rack-name <rack> parameters, which assign the node to a data center and a rack. For the parameter descriptions, go to Managed Cluster installation parameters.

Target-DC's rack layout comes from the --rack-name value you pass for each node, not from Source-DC. Target-DC can become rack-aware whether Source-DC is rack-aware or not. For the full procedure, go to Combine Premium High Availability with rack awareness.

The node installation takes about three to five minutes. The expected output is:

Installation in new data center completed successfully after 2 minutes 51 seconds.

Check Nodekeeper in the target data center

Run the following Cluster API call only on the seed node when all nodes in Target-DC finish installing:

curl -ikS https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/nodekeeper/healthCheck?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Step 5 Migrate Cassandra

Reconfigure Cassandra in Source-DC and Target-DC for cross data center replication. Then trigger data synchronization, rebuild Cassandra data, and verify Cassandra state.

The Cassandra migration can take minutes to hours, depending on your metric storage size.

Trigger migration

To start migration of Cassandra in Target-DC, run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200 and the response body contains a request ID. Use this ID to check the migration status. Set the request ID environment variable only on the seed node.

REQ_ID=<migration-new-datacenter-request-id>

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Check migration status

To check the migration status, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/newDc/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Start the Cassandra rebuild

Depending on the size of your Cassandra database, the rebuild can take several hours.

To rebuild Cassandra data in Target-DC, run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200. If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Check the rebuild status

To check the rebuild status, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/cassandra/rebuild?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after approximately 15 minutes.

If the response has an error flag set to true, contact a Dynatrace product expert via live chat.

Step 6 Migrate Elasticsearch

Migrate Elasticsearch to Target-DC, and then verify the configuration and data migration. The Elasticsearch migration can take minutes or hours, depending on your Elasticsearch storage.

Migrate Elasticsearch to the target data center

Start Elasticsearch. Run the following command on every node in Target-DC in sequence:

sudo $DT_DIR/launcher/elasticsearch.sh start

To start migration of Elasticsearch to Target-DC, run the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/elasticsearch/recover?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200.

Verify progress and status

To check the migration status of Elasticsearch, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/elasticsearch/recover?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Verify data migration

To verify the Elasticsearch data migration, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/migration/elasticsearch/indexMigrationStatus?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Step 7 Migrate the server

Migrate the server, refresh the authorization tokens for OneAgent connectivity, and start NGINX in Target-DC. Then refresh the installers in Source-DC for adding nodes.

Start the Managed Cluster in the target data center

Start the Managed Cluster in Target-DC by running the following Cluster API call only on the seed node:

curl -ikS -X POST https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/server/recovery?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If successful, the status code is 200 and the response body contains a request ID. Use this ID to check Managed Cluster readiness. Set the request ID environment variable only on the seed node.

REQ_ID=<migration-server-request-id>

If the status code isn't 200 and the response doesn't suggest next steps, contact a Dynatrace product expert via live chat.

Check Managed Cluster readiness

To check whether the Managed Cluster is ready, run the following Cluster API call only on the seed node:

curl -ikS -X GET https://$SEED_IP/api/v1.0/onpremise/multiDc/restore/server/recovery/$REQ_ID?Api-Token=$API_TOKEN -H "accept: application/json" -H "Content-Type: application/json"

If the status code isn't 200, try again after a few minutes.

Step 8 Restore traffic and backup

  1. Restore OneAgent traffic.
    For details, see Cluster node capabilities.
  2. Turn on backup in one of the data centers. Migration turns off backup.
    For details, see Back up and restore a Managed Cluster.

Go to the Deployment status page in the Cluster Management Console and confirm that it lists both data centers with all nodes healthy.

Related topics

  • Multi-data center high availability
  • Multi-data center failover