Try it free

Back up and restore a Managed Cluster

  • How-to guide
  • 10-min read

Configure automatic backup

To configure automatic backup for a Managed Cluster, follow the steps below.

  1. Sign in to Cluster Management Console.

  2. Go to Settings > Backup.

  3. Enable cluster backup and choose the scope:

    • User sessions might contain sensitive information. Exclude user sessions from the backup to remain compliant with the General Data Protection Regulation.

    • Exclude time series metric data from the backup if your historical data isn't relevant and you only want to retain configuration data.

    • Include backup of Log Monitoring events.

  4. (Optional) Select the data center. This step applies only if you have a multi-data center deployment (Premium High Availability deployment). For more information, see Multi-data center high availability and Recover from a backup.

  5. Provide a full path to the mounted network file storage where backup archives are stored.

  6. Configure start time.

Automatic Managed Cluster backup

You can automatically back up Dynatrace Managed configuration data (naming rules, tags, management zones, alerting profiles, and more), time series metric data, and user sessions. To maximize resilience, save your backup files to an off-site location.

  • Each node needs to be connected to the shared file system—for example, Network File Sharing (NFS)—and the shared file system needs to be mounted at the same shared directory on each node.
    To test your user permissions, run the following command on each node:

    su - dynatrace -s /bin/bash -c "touch /nfs/dynatrace/backup/$(uname -n)"
    ls -ltr /path/to/backup/

  • The user running Dynatrace services needs read/write permissions for the shared file system.

  • The shared file system mount must be available at system restart.

  • You can add a mount point to fstab or use your disk management tool to make the shared file system mount persistent.

  • The protocol used to transmit data depends on your configuration. NFSv4 is the recommended protocol. CIFS is not recommended, due to its low performance and resilience.

Shared file system mount point on system boot

If the shared file system mount point is not available on system boot, Dynatrace services will not start on that node, which might make the Managed Cluster unavailable. You must disable backups manually to allow Dynatrace to start.

Metrics and configuration storage

Dynatrace keeps the previous backup until a new one is completed.

Backup characteristics for metrics and configuration storage

  • The snapshot is performed daily.
  • Any data that's replicated between nodes is also stored in the backup (there is no deduplication).
  • Dynatrace excludes the most frequently changing column families (excluded column families comprise about 80 percent of total storage) in addition to 1-minute and 5-minute resolution data.

Elasticsearch

Elasticsearch files are stored in uncompressed binary format. While the data is replicated across nodes and there are two replicas in addition to the primary shard, the backup excludes the replicated data.

Do not remove Elasticsearch backups

Elasticsearch backs up incrementally, so it needs to be able to append recent changes to the previous backup. Do not remove Elasticsearch backup files.

Elasticsearch backup characteristics

  • The snapshot is performed, by default, every two hours, and it's incremental.
    Initially, Dynatrace copies the entire data set and then creates snapshots of the differences. Older snapshots are removed gradually once they're five days old.

  • Since Dynatrace keeps some of the older snapshots, backup size grows regardless of the current size on disk. The snapshots are incremental, but Elasticsearch merges data segments over time, which results in certain duplicates in the backup.

No transaction storage backup

Transaction storage data is not backed up, so after a restore you might see gaps in deep monitoring data (for example, distributed traces and code-level traces). By default, Dynatrace retains transaction storage data for only 10 days, so including it in backups is not necessary for long-term retention.

Managed Cluster restore

To restore a Managed Cluster, follow the steps below.

Before you begin

  • To restore a Managed Cluster on the same host as the source Managed Cluster, make sure to uninstall it first.

  • Make sure the machines prepared for the Managed Cluster restore have a similar hardware and disk layout as the original Managed Cluster and sufficient capacity to handle the load after the restore.

    Restore the Managed Cluster to the same number of nodes as the backed-up Managed Cluster whenever possible. In exceptional cases, you can restore to up to two nodes fewer, but a larger shortfall risks losing the Managed Cluster configuration.

  • Make sure the existing Managed Cluster is stopped to prevent two Managed Clusters with the same ID from connecting to Dynatrace Mission Control. See Start/stop/restart a Managed Cluster.

  • Make sure that system users created for Dynatrace Managed have the same UID:GID identifiers on all nodes.

  • On each target node, mount the backup storage to, for example, /mnt/backup. This path is referred to as <path-to-backup> in the steps below.

  • Ensure the installer has read permissions to the NFS. For example: sudo adduser dynatrace && sudo chown -R dynatrace:dynatrace <path-to-backup>

  • Create your Managed Cluster inventory. You'll need this information during the restore.

    • IDs of nodes in the Managed Cluster. The backup of each node is stored in a dedicated directory whose name is based on the node identifier, in the format node_<node-id> (for example, node_1 and node_5).
    • IPv4 addresses of the new machines.
    • The IP address of the target machine for each node.
    • The node ID of the node that will become the seed node in the Managed Cluster.

Restore from backup

To restore a Managed Cluster, follow the steps below.

  1. Copy the installer to target nodes
    To restore the Managed Cluster, you need to use the exact same installer version as in the original one. Copy the installer from <path-to-backup>/<UUID>/node_<node-id>/ to a local disk on each target node.

    For example cp <path-to-backup>/<UUID>/node_<node-id>/files/<backup-version-number>/backup-001-dynatrace-managed-installer.sh /tmp/

  2. Launch Dynatrace restore on each node
    In parallel, on each node, run the Dynatrace Managed installer using the following parameters:

    • --restore - switches the installer into the restore mode.
    • --cluster-ip - IPv4 address of the node on which you run the installer.
    • --cluster-nodes - the comma-delimited list of IDs and IP addresses of all nodes in the cluster, including the one on which you run the installer, in the following format <node-id>:<node-ip>,<node-id>:<node-ip>.
    • --seed-ip - IPv4 address of the seed node.
    • --backup-file - the path to the backup *.tar file, which includes the path to the shared file storage mount, the cluster ID, the node ID, the backup version, and the backup *.tar file in the following format:
      <path-to-backup>/<UUID>/node_<node-id>/files/<backup-version-number>/<backup-file>
    Backup path example

    In this example path:

    /mnt/backup/bckp/c9dd47f0-87d7-445e-bbeb-26429fac06c6/node_1/files/19/backup-001.tar

    the parts of the path are as follows:

    • <path-to-backup> = /mnt/backup/bckp/
    • <UUID> = c9dd47f0-87d7-445e-bbeb-26429fac06c6
    • <node-id> = 1
    • <backup-version-number> = 19

    While the backup is in progress, two backup directories might be present with different backup version numbers:

    • The directory with the lower version number contains the old backup. It will be removed after the backup is completed.
    • The directory with the higher version number contains the backup that is in progress.

    The backup version number increments automatically with each backup run.

    Get the IDs and IP addresses from the inventory you created before you started.

    For example:
    10.176.41.168 - The IP address of the node to restore
    1: 10.176.41.168, 3: 10.176.41.169, 5: 10.176.41.170 - Node IDs and new IP addresses of all nodes in the cluster

    sudo /tmp/backup-001-dynatrace-managed-installer.sh
    --restore
    --cluster-ip "10.176.41.168"
    --cluster-nodes "1:10.176.41.168,3:10.176.41.169,5:10.176.41.170"
    --seed-ip "10.176.41.169"
    --backup-file /mnt/backup/bckp/c9dd47f0-87d7-445e-bbeb-26429fac06c6/node_1/files/19/backup-001.tar

  3. Start the firewall, Cassandra, and Elasticsearch
    On each node successively, start the firewall, Cassandra, and Elasticsearch using the launcher script:

    /opt/dynatrace-managed/launcher/firewall.sh start
    /opt/dynatrace-managed/launcher/cassandra.sh start
    /opt/dynatrace-managed/launcher/elasticsearch.sh start

  4. Verify Cassandra state
    On each node, check whether Cassandra runs. Run the command: <dynatrace-install-dir>/utils/cassandra-nodetool.sh status

    All the nodes of the restored Managed Cluster should be listed in the response with the following values:
    Status = Up State = Normal

  5. Verify Elasticsearch state
    On each node, check whether Elasticsearch runs. Run the command:
    curl -s -N -XGET 'http://localhost:9200/_cluster/health?pretty' | grep status

    You should get the following response:
    "status" : "green"
    or for one node setup:
    "status" : "yellow"

  6. Restore the Elasticsearch database
    On the seed node, run the following command: <dynatrace-install-dir>/utils/restore-elasticsearch-data.sh <path-to-backup>/<UUID>

  7. Restore metrics and configuration data files
    On each node successively, starting with the seed node, run the following command: <dynatrace-install-dir>/utils/restore-cassandra-data.sh <path-to-backup>/<UUID>/node_<node-id>/files/<backup-version-number>/backup-001.tar

    Wait until Cassandra has its cluster fully set. Use the command: <dynatrace-install-dir>/utils/cassandra-nodetool.sh status

    • You should get the following response: Status = Up State = Normal

  8. Optional Repair Cassandra

    You can run the repair only on clusters with more than one node.

    Sequentially on all nodes, initiate the Cassandra repair:

    <dynatrace-install-dir>/utils/repair-cassandra-data.sh

    The repair ensures data consistency between the nodes. This step might take several hours to complete.

    To restore only the configuration data, without metric time series, run the following command:

    <dynatrace-install-dir>/utils/repair-cassandra-data.sh 1

  9. Start Dynatrace

    On each node successively, starting with the seed node, run the following command:

    <dynatrace-install-dir>/launcher/dynatrace.sh start

    Wait until you can sign in to Cluster Management Console.

  10. Optional Remove remaining references to old nodes
    If you restore fewer nodes than in the original Managed Cluster, remove the nodes marked as Offline in the Cluster Management Console. For more information, see Remove a Cluster node.

  11. Switch OneAgents to the new Managed Cluster address
    If you originally configured the Managed Cluster with the DNS for OneAgents, you only need to update the DNS records as explained in the next step.

    Otherwise, you need to configure Cluster ActiveGates (or OneAgents, if no ActiveGates are used) with the new target address and restart them. If there are no Cluster ActiveGates but there are Environment ActiveGates, this should be done on the Environment ActiveGates.

    To configure a new target address on an ActiveGate, follow these steps:

    1. Stop the ActiveGate service.
    2. Update the seedServerUrl parameter in config.properties.
    3. Update the cluster.properties with new URLs.
    4. Remove the connectivity_history.properties file.

    Run the following Cluster API call for each node, replacing <node-id> with the node identifier, <node-ip> with the node IPv4 address, and <api-token> with a valid Cluster API token.

    curl -ikS -X PUT -d <node-ip> https://<node-ip>:8021/api/v1.0/onpremise/endpoint/publicIp/agents/<node-id>?Api-Token=<api-token> -H  "accept: application/json" -H  "Content-Type: application/json"

    You should receive the 200 response as in the example below:

    HTTP/1.1 200 OK
    Date: Tue, 19 Feb 2019 17:49:06 GMT
    X-Robots-Tag: noindex
    Server: ruxit server
    Content-Length: 0

  12. Optional Update Managed Cluster DNS records
    If the Managed Cluster restore changed the IP addresses, update the DNS records.

    • If you use automatic domain and certificate management, run the following Cluster API call for each node, replacing <node-id> with the node identifier, <node-ip> with the node IPv4 address, and <api-token> with a valid API token.
    curl -ikS -X PUT -d <node-ip> https://<node-ip>:8021/api/v1.0/onpremise/endpoint/publicIp/domain/<node-id>?Api-Token=<api-token> -H  "accept: application/json" -H  "Content-Type: application/json"

    You should receive the 200 response as in the example below:

    HTTP/1.1 200 OK
    Date: Tue, 19 Feb 2019 17:49:06 GMT
    X-Robots-Tag: noindex
    Server: ruxit server
    Content-Length: 0
    • If you use your own DNS, update your Managed Cluster domain to a new IP address.

  13. Enable the backup
    To prevent overwriting the previous snapshot, the backup is automatically disabled after the restore. Once you have finished restoring, you should enable the backup again.

    1. Sign in to Cluster Management Console.

    2. Go to Settings > Backup.

    3. Turn on Enable cluster backup and confirm the full path to the backup archive and schedule daily backup time.

Disable backups manually

Certain situations require that you manually disable Managed Cluster backup. For example, if the shared file system mount point isn't available on system boot, Dynatrace won't start on that node. In this case, you need to disable backups manually to allow Dynatrace to start.

  1. Edit the <install-dir>/elasticsearch/config/elasticsearch.yml file.

  2. Remove the line with the path.repo: parameter. For example:

    network.host: [ _local_, "10.10.10.10" ]
    network.publish_host: 10.10.10.10
    path.data: /var/opt/dynatrace-managed/elasticsearch
    path.repo: <REMOVE THIS LINE>

  3. Save the file and restart the system. See Start/stop/restart a Cluster node.