Add a Managed Cluster node

How-to guide
5-min read

To add a node to your Managed Cluster, follow the steps below.

Install the node

After removing a Cluster node, wait 72 hours before installing a new node on a host with the same IP address (for example, when reinstalling on the same host).

Sign in to the Cluster Management Console.
Go to Deployment Status and select Install cluster node.
Copy the wget command line from the Run this command on the target host text field and paste it into your terminal window. Wait for the download to complete.
Run one of the following commands from the directory where you downloaded the installation script. Root access is required.

Replace <version> with your Dynatrace Managed version.
- Ubuntu Server
  sudo /bin/sh dynatrace-managed-<version>.sh
- Red Hat Enterprise Linux
  su -c '/bin/sh dynatrace-managed-<version>.sh'
- Other Linux distributions with root session
  /bin/sh dynatrace-managed-<version>.sh
Enter Accept to agree to the Dynatrace Managed Terms of use. Installation will not continue until you complete this step. To quit installation, press Ctrl+C.
Accept the prompts for values such as installation path and user account by pressing Enter. To override a value, enter your choice and press Enter.

The new node appears in the Cluster Management Console with the status joining.

Monitor synchronization

Run the following command to monitor Cluster node synchronization progress. Full data synchronization can take a couple of hours.

[root@host]# /opt/dynatrace-managed/utils/cassandra-nodetool.sh status

Example response:

Datacenter: datacenter-1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load     Tokens       Owns (effective)  Host ID                               Rack
UN  1.6.1.6  349.88 GiB       256          100.0%           aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee  rack1
UN  1.6.2.2  358.2  GiB       256          100.0%           zzzzzzzz-xxxx-cccc-vvvv-ffffffffffff  rack1
UJ  1.6.3.9  278.75 GiB       256          ?                qqqqqqqq-wwww-eeee-rrrr-rrrrrrrrrrrr  rack1

Where UJ marks the node as joining.

When synchronization completes, the new node shows UN (Up, Normal), like the other nodes.

You can't add more Cluster nodes until synchronization completes.

Read the response

A value of 100.0% in the Owns (effective) column is only visible for Managed Clusters that have up to three nodes.

If there are more than three nodes, the percentage is calculated as follows:

(3/number_of_nodes)*100%

While a node joins, the Owns (effective) column shows a question mark (?). Once you scale out from two to three nodes, the value changes to 100.0%. For additional nodes, the value is calculated with the previous formula.

In a multi-rack setup, the formula for the Owns (effective) column changes to (100 / number_of_nodes)%. The first node in a rack shows 100.0%. When you add a second node, it shows 50.0%, and a third node shows 33.3%.

Review the data distribution

It's possible to see unequal values in the Owns (effective) column, such as 75.1% instead of 75.0%. This is normal, as data can't be divided into equal-sized parts between the nodes. As a result, the value for Load can slightly differ between the nodes.
Cassandra doesn't automatically balance the data on disk in the entire cluster after a node is added. The value for Owns (effective) falls as more nodes are added to a cluster. This is expected and shows that the data responsibility per node is reduced.
Data on disk (Load) isn't automatically reduced on the Cluster nodes. To remove the data that a node is no longer responsible for, run the cleanup.

Clean up data on the Cluster nodes

Run the cleanup command on each Cluster node after you add all the nodes you plan to add. Cleanup is a node-local operation, so it removes only the data that the node it runs on is no longer responsible for.

The cleanup can run for several hours, because Cassandra needs to rewrite all data on the local disk. CPU and disk utilization increase locally on the node that runs the cleanup, but performance isn't affected.
To avoid rewriting the data several times, run the cleanup only after you add all the nodes you plan to add. For example, if you plan to scale out your Managed Cluster from three to six nodes, run the cleanup only after the sixth node has been successfully added. Run the cleanup on all five remaining nodes.

sudo /opt/dynatrace-managed/utils/cassandra-nodetool.sh cleanup

Frequently asked questions

Can you add a node using the Managed Cluster installation procedure?

Yes, you can add a node using the same procedure as for installing a Managed Cluster. The difference is that the installer asks whether to add the node to an existing Managed Cluster (you must enter the IP address of an existing Managed Cluster node) or create a new Managed Cluster. Alternatively, use the --seed-auth parameter to skip this prompt and automatically add the node to the existing Managed Cluster.

Can you use a privilege management system other than sudo?

Yes, you can use pbrun, but you must grant the Dynatrace user permission to run /opt/dtrun/dtrun *. Specify the user who installs Dynatrace Managed and the command that replaces sudo. <version> is the Dynatrace Managed version number.

/bin/sh dynatrace-managed-<version>.sh --system-user dynatrace:dynatrace --sudo-cmd  "/usr/bin/pbrun \$CMD"

For maintenance purposes, add the following script paths to your privilege management configuration:

/opt/dynatrace-managed/uninstall-dynatrace.sh
/opt/dynatrace-managed/launcher/*
/opt/dynatrace-managed/utils/*

Run this command to stop all Dynatrace Managed processes on a node:

pbrun /opt/dynatrace-managed/launcher/dynatrace.sh stop

Don't remove or overwrite dtrun, as it's required by installation and update procedures. The installer calls dtrun without arguments to validate that the user has administrative privileges; during normal operation, Dynatrace calls dtrun with arguments to run commands.