Cassandra Cluster Setup

Adding Cassandra Nodes

If you need to expand the Cassandra cluster on the same machine, you can launch a new Cassandra container and configure it to point to the seed container. Make sure that you provide a valid seed container so the new node can bootstrap properly.

Launch the new Cassandra container.

Note

If the cluster needs to run on a separate machine, you must specify the option -e CASSANDRA_BROADCAST_ADDRESS=<public-ip> on each node that is launched, where public-ip is the IP of the machine on the network. It is not possible to combine the approach of having multiple nodes on the same box and nodes on a remote box for one cluster.

docker run -d -e CASSANDRA_SEEDS=mwmanager-cassandra -d \
              --link mwmanager-cassandra  \
              registry.access.redhat.com/jboss-mm-7-tech-preview/middleware-manager-datastore:latest

Verify the container has launched using the docker ps command.

Verify the status of the node by running the nodetool command inside the container.

docker exec -it mwmanager-cassandra /opt/apache-cassandra/bin/nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.2  490.35 KB  256          50.2%             ead3a0ee-b040-4873-8b62-aa700d02b0c1  rack1
UN  172.17.0.4  312.28 KB  256          49.8%             8e4a73eb-5545-48e1-9464-ef2728f8852e  rack1

After all nodes appear UP you should run the cleanup process in each of the other nodes of the cluster. This operation removes all unnecessary keys that don’t belong to the node itself. You can perform this operation using nodetool cleanup inside each container.

Note
The cleanup process needs temporary disk space (proportional to the amount of data stored) and is an I/O intensive operation, so it should be postponed to lower usage hours.
```
docker exec -it <container_id> /opt/apache-cassandra/bin/nodetool cleanup
```
Each time you want to add a new node to the cluster, you should repeat this entire process.

Removing a Cassandra Node.

Select the node to be removed from the cluster. You can obtain a list of nodes by running docker ps.
After you have selected the node, check whether the node is up or down using nodetool status.
Run the decommission process inside the container. This process will move data to the other nodes and replicate the appropriate data.
```
  docker exec -it <my_container_id> /usr/bin/nodetool decommission
```

You can monitor the progress of the process using nodetool netstats.

docker exec -it <container_id> /opt/apache-cassandra/bin/nodetool netstats

Mode: DECOMMISSIONED
Not sending any streams.
Read Repair Statistics:

Once the process finishes, you can delete the container.

docker stop <my_container_id>
docker rm <my_container_id>

docker rm --force my_container_id

Replacing a Cassandra Node

If something goes wrong with one node, you can replace it with a new one.

Get the status of the cluster using nodetool status.

docker exec -it mwmanager-cassandra /opt/apache-cassandra/bin/nodetool status

=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.3  179.25 KiB  256          67.6%             3dc29aa0-5cb7-4352-a45d-72600f87ee48  rack1
DN  172.17.0.2  166.47 KiB  256          63.4%             b73c89e5-e2e1-470e-874b-f926b7243b49  rack1
UN  172.17.0.4  83.71 KiB  256          69.0%             fe1e6949-1fc7-496a-a572-a3416b47d16f  rack1

If there is a dead node, get the IP address of that node.

Start a new node container using the following command, where node_replaced_ip is the IP of the node you want to replace.

docker run -d -e JVM_OPTS="-Dcassandra.replace_address=<node_replaced_ip>" \
           -e CASSANDRA_SEEDS=mwmanager-cassandra\
           -e CASSANDRA_START_RPC=true \
           --link mwmanager-cassandra \
           registry.access.redhat.com/jboss-mm-7-tech-preview/middleware-manager-datastore:latest

Run nodetool status again to see if the node was replaced.
After the new node finishes the bootstrap process, you can remove the old node.