In this step, we will create a Portworx volume (PVC) for Cassandra.

Step: Create StorageClass

Take a look at the StorageClass definition for Cassandra: cat cassandra-sc.yaml. Note that we define a replication factor of 2 to accelerate Cassandra node recovery and we also defined a group name for Cassandra so that we can take 3DSnapshots which will be consistent across the whole Cassandra cluster. In production environment which larger clusters you would also add the "fg=true" parameter to your StorageClass to ensure that Portworx places each Cassandra volume and their replica on separate nodes so that in case of node failure we never failover Kafka to a node where it is already running. To enable this feature with a 3 volume group and 2 replicas you need a minimum of 6 worker nodes.

The parameters are declarative policies for your storage volume. See here for a full list of supported parameters.

Create the storage class using:

kubectl create -f cassandra-sc.yaml

Now that we have the StorageClass created, let's deploy Cassandra!

In this step, we will deploy a 3 node Cassandra application using a stateful set. To learn more about stateful sets follow this link.

Step: Create the Cassandra StatefulSet

Create a Cassandra StatefulSet that uses a Portworx PVC.

Take a look at the yaml: cat cassandra.yaml

Observe that the stateful set is exposed through a headless service. Also note how PVCs will be dynamically created with each member of the stateful set based on the volumeClaimTemplates and it's storage-class sections. Finally, you will also see that we are starting with a single node (replicas: 1).

Now use kubectl to deploy Cassandra.

kubectl create -f cassandra.yaml

Step: Verify Cassandra pod is ready

Below commands wait till the Cassandra pod are in ready state. Take note of the node it's running on.

watch kubectl get pods  -o wide

This takes a few minutes, when the cassandra-0 and cqlsh pods are in STATUS Running and READY 1/1 then hit clear to ctrl-c and clear the screen.

In this step, we will use pxctl to inspect the volume

Step: Inspect the Portworx volume

Portworx ships with a [pxctl](https://docs.portworx.com/reference/cli/basics/ command line that can be used to manage Portworx.

Below we will use pxctl to inspect the underlying volumes for our Cassandra pod.

VOLS=`kubectl get pvc | grep cassandra | awk '{print $3}'`
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume inspect $VOLS

Make the following observations in the inspect output

• State indicates the volume is attached and shows the node on which it is attached. This is the node where the Kubernetes pod is running.
• HA shows the number of configured replicas for this volume
• Labels show the name of the PVC for this volume
• Replica sets on nodes shows the px nodes on which volume is replicated

Now that we have Cassandra up, let's proceed to run some tests!

In this step, we will initialize a sample database in our cassandra instance.

Step: Create a table and insert data

Start a CQL Shell session:

kubectl exec -it cqlsh -- cqlsh cassandra-0.cassandra.default.svc.cluster.local --cqlversion=3.4.4

Create a keyspace with replication of 3 and insert some data:

CREATE KEYSPACE portworx WITH REPLICATION = {'class':'SimpleStrategy','replication_factor':3};
USE portworx;
CREATE TABLE features (id varchar PRIMARY KEY, name varchar, value varchar);
INSERT INTO portworx.features (id, name, value) VALUES ('px-1', 'snapshots', 'point in time recovery!');
INSERT INTO portworx.features (id, name, value) VALUES ('px-2', 'cloudsnaps', 'backup/restore to/from any cloud!');
INSERT INTO portworx.features (id, name, value) VALUES ('px-3', 'STORK', 'convergence, scale, and high availability!');
INSERT INTO portworx.features (id, name, value) VALUES ('px-4', 'share-volumes', 'better than NFS, run wordpress on k8s!');
INSERT INTO portworx.features (id, name, value) VALUES ('px-5', 'DevOps', 'your data needs to be automated too!');

Select rows from the keyspace we just created:

SELECT id, name, value FROM portworx.features;

Now that we have data created let's quit the cqlsh session.

Step: Flush data to disk

Before we proceed to the failover test we will flush the in-memory data onto disk so that when the cassandra-0 starts on another node it will have access to the data that was just written (Cassandra keeps data in memory and only flushes it to disk after 10 minutes by default).

kubectl exec -it cassandra-0 bash
nodetool flush
exit

In this step, we will simulate failure by cordoning the node where Cassandra is running and then deleting the Cassandra pod. The pod will then be resheduled by the STorage ORchestrator for Kubernetes (STORK) to make sure it lands on one of the nodes that has of replica of the data.

Step: Simulate a node failure to force Cassandra to restart

First we will cordon the node where Cassandra is running to simulate a node failure or network partition:

NODE=`kubectl get pods -o wide | grep cassandra-0 | awk '{print $7}'`
kubectl cordon ${NODE}

Then delete the Cassandra pod:

POD=`kubectl get pods -l app=cassandra -o wide | grep -v NAME | awk '{print $1}'`
kubectl delete pod ${POD}

Once the cassandra pod gets deleted, Kubernetes will start to create a new cassandra pod on another node.

Step: Verify replacement pod starts running

Below commands wait till the new cassandra pod is ready.

watch kubectl get pods -l app=cassandra -o wide

When the pod come back up it will be in the Running and READY(1/1) state. When it does hit clear to ctrl-c and clear the screen.

Before you proceed you should uncordon your node:

kubectl uncordon ${NODE}

Now that we have the new cassandra pod running, let's check if the database we previously created is still intact.

In this step, we will check the state of our sample Cassandra database.

Step: Verify data is still available

Start a CQL Shell session:

kubectl exec -it cqlsh -- cqlsh cassandra-0.cassandra.default.svc.cluster.local --cqlversion=3.4.4

Select rows from the keyspace we previously created:

SELECT id, name, value FROM portworx.features;

Now that we have verify our data survived the node failure let's quit the cqlsh session before continuing to the next step.

In this step, we will scale our Cassandra stateful set to 3 replicas

Step: Scale the cluster

Run this command to add two nodes to the Cassandra cluster:

kubectl scale sts cassandra --replicas=3

You can watch the cassandra-1 and cassandra-2 pods get added:

watch kubectl get pods -o wide

You can clear to exit the watch and clear the screen. Now, to verify that Cassandra is in a running state you can run the nodetool status utility to verify the health of our Cassandra cluster

kubectl exec -it cassandra-0 bash
nodetool status
exit

It will take a minute or two for all three Cassandra nodes to come online and discover each other. When it's ready you should see the following output in from the nodetool status command (address and host ID will vary):

[email protected]:/# nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.32.0.4  153.59 KiB  32           100.0%            2fb16c55-1337-4b04-a4a4-13da82cca0cf  Rack1-K8Demo
UN  10.38.0.3  178.86 KiB  32           100.0%            ee7f6cb5-a631-4987-8888-28d008cfb959  Rack1-K8Demo
UN  10.40.0.5  101.46 KiB  32           100.0%            e2adf023-04f7-44a4-824b-55e75be7d74c  Rack1-K8Demo

When you see your Cassandra node is in Status=Up and State=Normal (UN) that means the cluster is fully operational.

Pro Tip: Use jq to get useful cluster configuration summary

Install jq - it's awesome and very useful with Kubernetes and Portworx!

apt-get -qq install jq

Get the pods and the knowledge of the Hosts on which they are scheduled.

kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'