Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. -- Principles of Chaos Engineering
There are several frameworks that you can add to Kubernetes to provide chaos testing. However, creating a simple chaos tester does not mean it has to be complicated. With just a few tricks a simple chaos tester can run amuck in your cluster. Relentlessly messing with your precious snowflakes.
You will learn how to:
- Use the Kubernetes API with Python.
- Install a container with a simple script that accesses the Kubernetes API.
- Invoke a Cron Job with the Chaos script to randomly melt away Pods.
- Use annotations to define chaos targets.
You don't need a fancy system to carry out the essence of chaos testing. You leveraged the basics of Kubernetes to implement a core Pod destructor using just a Python script in a small custom container then ran it as a CronJob.
With these steps you have learned:
- ✔ Use the Kubernetes API with Python.
- ✔ Install a container with a simple script that accesses the Kubernetes API.
- ✔ Invoke a Cron Job with the Chaos script to randomly melt away Pods.
- ✔ Use annotations to define chaos targets.
In the last year we've seen Chaos Engineering move from a much talked-about idea to an accepted, mainstream approach to improving and assuring distributed system resilience. As organizations large and small begin to implement Chaos Engineering as an operational process, we're learning how to apply these techniques safely at scale. The approach is definitely not for everyone, and to be effective and safe, it requires organizational support at scale. -- ThoughtWorks Radar
- Principles of Chaos Engineering
- Fallacies of Distributed Computing Explained (PDF)
- Kubernetes Client Libraries
- Kubernetes CronJob
Your Kubernetes Cluster
For this scenario, Katacoda has just started a fresh Kubernetes cluster for you. Verify that it's ready for your use:
kubectl version --short && \
kubectl get nodes && \
kubectl get componentstatus && \
It should list a 2-node cluster and the control plane components should be reporting Healthy. If it's not healthy, try again in a few moments. If it's still not functioning refresh the browser tab to start a fresh scenario instance before proceeding.
The Helm package manager used for installing applications on Kubernetes is also available:
helm version --short
You can administer your cluster with the
kubectl CLI tool or use the visual Kubernetes dashboard. The Dashboard can be accessed from the tab labeled Kubernetes Dashboard above the command line. When the Dashboard first appears, it will prompt for an access token. At any time you can run this script to access the Dashboard token:
This script will display the token in the terminal. Copy the green text using your browser's copy feature then paste the token into the prompt when the Dashboard is accessed. If the Dashboard is still starting up, then Katacoda will report the access error. Once the dashboard Pod reports the status Running it can be accessed:
kubectl get pods -n kube-system -l app.kubernetes.io/name=kubernetes-dashboard