Lecture
Chaos engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.
In software development, the ability of a program to withstand failures while maintaining an adequate quality of service — often called fault tolerance — is usually specified as a requirement. However, development teams may fail to meet this requirement due to factors such as tight deadlines or a lack of domain knowledge. Chaos engineering includes techniques aimed at meeting fault-tolerance requirements.
Chaos engineering can be used to achieve resilience against infrastructure failures, network failures and application failures; the latter includes resilience to unhandled exceptions.
Chaos Engineering directly verifies:
To assess the level of confidence in the reliability of interconnected complex systems deployed to a production environment, operational readiness indicators are needed. Operational readiness can be assessed using simulation within chaos theory. Solutions for increasing fault tolerance and operational readiness of a platform include strengthening backup, recovery, network file transfer, failover and overall environment security capabilities.
An assessment designed to induce chaos in a Kubernetes environment resulted in the termination of random pods receiving data from edge devices in data centers during the processing of analytics on a big data network. Pod recovery time was a fault-tolerance indicator that assessed response time.
1983 – Apple
While MacWrite and MacPaint were being developed for the first Apple Macintosh computer, Steve Capps created «Monkey» — a desk accessory that randomly generated user-interface events at high speed, simulating a monkey desperately banging on the keyboard, moving and clicking the mouse. It was quickly put to use for debugging, generating errors for programmers to fix, since automated testing was not possible; the first Macintosh had too little free memory for anything more sophisticated.
1992 – Prologue. While ABAL2 and SING were being developed for the first graphical versions of the PROLOGUE operating system, Ian James Marshall created «La Matraque» — a desktop device that, at high speed, generated random sequences of both valid and invalid graphical-interface events, thereby testing the behavior of critically important graphics libraries at partition boundaries. This program was run before deployment to production and worked for several days in a row, providing the necessary degree of complete fault tolerance. The tool was later extended to include database access instructions and other ABAL language files to verify and ensure their subsequent fault tolerance. A variant of this tool is currently used to test the modern version known as OPENABAL.
2003 – Amazon
While working on improving the reliability of Amazon's websites, Jesse Robbins created «Game Day» — an initiative that increases reliability through the deliberate, regular creation of serious failures. Robbins said he was inspired by firefighter training and research in other fields, lessons on complex systems and reliability engineering.
2006 – Google
Working at Google, Kripa Krishnan created a program similar to Amazon's Game Day program (see above), called "DiRT" (Disaster Recovery Testing). Jason Cahoon, a site reliability engineer at Google, wrote a chapter on Google DiRT in the book "Chaos Engineering" and described the system at the GOTOpia 2021 conference.
2011 – Netflix
In 2011, while overseeing Netflix's migration to the cloud, Nora Jones, Casey Rosenthal and Greg Orzell expanded this discipline, working together at Netflix to create a tool that induced failures in their production environment used by Netflix customers. The goal was to move from a development model that assumed the absence of failures to a model in which failures were considered inevitable, prompting developers to treat built-in fault tolerance as a duty rather than an option:
«At Netflix, our culture of freedom and responsibility led us to not force engineers to design their code in a particular way. Instead, we found that we could rally our teams around the idea of infrastructure fault tolerance, isolating the problems caused by neutralizing servers and pushing them to the extreme. We created Chaos Monkey — a program that randomly picks a server and shuts it down during normal working hours. Some may find this crazy, but we could not rely on the random occurrence of an event to test our behavior in the face of the consequences of that event. Knowing that this would happen often created strong cooperation among engineers to build redundancy and process automation in order to survive such incidents without affecting millions of Netflix users. Chaos Monkey is one of our most effective tools for improving the quality of our services».
By regularly "shutting down" random instances of a software service, it was possible to test redundant architecture and make sure that the failure of a server did not have a noticeable effect on customers.
The concept of chaos engineering is close to the concept of Phoenix servers, first introduced by Martin Fowler in 2012.
Chaos Monkey, Gorilla
Latency Monkey
Exception Monkey
Security Monkey
Kong / Gorilla
Chaos engineering ≠ random chaos
It is closer to:
For example, Chaos Kong / Chaos Gorilla are ideally tested on real production systems, but in a very controlled and staged manner.
The main rule of Chaos Engineering
The test must be as close to reality as possible,
but the risk must be manageable
Chaos Kong (region outage) This is already a very high level of maturity
The real-world approach:
Chaos Monkey — is a tool developed by Netflix in 2011 to test the resilience of its IT infrastructure. It works by deliberately shutting down computers in Netflix's production network to test how the remaining systems respond to the failure. Chaos Monkey is now part of a larger toolset called the Simian Army, designed to simulate and test the response to various system failures and edge cases.
The code underlying Chaos Monkey was released by Netflix in 2012 under the Apache 2.0 license.
The name «Chaos Monkey» is explained in Antonio Garcia Martinez's book «Chaos Monkeys»:
Imagine a monkey getting into a «data center» — those «farms» of servers that host all the mission-critical functions of our online activity. The monkey randomly rips out cables, breaks devices and returns whatever it gets its hands on [that is, throws around excrement]. The task of IT managers is to design the information system they are responsible for in such a way that it can operate despite these monkeys, since no one ever knows when they will show up and what they will destroy.
What it does:
Randomly shuts down instances (servers)
Goal:
Verify that the system:
The «Simian Army» is a set of tools developed by Netflix to test the reliability, security or fault tolerance of its Amazon Web Services infrastructure, and includes the following tools:
What it does:
Adds delays (latency)
Goal:
Verify:
What it does:
Generates errors (exceptions)
Goal:
What it does:
Shuts down an entire data center / availability zone
Goal:
Verification of:
What it does:
Takes out an entire region
Goal:
What it does:
Looks for vulnerabilities (for example, open ports)
Goal:
What it does:
Removes unused resources
Goal:
«Chaos Day» at Voyages-sncf.com in 2017 turned failure simulation at the pre-production stage into a game, which was presented at the DevOps REX 2017 conference. Steadybit, founded in 2019, popularized chaos at the pre-production stage and reliability engineering. Its open Reliability Hub extends Steadybit's capabilities.
Proofdock can inject infrastructure, platform and application failures into Microsoft Azure DevOps. Gremlin is a «failure as a service» platform. Facebook's Project Storm simulates data-center outages to improve resilience to natural disasters.
Comments