Chaos engineering experiment simulator

The service is a web page whose front end calls an auth service, then a catalog and a recommendations service in parallel, and must answer within its deadline. Choose which dependency to break and how: latency adds a delay to every call, errors makes a share of calls fail, and a black hole accepts calls and never answers. The blast radius is the share of requests that get the fault; the same share again forms the control group.

Each change re-runs the experiment over the given number of requests, with latencies drawn from each dependency's median and p99. The chart compares the success rate of the two groups as the run goes on, and the panel on the right says whether the steady state held. Edit timeouts, retries and the recommendations fallback in the table to fix what the experiment finds.

Chaos engineering is the practice of experimenting on a system by injecting the failures it will meet in production, such as slow or failing dependencies, lost servers and network trouble, to build confidence that it keeps serving users. It grew out of Netflix's move to the cloud, where any instance could disappear; their Chaos Monkey terminated instances at random so that every service had to tolerate it.

The Principles of Chaos Engineering describe the method in four steps: define the steady state as a measurable output of the system that shows normal behaviour; hypothesise that it will continue in both a control group and an experimental group; introduce variables that reflect real-world events; and look for a difference between the two groups. Disproving the hypothesis means a weakness has been found. Netflix engineers described the approach in IEEE Software in 2016.

The steady-state metric should be something users feel, not a server statistic: Netflix uses stream starts per second. Here it is the share of page requests that succeed within the front end's deadline. Comparing the experiment group with a control group drawn from the same traffic at the same time separates the fault's effect from everything else going on: a deploy, a traffic peak, a slow database. The hypothesis in this simulator holds if the experiment group is no more than 0.5 percentage points below control.

Netflix's Chaos Automation Platform, ChAP, works this way: Basiri and colleagues describe putting about 1% of users into an experiment group and another 1% into a control group, routing each to its own small copy of the service, and comparing the two.

An experiment in production that finds a weakness has, by definition, hurt someone. The blast radius limits how many: only the experiment group sees the fault, and an automatic check stops the run once that group does clearly worse than control. ChAP stops experiments early when it detects excessive customer impact. The difference is large. With 900 ms added to recommendations:

Recommendations are optional, so a slow recommendations service should not break the page. The experiment shows that it does: the front end has 1,000 ms in total, but the recommendations call is allowed to wait up to 1,000 ms and has no fallback. With 900 ms added, only 45.8% of experiment requests succeed, against 100% in control, and 545 of the failures are the page running out of time. A timeout that is longer than the caller's own deadline never fires in time to help.

Setting the recommendations timeout to 250 ms and serving a fallback (a generic list, say) when it fails brings the experiment group to 100%. The retries are still a problem, though: each of the 2 retries waits another 250 ms for a service that is slow on every call, so the p99 is 828 ms against 361 ms in control, and the slow service gets 3 calls per request, tripling the load on the thing that is already struggling. With no retries the p99 is 328 ms and the load 1 call per request.

Other faults find other gaps. With 20% of auth calls failing, auth's single retry leaves 95.2% of requests succeeding, and three retries 99.8%, because retries do help against failures that come and go. A catalog that never answers, with a 500 ms timeout and one retry, uses up the whole page deadline: 0% succeed, and the run is stopped after 100 requests.

• Latency on a dependency. The most revealing fault, because slow is harder to handle than dead: it tests timeouts, deadlines and thread or connection pools.
• Errors from a dependency, all or a share. Tests retries, fallbacks and whether errors are reported to users sensibly.
• Black holes: connections that are accepted and never answered, or packets dropped. Without a timeout, callers wait forever.
• Lost instances or zones: Chaos Monkey's original fault. Tests redundancy, health checks and how fast load moves elsewhere.
• Resource pressure: full disks, CPU or memory exhaustion, clock skew. Often finds monitoring and alerting gaps rather than code bugs.

Common tools include Netflix's Chaos Monkey for instance termination, Shopify's Toxiproxy for network faults between services in tests, Chaos Mesh and LitmusChaos for Kubernetes, and managed services such as AWS Fault Injection Service.

Teams usually start with a game day: a planned session in which people inject a failure into a staging or production system while everyone involved watches, often finding as many problems in dashboards, alerts and runbooks as in the code. Once experiments are routine they are automated and run continuously, so that a regression (a new timeout, a removed fallback) is caught by the next run rather than by the next outage.

Some rules carry over from the method: state the hypothesis before injecting anything; choose a metric that reflects users; keep the blast radius small and widen it only when the results are clean; have a way to stop at once; and fix what you find before moving on to the next experiment.