Chaos engineering is not breaking things

Free preview 3 min to read

The name is unhelpful. Chaos engineering isn't randomly killing processes to see what happens. It's a method with a specific shape, and the shape is what makes it useful instead of reckless.

The four steps

State a hypothesis about steady state. Not "the system works", but something measurable: "99% of roadmap requests return 200 within 500 milliseconds."

Inject a specific failure: one thing, chosen deliberately.

Observe whether the hypothesis still holds.

Stop if it doesn't, and you've learned something you didn't know.

flowchart TB
  H["a measurable hypothesis about steady state"] --> I["inject one specific failure"]
  I --> O["observe the same measurement"]
  O --> Q{"does the hypothesis still hold?"}
  Q -->|yes| L["confidence, about that failure only"]
  Q -->|no| F["a finding. stop the experiment"]
  H --> N["without a hypothesis, this is just an outage"]

Why the hypothesis comes first

Without it, you can't tell success from failure.

If you kill the database and the site breaks, you've learned nothing, because everyone expected that. If you kill the database and the site keeps serving cached pages for four minutes, that's a finding. The hypothesis is what makes the difference between those two visible.

A hypothesis you're confident about is the most valuable kind to test. A confident hypothesis that fails shows where your mental model was wrong, and that's the entire product of this practice.

What it's for

It's for finding the failures nobody wrote down.

A system has documented dependencies and real ones, and the real set is always larger: a library that retries silently, a timeout that's longer than you think, a health check that reports success for the wrong reason.

Testing verifies what you specified. Chaos engineering discovers what you didn't.

Why the name persists

It came from Netflix, from a tool that killed production instances at random, and it stuck.

The random killing was the least important part and the part that transferred worst. It worked there because that system was already redundant enough for a random loss to be a non-event. For a system that isn't, the same act isn't an experiment.

Key takeaways

  • Chaos engineering is hypothesis, injection, observation and stopping, not random destruction.
  • The hypothesis must be measurable, or you can't tell success from failure.
  • A confident hypothesis that fails is the most valuable result available.
  • Testing verifies what you specified, and chaos engineering discovers what you didn't.
  • Random instance killing worked at Netflix because that system was already redundant.
Lesson 1 of 7 · free

Keep going with lesson 2

You cannot do this blind

Observing steady state is the prerequisite, and the available logs need a steady-state number built from them.

A free account also gives you the quizzes, your progress and notes, and the discussion.

Type to start searching...