Planning is critical to running a successful War Game. One of the core goals is for the incident to feel real, so expect to spend 3-4x the amount of time planning the War Game as you spend running it.

In the first article of this series, we identified a few challenges of production incidents. They’re fast, filled with pressure, and are (hopefully) brand new failures. If the best-case scenario is a new failure (remember: repeated failures mean we never solved it the first time), how can we practice?

It’s 3 AM and your phone is ringing. There’s only one number you let ring through your Do Not Disturb settings. You open one eye and look at the first of 12 on-call notifications.

“Database down. Need help.”

It’s gonna be a long night.