12 videos · 7 hr 16 minHow Things Fall Over
HN's reading list for the day after an outage. It opens with short, funny disaster stories, moves into detailed incident accounts and the craft of debugging under pressure, then to designs that limit the damage. It ends with Richard Cook's two short Velocity talks, which several commenters say they rewatch every few months.
Play all on YouTube- 12:26
How Toy Story 2 Almost Got Deleted: Stories From Pixar Animation: ENTV
@HOLLYWOOD · Picked by 2 · 2016–2026Two minutes to start: Pixar nearly lost a film to one command and was saved by a home-computer backup, a cautionary tale HN tells to anyone in storage or SRE.
- 210:20
Dev Deletes Entire Production Database, Chaos Ensues
Kevin Fang · Picked by 1 · 2023–2024Kevin Fang's comic but accurate retelling of GitLab's 2017 incident, where every backup layer failed in turn, which HN prescribes for a bad ops day.
- 327:21
dotScale 2014 - Robert Kennedy - Life in the Trenches of healthcare.gov
dotconferences · Picked by 3 · 2014–2015A rescue engineer's first-hand account of a failure that was organizational, not technical: nobody owned the whole site.
- 437:07
Who Destroyed Three Mile Island? - Nickolas Means | The Lead Developer Austin 2018
LeadDev · Picked by 4 · 2020–2024A minute-by-minute account of a nuclear accident, which HN uses to push back whenever an outage gets blamed on human error.
- 5
52:35Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
GOTO Conferences · Picked by 8 · 2018–2025The postmortem HN rates above almost every written one: a mistyped command reboots a whole Joyent datacenter, and Cantrill walks through what came after.
- 6
49:09Debugging with the Scientific Method - Stuart Halloway
ClojureTV · Picked by 4 · 2015–2023A calmer counterpart on method (form hypotheses, try to falsify them, write everything down), which several commenters say changed how they debug.
- 7
46:37Zebras All the Way Down - Bryan Cantrill, Uptime 2017
Code & Supply · Picked by 6 · 2018–2025Cantrill again, on the rare failures (firmware, disks, opaque hardware) that keep turning out to be the real cause, a talk HN calls entertaining and sobering.
- 8
55:44AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)
Amazon Web Services · Picked by 2 · 2019–2023Moving from response to design: cell-based architecture and shuffle sharding, which HN links whenever someone asks how to contain an outage.
- 9
45:31The Do's and Don'ts of Error Handling • Joe Armstrong • GOTO 2018
GOTO Conferences · Picked by 2 · 2018–2022Erlang's creator makes the case for letting things crash and recover, a conceptual talk rather than a code-heavy one.
- 10
1:01:59TigerStyle! (Or How To Design Safer Systems in Less Time) by Joran Dirk Greef
TigerBeetle · Picked by 2 · 2023–2025The most demanding answer here: TigerBeetle's method of assertions, up-front design and deterministic simulation for software meant to be defect-free.
- 11
28:21Velocity 2012: Richard Cook, "How Complex Systems Fail"
O'Reilly · Picked by 3 · 2013–2021A patient-safety physician explains to ops engineers why complex systems always run partly broken, and one HN commenter rewatches it every couple of months.
- 12
19:10Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"
O'Reilly · Picked by 4 · 2017–2021The capstone, in 19 minutes: Cook's model of pressures pushing a system toward the edge of failure, which one commenter called among the most important talks they had seen.