Dev Deletes Entire Production Database, Chaos Ensues
Kevin Fang1 HN recommenders · 2023–2024
3.6M views · comedy · evergreen · Uploaded 2023-04-26
Kevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.
Shared as a funny, accurate retelling of a classic outage, especially on a bad ops day.
40:20"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson
Strange Loop Conference
31:44The Value of Values with Rich Hickey
InfoQ
19:10Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"
O'Reilly
52:35Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
GOTO Conferences
51:22USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
USENIX
59:56AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401)
Amazon Web Services
2:58The Loving Trap
Ben Woodhams
47:43"Turning the database inside out with Apache Samza" by Martin Kleppmann
Strange Loop Conference
4 comments from Hacker News
Here is a retelling of the time when gitlab went down because someone deleted a production database: "Dev Deletes Entire Production Database, Chaos Ensues" [VIDEO]
"When a program has nothing surprising to say, it should say nothing." Which sounds good in theory, until you run into issues like this: [VIDEO] TLDW: Most of the Gitlabs outage where they deleted the primary prod and backup prod database could have been solved if they just waited for the command line op to complete - but because of a lack of feedback they thought it had frozen, leading to several other plans which made the situation worse.
If the primary fails, will the replica fail over? How do you know? Are the backups working? Who's checking? Can anyone delete a directory when tired at 6PM on a Friday? [VIDEO]
Every time GL is mentioned, I am reminded of the 2017 incident where a person 'rm -rf' the primary pg data logs. The reenactment by Kevin Fang is always a good source of comedy on a bad day. [VIDEO] It’s a shame they are selling out, but I suppose it was bound to happen sooner or later. GH is vacuuming up all of the market share. I kind of liked the ability to self host my own GL. But for small/personal use cases. It was a bit overkill.