HackerTube

NYC Tech Talk Series: How Google Backs Up the Internet

Google TechTalks1 HN recommenders · 2014–2021
41K views · talk · dated · Uploaded 2014-01-31

A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.

Cited in backup and data-loss threads as a rare inside look at Google's backup practices, including its use of tape.

Cloud, DevOps & reliabilityDatabases & storageDistributed systems

Watch on YouTube

5 comments from Hacker News

@timothya2014 · in “Dropbox confirms that a bug within Selective Sync may have caused data loss”

There may occasional bugs, but I don't know of any cases when Gmail actually lost emails. According to this talk [0], the bug referred to by that article caused some users' emails to be temporarily inaccessible, but all the emails were eventually recovered. [0]: [VIDEO]

CitesView on HN
@drieddust2016 · in “Code hosting provider ceases trading after “well-orchestrated” DDoS attack”

This should serve as warning to anyone relying blindly on cheap cloud to keep themselves up and running. This also remind me of importance of offline tape backup. Google still uses tapes[1], yet some of the companies have started thinking they can eliminate tapes and move their backups to cloud. [1] [VIDEO]

CitesView on HN
@stegro322016 · in “Don’t Mess with The Google”

I also remember that talk. I haven't verified by watching again, but if anyone's interested I suspect this is the one: [VIDEO]

CitesView on HN
@carapace2019 · in “Game developer loses multiplayer service code”

FWIW... "Backups are a tax you pay for the luxury of restore" "How Google Backs Up the Internet" [VIDEO] Detailed notes: [LINK]

RecommendsView on HN
@twotwotwo2021 · in “Unreliability at Scale”

The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: [LINK] . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if you're trying to be more cautious than you usually want for commercial/academic online storage or even backup, what do you do? And would it work?" A lot (but not all) of the author's Queue piece talks about stats from online storage, which doesn't have some wins you can get if you're entirely about long-term durability. In online systems, heavy ECC seems out of favor compared to replication for performance reasons, but additional LDPC or RS at the app layer can absorb a substantial % of your volumes having problems, or a ton of random bad blocks. (In a more near-line context, Backblaze uses RS: [LINK] ) Same for tape -- offline LTO is slow and drives are costly, but the cost/TB and the rated lifespan seem like advantages over HDDs for this specific goal, at least in a narrow engineering sense. A pile of cryptographic hashes that fits on one device can help you check for and localize errors without sending everything over a network/doing the full ECC dance. If the hash being broken and data tampered with is in your threat model (hey, weird things can happen in a century), you can also

CitesView on HN