"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson

I'll give you some brief thoughts from my experience working at FoundationDB, but if you really want the in-depth answer to what makes our simulation framework an enabling technology for everything we do, you should take a look at the talk my colleague Will Wilson gave at Strange Loop [0]. Everything in our core is written in Flow, our asynchronous programming environment. The architecture of our server is essentially single threaded, with (pseudo-)concurrent actors allowing a single process to satisfy multiple roles at once. The interfaces that these actors use to communicate -- with the disk, over the network, with the OS in general -- are abstracted and implemented in Flow as well. Each of these interfaces not only has at least one "real" implementation, but at least one, and sometimes more, simulated implementations. Since we run multiple actors in a single process all the time, running yet more actors, pretending to be different machines, all still in the same process, was an obvious step. These workers communicate with each other via the _same_ network interfaces that real-world workers do in real clusters. In the end we are able to become our own adversaries, pushing the limits of the system in ways that just don't happen often enough in the real world to test and debug in the wild. Our simulated implementations are allowed to present any behavior that can be observed in the real world, or justified by the spec, or implied by the contract. And they can do so much more

One of their engineers gave a great talk at Strangeloop ( [VIDEO] ) that shows some of the amazing simulations they ran FoundationDB through. It's well worth a watch if you want to understand the punishment that they put this database through to make it stand out.

I'm very excited by this work. As they say, testing for distributed systems leaves a great deal to be desired, but there work is a great step! I was excited for FoundationDB, who put a great deal of focus on testing[1], but they were acquired into mystery by Apple. To give you an idea of how well tested they were, Aphyr of "Call Me Maybe" fame didn't test them "because their testing appears to be waaaay more rigorous than mine"[2]. FoundationDB ran Jepsen internally anyway[3]. DEMi is a really nice tool that has me excited. I've worked with similar test systems for non-blocking IO models and they're hugely useful. For non-blocking IO, you can artificially accelerate time as you can leap to the next interrupt, speeding up fuzzing substantially (see: Flow from FoundationDB in [1]). @ikneaddough, the demi-applications repo notes Spark, have you tested it so far? I quite like Spark but had some very temperamental issues at scale. [1]: [VIDEO] [2]: [LINK] [3]: [LINK]

This talk was about Foundation DB was brought up recently, and it's pretty amazing. I recommend watching the whole thing, but to be brief they are taming the "infinite interleavings" problem through determinism. "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson [VIDEO] They wrote an interesting Actor DSL that compiles to C++ and is completely deterministic, and they torture this deterministic engine with generated test cases on a cluster every night. I guess you could say that the whole cluster is necessarily non-deterministic, but an individual node is deterministic, given an ordering of the messages it receives.

>>> But for the Web site for my startup, early on it was clear that I needed a good key-value store as a Web user session state store (server). So, I designed one, coded it up, and have been using it. It seems to work fine. Well depending on what error scenarios you might run into your code will work until it runs into those scenarios. Few ideas what could go wrong you can get from this video: [VIDEO]

Soooo stoked. This should make futzing with it to get started a lot easier. I've been thinking about going to the FoundationDB summit to hopefully learn more... Does anyone have any good resources for learning about it? I've tried a few times but find the documentation a bit hard to grok. Maybe I'm just slow. Also if you haven't seen it, this is one of the most amazing talks on distributed systems (because it's about testing them) I've ever seen (by Will Wilson from FoundationDB pre-Apple merger): [VIDEO] Truly mind blowing how good FoundationDB is supposed to be for its use cases, and this Mongo layer is just the business when it comes to helping adoption.

Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this: [VIDEO] Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation. In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security: [LINK] [LINK] Plus some of their other tools in various states of usability: [LINK]

There is a great talk about this testing, it goes all the way to deterministic seeds and other basics. The level of determinism is admirable. [VIDEO]

Neat to learn about thread sanitizer. It sounds similar to another tool from Microsoft Research called Torch ( [LINK] ) which automatically instruments binaries to detect data races. Coyote is similar in some ways but different in others. Coyote serializes the execution of the entire program (running one task at a time), exploring one set of interleavings and then rewinding, and then exploring another set of interleavings, hoping to hit hard-to-find safety and liveness bugs. In addition to finding concurrency bugs in one isolated process, we use it to find bugs in our distributed system by effectively running our entire distributed system in one process and having Coyote explore the various states our system can be in. It sounded mind-boggingly cool when I first came across this way of testing distributed systems through Foundation DB ( [VIDEO] we're emulating this kind of testing in our distributed system through Coyote. And unlike Foundation DB which had to develop their own variant of C++ to be able to do this kind of testing (kudos to them for doing it), Coyote allows us to do it on regular C# programs written using async/await asynchrony and benefit from decades of Microsoft Research in exploring large state spaces effectively.

Here's another excellent talk at Strangeloop on FoundationDB's simulation testing by Will Wilson in 2014: [VIDEO]

The FoundationDB team took a different approach, that I found pretty interesting. They created a fully deterministic (single threaded) simulation of a distributed database, and then use that simulation to test their implementation of the database under difference scenarios. They'd probably be interested in something like what you describe, as the bulk of their work seems to be rooting out and squashing bugs caused by factors out of their control (dropped connections, etc.) [VIDEO]

I was watching the old 2014 video: [VIDEO] I've included the timestamp of the relevant bit. He is talking about swizzling, which is like clogging++. He says they take a subset of the network connections, and have them timeout, but then bring them all back in reverse order, and that this somehow exposes far more bugs than other methods. This seems like a very interesting "well, that's funny!" moment that might lead to understanding something foundational about how distributed systems fail. Does anyone have more insight on this?

AHA! I should have known that generation-reference business was intended for games all along. Reminds me of the time the FoundationDB team wrote a compiler to write a simulator to test their database: [VIDEO]

What always fascinated me is they built the simulator for the database first(ish) and relied on it as a first class citizen while building the DB: [VIDEO] > We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing. [LINK]

It is a tall order, but but it's doable. The most important thing is to reduce the surface area of your application. Our business logic is almost entirely pure. The 'procs' have no side effects, except for things that happen on the other side of a clojure protocol (java interface). That means all side effects can be stubbed out during tests. Our "user" code has no access to threading libraries. Threading happens in "kernel" code. Watch [VIDEO] for a good example of how this has already been done in practice.

Tangent about FoundationDB: this is a great video that explains how the team tested it [VIDEO] Spoiler: they even had their own custom power supplies used to test against power failures.

"FDB can tolerate f failures with only f+1 replicas." Wait a minute, I know that formula... Viewstamped Replication?? I need to read the foundation DB paper. (I mainly read CRDT stuff so hopefully it's understandable). --- In general I'm really impressed foundation DB folks. The talk "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson blew my mind. TL;DR they spent the majority of their initial dev effort into making a simulation of the database, then when they were happy with that plugged in real storage, time and networks at the end. Well worth a watch for anyone interested in distributed systems & reliability. [VIDEO]

The great thing about FDB is that they have a testing infrastructure that is unlike any other database: [VIDEO] If there is a Swift compiler bug that causes data loss I will bet money they will catch it (I will also bet they will find at least one Swift compiler bug)

> We would like to combine Jepsen’s workload generation and history checking with Antithesis’ deterministic and replayable environment to make our tests more reproducible. For those unaware, Antithesis was founded by some of the folks who worked on FoundationDB - see [VIDEO] for some of their handiwork. A Jepsen + Antithesis team up is something the world needs right now, specifically on the back of the Horizon Post Office scandal. Thanks for all your work highlighting the importance of db safety Aphyr

FondationDB is such an incredible resource, but it’s so hard to access for the general public. It’s no wonder so few people know about it. I’ve always thought that if there were easier ways to interact with it, or some dev-friendly layers built on top (or public) it would become way more popular. it has shown how well it scales with Apple and snowflake, but is not an obvious choice for non-internet scale applications. If you don't know about FDB; there's an amazing video about how they test it that really got me into learning more: [VIDEO]