Unix and Google

This video shows how Unix was mathematically more efficient than Multics, which goes a very long way at explaining its success and longevity: [VIDEO]

I made a simulation and video which shows why the pipe is so powerful: [VIDEO] I showed it to Doug McIlroy, and while he thought there was more to the story, he didn't disagree with it.

This video explains why UNIX was a moonshot, and why it could be developed by a small number of people because of it: [VIDEO]

I wrote a non-programmer summary to some of my friends, because my mind is blown. Here it is (warning: it's very informal). This is some next level 80/20 stuff, I'd argue it's even 99/1 (consultant speech for "being efficient" by seeing if you can exploit Pareto's principle [0]). A guy on YT [1] simulates in 7 minutes how the two biggest operating systems were built in the sixties (Multics by IBM and Unix by 2 guys in a garage). Nowadays we only have descendants of Unix, this simulation shows IMO two things: 1. How a chain of command screws up productivity (I doubt that anyone at IBM had the power/autonomy to change complete structure of the product. This was needed, but such change management is insane). 2. How the pipe command in Unix changed the mathematical nature of developing an operating system (in fact, it's even better than 99/1, when you think about it). In normal human language, the difference is: IBM programmed modular blocks. Unix programmed modular blocks. IBM then programmed how those modular blocks should interact. Unix created a pipe command and allowed computer users themselves to implement how these modular blocks should interact. At the time, computer users were all savvy enough like that. Result: when IBM created a feature, it needed to integrate it with all the other features that came before. When those 2 guys in a garage (Unix) created a feature, all they needed to do was create that feature. So mathematically you could say (where each term is the amou

I was going to comment on this very thing in the other thread about software glue[1]. In that article there is a youtube video on Multics vs Unix[2] which really outlines why microservices were always doomed. Someone should coin a new law for how programmers have to rediscover Brooks's law every 5-10 years. The issue with microservices, as it has always been, is that you need an enormous company to brute force the communication pathways and maintenance overhead for it to all work. And by work, I don't mean function efficiently (as the Multics vs Unix video shows). I mean just function . Just work at all. The Multics team had all the devs and the Unix team was two guys doing laps around the Multics team. Because they had the mathematics on their side. Remember the bad old days of memory thrashing? That's what happens to teams that do not have enough bandwidth to properly maintain the dozens of services they are responsible for. Your organization gets frozen. This is what we all get for taking advice from the Googles and Facebooks of the world. Google has like a billion lines of code in a monorepo. They do not do things remotely like 99% of the businesses out there. They are sitting on huge piles of money that lets them be incredibly inefficient for decades . [1] [LINK] [2] [VIDEO]

No. pg has a good article on doing things that don't scale[0] and showed how the AirBnB guys went and setup the first listings manually, but it's actually far more general than just kicking off the network effect. Moats are things that don't scale. Open Source software scales perfectly. Billion dollar companies are billion dollar companies because they can use Open Source software wrapped around some Moat-able structure. The more Open Source Software that exists, the bigger the walled garden. When Google spends a billion dollars sending cars all around the world taking pictures, they are doing so in a way that the Open Source world can't centrally compete against, as they don't have the billions to get that done. This means that all Open Source mapping software will be missing a crucial feature that users rely on, and will therefore go unused. But meanwhile the libraries that those Open Source Software use, Google can freely use within their walled garden. On the other hand, Microsoft paid a great deal of money to researchers with Encarta and had this same exact structure and Wikipedia came in and disrupted them. It takes more time for the decentralized approach to come online, but as long as it's possible and the data is kept high quality Google Maps will slowly lose market share to OSM. Open Source software is generally created as libraries, whereas most companies build frameworks. Frameworks are easier to setup and give control to the makers, but they do not scale. Why is

I posted a comment on libraries vs frameworks yesterday[0], here's the relevant section. > Frameworks are easier to setup initially, but they do not scale. Why is it that Microsoft Windows has 13 different dialog generations? Because each is a framework on top of a framework on top of a framework. It's amazing that they can even get that done. > On the other side, OSS is generally built on libraries. When the 2 UNIX devs were in a basement building UNIX and were able to out-compete Multics[1], they did it because they were building libraries that could talk with each-other using pipes around the boundary. Applications that communicate based on input-output with no internal state behave just like pure functions do. Pure functions compose. When Linus built git in 10 days, he was able to do this because the core idea of git isn't actually that much work. The library is built out of composable blocks that neatly come together. Microsoft's TFS Source Control is a framework that acts on your behalf and therefore the bigger the project gets, you need n^2 people to work on it. Amazon's API mandate is the same exact unification as UNIX's pipe effect was to Operating Systems. Amazon's internal teams are building libraries whereas all other companies started at the same time were building frameworks. Jeff was brilliant to see this at the time, and I expect that this memo will be seen as just as pivotal as the Toyota Production System, and it might already have that prestige to some. Unf

This is my take as well. If you are building a framework and its sub-components are frameworks that act on their own behalves, you better build exactly what you need in an opinionated style. But if you are building a framework and its sub-components are libraries, those libraries can be as general as possible, and your framework can then call those libraries in just the way that you need to. The library strategy will always out-compete in the end if you can handle it properly. If you try to do it half-way like these developers did, you will be ruined just like they found. Reposting from a similar comment yesterday - > Frameworks are easier to setup initially, but they do not scale. Why is it that Microsoft Windows has 13 different dialog generations? Because each is a framework on top of a framework on top of a framework. It's amazing that they can even get that done. > When the 2 UNIX devs were in a basement building UNIX and were able to out-compete Multics[0], they did it because they were building libraries that could talk with each-other using pipes around the boundary. Applications that communicate based on input-output with no internal state behave just like pure functions do. Pure functions compose. When Linus built git in 10 days, he was able to do this because the core idea of git isn't actually that much work. The framework is built out of composable blocks that neatly come together as libraries. Microsoft's TFS Source Control is a framework that acts on your behal

> The whole thing was invented by a "software consulting" firm I don't know the whole origin story. It definitely does feel like something Martin Fowler would come up with. But I blame Google for really making it a trend: [VIDEO] And you can see they understand the whole problem with microservices. It's the same thing The Mythical Man-Month was trying to tell everyone decades ago[1] > When n people have to communicate among themselves, as n increases, their output decreases Microservices exasperates this. It is the Multics model. Each microservice implements its own, often wildly different, API. Which every bit of code that needs to use that microservice has to go and implement. [1] [LINK]

Programs/functions like left-pad do not follow the unix philosophy (which the author did say). The idea of the unix philosophy is to write less programs. Not write a ton of tiny programs, but a small number of composable programs. So instead of left-pad, right-pad, middle-pad, and anything else to get specific need you can use sprintf which can format a string in general, one program that can do implement the features of all of those. It is about “coding the permitter, not the area”, so if you have NxM features to build, don’t write N*M individual programs/functions, instead write N+M programs/functions that can be composed together to get all NxM features. This video explains it well: [VIDEO]

Sort of related but an interesting idea is take it one step further with the Feature Oriented Active Modeler (FOAM) [1,2] paradigm and use code to model your whole system, which generates diagrams model, and runnable code in whatever language needed. The project is still young and it may not be practical today with the available tooling but it seems like a cool idea and project. It is influenced by the unix principle of “coding the perimeter not the area” which is essentially factoring your dev tasks into building NxM capabilities, but instead of building NxM things individually build N+M tools that can be composed into NxM capabilities [2]. So with FOAM the idea is if we want to maintain a model of our software, and build it as well, what if we can use one composable tool to generate both, rather than model everything and code it separately. [1] [LINK] [2] [LINK] [3] [VIDEO]

I would say that the best programmers don't change the multiplier, but the exponent, so that the factor by which they're better than the average increases with the size of the problem. This is why you might not see as much of a big difference with a test lasting a few hours, as you would with one lasting weeks, months or years. I explain this a bit by means of a simulation in this video: [VIDEO]

If we analyze the computational complexity of software architectures the way we do regular algorithms, we see that UNIX was much more efficient than previous OSes/approaches. It's demonstrated in this short video: [VIDEO]

Unlike previous OSes, UNIX provided the pipe facility to allow you to combine the functionality of many small programs. Ex. ls | grep ... | sort ... | more Before that every command-line program had to support the full suite of additional functionality like search, filtering, paging, etc. This UNIX/Pipe approach allowed for greater reuse and much less code. This is explained and simulated in this video: [VIDEO]