Refterm Lecture Part 1 - Philosophies of Optimization

> The overwhelming majority of slow code isn't slow because it's failed to do this kind of microoptimization, it's slow because it used the wrong algorithm. I disagree. I believe the majority of code is slow because it's written without any consideration at all to performance. I like Casey Muratori's philosophy of non-pessimization where true optmization (measuring, working hot spots) is rare and rarely necessary but massive speedups compared to the general state of the art are achievable by simply not writing code using patterns that are inherently slow. This isn't deep algorithmic stuff it's just avoiding copies and/or pointer chasing. Edit: [VIDEO] <- Casey's most recent intro to non-pessimization > For a realistic-sized problem, if you write solutions in C and Python with the same amount of effort from equally skilled programmers, the Python version will almost certainly be faster The Advent of Code runs every year and I'm not sure about C (not something I track) but there are plenty of Rust submissions and while the Rust submissions take longer to come in, it's not THAT much longer. From memory it's like 2x on a timescale of minutes. The Python programs are not faster.

Casey Muratori's refterm video series is a great example of how to do application design. It focuses on writing a terminal emulator for Windows, but almost all the concepts apply to nixes as well -- even down to the level of OS primitives like memory mappings to create a ring buffer, etc. He builds a nearly complete terminal that appears to better support unicode and escape sequences than windows-terminal, in drastically less code and with literally orders of magnitude better performance. In particular, I like his focus on experimentation to figure out a reasonable upper bound on performance (so you can measure success/failure), "non-pessimisation" (don't do things extrinsic to the actual task), understanding the actual machine and its capabilities (he doesn't use the term, but sometimes called "mechanical sympathy"), and tactics for isolating "bad code" that you probably have to use (at least to get started). [VIDEO]

The fallacy of premature optimization rears its ugly head again. When Knuth wrote that quote, he meant something very specific with the word optimization: low level micro optimizations. Spending a lot of time trying to get every little ounce of performance from every little CPU instruction. High level reasonable design decisions are not "optimizations" in this sense at all. You should think of them as non-pessimization. Choosing Python or another slow language is premature pessimization. You just make your program slower for no reason. So choosing a language based on how programs written in it perform is simply non-pessimization. If you are interested in an expansion on this idea, checkout Casey Muratori's lecture about philosophies of optimization: [VIDEO] The article makes claims about software being "fast enough" and not needing further optimization. But what you call "fast enough" is probably 10000 times slower than what it could be. I'm not joking. People who program in slow languages have a really skewed perception about performance. If a page takes 5 seconds to load, they don't see that as a problem. If a server takes 500ms to respond to a request, they don't see that as a problem. This is simply unacceptable.

I’ve found this to be an unsuccessful approach in practice. Performance is a complex, many-faceted thing. It has hidden costs that are hard to quantify. Customers leave in disgust because the site is slow. No amount of “throwing more cores at it” will help if there’s a single threaded bottleneck somewhere. Superlinear algorithms will get progressively worse, easily outpacing processor speed improvements. Notably this is a recent thing — single threaded throughout was improving exponentially for decades so many admins internalised the concept that simply moving an app with a “merely quadratic” scaling problem to new hardware will always fix the problem. Now… this does nothing. I’ve turned up at many sites as a consultant at eyewatering daily rates to fix slow apps. Invariably they were missing trivial things like database indexes or caching. Not Redis or anything fancy like that! Just cache control headers on static content. Invariably, doing the right thing from the beginning would have been cheaper. Listen to Casey explain it: [VIDEO] You need to have efficiency in your heart and soul or you can’t honestly call yourself an engineer. Learn your craft properly so you can do more with less — including less developer time!

I think this is a valid critique. But then, most of the development (C aside) has been a victim to OOP. I have begun to think that OOP creates more problems than it solves. Added complexity and performance loss being the biggest. It's insane if you peek into a big projects where many developers came and went and each of them read an article about what it seemed an interesting design pattern and tried to implement them. You can have some kind of procedural programming if you use classes just to store data and use static methods. But the frameworks are OOP and there is no way around that unless you write your own frameworks. I am a big fan of Mike Acton's Data Oriented Architecture speeches, but I didn't find a way to implement something similar in C#. For those interested in something better than OOP, here's a few links: [LINK] [LINK] [VIDEO]

Honestly you don't even have to think about optimization to make your software not horrifically slow on modern hardware, instead you have to think about non-pessimization. Simply put, avoid needless complexity and gratuitous waste in the first place. See Refterm Lecture Part 1 - Philosophies of Optimization by Casey Muratori: [VIDEO]

>most of the code in the parser/renderer part of the terminal is unnecessary and just slows things down. What this code needs to do is extremely simple and it seems like it has been massively overcomplicated He means exactly what he says. Notably, refterm had almost no optimization applied to it.[0] The massive performance increase was entirely due to non-pessimization.[1] >refterm actually isn't very fast. Despite being several orders of magnitude faster than Windows Terminal, refterm is largely unoptimized and is much slower than it could be. It is nothing more than a straightforward implementation of a tile renderer, with a very simple cache to ensure that glyph generation only gets called when new glyphs are seen. It is all very, very simple. A more complex codebase that parsed Unicode and rendered glyphs itself would likely be much faster than refterm for many important metrics.[0] [0] - [LINK] [1] - Refterm Lecture Part 1 - Philosophies of Optimization - [VIDEO]

I like Casey Miratory's "non-pessimization" approach in Philosophies of Optimization [1] [2]. He breaks "making a program go faster" into 3 modes: (1) "optimization" which includes counting expected cycles or memory bandwidth, careful benchmarking, designing custom algorithms, etc which can be fun but is expensive and is only rarely applied; (2) "non-pessimization" which is simply avoiding adding extra unnecessary work for the cpu, which should be done 100% of the time but isn't in the industry today; (3) and "fake optimization" which is shallowly repeating 'optimization tips' and conclusions from unrelated cases of (1) without actually measuring it. [1]: [VIDEO] [2]: [LINK]

But perfect performance isn't even the benchmark, it's "not ridiculously slow". This is what is meant by "Computers are fast, but you don't know it", you don't even know how ludicrously fast computers are because so much stuff is so insanely slow. They're so fast that, in the vast majority of cases, you don't even need optimization, you just need non-pessimization: [VIDEO]

I under the impression that we are building software the wrong way. This video from Casey Muratori helped me point to some reasons: [VIDEO]

I'm going to pre-emptively link Casey Muratori's Refterm Lecture Part 1[0], because invariably when this topic comes up people are all "but optimization takes time and we gotta push the new features right now or the market will eat us." Computers are so ludicrously fast these days that in the majority of cases you do not need optimization at all. You do need de-pessimization though. [VIDEO]

I realised when I implemented EdDSA for Monocypher that optimisations compound. When I got rid of a bottleneck, I noticed that another part of the code was the new bottleneck, and some of the optimisations compounded multiplicatively . It took many changes before I finally started to hit diminishing returns and stop. All while restricting myself to standard C99, and trying fairly hard not to spend too many lines of code on this. My point being, if most of the program is slowed down by a slew of wasted CPU cycles (costly abstractions, slow interpreted language…), there's a good chance what should have been obvious bottlenecks get drowned in a see of underperformance. They're harder to spot, and fixing them doesn't change much. So before you even get to actual optimisation, your program should be fast enough that actual optimisations have a real impact. And yes, actual optimisation should be done quite rarely. But first, we need to make sure our programs aren't as slow as molasses. See [VIDEO]

Agreed, but most people here haven't paid for the rest of the course, and Casey sometimes forgot that here he's addressing a wider audience. For instance he fails to explain why he didn't bother addressing the tail of his unrolled loop. He does in the course, but here he's just assuming it's irrelevant, and doesn't address again potential criticism like "he doesn't even bother to write correct code, look at that lazy unrolling!" . Thankfully there's a free lecture that explains the broad concept with an example. It's this lecture that convinced me to try out his course, where I hope he'll go into more details: [VIDEO]

Probably inspired by the performance problems with the windows terminal [1] and the accelerated terminal [2] developed by Molly Rocket as 'answer'? A series of videos presenting the poc [3] [1] [LINK] (It takes a PhD to develop that) [2] [LINK] [3] [LINK] , [VIDEO]

> For example they can write some code and say, "adjust for best practice". And they’ll get code which isn’t good, just a parroting of bad ideas widely disseminated. A bit like Casey Muratori’s concept of fake optimisation: [VIDEO]

He isn't saying that you should spend time on optimising your code, rather that you should write performant code from the beginning by not giving the cpu more work than it needs to do to perform the task, typically by avoiding OO paradigms, followings things like SOLID, etc. Then if needed, real optimisation can handle the hotspots in your code and is typically something that only a few of an orgs best engineers can do properly, but if everything is slow then its an insurmountable task for these few engineers. If you want to understand what he means with this watch the refterm series. [VIDEO]

Anything object oriented is slower than anything procedural. Boxing and unboxing is expensive, running constructors and destructors is expensive, copying objects is expensive, virtual tables are expensive. And stuff isn't aligned greatly in memory, we can expect lots of cache misses and branch mispredictions. The fastest possible way is to have C strings in an array and run a function over it. If there are advantages to OOP, speed is not one of them. Here's a great talk about software speed: [VIDEO]

> Casey Muratori has a real in-depth course on why Python is slow and how to make it fast I don’t think that’s a fair characterisation of the course and that description does it a disservice. The course is not about Python, it’s about programming with performance in mind. The video on Philosophies of Optimisation is a good primer on the tenets of the course. [VIDEO]

>You can't get 2000x faster without actually eliminating some large percentage of the work Yeah, exactly. That's how optimization works! Recognizing that in many cases, the work is that was being done is not necessary to produce the result you actually need, that it could have been in a different way. This can sometimes involve adding complexity (by means of a better data structure or algorithm, e.g. a quadtree), and sometimes involve removing it (non-pessimization[0]). That being said, it's certainly also possible to achieve a speedup by breaking things, which you are suggesting is the case here. I can't comment on that part. [0] Casey Muratori - Refterm Lecture Part 1 - Philosophies of Optimization [VIDEO]

I was working on a multiplayer game a while ago, and one of the iterations of the netcode was "thin client" where clients just sent input, server simulated the game, and it dumped world state onto the pipe at 60hz. I didn't ship that version but I estimated a $3000 bandwidth bill with that approach! I started looking into diffing the state, compression, etc... until I realized, wait a minute! My player movement is linear so I only need a packet for start and stop! And so I achieved near infinite efficiency improvement :) I think the word is... a specialized solution can beat a general one. Also, "remembering what the program actually needs to do, and just making it do that"... I de-pessimized the netcode: [VIDEO]