Understanding the Python GIL

"I never actually got around to learning -- or even thinking about!-- what exactly the GIL is" David Beazley does a good job on explaining the GIL. There are multiple videos of his GIL talks on YouTube. This one, for instance: [VIDEO] He has a page about this at [LINK]

"The point of such a presentation is to take a piece of code that the presenter has understood deeply and for them to help the audience understand the core ideas..." I do get lots of value out of that. My favorite example is Beazley's GIL talk: [VIDEO]

> If you want speed you go to any other scripting language (other than Ruby). Ruby has historically had the same issues. Most Pythonistas I know aren't so evangelical. It's mostly a question of how to go about integrating C/C++ code. Many people complaining about the GIL (and the like) have some naive microbenchmark, don't understand the trade-offs/limitations of their runtime etc. That doesn't mean critique isn't important and required, but it's going to be better when it's properly researched and improves on the body of work out there ( [VIDEO] ). > Transpiling it to another language like Google did, shows that the underlying technology is not worth much. How is this any general indicator of the worth of the language? It shows for some cases, that Google thought this was a worthwhile investment. Google has experimented for a long time with ways to improve how Python code can be run. They ran the Unladen Swallow project, but spent more time on LLVM issues at the time making it infeasible to continue the project. They'll discontinue one path and try another. None of this is really a commentary from Google on CPython, the community, or the value that it has for most people. The people working on this stuff interact in a pretty friendly basis.

I would recommend you use coroutines rather than multiprocessing.dummy for balancing blocking operations (like IO). I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1]. Python3 has asyncio as part of the stdlib but for Python2 there are a variety of coroutine libraries out there (I think the most widely recognized being gevent). [1] [VIDEO]

David probably isnt the best guy for that. But here's one of the famous ones on GIL. [VIDEO] Beware though, his talks are known to be devastating to the programmer, resulting in serious head injuries, trauma, headaches and other fun stuff.

> > Python's multiprocessing library is needed to overcome the GIL > No it's not, just use threads. I just wanted to expand on this a little to describe some of the downsides to threads in Python. Multi-threaded logic can be (and often is) slower than single-threaded logic because threading introduces overhead of lock contention and context switching. David Beazley did a talk illustrating this in 2010: [VIDEO] He also did a great talk about coroutines in 2015 where he explores threading and coroutines a bit more: [LINK] In workloads that are often "blocked" like network calls our I/O bound work loads, threads can provide similar benefits to coroutines but with overhead. Coroutines seek to provide the same benefit without as much overhead (no lock contention, fewer context switches by the kernel). It's probably not the right guidelines for everyone but I generally use these when thinking about concurrency (and pseudo-concurrency) in Python: - Coroutines where I can. - Multi-processing where I need real concurrency. - Never threads.

I am not expert, and this might be dated information, but historically there is a "thrashing" introduced between the GIL and OS scheduled threads which has surprising performance implications. David Beazley does a much better illustrating the nuances than I ever could: [VIDEO] Including a wild phenomenon where a threaded Python program's performance can degrade by increasing number of CPU cores available to a system. Coroutines avoid this by not requiring individual thread scheduling by the OS. So they effectively release the GIL like threads on IO but avoid OS level thread switching.

There's a GIL presentation [1] from 2010 which shows Python 3.2 on a 1 CPU machine was faster than on a dual core. In all, there might be reason to expect GILless python to be faster single core in certain scenario's. [1] [VIDEO]