Modern Dictionaries by Raymond Hettinger

Raymond Hettinger gave a presentation a few weeks ago on the evolution of the Python dict implementation - one of the most insightful videos I've seen in a long time. [VIDEO]

Raymond Hettinger made an interesting talk about Python's dict (hash table) and how Python's core development team evolved it over time. He's giving the polished version of talk at PyCon this May but if you can't wait for it, here's a rough version: [VIDEO]

This has been changed in Python 3.6 only, due to a change in dictionary implementation to make them more efficient. "Modern Dictionaries by Raymond Hettinger" [1] is a quite interesting and technical talk about these changes, explaining also the change in iteration order IIRC, worth a watch in my opinion! [1] [VIDEO]

Python dictionaries: A Confluence of Great Ideas (also known as Modern Dictionaries) by Raymond Hettinger [VIDEO]

Actually they optimized the hashmap, and it turned out a side effect of the optimization was insertion order. First implemented in pypy, then ported to cpython 3.6. Talk by Raimond Hettinger about the new implementation with several other implementations: [VIDEO]

I don't know if it is part of the bests, but very interesting anyway: Modern Dictionaries by Raymond Hettinger [1] It's about Python 3.6 dictionaries improvements. [1] [VIDEO]

I think this behavior will only be working starting from Python 3.6, as the dictionary obtained by calling `dict.fromkeys()` before this would not keep the ordering [1] [1] [VIDEO]

As far as I know, it was implemented by Raymond Hettinger. This is a very interesting talk about the new tech: [VIDEO]

Raymond Hettinger does or did a few talks on dictionaries too. He covers the history of dictionaries in python and how they evolved. It was mentioned in the article's comment section. [VIDEO]

The talks by Brandon Rhodes are these two: * The Mighty Dictionary (PyCon 2010): [LINK] * The Dictionary Even Mightier (PyCon 2017): [LINK] There's also one by Raymond Hettinger, which seems to have been given first at a meetup and then, in shorter form, at PyCon 2017: * Modern Dictionaries: [VIDEO] * Modern Python Dictionaries - A confluence of a dozen great ideas (PyCon 2017): [LINK]

3.6 had a couple of big improvement - f strings, annotation, and if I'm not mistaking, the devs did something to improve the dictionaries; from 3.6 on dicts are ordered. I don't know too much about it but raymond hettinger gave a talk about it. [VIDEO]

[VIDEO] Modern Dictionaries by Raymond Hettinger The official description of this talk is below, but personally, I'd watch any video with "Raymond Hettinger" in the title. > Python's dictionaries are stunningly good. Over the years, many great ideas have combined together to produce the modern implementation in Python 3.6. This fun talk is given by Raymond Hettinger, the Python core developer responsible for the set implementation and who designed the compact-and-ordered dict implemented in CPython for Python 3.6 and in PyPy for Python 2.7. He will use pictures and little bits of pure python code to explain all of the key ideas and how they evolved over time. He will also include newer features such as key-sharing, compaction, and versioning. This talk is important because it is the only public discussion of the state of the art as of Python 3.6. Even experienced Python users are unlikely to know the most recent innovations.

You're welcome! I deliberately didn't go in to that for a few reasons. 1. It would have made this article very, very long. 2. It's a bit out of scope for an article on hashing. 3. I think I might give hash maps their own article in future. Hash maps are fantastically deep. So many different ways to do it. You'll find a lot of material online but I'd recommend Raymond Hettinger's talk on how Pythons hash map data structure has evolved over time: [VIDEO] .

This implementation was used from 3.6, right? It's interesting that the idea mail mentions that nothing changes about the implementation (including order) but the memory layout. Which would imply insertion order was already preserved in older versions (not the case IIRC) or the idea underwent a few more changes that did in fact impact order. EDIT: I couldn't quite find an answer but [LINK] mentions the behavior happens since then because the implementation only tracks indices in the hash table itself and relies on maintaining entries separately in a second array that gets expanded in insertion order. This would also seem straightforward but it raises a few questions such as how deletion is implemented (efficiently). EDIT2: Okay, the talk ( [VIDEO] ) mentions they just leave holes until the next resize (at around 42:00). Raymond also mentions there that his original idea didn't preserve ordering but happened due to an additional compacting optimization? Should probably watch the whole thing some time to get the history. Sounds like a fun talk.

I was going to make a tongue-in-cheek suggestion about randomizing undefined behaviour, but I suppose that's common in debugging and even seen in some APIs. At one time, in Python, dictionary (hashmap) ordering was randomized so users wouldn't rely on non-specified ordering behaviour. Eventually though, dictionaries were guaranteed insertion ordering. [VIDEO]

The python dict() is already high-quality and high-performance. You're probably not going to be able to do much better than the current state. If you want faster you end up moving your data into an environment that lets you make more assumptions and be more restrictive with your data and operate on it there. It's how numpy, polaris, and pandas work. Everything in Python is a dict, there's no data structure that's been given more attention. [VIDEO]

I think Raymond Hettinger is called out specially here because he did a well known talk called [Modern Dictionaries]( [VIDEO] ) where around 32:00 to 35:00 in he makes the quip about how younger developers think they need new data structures to handle new problems, but eventually just end up recreating / rediscovering solutions from the 1960s. “What has been is what will be, and what has been done is what will be done; there is nothing new under the sun.”