Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Sasha Rush3 HN recommenders · 2024–2024
35K views · lecture · dated · Uploaded 2023-07-05
Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.
HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.
2:25:52The spelled-out intro to neural networks and backpropagation: building micrograd
Andrej Karpathy
18:40But what is a neural network? | Deep learning chapter 1
3Blue1Brown
1:30:28AlphaGo - The Movie | Full award-winning documentary
Google DeepMind
27:14Transformers, the tech behind LLMs | Deep Learning Chapter 5
3Blue1Brown
51:22USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
USENIX
1:08:40Lecture 1 | Machine Learning (Stanford)
Stanford
5:58MarI/O - Machine Learning for Video Games
SethBling
3:31:24Deep Dive into LLMs like ChatGPT
Andrej Karpathy
3 comments from Hacker News
Would recommend Sasha Rush's lecture [VIDEO] , it provides the intuition for state space models via linear RNNs
Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: [VIDEO] Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).
To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: [VIDEO]