HackerTube

The spelled-out intro to language modeling: building makemore

Andrej Karpathy2 HN recommenders · 2024–2024
1.3M views · tutorial · evergreen · Uploaded 2022-09-07

Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

HN recommends it as a long but worthwhile way to demystify how language models work technically.

Machine learning & deep learningLLMs & generative AI

Watch on YouTube

4 comments from Hacker News

@islewis2024 · in “Markov Chains are the Original Language Models”

I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [VIDEO]

Strongly recommendsView on HN
@chrsig2024 · in “GPT in 500 Lines of SQL”

This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [VIDEO]

CitesView on HN
@georgehill2024 · in “Show HN: Next-token prediction in JavaScript”

You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [LINK] - Highly recommend this one [VIDEO] - [LINK] - [LINK] 1: ex: [LINK]

Strongly recommendsView on HN
@caprock2024 · in “The n-gram Language Model”

For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [LINK] It appears to be a generalized version of the bi-gram project, which he links to here: * [VIDEO] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.

CitesView on HN