[1hr Talk] Intro to Large Language Models

Seems like they have made progress in combining reinforcement learning and LLMs. Andrej Karpathy mentions it in his new talk (~38 minutes in) [1], and Ilya Sutskever talks about it in a lecture at MIT (~29 minutes in) [2]. Finding a proper reward function to train LLMs in a reinforcement learning setup, and using it to train a model to solve math problems in a similar fashion to how AlphaGo used self-play to learn Go, would indeed be a tremendous breakthrough. [1] [VIDEO] [2] [LINK]

Current architectures have a lot of juice left across several axes, using extraordinarily accurate scaling laws. We're nowhere close to a wall. The industry has been focusing hard on 'System Two' approaches to augment our largely 'System One'-style models, where optimal decision policies make Q-learning a natural approach. The recent unlock here might be related to using the neural nets' general intelligence/flexibility to better broadly estimate their own states/rewards/actions (like humans). [EDIT: To be clear, the Q-learning stuff is my own speculation, whereas the 'System Two' stuff is well known.] Serendipitously, Karpathy broadly discussed these two issues yesterday ! Toward the lecture's end: [VIDEO]

Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy. [1]: [VIDEO]

Seems like he actually disagrees here: If you train a bigger model on more text, we have a lot of confidence that the next-word prediction task will improve. So algorithmic progress is not necessary, it's a very nice bonus, but we can sort of get more powerful models for free, because we can just get a bigger computer, which we can say with some confidence we're going to get, and just train a bigger model for longer, and we are very confident we are going to get a better result. [VIDEO] (23:43)

Andrej Karpathy's "Zero to Hero" series on YouTube is the ultimate guide to building LLMs. Extremely information-dense but as complete as it gets: [LINK] Also, an amazing high-level overview of LLMs, including extensive discussion about attack vectors, that he published a couple days ago: [VIDEO]

Are you sure? In Andrej Karpathy's intro to LLMs[1], he says pre-training uses about 10TB of text from the web. It's hard to believe that a comparable amount of text had been created, even across all languages, in the entire history of humanity up to the 17th century. [1] [VIDEO]

Well, I was discussing Large Language Models (LLMs) as a technology, and esp, via this video [0] with a friend, just now. He made a striking remark. LLMs compress the information they ingest in a variably-lossy way, and when you query them, they rebuild a representation from whole or part of this compressed data in a statistical manner. As a result they're a lossy storage medium. Not unlike some music formats which store audio data in a lossy way. You know, this data can be restored with mathematical and statistical wizardry. You don't get everything you put in, but 97%-99% out of it, and companies and RIAA went bonkers for years, because even if it's not an exact reproduction, it was a reproduction enough, and this is a copyright violation. If an LLM can reproduce what I have written, or coded with 97%-99% accuracy without any license information, and I licensed this thing with less than permissive licenses, and sue the maker of that LLM, what will happen? - Will it be a copyright infringement? - Will it be fair use? It's the first if you look at it fairly, but it'll be probably be the latter, because money, fame and other corporate points will be at stake, otherwise. [0]: [VIDEO] Edit: I forgot to add the video. :)

I watched the latest Karpathy video, where he shows that you can create a prompt suffix that looks like jibberish, using an iterative approach. It looks like this technique will be impossible to defend against, because it discovers for an arbitrary string that causes the jailbreak. Looks like this paper is building on the token level attack from that work. The base64 encoding technique was amusing as well [VIDEO]

Hey, imho best overall technical intro to LLMs (I guess that´s your main interest as you mentioned qlora + llama) is by Simon Willis [1]. Additionally or if you prefer videos, the recent 1h "busy persons intro" by Andrei Karpathy is great + dense as well [2]. [1] [LINK] [2] [VIDEO] EDIT: Maybe I misunderstood as you asked about papers, not general intros. I don´t think that reading papers is the best way to "catch up" as the pace is rapid and knowledge very decentralized. I can confirm what Andrej recently wrote on X [3]: "Unknown to many people, a growing amount of alpha is now outside of Arxiv, sources include but are not limited to: - [LINK] - HN - that niche Discord server - anime profile picture anons on X - reddit" [3] [LINK]

My view is to focus on doing stuff. That's what I did. Pick up some task you want the model to do, try finetuning llama, playing with APIs from OpenAI, etc. Googling and asking GPT along the way. Foundational model training got so expensive that unless you can get hired by "owns nuclear power plant of GPUs" you are not going to get any "research" done. And as the area got white-hot those companies have more available talent than hardware nowadays. So just getting into the practitioner area is the best way to get productive with those models. And you improve as a practitioner by practicing, not by reading papers. If you're at the computer, your time is best spent writing code and interacting with those models in my opinion. If you cannot (e.g. commute) I listen to some stuff (e.g. [VIDEO] - Anything from Karpathy on youtube, or [LINK] channel).

Short answer is No. I highly recommended watching Andrej Karpathy's Intro to LLMs talk, particularly the section on System 1 vs System 2 thinking. Long story short, what you are describing, using more processing to prepare a better response, is something that is an area of interest, but is not currently part of ChatGPT (or any other LLM that I am aware of). See: [VIDEO]

Look into the training process - [VIDEO] The goal of the initial pretraining phase is to make it good at predicting the next word. The rest of the training process is aimed at making it (1) helpful and (2) as correct as possible. I think some people oversimplify things by calling LLMs "next token predictors" and they leave out the tuning towards helpfulness and correctness.

In addition to the other great suggestions, point him to Karpathy's YouTube channel[1]. Karpathy has an approachable communication style. Here's his "1 hour intro to LLMs" video: [VIDEO] 1. [LINK]

I suspect these videos would be higher quality than most offerings from universities or business schools (given that it's a fast moving field, and most of the true experts are practitioners, and so if they're teaching, they're not teaching actual classes, at most they're doing youtube videos or interviews): - [LINK] (Jeremy Howard Hacker's Guide to LLMs) - [VIDEO] (Karpathy Intro to LLMs)

All of Andrej Karpathy's stuff on YouTube is absolutely top tier. He's a rare example of a brilliant contributor in his field who is an equally brilliant educator. Start with his short intro to LLMs, only 1 hour long but is dense with valuable stuff: [VIDEO]

Actually the improvement graphs are still scaling exponentially with training/compute being the bottleneck. So there isn't yet any evidence of diminishing returns. source: [VIDEO]

It might be a preview or something because I have YT premium and doesn't show up that anywhere. Can you share a video that works for that? Like this one. [VIDEO]

I watched Karpathy's Intro to Large Language Models [0] not so long ago and must say that I'm a bit confused by this presentation, and it's a bit unclear to me what it adds. 1,5 years ago he saw all the tool uses in agent systems as the future of LLMs, which seemed reasonable to me. There was (and maybe still is) potential for a lot of business cases to be explored, but every system is defined by its boundaries nonetheless. We still don't know all the challenges we face at that boundaries, whether these could be modelled into a virtual space, handled by software, and therefor also potentially AI and businesses. Now it all just seems to be analogies and what role LLMs could play in our modern landscape. We should treat LLMs as encapsulated systems of their own ...but sometimes an LLM becomes the operating system, sometimes it's the CPU, sometimes it's the mainframe from the 60s with time-sharing, a big fab complex, or even outright electricity itself? He's showing an iOS app, which seems to be, sorry for the dismissive tone, an example for a better looking counter. This demo app was in a presentable state for a demo after a day, and it took him a week to implement Googles OAuth2 stuff. Is that somehow exciting? What was that? The only way I could interpret this is that it just shows a big divide we're currently in. LLMs are a final API product for some, but an unoptimized generative software-model with sophisticated-but-opaque algorithms for others. Both are utterly in need fo

* "[1hr Talk] Intro to Large Language Models" by Andrej Karpathy: [VIDEO] from [LINK] * Andrej Karpathy: [LINK]

The weights are just numbers. I don't what technical background you have in other areas of computing, but I think that this is a good, short introduction that doesn't assume too much: [LINK] To quote part of it, Training a model can be thought of as tuning the dials on a really big machine. The way that a language model behaves is entirely determined by these many different continuous values, usually called parameters or weights. Longer and slightly more technical, "Intro to Large Language Models" by Andrej Karpathy: [VIDEO]