State of GPT | BRK216HFS

Chat GPT 4 has ongoing training, such as using Reinforcement Learning from Human Feedback (RLHF) to tune it to provide "better" responses, "safer" answers, and to generally obey the system prompts. There's a release every few weeks. Yes, I've noticed too that recently it has become very "cagey", qualifying everything to death with "As an AI model...". A paper[1] that took snapshots monthly mentioned that as the initial bulk self-supervised learning went on, the model became smarter, as expected. However, once the "clicker training" was imposed on top to make it behave, its peak capabilities were reduced. I'm not sure if it's in the paper or the associated video, but the author mentioned that the original unrestricted model would provide probability estimates using percentages, and it was a very accurate predictor. The later versions that were adjusted based on human expectations used wishy-washy words like "likely" or "unlikely", and its estimation accuracy dropped significantly.[3] At Build 2023, Andrej Karpathy outlined[2] how they trained GPT 4. Essentially, the raw model during training builds its own intelligence. Then there's three stages of "tuning" to make it behave, and all three are based on human input. Essentially, they had contractors provide samples of "ideal" output. Similarly, end-users could up-vote or down-vote responses, which also got fed in. My personal theory is that the raw models can get about as intelligent as the average of the consistent and coheren

I share many of your concerns and frustrations, although I suspect what you're asking for their consider a moat along the lines of a trade secret, rivaled only by the collection of performance improvement techniques they've amassed in 1000s-10,000s of training runs, 100s of engineers, and (hundreds of?) millions spent on compute. People are hired and praised in the community for their skill in cleaning data. A non-answer for you, but for curious others, [State of GPT] 10 days ago provides a through introduction to the process used to train, Karpathy speaking at a Microsoft event providing a deep summary review of concepts, training phases, and techniques proving useful in the world of Generative Pretrained Transformers. [State of GPT]: [VIDEO]

I recognize there's plenty of catnip here when it comes to calling this "engineering" or not, however, whatever you want to call it (prompt fiddling?), the techniques are crucial if you want to achieve reasonably consistent output from current-state LLMs. As models improve concerns about context window limitations will be reduced and it will be easier to discern user intent. These are good straight-to-the-point guides: - Prompt Engineering by BrexHQ: [LINK] - OpenAI guidance: [LINK] - [LINK] - (great examples): [LINK] - (~hr) Karpathy talk: [VIDEO] tl;dr: - Begin with the best and most forgiving model available, optimizing it later if necessary. - Write your prompts in a clear, specific, and detailed manner - Be mindful, however, of the tradeoff between including more detail and increasing latency or cost. - If you're doing any complex reasoning, ask the model to show its work, help "stretch out" computation over more tokens - "Escape" any included or quoted text properly to avoid confusing the model. - Keep track of your token budget, considering the context window in conversations and response length. - Employ an iterative process of measuring, adjusting, and improving your prompt engineering techniques.

I would highly recommend watching - State of GPT video by Karpathy at Microsoft Build to get an idea of how LLM’s work. There are some really good prompting ideas. It changed the way I use GPT-4. Video: [VIDEO]

Kaparthy explained how LLMs can retrospectively assess their own output and judge if they were wrong. Source: [VIDEO]

Karpathy explains it far better than I did in his "State of GPT" talk about a month ago: [VIDEO] Yes, language can do that, but books and other texts, on average, leave out a lot of steps that humans have learned to do naturally. They can be prompted to do them, and their answers improve, indicates that there is definitely some level of "reasoning" there. But it has weird limitations in its current forms. I expect that to improve pretty quickly, tbh.

Promises are cheap - even if made by researchers and engineers deeply believing that they can beat OpenAI. Let alone any assurances given by PR or marketing departments. And don't fool ourselves - everything announced publicly is prepared or at least approved by such. Computational power alone is not the only resource. It is also the training process itself (see this talk by Andrej Karpathy, [VIDEO] , at 1 min) and, obviously, data and its quality. I will be convinced only after Google demonstrates that Gemini is better than GPT4 (in some, or all, tasks).

> They just produce something that looks like a human wrote it real good. That's what the base model does. To get an LLM assistant there are additional training phases to make it conversational, helpful and more correct - [VIDEO]

Last May, Karpathy mentioned that the next step in LLM performance would likely take inspiration from AlphaGo's Monte Carlo-style tree traversal to enable deeper and more self-correcting reasoning: [VIDEO] But if this approach holds up, it suggests that the most valuable part of the AlphaGo project that applies to LLM development is in fact reinforcement learning through self-play. Why not create a leaderboard of reward-generating language models that constantly "play" against each other, where model selection frequency is based on ELO, and update on the results? What if the Open LLM leaderboard ( [LINK] ) evolved into a constantly improving pool of such models? This also alleviates data scaling issues by providing a diverse and continuously changing distribution of new training inputs.

It sounds like you want more broad stuff, not necessarily learning how to train models. More like learning to use them and how they work. [LINK] and Hacker's Guide to LLMs by Jeremy from Fast.ai - [LINK] State of GPT by Karpathy - [VIDEO] LLMs by 3b1b - [LINK] Visualizing transformers by 3b1b - [LINK] How ChatGPT trained - [LINK] AI in a nutshell - [LINK] How Carlini uses LLMs - [LINK] For staying updated: X/Twitter & Bluesky. Go and follow people that work at OpenAI, Anthropic, Google DeepMind, and xAI. Podcasts: No Priors, Generally Intelligent, Dwarkesh Patel, Sequoia's "Training Data"

Thanks. Here is presentation by Karpathy explaining different stages of LLM training. Explains many details in a form suitable for beginners. [VIDEO]