Andrej Karpathy — Intro to Large Language Models
A One-Hour General-Audience Talk · Recorded November 2023 · Released Freely for Education
Andrej Karpathy · Intro to Large Language Models (2023)
Key Themes from the Talk
An LLM Is Two Files
Karpathy's famous simplification: a large language model is a parameters file — the learned numbers, which for a big open model run to hundreds of gigabytes — plus a comparatively tiny program that runs them. The mystery has a file size, and he gives it.
Training Compresses the Internet
Pretraining squeezes a vast slice of the internet into the parameters by teaching the model to predict the next word, billions of times. What results is a lossy compression of human text — which is why the base model dreams plausible continuations instead of giving reliable answers.
Fine-Tuning Directs the Dream
An assistant is made, not born: supervised fine-tuning on labeled conversations, then reinforcement learning from human feedback. Karpathy's own caution is the point — fine-tuning directs the dreaming into helpful shapes; it does not remove the dreaming. Verify what matters.
The LLM Operating System
His forward look: models that use tools, see and hear, reason more slowly in a System 2 style, improve themselves, and sit at the center of computing the way an operating system does — coordinating processes, memory, and tools. A new paradigm, arriving in pieces.
A New Platform, A New Attack Surface
The final part covers jailbreaks, prompt injection, and data poisoning — attacks with no real precedent in earlier software, aimed at a system that reads instructions and data in the same stream. His conclusion is a planner's, not a panicker's: treat model output the way you would treat any untrusted source.
Connected to the Voice Up Trilogy
The Naming Gap — Naming What Matters
Parameters, inference, fine-tuning, RLHF, prompt injection: this talk is a naming ceremony for the whole stack. Karpathy believes a general audience can hold the real vocabulary, and an hour later, his audience can.
A Time for Purpose — Living What You Name
Karpathy left the center of the industry and became one of its teachers. Purpose, in his case, looks like mastery turned outward: the person who helped build the models now spends his time making them legible to everyone else.
Miracle Has a Monster — Facing What Stands in the Way
The monster named here is unearned trust — fluent answers consumed as facts. Karpathy faces it in the open, from inside the field: the dreams are a feature of the compression, and the defense is a habit of verification.