Fundamentals
GPT's Leap
Why PreTraining Changed Everything: from CNN/RNN/BERT to GPT — the most important step in the history of large language models.
History of Large Language Models
Before GPT, everyone was building task-specific models.
GPT's arrival was a true leap forward.
GPT's arrival was a true leap forward.
CNN, RNN, and BERT all required defining a task first, then training a model for it.
GPT flipped the script: pre-train on massive text corpora first, then adapt to any task.
GPT flipped the script: pre-train on massive text corpora first, then adapt to any task.
Before
CNN / RNN / BERT
→
Leap
GPT (PreTraining)
The Essence of PreTraining
Training on almost all text on the internet to predict the next Token —
the model was forced to learn grammar, common sense, facts, logic, style…
All of it emerged as a byproduct of next-token prediction.
the model was forced to learn grammar, common sense, facts, logic, style…
All of it emerged as a byproduct of next-token prediction.
Why Is This a Leap?
Before: train from scratch for each task — switch tasks, swap models
GPT: pre-train once, transfer capabilities to any task
This is the core idea behind Foundation Models
GPT: pre-train once, transfer capabilities to any task
This is the core idea behind Foundation Models
Interactive Demo: Experience Four Generations of Models Hands-On
Window Size 3
CNN's Limitation: Only tokens within the window are visible; relationships outside are completely invisible.
The relationship between「哥哥」(brother) and「紫霞」(Zixia) at the start? CNN cannot capture it in one step.
(Example: Chinese sentence "紫霞捧着月光宝盒，轻声问：哥哥" — "Zixia, cradling the Moonlight Treasure Box, softly asked: Brother")
The relationship between「哥哥」(brother) and「紫霞」(Zixia) at the start? CNN cannot capture it in one step.
(Example: Chinese sentence "紫霞捧着月光宝盒，轻声问：哥哥" — "Zixia, cradling the Moonlight Treasure Box, softly asked: Brother")
Residual memory strength per token (as RNN processes to current position)
← Earlier tokens have weaker memory; current token = 100%
RNN's limitation: the hidden state is overwritten at each step; early tokens fade from memory. With long texts, "vanishing gradients" cause information from the beginning to nearly disappear by the end.
Click any token → heatmap brightness = attention weight, yellow=left green=right purple=self
← Click any token above to view the bidirectional attention heatmap
Left tokens (BERT can see)
Right tokens (BERT can see)
Self
BERT's Limitation: Strong bidirectional understanding, but its pre-training objective is "fill in the blank" — it is not good at generation and cannot directly continue text.
GPT Unidirectional Causal Generation · Looks left only, generates step by step
Next Token Probability
Click "Start Generation"
GPT's leap: the causal pretraining objective — predicting the next token — is naturally aligned with generation. No task-specific data needed; the larger the scale, the more surprising the emergent capabilities.