diff --git a/docs/knowledge/machine-learning/from-nlp-to-llms-gpt-3-5.md b/docs/knowledge/machine-learning/from-nlp-to-llms-gpt-3-5.md new file mode 100644 index 0000000..853fd23 --- /dev/null +++ b/docs/knowledge/machine-learning/from-nlp-to-llms-gpt-3-5.md @@ -0,0 +1,733 @@ +--- +tags: + +- machine-learning +- nlp +- transformers +- llms +- deep-learning +- intermediate + +--- + +# From NLP to LLMs: the Mathematical and Architectural Path to GPT-3.5 + +Large language models did not appear out of nowhere. Every ingredient — token +representations, sequence modeling, attention, scale, and alignment — was built on top of +the ingredient before it. This article reconstructs that chain step by step, aimed at +someone who already knows attention/Transformers at a conceptual level but wants the +mathematical bridge connecting classical NLP all the way to ChatGPT. + +The narrative in one line: + +```text +classical NLP → word representations → RNN/LSTM → attention → Transformer → +self-supervised pretraining → GPT → scaling (GPT-2 → GPT-3) → +instruction tuning → RLHF → GPT-3.5 / ChatGPT +``` + +## 1. Classical NLP Foundations + +Before learned representations, NLP relied on **sparse, hand-built features** counting how +words appear in documents. + +**Term Frequency–Inverse Document Frequency (TF-IDF)** weights a term by how often it +appears in a document, discounted by how common it is across all documents: + +$$ +TF(t,d) = \text{count of term } t \text{ in document } d +$$ + +$$ +IDF(t) = \log\frac{N}{df(t)} +$$ + +$$ +TFIDF(t,d) = TF(t,d) \cdot IDF(t) +$$ + +- $N$ — total number of documents. +- $df(t)$ — number of documents containing term $t$. + +A document becomes a vector over the whole vocabulary, mostly zeros (**sparse**), with +non-zero TF-IDF weights for the words it actually uses (Bag-of-Words, with TF-IDF as a +smarter weighting on top). The limitation: the vector for "car" and the vector for +"automobile" share no dimensions and look completely unrelated, even though the words are +near-synonyms. Sparse representations carry no notion of semantic similarity — they only +capture *which* words occurred. + +**Word embeddings** fix this by learning a dense vector $x \in \mathbb{R}^d$ (typically +$d \approx 100$–$300$) per token, trained so that words used in similar contexts end up +with similar vectors (the distributional hypothesis: "a word is characterized by the +company it keeps"). Because the vectors are dense and learned, semantic relationships +become geometric — synonyms cluster, and directions in the space can even encode +relationships (`king - man + woman ≈ queen` was the canonical Word2Vec demonstration). + +Similarity between two embeddings is measured with **cosine similarity**, which compares +direction rather than magnitude: + +$$ +\cos(x,y) = \frac{x^Ty}{\|x\|\|y\|} +$$ + +This is the first big shift: **from counting to learning a geometry of meaning.** +Everything downstream — RNNs, attention, Transformers — operates on dense vectors like +these rather than sparse counts. + +## 2. Neural Sequence Models: RNNs/LSTMs + +Bag-of-Words representations throw away word order entirely. Language is sequential — +"dog bites man" and "man bites dog" have the same bag of words but opposite meaning — so +the next step was models that consume a sequence and carry information forward through it. + +A **Recurrent Neural Network (RNN)** keeps a hidden state $h_t$ that summarizes everything +seen so far, updated at each time step from the current input and the previous state: + +$$ +h_t = \phi(W_x x_t + W_h h_{t-1} + b) +$$ + +$$ +y_t = \text{softmax}(W_y h_t + b_y) +$$ + +- $x_t$ — the input at step $t$ (e.g. a word embedding). +- $h_{t-1}$ — the hidden state carried over from the previous step (the model's "memory"). +- $\phi$ — a non-linearity (typically $\tanh$). +- $W_x, W_h, W_y$ — weight matrices shared across *every* time step, which is what makes + this recurrent rather than a separate network per position. + +The same weights are reused at every step, so an RNN can in principle process a sequence +of any length. In practice, training runs into **vanishing/exploding gradients**: +backpropagating the loss through $t$ time steps means multiplying $t$ Jacobians together +(this repeated application is called *backpropagation through time*), and if their norms +are consistently below or above 1, the product shrinks to ~0 or blows up exponentially in +$t$. Vanishing gradients mean the network effectively can't learn dependencies that span +more than a handful of steps — a pronoun 40 words back has almost no gradient signal +reaching it. + +**LSTM** (Long Short-Term Memory) and **GRU** (Gated Recurrent Unit) address this with +*gates* — learned vectors in $(0,1)$ that control how much of the previous state to keep, +how much new information to write, and how much to expose as output. This gives the +network a more direct path for gradients to flow through the memory cell across many time +steps, mitigating (not eliminating) vanishing gradients. The mechanism matters less here +than the motivation: **controlling information flow through a sequence was already the +central problem attention would later solve more directly.** + +## 3. Sequence-to-Sequence Models and the Need for Attention + +Tasks like machine translation need to map an input sequence to an output sequence of +possibly different length. The classical solution was **encoder-decoder**: an encoder RNN +reads the source sentence into a single fixed-size vector (the final hidden state), and a +decoder RNN generates the target sentence conditioned on that one vector. + +The problem is the bottleneck: a 50-word sentence and a 3-word sentence get compressed into +the *same-sized* vector. Long sequences lose information — the decoder has to reconstruct +everything from one summary, no matter how much it needed to remember. + +**Attention** removes the bottleneck by letting the decoder look back at *all* encoder +hidden states at every generation step, dynamically deciding which ones are relevant right +now, rather than relying on one fixed summary: + +$$ +e_{ij} = \text{score}(s_{i-1}, h_j) +$$ + +$$ +\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})} +$$ + +$$ +c_i = \sum_j \alpha_{ij} h_j +$$ + +- $s_{i-1}$ — the decoder's hidden state from the previous step (what it's currently + trying to produce). +- $h_j$ — the encoder's hidden state for source position $j$. +- $e_{ij}$ — a raw compatibility score between "what the decoder wants" and "what encoder + position $j$ offers." +- $\alpha_{ij}$ — that score normalized into a probability distribution over source + positions (softmax, so all $\alpha_{ij}$ for a given $i$ sum to 1). +- $c_i$ — the **context vector**: a weighted average of all encoder states, where the + weights are the attention distribution. + +Intuitively, $\alpha_{ij}$ answers "how much should output position $i$ care about input +position $j$ right now?" When translating a word, attention lets the decoder focus mostly +on the corresponding source word (and a bit of surrounding context), rather than fishing +it out of one compressed vector. This is **dynamic, content-based retrieval** from the +encoder's memory — a preview of everything that follows. + +## 4. Self-Attention + +The attention above relates a decoder state to encoder states — two *different* +sequences. **Self-attention** applies the same idea *within* one sequence: every token +attends to every other token in the same input, including itself. + +Instead of hand-designing the "query," self-attention learns three linear projections of +the input matrix $X$ (one row per token): + +$$ +Q = XW_Q, \quad K = XW_K, \quad V = XW_V +$$ + +- $Q$ (query) — "what am I looking for?" +- $K$ (key) — "what do I contain, for others to match against?" +- $V$ (value) — "what do I actually offer, once someone attends to me?" + +Then **scaled dot-product attention**: + +$$ +\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V +$$ + +Working through the shapes makes this concrete. For a sequence of length $n$ and key/query +dimension $d_k$: $Q \in \mathbb{R}^{n\times d_k}$, $K \in \mathbb{R}^{n\times d_k}$, so +$QK^T \in \mathbb{R}^{n\times n}$ — one raw compatibility score between every pair of +tokens. Softmax (applied row-wise) turns each row into a probability distribution over +*all* tokens — this replaces the hand-derived $\alpha_{ij}$ from the previous section with +a learned equivalent. Multiplying by $V \in \mathbb{R}^{n\times d_v}$ produces an output +in $\mathbb{R}^{n\times d_v}$: for each token, a weighted blend of every other token's +value vector, weighted by how relevant that token's key was to this token's query. + +**Why divide by $\sqrt{d_k}$?** Assume $Q$ and $K$ entries are roughly independent with +mean 0, variance 1. The dot product $q \cdot k = \sum_{i=1}^{d_k} q_i k_i$ sums $d_k$ +such terms, so its variance grows proportionally to $d_k$ — for large $d_k$, raw dot +products can become very large in magnitude. Feeding large values into softmax pushes it +into regions where the gradient is nearly flat (the largest logit dominates and the +softmax saturates), which makes learning unstable. Dividing by $\sqrt{d_k}$ rescales the +dot product back to roughly unit variance regardless of $d_k$, keeping softmax in a +well-behaved range. + +The key structural result: **every token can attend directly to every other token in a +single operation**, with a path length of 1 between any two positions — no matter how far +apart they are in the sequence. Compare this to an RNN, where information from position 1 +has to pass through $t-1$ recurrent steps to reach position $t$. This is exactly what +solves the vanishing-gradient/long-range-dependency problem from Section 2, and it does so +without any recurrence at all. + +## 5. Multi-Head Attention + +A single attention operation computes one weighted average per token — one "view" of +which tokens matter to which. But relevance is multi-faceted: one relationship might be +syntactic (subject-verb agreement), another semantic (coreference — which noun a pronoun +refers to), another positional (nearby words). A single softmax distribution can't +represent all of these at once — it's one weighting scheme, and averaging together +signals with different purposes tends to blur them into a single, unhelpfully generic +pattern. + +**Multi-head attention** runs several attention operations ("heads") in parallel, each +with its own learned projections, so each head is free to specialize in different +relational patterns: + +$$ +head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) +$$ + +$$ +\text{MultiHead}(Q,K,V) = \text{Concat}(head_1,\ldots,head_h)W^O +$$ + +Each head projects into a smaller subspace (if the model dimension is $d_{model}$ and +there are $h$ heads, each head typically uses $d_k = d_{model}/h$), so the total compute is +comparable to one full-size attention operation, just split across $h$ independent +"perspectives." The outputs are concatenated back to $d_{model}$ and mixed by output +projection $W^O$. + +In practice, some heads do turn out to specialize in interpretable ways (e.g. attending to +the previous token, or to a syntactic dependency), but this isn't guaranteed or uniform +across a whole model — treat "heads learn distinct roles" as a tendency the architecture +enables, not a designed guarantee for every head. + +## 6. The Transformer Architecture (2017) + +Vaswani et al.'s "Attention Is All You Need" assembled self-attention, multi-head +attention, and a feed-forward network into a repeatable block, and showed that stacking +these blocks — with no recurrence at all — was enough to beat RNN-based +sequence-to-sequence models. + +One Transformer block, in order: + +```text +input embeddings + positional encoding + → multi-head self-attention + → residual connection + layer normalization + → position-wise feed-forward network + → residual connection + layer normalization +``` + +The **feed-forward network** is applied independently to each position (hence +"position-wise") — it doesn't mix information across tokens (attention already did that); +it just transforms each token's representation: + +$$ +FFN(x) = \sigma(xW_1 + b_1)W_2 + b_2 +$$ + +Typically $W_1$ expands to a larger inner dimension (e.g. $4\times d_{model}$) and $W_2$ +projects back down, with $\sigma$ commonly ReLU or GELU. + +Residual connections ($x + \text{Sublayer}(x)$, wrapped by layer normalization) let +gradients skip past each sublayer, which is what makes stacking many blocks (dozens, in +large models) trainable at all — without them, deep stacks would suffer their own version +of the vanishing-gradient problem. + +**Positional encoding** is necessary because self-attention itself has no notion of order +— $QK^T$ treats the sequence as a *set*; permuting the tokens permutes the attention +matrix's rows/columns but changes nothing about how any single token is computed relative +to the others. The original paper injects a fixed sinusoidal pattern per position, added +to the token embedding before the first block: + +$$ +PE(pos, 2i) = \sin\left(pos / 10000^{2i/d_{model}}\right) +$$ + +$$ +PE(pos, 2i+1) = \cos\left(pos / 10000^{2i/d_{model}}\right) +$$ + +Each dimension pair oscillates at a different frequency, so the combination of all +dimensions gives every position a unique, smoothly-varying signature — and because sine +and cosine are periodic, relative offsets between positions can be expressed as linear +functions of the encoding, which the theory suggested would help the model generalize to +relative positions. (Many modern models use learned or relative positional schemes +instead, but the goal — giving otherwise order-blind self-attention some sense of +sequence position — is the same.) + +**Why this mattered practically:** removing recurrence means every token's representation +at a given layer can be computed *independently and in parallel* — there's no "wait for +$h_{t-1}$" dependency. On a GPU/TPU, an entire sequence's self-attention is one batched +matrix multiplication. This is what made training on vastly larger datasets and models +computationally feasible: Transformers parallelize across the sequence dimension in a way +RNNs fundamentally cannot. + +## 7. Encoder-Only, Decoder-Only, and Encoder-Decoder Transformers + +The Transformer block from Section 6 is a reusable unit; models differ in how they arrange +it and whether attention is allowed to look at the whole sequence or only the past: + +- **Encoder-only (BERT-style):** every token attends to every other token, in both + directions. Good for building representations of a full input (classification, span + extraction) but not naturally suited to generating text left-to-right. +- **Decoder-only (GPT-style):** every token can only attend to itself and earlier tokens — + **causal** (masked) self-attention. Naturally suited to autoregressive generation, since + at inference time future tokens don't exist yet anyway. +- **Encoder-decoder (T5-style):** an encoder processes the full input bidirectionally, and + a decoder generates the output autoregressively, attending both to its own previous + tokens (causal self-attention) and to the encoder's output (cross-attention, structurally + the same mechanism as Section 3's attention). + +Causal masking is implemented by adding a mask matrix $M$ to the raw attention scores +before softmax: + +$$ +M_{ij} = \begin{cases} 0, & j \le i \\ -\infty, & j > i \end{cases} +$$ + +$$ +\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V +$$ + +Adding $-\infty$ to the score for any future position $j > i$ makes that entry's +contribution to the softmax exactly zero (since $e^{-\infty} = 0$) — token $i$'s output can +only be a weighted combination of positions $\le i$. This is the mechanism that turns +self-attention into something compatible with next-token prediction: at training time the +model sees the whole sequence at once (for parallelism), but the mask ensures the +prediction for position $i$ never has access to positions after it, which is exactly the +information it wouldn't have at generation time either. GPT is built entirely from this +decoder-only, causally-masked stack. + +## 8. The Key Paradigm Shift: Self-Supervised Language-Model Pretraining + +Pre-Transformer NLP mostly followed a **task-specific** pattern: label a dataset for +sentiment analysis, train a model for sentiment analysis; label a dataset for named-entity +recognition, train a separate model for that. Every task needed its own labeled data and +its own model. + +The shift: train one large model on a task that requires *no labels at all* — +**predicting the next token** — over enormous amounts of raw, naturally-occurring text. +This is **autoregressive language modeling**: the model learns the joint probability of a +sequence by factoring it into a product of conditional next-token probabilities (the chain +rule of probability): + +$$ +P(x_1,\ldots,x_T) = \prod_{t=1}^{T} P(x_t \mid x_{