From Hand-Written Rules to LLMs: How AI Learned to Learn

For thirty years, "AI" mostly meant a human writing down what the machine should do. If an email contains "free money", flag it as spam. The rules worked until they didn't, because the world has more edge cases than anyone can enumerate by hand.

The shift that produced ChatGPT wasn't a bigger rulebook. It was giving up on rules entirely and letting the machine infer the pattern from examples. Show it 60 labelled emails and it discovers for itself which words signal spam — including combinations no human wrote down.

This walks the four steps the interactive deep dive lets you run live: why learning beat rules, how words became vectors, how attention lets a word see its whole sentence, and how predicting the next word turns into something that looks like understanding.

Why did hand-written rules hit a wall?#

Take spam filtering. The rule-based version is one line a person writes: does the text contain the adjacent tokens "free" and "money"? If yes, spam.

It fails in both directions: "the free webinar on money management starts at noon" is a normal work email that trips the rule, while "wire transfer five thousand dollars to claim your reward" is obvious spam that never says "free money" and slips through. Every patch you add spawns two new exceptions.

The learned alternative is a multinomial Naive Bayes classifier. Instead of one rule, it counts how often every word appears in spam versus ham across a labelled corpus, then scores a new email by the total evidence:

The + 1 is Laplace smoothing — it stops any single unseen word from zeroing out the whole product. Trained on just 30 spam and 30 ham lines, this reaches about 88% leave-one-out accuracy, and it handles both traps the rule missed: "management" and "webinar" pull the first email back toward ham, while "wire", "claim", and "reward" flag the second. Nobody told it those words mattered — it learned the weights from data, and it can still be wrong on adversarial text. Learning trades brittle certainty for calibrated, improvable guesses.

Words as numbers: can you do math on meaning?#

A classifier that counts words never knows "king" and "queen" are related — to a bag-of-words model they're just two different strings. The fix that unlocked modern NLP is the embedding: represent each word as a vector, a point in a space where distance means similarity of meaning.

In the deep dive the vectors are hand-built along 10 interpretable axes — royalty, gender, adult, capital-city, fruit, and so on — so you can read them directly. "king" scores high on royalty and positive on gender; "queen" is identical except gender flips negative. Real trained embeddings (word2vec, GloVe) discover axes like these on their own from raw text; the hand-built version just makes them visible.

Once meaning is geometry, you can do arithmetic on it. The famous example:

Subtract the "male" direction, add the "female" direction, and you land next to "queen". You measure "next to" with cosine similarity — the dot product of two L2-normalized vectors, which is just the cosine of the angle between them:

Rank the whole vocabulary by cosine to king − man + woman, exclude the three input words, and "queen" comes out #1 on the actual numbers. The analogy isn't hard-coded; it falls out of the geometry — meaning as position in a vector space is what lets a neural network operate on language at all.

Attention: how does "it" find what it refers to?#

One vector per word has a ceiling. "Bank" is one point, but a riverbank and a money bank are different meanings, and a fixed vector can't be both — the word needs to change depending on its neighbors. That's what attention does: it lets every word look at every other word and pull in whatever resolves its meaning.

Consider the sentence "The animal didn't cross the street because it was too tired." What does "it" refer to — the animal or the street? A human knows instantly; the machine has to compute it. Scaled dot-product attention is the mechanism:

Each token is projected into three vectors: a query (what am I looking for?), a key (what do I offer?), and a value (what content do I carry?). To resolve "it", you dot its query against every token's key, producing a raw alignment score for each. Divide by √dₖ — here the vectors are 9-dimensional, so you divide by 3 — which keeps scores from exploding as dimension grows and softmax from saturating. Softmax turns the score row into a probability distribution that sums to 1, and the context vector is the value-weighted blend.

The result: attention from "it" peaks on "animal". Change one word — "too tired" to "too wide" — and because "wide" describes extents, not animate things, the same computation flips the peak to "street". Nothing was rewired; the geometry of the new word redirected where attention pointed. That's coreference resolved by arithmetic, live in the deep dive's spotlight demo.

How ChatGPT actually works: predict, then loop#

Stack attention into a Transformer, train it on a slice of the internet with one objective — given the words so far, predict the next word — and generation is just that prediction run in a loop. There is no plan, no database lookup. Each step:

The model outputs a raw logit for every word in its vocabulary. Divide the logits by a temperature knob, softmax them into probabilities, keep only the top-k most likely, renormalize, pick a word, append it, and run again. Feed "the model predicts the next" and it returns "word"; feed that back and it keeps going.

The interactive version uses a real count-based bigram model, so every probability bar is computed live. Temperature is the dial worth understanding — trace exactly what it does to one distribution below.

Walkthrough: how temperature reshapes one distribution#

Suppose after some prompt the model produces three candidate next words with these logits: cat = 3.0, dog = 1.0, mat = 0.2. Temperature divides every logit before softmax, so it's the same three numbers reshaped three ways.

At T = 1.0, softmax of the raw logits gives cat a commanding 83.6%. Lower the temperature to 0.5 and you divide each logit by 0.5 — doubling the gaps — so cat balloons to 97.8% and the greedy pick becomes nearly deterministic and repetitive. Raise it to 2.0 and you halve the gaps; cat drops to 61.9% while dog and mat climb, so the distribution flattens toward less predictable, more varied picks (and eventually incoherent ones). In the demo, T = 0.2 sharpens the top word to ~70% of the mass while T = 2 flattens it to ~3% — nearly uniform over 131 words. Top-k then trims the long tail: keep the k highest-probability tokens, renormalize so they sum to 1, and pick only from those — why raising temperature without a top-k cap sends output off the rails.

What to remember#

The whole arc is one idea applied four times: stop specifying the answer, specify a way to learn it from data.

• Rules don't generalize; learned weights do. Naive Bayes beats a keyword rule because it weighs all the evidence, not one trigger phrase.
• Meaning becomes geometry. Embeddings turn words into vectors, so similarity is distance and analogies are vector arithmetic — cosine similarity does the ranking.
• Attention is context-dependent meaning. softmax(QKᵀ/√dₖ)·V lets each word attend to the tokens that disambiguate it, which is how "it" resolves to the right noun.
• Generation is prediction in a loop. Logits → temperature → softmax → top-k → greedy pick, one token at a time. Understanding is an emergent side effect of getting good at next-word prediction.

Keep going#

This piece is the map; the mechanics get deeper from here. To go one level down into each part:

• Neural Networks from Scratch: How Machines Actually Learn — what "learning the weights" actually means, gradient by gradient.
• Tensors and PyTorch: The Data Structure That Runs Deep Learning — the array type every embedding, logit, and attention matrix is made of.
• Transformers, Part 1: Tokenization, Embeddings & Positional Encoding — how raw text becomes the vectors this article started from.
• Transformers, Part 2: Self-Attention & Multi-Head Attention — the full attention stack, beyond the single spotlight demo.

Then open the interactive deep dive to type your own email into the spam race, run King − Man + Woman on the real embeddings, watch "it" flip its referent, and step the next-word loop with your own temperature and top-k.

FAQ#

What is the difference between rules-based AI and machine learning?#

Rules-based AI executes conditions a human wrote by hand: if the text contains "free money", flag it. It's transparent but brittle — it fails on any case the author didn't anticipate. Machine learning instead infers the rule from labelled examples: a Naive Bayes filter counts word frequencies across spam and ham, learning which words carry evidence and catching spam that never uses the obvious trigger phrase. The payoff is a calibrated probability you can improve with data, not a rigid rule a human keeps patching.

How can you do arithmetic like "King − Man + Woman = Queen" on words?#

Each word is an embedding — a vector positioned so similar meanings sit close together in space. Directions in that space correspond to concepts: roughly a "gender" direction and a "royalty" direction. Subtracting "man" and adding "woman" moves you along the gender axis while leaving royalty untouched, landing next to "queen". Cosine similarity — the angle between vectors — confirms it: "queen" ranks first once the input words are excluded. The relationship isn't programmed in; it emerges from how the vectors were learned.

What does the √dₖ scaling in attention actually do?#

Attention scores each query-key pair with a dot product, and dot products grow larger as the vector dimension dₖ increases. Feed those large raw scores straight into softmax and one score dominates, collapsing the distribution to nearly all-or-nothing — which kills the gradients needed for training. Dividing every score by √dₖ normalizes the magnitude back to a stable range so softmax stays smooth and differentiable — exactly why the formula is softmax(QKᵀ/√dₖ)·V and not just softmax(QKᵀ)·V.

How does temperature change what an LLM generates?#

Temperature divides the model's logits before softmax, stretching or compressing the gaps between candidate probabilities. A low temperature like 0.2 widens the gaps, so the top word dominates and output becomes deterministic and repetitive. A high temperature like 2.0 shrinks the gaps toward uniform, so lower-ranked words get a real chance and text becomes more varied but risks incoherence. The words never change — only how confidently the model commits to its favorite. Top-k truncation usually pairs with it, trimming to the k most likely tokens before the pick, so a high temperature can't drag absurd words into the output.