How far is next-token prediction from answering a person?

What is geometric about a word embedding?

Coordinates describe a vector relative to a basis. Embeddings use learned vectors whose directions can capture relationships among words.

What does attention add to layered activations?

A feedforward network propagates activations through its layers. Causal attention lets each position combine information from earlier tokens.

An attention map shows where to look. What comes back?

Take “fluffy blue creature.” To enrich the representation of “creature,” a head needs both to find relevant positions and to bring something back from them. Query–key dot products help with the first job. The weighted combination of value vectors does the second. Two identical attention maps could write different information into “creature” if the values differed. The adjectives illustrate the distinction; they do not assign a grammatical job to every head.

How does a network change after a wrong answer?

Forward propagation computes a prediction. In micrograd, a loss and its gradients connect that prediction to the next parameter update.

What change does a single handwritten 2 ask for?

Show the network one picture. It guesses the wrong digit, so we would like the score for “2” to rise. But the knobs are on the connections, not on the scores. Turning a weight barely helps if the unit feeding it is nearly silent on this picture. Backpropagation translates a desired change at the output into the local influence of each knob.

The next picture may ask the same knob to turn the other way. The landscape in the gradient descent talk is the average cost across training examples. That separates three jobs: the labels supply the targets, backpropagation computes sensitivities, and the optimizer chooses the step to take.

What does dropout do inside a GPT training loop?

The subnetworks interpretation explains why units are omitted during training. The implementation shows where the random masking happens in an actual model.

How can many subnetworks share one set of weights?

Dropping units during training can be viewed as training different subnetworks with shared parameters. The test-time prediction is not necessarily their exact arithmetic mean.

From predicting a token to using a language model

Building a predictor makes its training target concrete. Using a trained model shows how context and instructions put that predictor to work.

When does a text predictor become an assistant?

Next-token prediction teaches continuation. Compare how later training shapes responses to a user; the distinction does not prescribe a single post-training algorithm.

What is the next unit, and how far back can prediction look?

Karpathy's name generator looks back one character and predicts the next. It is scored by the probabilities assigned to the actual sequence. His tokenizer demonstration changes the unit under prediction by merging frequent adjacent bytes. Sutskever hypothesizes that sufficiently good next-token prediction may require modeling what produced the text. Test the boundary with names: learning frequent letter pairs improves the score without identifying a person. Testing a stronger claim requires contexts that local frequencies alone cannot resolve.