How a language model picks the next word

A language model does one thing, over and over: it predicts what comes next.

Give it the start of a sentence and it works out how likely every possible next piece of text is. It picks one, adds it to the text, and does the same again. A long answer is just that loop running many times.

That sounds too simple to produce anything useful. It is worth looking at how the loop works, because the details explain both what these models are good at and where they go wrong.

Text becomes tokens

A model does not read letters or whole words. It reads tokens, which are small pieces of text. Common words are often one token. Rarer words are split into several. "Understanding" might be one piece, while an unusual name might be three or four.

Each token is a number, and each number is looked up to become a long list of values the model can calculate with.

One step of the loop

For every step, the model reads all the tokens so far and produces a score for every token in its vocabulary, which can be tens of thousands of options or more. Those scores are turned into probabilities that add up to one.

If the text so far is "The capital of France is", the token for " Paris" gets a very high probability, and most other tokens get close to zero.

Then the model picks one. It does not always take the single most likely token. A setting called temperature controls how adventurous it is. At low temperature it almost always takes the top choice, so its answers are steady and repetitive. At higher temperature it will sometimes take a less likely token, so its answers are more varied and more likely to wander.

The chosen token is added to the text, and the whole step runs again.

How it learned

The model starts with billions of numbers, called parameters, set to nearly random values. Training adjusts them.

The process is simple to describe. Take a piece of real text. Hide the next token. Ask the model to predict it. Compare its prediction with the real one. Nudge every parameter slightly so that the right answer would have been a little more likely. Repeat this across a huge amount of text, an enormous number of times.

Nobody writes rules for grammar, facts or code. Whatever the model ends up knowing, it picked up because it made its predictions better.

What attention does

Predicting well often means looking back at an earlier part of the text. In "The trophy would not fit in the suitcase because it was too big", the word "it" refers to the trophy, and the model has to notice that.

Most modern models are built on an idea called attention, which lets each token look at the earlier tokens and decide which ones matter for the current prediction. That is how a model can connect a word to something said many sentences before.

From text-continuer to assistant

A model trained only to continue text is not yet an assistant. Ask it a question and it might continue with more questions, because that is what text often does.

To make it helpful, it gets more training on examples of good answers to instructions, and on human feedback about which answers are better. This stage shapes how it behaves, not just what it knows.

Why it can be wrong and sound sure

The training goal is to produce likely text, not to check what is true. When a model does not know something, a fluent, plausible-sounding answer is still the most likely kind of continuation, so it can state something false in a confident voice.

Models also only know what was in their training data unless they are given tools to look things up, and they can only consider a limited amount of text at once, called the context window.

What I'm not sure about

People still argue about what this adds up to. The model learns to predict, and to predict well it seems to build useful internal structure for things like grammar, facts and logic. Whether that deserves the word "understanding" is an open question, and so is exactly what the internal numbers represent. Researchers are working on reading them, but we cannot yet look inside and read them like code.

What is clear is the mechanism: predict the next token, check, nudge, repeat. Almost everything else comes from doing that very well, at huge scale.