How LLMs Work: A Beginner’s Guide to ChatGPT and Text Generation

When you type a message into ChatGPT, a language model uses the text available to it to predict a small piece of the reply. It adds that piece, predicts another, and keeps going until the response ends.

Those small pieces are called tokens. A token can be a word, part of a word, or punctuation.

Predicting one piece at a time may sound too basic to produce a useful answer. But before choosing each piece, the model processes the context through many layers of calculations. Those calculations use patterns learned during training to help it write explanations, answer questions, and generate code.

Let’s follow a message through that process. You do not need to know machine learning or matrix maths to follow along.

1. Start with the idea of completing a sentence

Imagine someone asks you to finish this sentence:

I drink a cup of

You might say tea, coffee, or water. All three fit. Your choice would probably change if the earlier conversation mentioned a coffee shop.

A language model also uses the available text to work out possible continuations. It assigns scores to the tokens that could come next.

For an easier-to-follow example, suppose the choices are whole words:

These numbers are invented. They show that the model can consider several possible continuations instead of having only one fixed answer.

After choosing tea, it works with the longer sentence:

I drink a cup of tea

Now it predicts what comes after tea. Perhaps a full stop, perhaps every morning over several steps.

The technical name is next-token prediction. It is the repeated operation behind text generation.

2. Tokens are the pieces the model works with

A language model does not always read or write one whole word at a time. Its text pieces come from a vocabulary created for that model.

A tool called a tokenizer splits your text into those pieces and gives each one a number.

For example:

Your text:   "I like tea."

Pieces:     ["I", " like", " tea", "."]

Numbers:    [12, 48, 73, 9]

The split and numbers here are made up. Real results depend on the tokenizer. Notice that a space can belong to the same token as the word after it.

A less familiar word might be split into smaller pieces. Punctuation can be a separate token. Different models can split the same text differently.

The number is an ID, like a product code in a shop. It identifies a piece of text; its size does not describe the piece’s meaning. Token 73 is not “more meaningful” than token 12.

Why tokens matter when you use an AI tool

AI services often measure input limits and pricing in tokens. A long conversation uses more tokens than a short question. The reply also uses tokens.

One hundred words will not necessarily equal one hundred tokens. Code, punctuation, and different languages can produce different token counts.

The model’s context window is the amount of text it can work with in a request, usually measured in tokens. Instructions and earlier messages can take up space too, so the latest question is only part of the input.

3. Embeddings give each token a numerical representation

A token ID tells the model which piece of text it has. The next step gives that piece a representation the model can calculate with.

For each ID, the model looks up a list of numbers. That list is called an embedding. Another name for a list of numbers is a vector.

Token ID:    73

Embedding:  [0.2, -0.7, 0.4, ...]

Real embeddings can contain thousands of numbers. You do not need to inspect those numbers to understand their purpose: they give the model a numerical starting point for processing each token.

During training, the model learns values that help it predict text. Related words and ideas can develop useful relationships in these numerical representations.

For example, patterns involving tea and coffee often involve drinks, cups, or cafés. The model can learn from those patterns. It does not need a programmer to write a separate rule for every pair of related words.

The individual numbers do not usually have tidy labels such as “drink” or “hot.” Meaning-related information is spread across the representation.

The same word can mean different things

Consider:

She deposited money at the bank.
She sat beside the river bank.

The word bank appears in both sentences, but the surrounding words change its meaning.

If both occurrences use the same token ID, they start with the same embedding. Later steps combine that starting representation with context. This lets the model represent the two uses differently.

An embedding is the starting point. It is not the model’s final interpretation of a word in a sentence.

4. The model also needs word order

Compare:

The dog chased the cat.
The cat chased the dog.

The words are the same. Who chased whom changes.

The model therefore needs information about where tokens appear. Different models supply this positional information in different ways, but the purpose is to make order available during the calculations.

At this stage, the model has numerical representations of the text and information about its order. It still needs to work out how the pieces relate.

5. Attention connects a token with its context

Attention lets the model combine information from different positions in the text. It calculates how strongly other available tokens should contribute to a token’s current representation.

Return to the bank example:

She deposited money at the bank.

Words such as deposited and money provide clues about bank. In the river sentence, river provides a different clue.

Attention helps pass that contextual information between positions. The model learns how to calculate these connections during training.

For the text-generating models discussed here, a position can use information from itself and earlier positions. It cannot look ahead at text that has not been generated yet.

Query, key, and value without the maths

An attention calculation uses three kinds of numerical lists, called query, key, and value. You will often see them shortened to Q, K, and V.

A library search is a useful analogy:

• The query is what you search for.
• The key is the information used to decide how well an item matches.
• The value is the information you receive from the matching item.

In attention, the model compares a query with keys to calculate matching scores. It then uses those scores to decide how much information to take from the values.

No written search question exists inside the model. The analogy describes the roles; the actual process uses calculations on numbers.

Some context contributes more than other context

Imagine a small attention calculation gives three earlier positions these weights:

Position A: 10%
Position B: 70%
Position C: 20%

The output combines information from all three, with position B contributing the most in this calculation.

Another calculation can give different weights. A model performs many such calculations, so one attention result does not explain the whole answer.

Models commonly use several attention heads. Each head performs its own attention calculation, and the model combines their results. This gives it several ways to connect information in the same text.

6. Transformer layers process the information repeatedly

A Transformer is the type of neural network used by many modern language models. A neural network is a system of calculations whose adjustable values are learned during training.

A Transformer contains repeated processing blocks, often called layers. Each block takes numerical representations, updates them, and passes them onward.

A simplified view is:

Token representations
        ↓
Attention combines context
        ↓
A small network transforms the result
        ↓
The next block repeats the process

The small network is called a feed-forward network. While attention exchanges information between positions, this network further transforms the information at each position.

Blocks also use supporting operations to keep the values manageable and carry earlier information forward. You may encounter the names normalization and residual connections when reading more technical explanations.

After many blocks, each position’s representation has been updated using the available context. The model uses the representation at the last position to score possible next tokens.

There is no fixed rule that the first layer learns grammar and the next learns meaning. The layers work together, and their roles are more mixed than that.

7. Scores become chances for the next token

After processing the input, the model produces a score for every token in its vocabulary. These raw scores are called logits.

Suppose the prompt is:

The capital of France is

For a tiny example with only three candidates:

A score of 3 does not mean a 3% chance. The raw scores are not percentages.

An operation called softmax converts the scores into probabilities that add up to 100%. For these example scores, the result is approximately:

A real model scores its full vocabulary, not only three city names. This smaller example makes the step easier to see.

There are now probabilities for the next token. The system still needs a rule for choosing one.

8. The choice rule can change the answer

The rule used to choose a token is called a decoding strategy. Two common approaches are greedy decoding and sampling.

Greedy decoding picks the highest score

With greedy decoding, the system always chooses the highest-scoring next token. In the city example, it chooses Paris.

This is a straightforward rule, but it does not guarantee the best complete response. Choosing the strongest-looking option at each individual step can still produce a weak answer overall.

Sampling allows other choices

Sampling makes a random choice according to the probabilities.

With the city example, Paris would be chosen most often across many draws. Lyon and London would still sometimes be chosen.

Once a different token is chosen, the next prediction uses a different context. That can change the rest of the answer. This is one reason the same prompt can produce different responses.

Temperature adjusts how spread out the chances are

Temperature is a setting that changes the probabilities before sampling.

A lower temperature makes the highest-scoring tokens more dominant. A higher temperature gives lower-scoring tokens a better chance.

Using the same example scores:

Higher temperature can produce more varied wording. It can also produce irrelevant or incorrect text. Lower temperature can make answers more consistent, including consistently wrong answers.

Temperature changes how tokens are selected. It does not add knowledge or check facts. If the system always chooses the highest score instead of sampling, changing a positive temperature alone will not change which token ranks first.

Some tools accept a temperature of zero as a request for greedy or near-greedy behavior. The exact behavior depends on the service.

9. The model adds a token and repeats

After choosing a token, the system adds it to the sequence and runs the next prediction.

Starting text:  The capital of France is
Added text:     Paris

Updated text:   The capital of France is Paris
Added text:     .

Updated text:   The capital of France is Paris.
Next action:    End the response

The token boundaries here are illustrative. The model may need several tokens to write a word, or one token may include a space and a word together.

The response ends when the model emits a special ending token or the system reaches another stopping condition. For example, the system may stop because it has reached the allowed output length.

Why replies appear a little at a time

Because generation happens in steps, the service can send part of the reply before the full answer is finished. This is called streaming.

The text you see arriving on screen is partial output. It does not mean the full answer was already written and the interface is only pretending to type it. Services can buffer output, though, so each visible update does not necessarily match one token.

To avoid repeating all earlier calculations, many systems keep reusable attention results in a key-value cache, or KV cache. The cache helps the model continue more efficiently, but it still uses memory, and longer conversations can require more work.

10. Training is how the model learns its patterns

Before you use the model, it goes through training. During next-token training, it processes examples of text and predicts what comes next. Its predictions are compared with the actual next tokens in those examples.

A training process adjusts the model’s internal numbers, called weights, to improve those predictions across many examples.

Additional training can help the model follow instructions and produce more useful responses.

Using the trained model to answer a request is called inference. During an ordinary chat, the model uses its learned weights. Your messages usually change its current context rather than directly changing those weights.

For example:

You: My project is called TeaTracker.
You: What is my project called?
AI: Your project is called TeaTracker.

The model can answer because the name is available in the conversation. This does not mean it has permanently learned that name for every future conversation. A product may offer separate memory features, but those are additional systems around the model.

Why a clear answer can still be wrong

Next-token prediction rewards patterns that fit the text. It does not automatically verify every statement against a trusted source.

A model can produce a natural-sounding answer containing a wrong date, an invented reference, or code that calls a function that does not exist. Such unsupported or incorrect output is often called a hallucination.

A high probability for the next token means the token fits the model’s prediction. It is not a certificate that the resulting claim is true.

For tasks where mistakes matter, check the output against something outside the generated text. Run the code. Open the cited source. Retrieve the current value from the relevant database. Changing temperature cannot replace those checks.

Follow one message through the whole process

Take this message:

Explain why tea gets cold.

Here is what happens at a high level:

1. The tokenizer splits the message into tokens and converts them to IDs.
2. The model looks up a numerical representation for each token.
3. Positional information makes the order available.
4. Attention connects information from different positions in the message and any earlier context.
5. Transformer blocks repeatedly update those representations.
6. The model scores possible next tokens, and the decoding strategy chooses one.
7. The system adds the chosen token and continues until it stops.

The reply might begin with Tea, then continue with pieces that form an explanation about heat moving into cooler surroundings. The exact wording depends on the model, the context, and the selection settings.

For a small experiment, ask an AI tool to complete I drink a cup of several times. Then change the prompt to At the coffee shop, I drink a cup of. Compare the continuations. You are changing the context used for the next prediction, even though the unfinished sentence stays the same.