Opening the black box · Behind the Secrets of LLMs

Marie and Martin. At the end of their story a model answers a question about Dresden in smooth, confident sentences. Marie asks the question this page is about: how would we know if it is right? Read the last scene

Two different black boxes#

The phrase is used for two things that have little to do with each other. It helps to keep them apart.

A box you cannot open

Many commercial models are only reachable through a website or an API. You send text and get text back. The weights stay with the company. All you can study is behaviour: what comes out for what goes in.

A box you can open but not read

For open models such as GPT-2 or Qwen you can download every parameter. Nothing is hidden. But half a billion numbers are not an explanation. Knowing all the weights does not tell you why the model wrote this word and not another.

This course works with the second kind. Every model you touch here is open: the bigram table, the tiny GPT you train yourself, GPT-2 and Qwen. So the question is never “what is inside?”. It is “how do I look at it so that I learn something?”.

What happens between text in and text out#

Each stage of the model produces something you can print and plot. None of it is secret.

Text becomes tokensYou can see: exactly how your text was cut up, and how many tokens it costs. Notebook 6, sections 1 and 2.
Tokens become vectorsYou can see: the embedding table, one learned vector per token. In notebook 4 the whole model is this one table.
Attention moves information between positionsYou can see: the attention weights of every head, as a picture of who looks at whom. Notebook 5, section 6.
The blocks rewrite the vectors, layer after layerYou can see: every intermediate vector. They are just long lists of numbers, and this is where reading them gets hard.
The last layer gives a probability for every next tokenYou can see: the full distribution, not only the winner. Notebook 6, section 3.
One token is pickedYou can see and control: temperature, top-k, top-p. This step is random on purpose, and it is not part of the model at all.

Six windows you open in this course#

Notebook 6 · sections 1, 2

The tokenizer

Before the model sees anything, the text is cut into pieces. For English that is little more than one token per word. For other languages it can be several times more, which makes them more expensive and leaves less room in the context.

Notebook 6 · section 3

Next-token probabilities

Ask the model for its ten most likely continuations of a prompt. You see what it considers plausible, how sure it is, and which alternatives it nearly chose.

Notebook 6 · section 6

Surprisal

For every token of a given text, how surprised was the model? You can point at the exact word where a sentence stops being what the model expected.

Notebook 6 · section 5

Perplexity

One number for how predictable a whole text is to the model. Scrambled word order makes it jump. Famous quotes get suspiciously low values, because the model has partly memorised them.

Notebook 5 · section 6

Attention maps

In the tiny GPT you trained, plot where each head looks. Usually one head watches the current or the previous character, and others spread out over the last few characters or back to the start of the word. Nobody programmed that.

Notebook 6 · section 8

Embeddings

Turn sentences into vectors and measure the angle between them. Sentences with a similar meaning point in a similar direction. This is how semantic search finds things.

What the windows do not show#

Each of these tools is real and useful, and each is easy to over-read. Some honest limits:

Attention shows where a head looked, not what it did with it. A bright square in an attention map is a fact about one step of the computation. It is not yet an explanation of the answer.
A low perplexity has more than one cause. The text may be fluent, or the model may have seen it during training. This is the idea behind detecting benchmark contamination: if a model finds a test set suspiciously easy to predict, it may have read it before.
The output depends on the wording and on chance. Change the prompt slightly, or sample again, and the answer can change. One example is never a measurement. Notebook 6 lets you test this in exercise 4.
Small models can prefer one label no matter what the input is. Section 7 of notebook 6 shows this and one way to correct for it.
A model's explanation of itself is generated text. When you ask a chat model why it answered as it did, the reply is produced the same way as any other reply. It may or may not describe what really happened inside.

How researchers go further#

Working out what the numbers inside a network mean is its own research field, usually called interpretability. A rough map of the approaches:

Approach	The question it asks	What you need
Behavioural testing	How does the output change when I change the input in a controlled way?	Only text in and out. Works on closed models too.
Looking at probabilities	How likely does the model find this text, token by token?	The logits. This is perplexity and surprisal from notebook 6.
Probing	Is some piece of information, such as the language or the grammar role, readable from the vectors inside?	The intermediate vectors and a small classifier trained on them.
Intervening	What happens to the output when I switch off or overwrite one part of the model?	Full access to the model while it runs.
Mechanistic interpretability	Can I describe the actual computation, part by part, in terms a person understands?	Full access, a lot of patience, and it is unfinished research.

The first two rows are what you do in this course. The exercises at the end of notebook 5 are small experiments of the fourth kind: you remove the causal mask or the position embeddings and watch what breaks.

Why it matters#

Trust. If a model is used for something that matters, “it usually works” is not enough. You want to know when it fails and why.
Fair testing. A benchmark score means little if the test questions were in the training data. Perplexity gives a first hint.
Fairness. A tokenizer that needs several times as many tokens for one language makes that language slower and more expensive, and less of it fits into the context. You can measure this in a few lines.
Fixing things. You can only repair what you can locate.

Experiments to try#

All of them use material from this lab.

Let the model cheat. Remove the causal mask in notebook 5 and train for 300 steps. The loss drops a lot. Then generate text. What went wrong? (Notebook 5, exercise 1.)
Take away the order. Remove the position embeddings. How much worse does the model get, and why can attention not tell positions apart by itself? (Notebook 5, exercise 2.)
Find memorised text. Look for a sentence of 15 or more words with a GPT-2 perplexity below 5. (Notebook 6, exercise 2.)
Break a classifier with wording. Change only the template of the zero-shot classifier and watch the accuracy. (Notebook 6, exercise 4.)
Find the Elbe. In notebook 6, ask GPT-2 for the ten most likely tokens after The Elbe flows through (section 3). Which of them are true? How much probability goes to wrong ones?
Ask in three languages. In exercise 6, do the German and Hindi prompts get answers in the same language? Count the tokens of each prompt first.
Compare two brains. Train the tiny GPT in notebook 5 a second time with another number in torch.manual_seed in the first cell. Does the loss end up in the same place? Do the attention maps look the same?