Marie and Martin · Behind the Secrets of LLMs

The cast#

sometimes too careful

Marie

New to the terminal. Reads the instructions twice before typing anything. When something takes long, Marie assumes it is broken.

sometimes too fast

Martin

Quick at the keyboard. Reads the instructions afterwards, if at all. When something looks unnecessary, Martin deletes it.

If the names look familiar: Marie and Martin are the two example users of the ZIH documentation. They are the Alice and Bob of the Dresden clusters, and you will meet them again in the Compendium, in lines like marie@login.alpha$.

The plan#

It is the first lab session. On the table are two laptops and one question:

What is the capital of Saxony, and what is it famous for?

They want a language model to answer it. Not a chatbot on some website, but a model they start themselves, on a GPU that belongs to their own university. The question is the second prompt in prompts.txt of the last exercise, and there are six steps before it.

A second thread runs through the story: one unfinished sentence, The Elbe flows through … It shows what happens inside the model, and it comes back at the end.

does not work what one of them tried first, and why it failed works what to do instead

Setting up#

scene 000_setupthe Setup page

Both are logged in on Alpha. They need Python with PyTorch.

Martindoes not work

Martin knows how to install things and starts right away.

on the login nodeterminal

$ pip install --user torch transformers
  (it downloads for a long time, into the home folder)

Why not: the login node is shared by everybody, and an installation is real work. Worse, everything lands in $HOME, which has a quota of 50 GiB. Add the caches of pip and a few models from Hugging Face and it fills up. With a full home folder you cannot even submit jobs any more.

Marieworks

Marie asks Slurm for a small machine and lets the setup script do the rest.

on the login nodeterminal

$ srun --account=p_lv_llmsecrets --nodes=1 --ntasks=1 --cpus-per-task=4 --mem-per-cpu=2G --gres=gpu:1 --time=00:45:00 --pty bash -l
$ cd ~/behind-llms-lab/hpc_lab
$ source 00_setup/setup_env.sh
workspace : /data/horse/ws/<zih-login>-llm-lab
  (a few minutes of installing)
environment ready
$ exit

The environment, the caches and the data now live in a workspace, not in the home folder.

Marie is not safe from mistakes either. The first attempt looked like this:

Mariedoes not work

terminal

$ bash 00_setup/setup_env.sh
  (the script runs and finishes)
$ echo $WS_DIR
  (an empty line)

Why not: bash starts a new shell, runs the script in it and closes it again. Everything the script switched on is gone with that shell.

Marieworks

terminal

$ source 00_setup/setup_env.sh
$ echo $WS_DIR
/data/horse/ws/<zih-login>-llm-lab

source runs the script inside the shell you are typing in, so the environment stays.

RememberHeavy work goes into a job, big files go into the workspace, and the setup script is sourced, not run.

The first job#

scene 101_helloexercise 1

The first exercise is one command. Martin does not need to read more than that.

Martindoes not work

terminal

$ cd ~/behind-llms-lab/hpc_lab/01_hello
$ sbatch hello.sbatch
Submitted batch job 1234567
$ squeue --me
  (no job. It is already gone.)
$ cat logs/hello-hpc-1234567.out
cat: logs/hello-hpc-1234567.out: No such file or directory

Why not: the job script sends its output to the folder logs. The folder did not exist, so the job ended at once and left no log behind.

Marieworks

terminal

$ mkdir -p logs && sbatch hello.sbatch
Submitted batch job 1234568
$ cat logs/hello-hpc-1234568.out
Hello from        : (the name of a compute node)
CPUs I may use    : 4   <- what Slurm allocated to me
  ...
pi ~ 3.14…   (20,000,000 samples on 4 CPU(s) in … s)

The job ran on a compute node, with exactly the 4 cores the script asked for.

RememberNo folder for the log, no log. Type mkdir -p logs before the first sbatch in a new folder.

Is it stuck?#

scene 202_cancelexercise 2

The second job is slow on purpose. This time it is Marie who gets nervous.

Mariedoes not work

terminal

$ sbatch sleepy.sbatch
Submitted batch job 1234570
$ squeue --me
  (the ST column says PD)
$ squeue --me
  (still PD. Something must be broken.)
$ sbatch sleepy.sbatch
Submitted batch job 1234571
$ sbatch sleepy.sbatch
Submitted batch job 1234572

Why not: PD means pending. The job waits for its turn, like everybody in the queue of the Mensa at noon. Nothing is broken. Now three copies are waiting instead of one.

Marieworks

terminal

$ squeue --me --start  # when will it probably start?
$ whypending 1234570  # a ZIH helper: why is it waiting?
$ scancel 1234571 1234572  # remove the two copies

Ask Slurm instead of guessing. And a batch job does not need you: close the laptop, go for lunch, it runs anyway.

Then Martin wants to stop the job that is running.

Martindoes not work

terminal

$ tail -f logs/sleepy-1234570.out
[sleepy] step    3  …  still working...
  (Martin presses Ctrl+C)
$ squeue --me
  (the job is still there, ST = R)

Why not: Ctrl+C stopped tail, the program that shows the log. The job runs on another machine and never noticed.

Martinworks

terminal

$ scancel 1234570
$ tail -n 2 logs/sleepy-1234570.out
[sleepy] received signal 15 (SIGTERM) -> cleaning up
[sleepy] saved checkpoint-1234570.json at step 4 (cancelled)

Slurm asks the program to stop before it kills it. This program uses the moment to save its work.

RememberPD is a queue, not an error. Ask whypending why, and stop jobs with scancel.

Where is my GPU?#

scene 303_gpuexercise 3

Martin tidies up the job script first. The lines that start with # are comments, after all, and comments can go.

Martindoes not work

Martin deletes the line #SBATCH --gres=gpu:1 and submits.

in the logterminal

CUDA available        : False
No GPU found. Did you ask for one with --gres=gpu:1 ?

Why not: lines that start with #SBATCH look like comments to the shell, but Slurm reads them as orders. Without that line the job runs on a machine full of GPUs and is not allowed to touch one.

Marieworks

Marie changes nothing and submits.

in the logterminal

CUDA available        : True
GPU 0                 : (name and memory of the GPU)
  (then three lines with the speed: CPU, GPU, and GPU with 16-bit numbers)

Now there is something to compare. How many times faster is the GPU?

Logging in to Barnard would not have helped Martin either. It is the biggest cluster, with 720 nodes, and it has not a single GPU. Every cluster has its own Slurm, so a job sent from Barnard stays on Barnard.

Remember#SBATCH lines are orders, not comments. No --gres=gpu:1, no GPU.

A tiny GPT learns Shakespeare#

scene 404_tinygptexercise 4

Two of the ZIH machines are called Romeo and Julia. That fits, because the tiny GPT learns from Shakespeare, and the first thing it is asked to continue is ROMEO:. The normal run works for both of them. Then Martin wants more.

Martindoes not work

More attention heads must be better.

terminal

$ sbatch train.sbatch --n-head 6
Submitted batch job 1234580
$ sacct -j 1234580 --format=JobID,State,Elapsed
  (State: FAILED, after a few seconds)
$ cat logs/tinygpt-1234580.err
  ...
    assert n_embd % n_head == 0
AssertionError

Why not: the width of the model, 128 numbers per character, is split evenly over the heads. 128 cannot be divided by 6.

Marieworks

Marie reads the last lines of the .err file and makes the model wider too. 384 divided by 6 is 64.

terminal

$ sbatch train.sbatch --n-head 6 --n-embd 384
$ tail -f logs/tinygpt-1234581.out
step   250 | train … | val … | …k tok/s | …s
step   500 | train … | val … | …k tok/s | …s

The two numbers after train and val go down. The model is learning.

Marie tries one more thing on purpose: a job with far too little time, sbatch --time=00:02:00 train.sbatch. If the training is not finished by then, this line appears in the log about a minute before the time is over:

in the logterminal

Got SIGTERM, saving a checkpoint and stopping

Running out of time is not a disaster when the script is prepared for it. The job script asks Slurm for a warning 60 seconds early, and the Python script uses it to save.

RememberWhen a job fails, the reason is almost always in its .err file. Read it before you ask anyone.

Four learning rates at once#

scene 505_sweepexercise 5

Which learning rate is best? Martin's plan is to copy the job script four times and change one number in each copy. Marie prefers the job array from the exercise, and opens a fresh terminal for it, to keep things tidy.

Mariedoes not work

in the new terminalterminal

$ cd ~/behind-llms-lab/hpc_lab/05_sweep
$ ./submit_sweep.sh
./submit_sweep.sh: line 7: WS_DIR: run source ../00_setup/setup_env.sh first

Why not: a new terminal is a new shell, and it knows nothing about the environment. $WS_DIR is empty, so the script stops before it sends results to nowhere.

Marieworks

terminal

$ source ../00_setup/setup_env.sh
$ mkdir -p logs && ./submit_sweep.sh
sweep (array) job : 1234590
collect job       : 1234591 (waits for 1234590, squeue shows it as Dependency)
$ squeue --me
  (four tasks, 1234590_0 to 1234590_3, and one job that waits for them)

One script and one line, --array=0-3. Four GPUs train at the same time, and a fifth job prints the table when they are done.

Martin's four copies would have worked too. But with four files there are four places for a typo, and with forty learning rates there are forty.

RememberEvery new terminal needs source …/setup_env.sh once. And when you copy a job script to change one number, you want a job array.

The question about Dresden#

scene 606_hf_generateexercise 6

The last step. A real language model, half a billion parameters, and the question they started with. Martin can already see the answer and skips a line.

Martindoes not work

terminal

$ mkdir -p logs && sbatch generate.sbatch
Submitted batch job 1234600
$ sacct -j 1234600 --format=JobID,State,Elapsed
  (State: FAILED, after a few seconds)

Why not: the job is offline on purpose (HF_HUB_OFFLINE=1), and the model was never downloaded. The .err file says that the model files cannot be found.

Marieworks

on the login nodeterminal

$ source ../00_setup/setup_env.sh
$ python generate.py --download-only
HF_HOME = /data/horse/ws/<zih-login>-llm-lab/.cache/huggingface
downloaded Qwen/Qwen2.5-0.5B-Instruct to (a folder in the workspace). Now submit the GPU job.
$ mkdir -p logs && sbatch generate.sbatch

Download once on the login node, into the workspace. Compute in the job.

When the job is done, the log holds what they came for:

logs/hf-generate-1234601.outterminal

>>> What is the capital of Saxony, and what is it famous for?
  (a few fluent sentences, written by the model)

An answer about Dresden, from a language model that runs on a GPU of TU Dresden. Marie reads it twice, of course. Is it right? A model of this size writes smooth sentences. It may mention the Frauenkirche, the Zwinger or the Semperoper, and it can say things that are simply wrong in the same smooth tone.

Check it yourself. The same file has a question with exactly one right answer: a train leaves Dresden at 9:15 and arrives in Berlin at 11:05. That is 1 hour and 50 minutes. Does the model agree? Run the job twice. Does it agree with itself?

RememberDownload on the login node, compute in the job. And do not trust an answer because it sounds good.

The same story, inside the model#

Why can a model be fluent and wrong at the same time? Follow the second thread: The Elbe flows through …

The sentence is cut into tokensThe, Elbe, flows, through. A rare word like Elbe may be cut into several pieces. Try the tokenizer.
Every token looks back at the ones before itTo guess what comes after through, the model has to find Elbe again. That is attention. See the mask.
The model gives a probability to every possible next tokenDresden, Hamburg and Magdeburg are all correct, the Elbe flows through each of them. Prague and Berlin sound just as good and are wrong.
One token is picked, with some randomnessUsually a likely one. Sometimes an unlikely one. If it is Prague, the model does not notice. It just continues the sentence. Play with the temperature.

The model never looks anything up. It continues text with what is likely. Likely is often true, and that is why these models are useful. Likely is not always true, and that is why looking inside matters.

What Marie and Martin learned#

Scene	What went wrong	The rule
0	Installing on the login node, into the home folder	Heavy work in a job, big files in the workspace.
0	`bash setup_env.sh`	`source` it, so the environment stays.
1	No `logs` folder, the job vanished	`mkdir -p logs` before `sbatch`.
2	Submitting again because the job was pending	`PD` is a queue. Ask `whypending`.
2	Ctrl+C to stop a job	`scancel` stops jobs.
3	Deleting the `--gres` line	`#SBATCH` lines are orders.
4	6 heads for a width of 128	Read the `.err` file.
5	A new terminal without the environment	`source …/setup_env.sh` in every new shell.
6	No download before an offline job	Download on the login node first.
6	Believing a fluent answer	Check it.

Now it is your turn. Start with the machines, or go straight to the setup.