Session 01 · the lab kit
Behind the Secrets of LLMs
A language model is often called a black box. In this lab you open it. You log in to a supercomputer, train a tiny GPT from scratch, and take real language models apart, one small script at a time.
Course at TU Dresden (ScaDS.AI). Everything on this page is in one GitHub repository.
text in
next token
Text goes in, a probability for every possible next token comes out. The numbers here are only an illustration.
What this is
Two parts, one story#
First you learn to drive the machine. Then you use it to build the thing everybody talks about, and you see that there is no magic inside.
Part 1 · on the cluster
The HPC lab
You log in to the ZIH clusters, send your first Slurm jobs, get a GPU, train a tiny GPT and run a real language model. Every exercise is one folder with a small Python script and the job script that runs it.
- your first batch job, and how to cancel one
- one GPU, and how much faster it is than a CPU
- a learning-rate sweep with a job array
- a pretrained LLM that answers a file of prompts
Part 2 · in a notebook
The crash course
Three notebooks that build a language model from scratch and then look inside real ones. They run in Google Colab and need no big GPU.
- tokens, a small BPE and a model that only counts
- self-attention, a Transformer block and a tiny GPT
- perplexity, surprisal, zero-shot labels and embeddings
- exercises at the end of every notebook
The running example
Meet Marie and Martin#
Two students at TU Dresden want a language model on a university GPU to answer one question: What is the capital of Saxony, and what is it famous for? They go through all seven exercises, and one of them gets it wrong first every time. Each scene shows what does not work, why, and what works.
One sentence travels with them through the whole site: The Elbe flows through …
Follow the storysometimes too careful
Marie
Sees a job waiting in the queue, decides it is broken, and submits it three more times.
sometimes too fast
Martin
Deletes the line that asks for a GPU because it looks like a comment, and wonders where the GPU went.
The first session
Follow the path#
The pages are in the order you need them. If you are new to all of this, go from 1 to 6. If you only need a command, the cheat sheet has it.
Meet the machines
What a cluster is, which five ZIH has, and which ones we use.
read firstGet set up
Log in, get the lab kit, and build your Python environment once.
do once · the install takes about 10 minDo the HPC lab
Seven exercises with all the code, from a hello job to a real LLM.
do · the main part of the sessionTake the crash course
From fitting a line to a Transformer, with things to play with.
read and playRun the notebooks
Build a bigram model and a tiny GPT, then use pretrained models.
do · 45 to 60 min eachOpen the black box
What you can see inside a model, and what stays hard to explain.
read last
Quick start
Three ways in#
Pick where you want to work. The cluster needs the university network or the VPN, and your ZIH login.
Log in to a GPU cluster and get the lab kit. Then continue with Setup.
ssh <zih-login>@login1.alpha.hpc.tu-dresden.de
git clone https://github.com/himanshubeniwal/behind-llms-lab.git
cd behind-llms-lab/hpc_labOpen a notebook with one click on the Notebooks page. Or, inside Colab: File > Open notebook > GitHub, paste himanshubeniwal/behind-llms-lab and pick a notebook.
For notebooks 5 and 6, switch to a T4 GPU under Runtime > Change runtime type. Notebook 4 is fine on a CPU.
The notebooks and the scripts also run on a normal laptop, only slower. You need a recent Python 3 (the cluster uses 3.12).
git clone https://github.com/himanshubeniwal/behind-llms-lab.git
cd behind-llms-lab
python3 -m venv .venv && source .venv/bin/activate
pip install torch numpy matplotlib tqdm transformers accelerate jupyter
jupyter lab notebooks/The repository
What is in the kit#
Clone it, download it as a .zip, or read every file right here on the site.
hpc_lab/ 00_setup/ workspace, modules and a venv in one script 01_hello/ your first batch job 02_cancel/ squeue, scontrol, scancel, sacct 03_gpu/ ask for a GPU and measure it 04_tinygpt/ train a tiny GPT as a batch job 05_sweep/ a learning-rate sweep with a job array 06_hf_generate/ run a pretrained LLM on a GPU node notebooks/ 4 - Tokens and Bigram LM.ipynb 5 - Attention and TinyGPT.ipynb 6 - Pretrained LLMs for Text Analysis.ipynb website/ the source of this site (python website/build.py)
When you are stuck
Help is close#
Something went wrong
No log file, no GPU, a full home folder, a job that waits forever. The usual causes and the fix for each.
Troubleshooting →ZIH HPC Compendium
The official documentation of the clusters: hardware, Slurm, workspaces, software modules, JupyterHub.
compendium.hpc.tu-dresden.de →HPC support
For questions about the cluster itself that the documentation does not answer.
hpc-support@tu-dresden.de →