Tokens and a Bigram Language Model
Tokenization, a small BPE, a counting language model, cross-entropy and perplexity.
- Text is just a long list of characters
- Tokenization: from text to integers
- A language model gives text a probability
- How good is a language model?
- The same model, learned with gradient descent
- Controlling the output with the temperature
- What is still missing?
Self-Attention and a Tiny GPT
Self-attention from scratch, a Transformer block, training a small GPT.
- What does a language model train on?
- Letting tokens talk to each other: averaging the past
- Scaled dot-product self-attention
- From one head to a Transformer block
- Training, with the same loop as for MNIST
- Looking inside: what do the heads attend to?
Pretrained LLMs as Tools for Text Analysis
Tokenizers across languages, decoding, perplexity, zero-shot labels, embeddings.
- A real tokenizer
- Are tokenizers fair across languages?
- Next-token prediction with a real model
- Decoding strategies
- Perplexity as a tool for text analysis
- Where exactly is the model surprised?
- Zero-shot classification by comparing likelihoods
- Sentence embeddings and semantic search
- (Optional) An instruction-tuned chat model
How to run them#
nothing to install
Google Colab
Press Open in Colab on a notebook. For notebooks 5 and 6, switch to a T4 GPU under Runtime > Change runtime type. Notebook 4 is fine on a CPU. You need a Google account.
on the cluster
JupyterHub
After the setup, open jupyterhub.hpc.tu-dresden.de, start a server and choose the kernel LLM Lab (venv-llm). The notebooks are in ~/behind-llms-lab/notebooks.
offline
Your laptop
Clone the repository and install torch transformers matplotlib jupyter. Without a GPU the notebooks choose smaller settings by themselves.
How they fit together#
Each notebook starts where the one before stopped, and each ends with exercises.
- Tokens and a bigram language modelText becomes numbers. A model that only counts which character follows which reaches a loss of about 2.5. You meet cross-entropy, perplexity and the temperature.
- Self-attention and a tiny GPTEvery position may look at all earlier ones. Same data, same loss, same training loop, and much better text. Then you look at what the attention heads learned.
- Pretrained LLMs for text analysisReal models from Hugging Face, used as measuring tools: tokenizers across languages, perplexity, surprisal, zero-shot labels and sentence embeddings.
They follow the warm-up notebooks 0 to 3b from the first crash course (regression, arbitrary functions, MNIST, autograd and neurons). If those ideas are new to you, read the crash course page first.
The same model runs on the cluster. The tiny GPT of notebook 5 is exercise 4 of the HPC lab, and the chat model at the end of notebook 6 is exercise 6. On an A100 or H100 you can train a model ten times bigger in the same time.