Pretrained LLMs as Tools for Text Analysis · Behind the Secrets of LLMs

Open in Colab View on GitHub Download .ipynb

No outputs here. This notebook is saved without its outputs, because the models are downloaded when you run it. Open it in Colab to see the results.

Behind the Secrets of LLMs, crash course part 2

In notebooks 4 and 5 we built language models ourselves. Now we load real pretrained models from Hugging Face and use the things we already know (tokens, logits, probabilities, cross-entropy) to analyse text.

section	what we do	model
1 and 2	real BPE tokenizers, and how fair they are across languages	GPT-2, XLM-R and Qwen tokenizers
3	next-token probabilities	GPT-2 (124M)
4	decoding: greedy, temperature, top-k, top-p	GPT-2
5 and 6	perplexity and surprisal per token	GPT-2
7	zero-shot classification by comparing likelihoods	GPT-2
8	sentence embeddings and semantic search	MiniLM (22M)
9	an instruction-tuned chat model and its chat template	Qwen2.5-0.5B-Instruct (optional)

Time: about 60 minutes. Everything runs on the free Colab CPU, and a T4 GPU makes section 9 faster. The models are downloaded the first time you use them (about 0.5 GB for sections 1 to 8, and 1 GB more for section 9).

In [1]python

# Colab already has these. Locally:  pip install torch transformers matplotlib
import math
import torch
import torch.nn.functional as F
import matplotlib.pyplot as plt
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModel

device = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(0)

LM_NAME   = "gpt2"                                       # 124 M parameters, trained on English web text
TOKENIZER_NAMES = ["gpt2", "xlm-roberta-base", "Qwen/Qwen2.5-0.5B"]
EMB_NAME  = "sentence-transformers/all-MiniLM-L6-v2"     # sentence-embedding model
CHAT_NAME = "Qwen/Qwen2.5-0.5B-Instruct"                 # small multilingual chat model
print("device:", device)

1. A real tokenizer#

GPT-2 uses byte-level BPE with 50,257 tokens. It is the same algorithm you wrote in notebook 4, just trained on much more text. The symbol Ġ marks a token that starts with a space.

In [2]python

tok = AutoTokenizer.from_pretrained(LM_NAME)

s = "Tokenization splits unbelievably long words into pieces."
ids = tok(s)["input_ids"]
print("vocabulary size:", len(tok))
print("ids   :", ids)
print("tokens:", tok.convert_ids_to_tokens(ids))
print("decode:", tok.decode(ids))

2. Are tokenizers fair across languages?#

Fertility is the average number of tokens per word. It matters because you pay per token, in compute, in money and in context length. A language with fertility 4 fits only a quarter as much text into the same context window as a language with fertility 1.

Below, one sentence is translated into six languages and measured with three tokenizers.

In [3]python

sentences = {
    "English": "The weather is nice today and we are going for a walk in the park.",
    "German":  "Das Wetter ist heute schön und wir gehen im Park spazieren.",
    "Spanish": "Hoy hace buen tiempo y vamos a dar un paseo por el parque.",
    "Russian": "Сегодня хорошая погода, и мы идём гулять в парк.",
    "Turkish": "Bugün hava güzel ve parkta yürüyüşe çıkıyoruz.",
    "Hindi":   "आज मौसम अच्छा है और हम पार्क में टहलने जा रहे हैं।",
}

fertility = {}
for name in TOKENIZER_NAMES:
    t = AutoTokenizer.from_pretrained(name)
    fertility[name] = {lang: len(t(s, add_special_tokens=False)["input_ids"]) / len(s.split())
                       for lang, s in sentences.items()}

langs = list(sentences)
w = 0.8 / len(TOKENIZER_NAMES)
plt.figure(figsize=(10, 4))
for j, name in enumerate(TOKENIZER_NAMES):
    plt.bar([i + j * w for i in range(len(langs))], [fertility[name][l] for l in langs], w, label=name)
plt.xticks([i + w for i in range(len(langs))], langs)
plt.ylabel("tokens per word"); plt.title("Tokenizer fertility (lower = cheaper)"); plt.legend(); plt.show()

print("Hindi with GPT-2:", AutoTokenizer.from_pretrained("gpt2").tokenize(sentences["Hindi"])[:12], "...")

GPT-2's BPE was learned almost only from English, so Hindi falls apart into byte pieces. Multilingual tokenizers like XLM-R and Qwen are much fairer, though rarely perfectly fair.

3. Next-token prediction with a real model#

This works exactly like our TinyGPT. The model returns logits for every position, and the last position gives the distribution of the next token:

P(x_{t+1} \mid x_{\le t}) = \text{softmax}(\mathbf{z}_t).

In [4]python

model = AutoModelForCausalLM.from_pretrained(LM_NAME).to(device).eval()
print(f"{sum(p.numel() for p in model.parameters()) / 1e6:.0f} M parameters")

def next_token_probs(prompt, k=10):
    enc = tok(prompt, return_tensors="pt").to(device)
    with torch.no_grad():
        logits = model(**enc).logits              # (1, T, vocab)
    probs = F.softmax(logits[0, -1], dim=-1)
    top = torch.topk(probs, k)
    return [tok.decode(int(i)) for i in top.indices], top.values.cpu()

words, p = next_token_probs("The capital of Germany is")
plt.barh(words[::-1], p.numpy()[::-1])
plt.xlabel("probability"); plt.title("P(next token | 'The capital of Germany is')"); plt.show()

4. Decoding strategies#

The model gives us a distribution, and we decide how to pick from it.

greedy: always take the most likely token. Deterministic, and often repetitive.
temperature $T$ : sample from $\text{softmax}(\mathbf{z}/T)$ .
top-k: keep only the $k$ most likely tokens and renormalise.
top-p (nucleus): keep the smallest set $S$ of most likely tokens with $\sum_{w \in S} P(w) \ge p$ , and renormalise.

In [5]python

def sample_next(logits, temperature=1.0, top_k=None, top_p=None):
    if temperature == 0:                                    # greedy
        return logits.argmax(dim=-1, keepdim=True)
    logits = logits / temperature
    if top_k is not None:
        kth = torch.topk(logits, top_k).values[..., -1, None]
        logits = logits.masked_fill(logits < kth, float("-inf"))
    if top_p is not None:
        sorted_logits, sorted_idx = torch.sort(logits, descending=True)
        probs = F.softmax(sorted_logits, dim=-1)
        # drop a token if the tokens BEFORE it already cover probability mass top_p
        drop = probs.cumsum(dim=-1) - probs > top_p
        sorted_logits = sorted_logits.masked_fill(drop, float("-inf"))
        logits = torch.full_like(logits, float("-inf")).scatter(-1, sorted_idx, sorted_logits)
    return torch.multinomial(F.softmax(logits, dim=-1), num_samples=1)

def generate(prompt, n_tokens=30, **kw):
    ids = tok(prompt, return_tensors="pt")["input_ids"].to(device)
    for _ in range(n_tokens):
        with torch.no_grad():
            logits = model(ids).logits[:, -1]               # (real systems cache keys/values here)
        ids = torch.cat([ids, sample_next(logits, **kw)], dim=1)
    return tok.decode(ids[0])

prompt = "Dresden is a city in Germany. It is famous for"
torch.manual_seed(0)
for kw in [dict(temperature=0), dict(temperature=0.7), dict(temperature=1.5), dict(top_k=5), dict(top_p=0.9)]:
    print(f"--- {kw}\n{generate(prompt, **kw)}\n")

In [6]python

# The library does the same (plus caching) in one call:
enc = tok(prompt, return_tensors="pt").to(device)
out = model.generate(**enc, max_new_tokens=30, do_sample=True, temperature=0.7, top_p=0.9,
                     pad_token_id=tok.eos_token_id)
print(tok.decode(out[0]))

5. Perplexity as a tool for text analysis#

From notebook 4, the perplexity is $\exp(\text{mean cross-entropy})$ . With a pretrained LM this gives us a fluency meter for any text:

\text{PPL}(x) = \exp\Big(-\frac{1}{n-1}\sum_{t=2}^{n}\log P(x_t \mid x_{<t})\Big)

If we pass labels=input_ids, Hugging Face computes exactly this cross-entropy. It shifts the labels by one position internally.

In [7]python

def perplexity(text):
    enc = tok(text, return_tensors="pt").to(device)
    with torch.no_grad():
        loss = model(**enc, labels=enc["input_ids"]).loss
    return math.exp(loss.item())

texts = [
    "The cat sat on the mat and looked out of the window.",
    "Window the of out looked and mat the on sat cat the.",
    "Colorless green ideas sleep furiously.",
    "To be, or not to be, that is the question.",
    "Sein oder Nichtsein, das ist hier die Frage.",
    "xq zvb plmr tkw oiu.",
]
for t in texts:
    print(f"PPL {perplexity(t):10.1f}   {t}")

A few things to notice:

Word order matters a lot. The same words in a scrambled order get a much higher perplexity.
Grammatical nonsense lands somewhere in between.
Famous text gets a much lower perplexity than ordinary sentences, because the model has partly memorised it. This is the basic idea behind detecting benchmark contamination: if a model finds a test set suspiciously easy to predict, it may have seen it during training.
An English-only model finds German surprising, even when the German is perfectly fluent.

6. Where exactly is the model surprised?#

The surprisal of a token in bits is $s_t = -\log_2 P(x_t \mid x_{<t})$ . The average surprisal over a text is $\log_2$ of its perplexity.

In [8]python

def token_surprisal(text):
    ids = tok(text, return_tensors="pt")["input_ids"].to(device)
    with torch.no_grad():
        logp = F.log_softmax(model(ids).logits[0, :-1], dim=-1)    # predictions for tokens 2..n
    s = -logp.gather(1, ids[0, 1:, None]).squeeze(1) / math.log(2)
    return tok.convert_ids_to_tokens(ids[0, 1:]), s.cpu()

fig, axes = plt.subplots(1, 2, figsize=(13, 3.5), sharey=True)
for ax, t in zip(axes, ["I take my coffee with milk and sugar.", "I take my coffee with milk and concrete."]):
    toks, s = token_surprisal(t)
    ax.bar(range(len(toks)), s)
    ax.set_xticks(range(len(toks))); ax.set_xticklabels([x.replace("Ġ", " ") for x in toks], rotation=45)
    ax.set_title(t); ax.set_ylabel("surprisal (bits)")
plt.tight_layout(); plt.show()

7. Zero-shot classification by comparing likelihoods#

No training at all. We write the task as text and check which continuation the model finds more likely:

\hat{y} = \arg\max_{y \in \{\text{positive}, \text{negative}\}} \log P(y \mid \text{prompt}).

Small base models often prefer one label no matter what the input is. A well-known fix, called contextual calibration, subtracts the score each label gets for an empty review: $\;\hat{y} = \arg\max_y \big[\log P(y \mid \text{prompt}) - \log P(y \mid \text{empty prompt})\big]$ .

In [9]python

reviews = [
    ("An absolute delight from start to finish, I loved every minute.", "positive"),
    ("The acting was wooden and the plot made no sense.", "negative"),
    ("A beautiful, moving story with a fantastic soundtrack.", "positive"),
    ("Two hours of my life I will never get back.", "negative"),
    ("Funny, warm and surprisingly clever.", "positive"),
    ("Boring, predictable and far too long.", "negative"),
    ("I would happily watch it again tomorrow.", "positive"),
    ("The worst film I have seen this year.", "negative"),
]
template = 'Review: "{}"\nSentiment (positive or negative):'

def logprob_of_continuation(prompt, continuation):
    p_ids = tok(prompt, return_tensors="pt")["input_ids"]
    c_ids = tok(continuation, return_tensors="pt")["input_ids"]
    ids = torch.cat([p_ids, c_ids], dim=1).to(device)
    with torch.no_grad():
        logp = F.log_softmax(model(ids).logits[0, :-1], dim=-1)
    lp = logp.gather(1, ids[0, 1:, None]).squeeze(1)
    return lp[-c_ids.shape[1]:].sum().item()               # only the continuation's tokens

labels = ["positive", "negative"]
prior = {lab: logprob_of_continuation(template.format(""), " " + lab) for lab in labels}   # the bias

n_raw = n_cal = 0
print(f"{'raw':10s}{'calibrated':12s}{'true':10s}review")
for text, gold in reviews:
    s = {lab: logprob_of_continuation(template.format(text), " " + lab) for lab in labels}
    raw = max(labels, key=lambda lab: s[lab])
    cal = max(labels, key=lambda lab: s[lab] - prior[lab])
    n_raw += raw == gold; n_cal += cal == gold
    print(f"{raw:10s}{cal:12s}{gold:10s}{text}")
print(f"\naccuracy  raw {n_raw}/{len(reviews)}   calibrated {n_cal}/{len(reviews)}")

8. Sentence embeddings and semantic search#

An encoder model turns a sentence into one vector. Here we take the mean of the token vectors. Sentences with a similar meaning point in a similar direction, which we measure with the cosine similarity

\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u}\cdot\mathbf{v}}{\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert}.

This is the retrieval part of retrieval-augmented generation (RAG).

In [10]python

etok = AutoTokenizer.from_pretrained(EMB_NAME)
emodel = AutoModel.from_pretrained(EMB_NAME).to(device).eval()

def embed(texts):
    enc = etok(texts, padding=True, truncation=True, return_tensors="pt").to(device)
    with torch.no_grad():
        h = emodel(**enc).last_hidden_state                 # (B, T, d)
    mask = enc["attention_mask"].unsqueeze(-1).float()      # ignore padding tokens
    e = (h * mask).sum(dim=1) / mask.sum(dim=1)             # mean pooling
    return F.normalize(e, dim=-1)                           # unit length -> dot product = cosine

docs = [
    "Submit your training script to the cluster with sbatch.",
    "Request a GPU by adding --gres=gpu:1 to your job.",
    "Use scancel to stop a job that is stuck.",
    "Knead the dough for ten minutes and let it rise.",
    "Bake the bread at 220 degrees for thirty minutes.",
    "The striker scored twice in the second half.",
    "The match ended in a draw after extra time.",
]
E = embed(docs)
S = (E @ E.T).cpu()
plt.figure(figsize=(6, 5)); plt.imshow(S, cmap="viridis", vmin=0, vmax=1); plt.colorbar()
plt.xticks(range(len(docs)), [f"d{i}" for i in range(len(docs))]); plt.yticks(range(len(docs)), [d[:40] for d in docs])
plt.title("cosine similarity"); plt.show()

query = "How do I get a graphics card for my experiment?"
scores = (embed([query]) @ E.T)[0].cpu()
print("query:", query)
for i in scores.argsort(descending=True)[:3].tolist():
    print(f"  {scores[i]:.3f}  {docs[i]}")

9. (Optional) An instruction-tuned chat model#

Chat models are base models that were fine-tuned on conversations. They expect a chat template, which uses special tokens to mark who is speaking. Here is the raw text the model actually gets:

In [11]python

ctok = AutoTokenizer.from_pretrained(CHAT_NAME)
cmodel = AutoModelForCausalLM.from_pretrained(CHAT_NAME)
cmodel = cmodel.to(device=device, dtype=torch.bfloat16 if device == "cuda" else torch.float32).eval()

messages = [
    {"role": "system", "content": "You are a concise, helpful assistant."},
    {"role": "user", "content": "Explain self-attention to a 10-year-old in three sentences."},
]
chat_text = ctok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(chat_text)

enc = ctok(chat_text, return_tensors="pt").to(device)
out = cmodel.generate(**enc, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.9)
print(ctok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Exercises

Your language. Add a sentence in a language you speak to sentences. Which tokenizer treats it most fairly? Roughly how much more would a 1,000-word document cost than in English?
Find memorised text. Find a sentence of 15 or more words with a GPT-2 perplexity below 5. Hint: try famous speeches, legal boilerplate or the opening of a constitution. Why does this matter when LLMs are tested on public benchmarks?
Decoding. Generate 5 samples with temperature=1.5 and 5 with top_p=0.5. Which settings would you use for code, for poems and for factual answers?
Prompt sensitivity. Change only the wording of template in section 7, for example to "The movie was". How much does the accuracy change? Try the same with the chat model from section 9.
Search across languages. Add German or Hindi sentences to docs. Does all-MiniLM-L6-v2 still find them? Try again with sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2.

Running it on the cluster

hpc_lab/06_hf_generate runs a chat model over a whole file of prompts as a GPU batch job (sbatch generate.sbatch) and prints the tokens per second. Compare an A100 with this Colab session.

Open in Colab View on GitHub Download .ipynb