Cheat sheet · Behind the Secrets of LLMs

Log in#

you need the university network or the VPNshell

ssh <zih-login>@login1.alpha.hpc.tu-dresden.de      # GPU cluster, 8 A100 per node
ssh <zih-login>@login1.capella.hpc.tu-dresden.de    # GPU cluster, 4 H100 per node
ssh <zih-login>@login1.barnard.hpc.tu-dresden.de    # CPU cluster

Every cluster has its own Slurm, so GPU jobs have to be sent from an Alpha or Capella login node. Our project (account) is p_lv_llmsecrets.

Slurm commands#

What you want	Command
see partitions and free nodes	`sinfo`, `sinfo -s`, `sinfo -o "%20P %5a %10l %6D %20G"`
send a batch job	`sbatch job.sbatch` Options on the command line win over `#SBATCH` lines.
a terminal on a compute node	`srun --account=p_lv_llmsecrets --nodes=1 --ntasks=1 --cpus-per-task=2 --mem-per-cpu=2G --gres=gpu:1 --time=00:30:00 --pty bash -l` On Alpha, for quick tests: `--partition=alpha-interactive` (1 GPU, 1 core, 20 GB)
the same on Barnard (CPU only)	leave out `--gres=gpu:1`
my jobs	`squeue --me` with the expected start time: `squeue --me --start`
why is my job waiting?	`whypending <jobid>` (a ZIH helper) or `scontrol show job <jobid>`
cancel	`scancel <jobid>` one array task: `scancel <jobid>_3` all of mine: `scancel -u $USER`
what did it use?	`sacct -j <jobid> --format=JobID,JobName,State,Elapsed,MaxRSS,ExitCode`
change the time limit	`scontrol update timelimit=00:20:00 jobid=<jobid>` Normally you can only make it shorter.
watch a log while the job runs	`tail -f logs/<name>-<jobid>.out` Ctrl+C stops `tail`, not the job.

Job states

Code	State	Meaning
`PD`	pending	Waiting in the queue. The REASON column says why.
`R`	running	It has its resources and is working.
`CG`	completing	Finished, Slurm is cleaning up.
`CD`	completed	Ended without an error.
`F`	failed	The program ended with an error. Read the `.err` file.
`CA`	cancelled	Stopped with `scancel`.
`TO`	timeout	It hit the time limit.
`OOM`	out of memory	It used more memory than it asked for.

A job script, line by line#

a GPU jobshell

#!/bin/bash
#SBATCH --job-name=tinygpt            # the name you see in squeue
#SBATCH --account=p_lv_llmsecrets      # the project that pays for the time
#SBATCH --nodes=1                     # one machine, ZIH wants this line in every job
#SBATCH --ntasks=1                    # one process
#SBATCH --cpus-per-task=6             # CPU cores that come with the GPU
#SBATCH --gres=gpu:1                  # this is the line that gets you a GPU
#SBATCH --mem=32G                     # memory for the whole job
#SBATCH --time=00:30:00               # hard time limit, hh:mm:ss
#SBATCH --signal=TERM@60              # send SIGTERM 60 s before the time limit
#SBATCH --output=logs/%x-%j.out       # %x is the job name, %j the job ID
#SBATCH --error=logs/%x-%j.err        # the logs folder has to exist before you submit

source ~/behind-llms-lab/hpc_lab/00_setup/setup_env.sh    # workspace, modules, venv

export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK            # pass all cores on to the program
srun python -u my_script.py                               # -u: print without buffering

Also useful	What it does
`--mem-per-cpu=2G`	Memory per core, instead of `--mem` for the whole job.
`--array=0-3`	Start the script 4 times. `$SLURM_ARRAY_TASK_ID` goes from 0 to 3. With `0-3%2` at most 2 run at the same time.
`--output=logs/%x-%A_%a.out`	For arrays: `%A` is the array job ID, `%a` the task number.
`--dependency=afterok:<jobid>`	Start only after another job has finished without errors.
`--begin=now+5minutes`	Do not start before this time.
`--parsable`	Make `sbatch` print only the job ID, to keep it in a shell variable.
`--export=ALL,NAME=value`	Hand an extra variable to the job.

Variables inside a job

Variable	Holds
`$SLURM_JOB_ID`	The job ID.
`$SLURM_CPUS_PER_TASK`	The number of cores you asked for.
`$SLURM_ARRAY_TASK_ID`, `$SLURM_ARRAY_JOB_ID`	The number of this task in an array, and the ID of the whole array.
`$SLURM_JOB_NODELIST`	The node or nodes the job runs on.
`$WS_DIR`	Your workspace. Set by `setup_env.sh`, not by Slurm.
`$HF_HOME`	Where Hugging Face keeps models. `setup_env.sh` points it into the workspace.

Workspaces#

What you want	Command
create or reuse	`ws_allocate -F horse llm-lab 30` prints `/data/horse/ws/<user>-llm-lab`
list	`ws_list`
extend	`ws_extend -F horse llm-lab 30` This sets the remaining time to 30 days. It does not add 30 days.
delete	`ws_release -F horse --name llm-lab`
e-mail reminder	`ws_allocate -F horse -r 7 -m <you>@tu-dresden.de llm-lab 30`

Keep data, models and caches out of $HOME. It is small and backed up, so it fills up fast.

The lab in ten lines#

after the setup, from behind-llms-lab/hpc_labshell

cd 01_hello  && mkdir -p logs && sbatch hello.sbatch
cd ../02_cancel && mkdir -p logs && sbatch sleepy.sbatch     # and then cancel it
cd ../03_gpu    && mkdir -p logs && sbatch gpu.sbatch
cd ../04_tinygpt && mkdir -p logs && sbatch train.sbatch
cd ../05_sweep  && mkdir -p logs && source ../00_setup/setup_env.sh && ./submit_sweep.sh
cd ../06_hf_generate && source ../00_setup/setup_env.sh && python generate.py --download-only
mkdir -p logs && sbatch generate.sbatch

When something goes wrong#

What you see	What it usually is
`This is a GPU cluster, only GPU jobs are allowed` and `QOSMinGRES`	The job asks for no GPU. On Alpha every job needs `--gres=gpu:1`, even one that only uses the CPU.
`Please always specify the node count`	Add `#SBATCH --nodes=1`. The job is still accepted, but ZIH wants the line.
`bad argument #2 to 'format' (no value)`	A small bug in the cluster's submit check. It comes with the two messages above and does no harm on its own; read the line before it.
The job ends at once and there is no log file.	The `logs/` folder is missing. Run `mkdir -p logs`.
`CUDA available: False`	You forgot `--gres=gpu:1`, or you sent the job from Barnard.
`Disk quota exceeded` in `$HOME`	The pip or Hugging Face caches are in your home folder. `source 00_setup/setup_env.sh` moves them to the workspace.
`module load` fails	Module versions change about twice a year. Run `module spider Python` and change `PY_MODULES` in `setup_env.sh`.
`OUT_OF_MEMORY` in `sacct`	Ask for more memory with `--mem`.
The job waits for a long time.	Ask for less time or fewer GPUs, and check `whypending <jobid>`.
The log stays empty while the job runs.	Python is buffering. Use `python -u` or `print(..., flush=True)`.
`WS_DIR: run source ../00_setup/setup_env.sh first`	`$WS_DIR` is not set in this shell. Do what the message says.
A model cannot be found in a job with `HF_HUB_OFFLINE=1`	It was never downloaded. On the login node: `python generate.py --download-only --model <name>`.
`ssh` does not connect at all	You are not on the university network. Start the VPN.

Documentation: compendium.hpc.tu-dresden.de. Support: hpc-support@tu-dresden.de.