# Training on the GPU cluster (template, not yet filled in) `train.sbatch` runs one ledger run on the `mx` partition. It needs these facts, which replace the `__PLACEHOLDERS__` in the script: | Placeholder | What | Value | | --- | --- | --- | | `__PARTITION__` | SLURM partition | `mx` | | `__ACCOUNT__` | SLURM account to charge | | | `__WALLTIME__` | wall-time limit for 400 epochs of yolo26n at 640 px | | | `__SHARED_FS__` | shared filesystem visible from login and compute nodes, holds the repo checkout | | | `__MODULES__` / `__VENV__` / `__CONTAINER__` | how Python, CUDA and ultralytics are provided on compute nodes (one of the three) | | | `__DEPLOY_KEY__` | SSH key on the cluster allowed to push to this repo, if the job reports back itself | | Still to confirm before the first job: the cluster and login node names; whether compute nodes reach gitea.psi.ch (git + LFS) and PyPI, or whether a container has to carry everything; the GPU type on the partition; who submits (a service account or a person). Until then the reference runs happen on the DGX Spark with the same `tools/train_run.py`. Workflow once filled in: ```bash python tools/ledger.py new --dataset dataset/ --tag --base yolo26n-seg.pt git commit -am "run : exported" && git push sbatch slurm/train.sbatch python tools/ledger.py show ``` ## The run that is waiting for the cluster `2026-09-29_v2-noflip-mask1` — dataset `dataset/2026-09-29_v2` (the whole reviewed pool as of 2026-09-29, gold grown to 200 frames), default recipe: no horizontal flip, full-resolution masks, base weights `models/base/yolo26n-seg.pt` (in the repository under LFS). Its twin with the older settings A trains on the Spark for the comparison. Once, on the login node (fill the placeholders in `train.sbatch` first): ```bash cd git clone git@gitea.psi.ch:mx/AareLC_train.git # or https, with a token cd AareLC_train git lfs install git lfs pull -I "data/pool/images/**,data/gold/**,models/base/**" python -m venv .venv && . .venv/bin/activate && pip install ultralytics opencv-python-headless numpy ``` Then, for this run and every later one: ```bash sbatch slurm/train.sbatch 2026-09-29_v2-noflip-mask1 squeue -u $USER # the job tail -f runs/2026-09-29_v2-noflip-mask1/train.log # the training python tools/ledger.py show # the state ``` The job trains, measures the weights on the gold set, gates against the active model and promotes on a pass (`models/active/production.json`); it commits the run directory and the registry entry locally. Pushing needs a key on the cluster that Gitea accepts: until then, `git push origin main` from the login node by hand, or the commit stays there for someone to push.