AareLC_train/slurm

Training on the GPU cluster (template, not yet filled in)

train.sbatch runs one ledger run on the mx partition. It needs these facts, which replace the __PLACEHOLDERS__ in the script:

Placeholder	What	Value
__PARTITION__	SLURM partition	mx
__ACCOUNT__	SLURM account to charge
__WALLTIME__	wall-time limit for 400 epochs of yolo26n at 640 px
__SHARED_FS__	shared filesystem visible from login and compute nodes, holds the repo checkout
__MODULES__ / __VENV__ / __CONTAINER__	how Python, CUDA and ultralytics are provided on compute nodes (one of the three)
__DEPLOY_KEY__	SSH key on the cluster allowed to push to this repo, if the job reports back itself

Still to confirm before the first job: the cluster and login node names; whether compute nodes reach gitea.psi.ch (git + LFS) and PyPI, or whether a container has to carry everything; the GPU type on the partition; who submits (a service account or a person). Until then the reference runs happen on the DGX Spark with the same tools/train_run.py.

Workflow once filled in:

python tools/ledger.py new --dataset dataset/<name> --tag <tag> --base yolo26n-seg.pt
git commit -am "run <id>: exported" && git push
sbatch slurm/train.sbatch <id>
python tools/ledger.py show

The run that is waiting for the cluster

2026-09-29_v2-noflip-mask1 — dataset dataset/2026-09-29_v2 (the whole reviewed pool as of 2026-09-29, gold grown to 200 frames), default recipe: no horizontal flip, full-resolution masks, base weights models/base/yolo26n-seg.pt (in the repository under LFS). Its twin with the older settings A trains on the Spark for the comparison.

Once, on the login node (fill the placeholders in train.sbatch first):

cd <shared filesystem>
git clone git@gitea.psi.ch:mx/AareLC_train.git        # or https, with a token
cd AareLC_train
git lfs install
git lfs pull -I "data/pool/images/**,data/gold/**,models/base/**"
python -m venv .venv && . .venv/bin/activate && pip install ultralytics opencv-python-headless numpy

Then, for this run and every later one:

sbatch slurm/train.sbatch 2026-09-29_v2-noflip-mask1
squeue -u $USER                                        # the job
tail -f runs/2026-09-29_v2-noflip-mask1/train.log      # the training
python tools/ledger.py show                            # the state

The job trains, measures the weights on the gold set, gates against the active model and promotes on a pass (models/active/production.json); it commits the run directory and the registry entry locally. Pushing needs a key on the cluster that Gitea accepts: until then, git push origin main from the login node by hand, or the commit stays there for someone to push.