AareLC_train/slurm

Training on the GPU cluster (template, not yet filled in)

train.sbatch runs one ledger run on the mx partition. It needs these facts, which replace the __PLACEHOLDERS__ in the script:

Placeholder	What	Value
__PARTITION__	SLURM partition	mx
__ACCOUNT__	SLURM account to charge
__WALLTIME__	wall-time limit for 400 epochs of yolo26n at 640 px
__SHARED_FS__	shared filesystem visible from login and compute nodes, holds the repo checkout
__MODULES__ / __VENV__ / __CONTAINER__	how Python, CUDA and ultralytics are provided on compute nodes (one of the three)
__DEPLOY_KEY__	SSH key on the cluster allowed to push to this repo, if the job reports back itself

Still to confirm before the first job: the cluster and login node names; whether compute nodes reach gitea.psi.ch (git + LFS) and PyPI, or whether a container has to carry everything; the GPU type on the partition; who submits (a service account or a person). Until then the reference runs happen on the DGX Spark with the same tools/train_run.py.

Workflow once filled in:

python tools/ledger.py new --dataset dataset/<name> --tag <tag> --base yolo26n-seg.pt
git commit -am "run <id>: exported" && git push
sbatch slurm/train.sbatch <id>
python tools/ledger.py show

The runs that are waiting for the cluster (2026-09-30)

All on dataset/2026-09-29_v3 (1 181 train / 210 val, the reviewed pool as of the evening of 2026-09-29; gold set 203 frames, never trained on) with the promoted recipe: no horizontal flip, full-resolution masks, overlap_mask off. The two levers the Spark cannot pull are resolution (frames are 2040 px, every run so far trained at 640) and model size (only the nano model has ever been trained). The base weights are in the repository under LFS.

Run	Base	Image size	Question it answers
2026-09-30_v3-n-1024px	yolo26n-seg.pt	1024	do small objects (crystal, ice, needle tip) gain from resolution alone
2026-09-30_v3-s-640px	yolo26s-seg.pt	640	how much capacity is missing at the same resolution
2026-09-30_v3-s-1024px	yolo26s-seg.pt	1024	both levers together

The image size lives in each run's train_args in the ledger, so the job needs no extra flags; --batch 8 after the run id halves the batch if the GPU runs out of memory at 1024 px. Their reference on the Spark is 2026-09-29_v3-noflip-mask1 (nano, 640 px, same recipe and dataset) and the quick signal 2026-09-29_v3-recipeA (gate passed, F1 0.855 on the unseen frames).

Once, on the login node (fill the placeholders in train.sbatch first):

cd <shared filesystem>
git clone git@gitea.psi.ch:mx/AareLC_train.git        # or https, with a token
cd AareLC_train
git lfs install
git lfs pull -I "data/pool/images/**,data/gold/**,models/base/**"
python -m venv .venv && . .venv/bin/activate && pip install ultralytics opencv-python-headless numpy

Then:

sbatch slurm/train.sbatch 2026-09-30_v3-n-1024px
sbatch slurm/train.sbatch 2026-09-30_v3-s-640px
sbatch slurm/train.sbatch 2026-09-30_v3-s-1024px      # add --batch 8 if the GPU is small
squeue -u $USER                                        # the jobs
tail -f runs/2026-09-30_v3-s-1024px/train.log          # a training
python tools/ledger.py show                            # the states

Each job trains, measures the weights on the gold set, gates against the active model and promotes on a pass (models/active/production.json); it commits the run directory and the registry entry locally. Pushing needs a key on the cluster that Gitea accepts: until then, git push origin main from the login node by hand, or the commit stays there for someone to push. Latency at the beamline must be measured again before a 1024 px or a small-model engine is deployed: the live server scales frames to 640 px today.

One environment per verdict

The gate compares two models only when both were measured with the same library versions. The first cluster run (2026-09-30_v3-s-640px, 2026-10-01) was gated against a baseline report written on the Spark: the same weights scored F1 0.840 and mask score 0.706 on the cluster and 0.861 and 0.757 on the Spark, and the gate failed a model that passes it in one environment. Since then every gold report records environment (ultralytics, torch, host); when the active model's report comes from other versions, train_run.py measures the active model again inside the job, which is why the job pulls models/registry/<active>/weights.pt. If those weights are missing the run ends as gate_failed with an environment_note and nothing is promoted. Keeping the cluster's ultralytics at the Spark's version removes the difference at its source.