GotthardGandClaude Fable 5.1
62f568b0f6
The cluster's first run was gated against a baseline measured on the Spark. The same weights score lower there (F1 0.840 vs 0.861, mask score 0.706 vs 0.757 on the 203 gold frames), so the gate failed a model that beats the active one on F1 in 98.7 % of bootstrap draws when both are measured on one machine. Gold reports now record the environment (ultralytics, torch, host). When the active model's report comes from other versions, train_run.py measures its weights again in the job and gates against that; the predictions feed the paired bootstrap. Without the weights there is no verdict and no promotion. The SLURM scripts pull the active model's weights. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
45 lines
2.4 KiB
Bash
45 lines
2.4 KiB
Bash
#!/bin/bash
|
|
# Train one ledger run on the GPU cluster. Placeholders __LIKE_THIS__ are filled in
|
|
# once the cluster facts are known (see slurm/README.md).
|
|
#
|
|
# sbatch slurm/train.sbatch <run_id> [extra train_run.py args] # run created with: python tools/ledger.py new ...
|
|
#
|
|
#SBATCH --job-name=aarelc-train
|
|
#SBATCH --partition=__PARTITION__ # mx
|
|
#SBATCH --account=__ACCOUNT__
|
|
#SBATCH --gres=gpu:1
|
|
#SBATCH --cpus-per-task=8
|
|
#SBATCH --mem=32G
|
|
#SBATCH --time=__WALLTIME__ # e.g. 08:00:00
|
|
#SBATCH --output=__SHARED_FS__/AareLC_train/runs/slurm-%j.out
|
|
set -euo pipefail
|
|
|
|
RUN_ID="${1:?usage: sbatch slurm/train.sbatch <run_id> [train_run.py args, e.g. --batch 8]}"
|
|
shift
|
|
REPO="${AARELC_TRAIN_REPO:-__SHARED_FS__/AareLC_train}" # a checkout on the shared filesystem
|
|
|
|
# --- environment: keep exactly one of the three ---------------------------------
|
|
# module load __MODULES__ # e.g. cuda/12 python/3.12
|
|
# source __VENV__/bin/activate # venv with ultralytics + opencv
|
|
# APPTAINER="apptainer exec --nv __CONTAINER__" # container with CUDA + ultralytics
|
|
APPTAINER="${APPTAINER:-}"
|
|
|
|
cd "$REPO"
|
|
git fetch -q --tags
|
|
git pull -q --ff-only
|
|
DATASET_TAG="$(python -c "import json;print(json.load(open('runs/$RUN_ID/run.json'))['dataset'])")"
|
|
git checkout -q "$DATASET_TAG" -- data/datasets # the dataset exactly as tagged
|
|
# the active model's weights too: the gate measures the baseline again in this job's own environment
|
|
ACTIVE="$(python -c "import json;print(json.load(open('models/active/production.json'))['version'])" 2>/dev/null || true)"
|
|
git lfs pull -q -I "data/pool/images/**,data/gold/**,models/base/**,models/registry/$ACTIVE/weights.pt" # the labelled pool, the gold set, the base weights
|
|
|
|
python tools/ledger.py set "$RUN_ID" queued --kv "job=$SLURM_JOB_ID" "host=$(hostname)" "partition=$SLURM_JOB_PARTITION"
|
|
$APPTAINER python tools/train_run.py "$RUN_ID" --device 0 --workers "$SLURM_CPUS_PER_TASK" "$@" 2>&1 | tee "runs/$RUN_ID/train.log"
|
|
STATUS=${PIPESTATUS[0]}
|
|
|
|
# --- report back: commit the run, the registry entry and, on a pass, the promotion --
|
|
git add "runs/$RUN_ID" models
|
|
git commit -q -m "run $RUN_ID: $(python -c "import json;print(json.load(open('runs/$RUN_ID/run.json'))['status'])")" || true
|
|
# git push -q origin HEAD:main # needs a deploy key on the cluster (__DEPLOY_KEY__)
|
|
exit "$STATUS"
|