454 of the 1 029 gold outlines are a proposal kept unchanged (loop 159 of 317, pin 140 of
224, face 140 of 295, crystal 14 of 149): the pre-labelling model's own masks. The
evaluation now ignores them object by object for every model, like a crowd region: they
leave the gold, and an unmatched prediction on one of them is not a false alarm.
tools/gold_kept.py finds them against the studio's batch proposals (mask IoU >= 0.98) and
writes data/gold/kept_machine.json; evaluate_image, evaluate_dirs and gate_stats.per_frame
read it. With it, the frame-level exclusion of 845bd87 is no longer applied (it stays as
the fallback without the file), so the gold set can grow with frames the active model
pre-labelled. Every registry model is measured again on this basis and the three v5 runs
are re-gated, without promotion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013uu3Bxsu5JqFhaWa9Ftu6z
Where a reviewer keeps an outline the model proposed, the gold holds the model's own mask,
which overlaps itself perfectly: on the 50 gold frames v4 had pre-labelled it scored a
matched IoU of 0.959 against 0.905 elsewhere, every other model 0.93, and that alone
produced the "mask dip" that failed all three v5 runs. tools/gold_prelabel.py records
which model proposed each gold frame (data/gold/prelabel.json, 265 of 335 frames);
gate_stats.without_own_proposals drops those frames from that model's measure; the unseen
comparison and the gate's paired bootstrap use it, so two models are compared on the
frames neither of them touched. The three v5 runs are re-gated on that basis, without
promotion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Frames of one mount are the same loop seen again, so the paired bootstrap now draws
mounts (data/gold/groups.json, built by tools/gold_groups.py from the mount ids the
studio writes in the gold manifest). The comparison also reports the unseen core without
the saturated frames, and frames the studio dropped from the gold set are pruned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The cluster's first run was gated against a baseline measured on the Spark.
The same weights score lower there (F1 0.840 vs 0.861, mask score 0.706 vs
0.757 on the 203 gold frames), so the gate failed a model that beats the
active one on F1 in 98.7 % of bootstrap draws when both are measured on one
machine.
Gold reports now record the environment (ultralytics, torch, host). When the
active model's report comes from other versions, train_run.py measures its
weights again in the job and gates against that; the predictions feed the
paired bootstrap. Without the weights there is no verdict and no promotion.
The SLURM scripts pull the active model's weights.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A paired bootstrap over the 149 unseen gold frames (2026-10-02) showed that a
difference between two models carries about +-0.023 of noise in F1, +-0.018 in
the mask score and +-0.10 in crystal F1, while the gate's tolerance was 0.005.
- tools/gate_stats.py: per-frame counts and a paired bootstrap (pure Python).
- train_run.py: when the baseline's gold predictions exist, the gate requires
the candidate to beat it on F1 or the mask score in 90 % of the draws and
not to lose the other in more than 90 %; the point rule is the fallback and
is always reported (pass_point_rule).
- tools/compare_unseen.py replaces the Spark's ~/fair_gate_all.py: all models
on the frames none trained on, on the frozen core (data/gold/unseen_core.json,
149 frames) and on all unseen frames once the gold set grows, with the
interval and win share against the active model.
- measure_weights.py keeps a version's gate verdict when it re-measures it.
- tools/spark_cycle.sh: gold growth, export, re-measure, train, gate, compare,
push, with what cycles 3 and 4 taught built in.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The exit code 2 of a failed gate made the cycle script retry and train
2026-10-01_v4-recipeA twice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three ledger runs for the cluster, exported on dataset/2026-09-29_v3 with the
promoted recipe: nano at 1024 px, small at 640 px, small at 1024 px. A run can
now carry its own training arguments (train_args in run.json, set with
ledger new --kv), which train_run.py applies under any --set; train.sbatch
forwards extra arguments after the run id (e.g. --batch 8). yolo26s-seg.pt
joins the base weights under LFS. slurm/README lists the runs, the questions
they answer and the commands.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
