454 of the 1 029 gold outlines are a proposal kept unchanged (loop 159 of 317, pin 140 of
224, face 140 of 295, crystal 14 of 149): the pre-labelling model's own masks. The
evaluation now ignores them object by object for every model, like a crowd region: they
leave the gold, and an unmatched prediction on one of them is not a false alarm.
tools/gold_kept.py finds them against the studio's batch proposals (mask IoU >= 0.98) and
writes data/gold/kept_machine.json; evaluate_image, evaluate_dirs and gate_stats.per_frame
read it. With it, the frame-level exclusion of 845bd87 is no longer applied (it stays as
the fallback without the file), so the gold set can grow with frames the active model
pre-labelled. Every registry model is measured again on this basis and the three v5 runs
are re-gated, without promotion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013uu3Bxsu5JqFhaWa9Ftu6z
Frames of one mount are the same loop seen again, so the paired bootstrap now draws
mounts (data/gold/groups.json, built by tools/gold_groups.py from the mount ids the
studio writes in the gold manifest). The comparison also reports the unseen core without
the saturated frames, and frames the studio dropped from the gold set are pruned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The cluster's first run was gated against a baseline measured on the Spark.
The same weights score lower there (F1 0.840 vs 0.861, mask score 0.706 vs
0.757 on the 203 gold frames), so the gate failed a model that beats the
active one on F1 in 98.7 % of bootstrap draws when both are measured on one
machine.
Gold reports now record the environment (ultralytics, torch, host). When the
active model's report comes from other versions, train_run.py measures its
weights again in the job and gates against that; the predictions feed the
paired bootstrap. Without the weights there is no verdict and no promotion.
The SLURM scripts pull the active model's weights.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A paired bootstrap over the 149 unseen gold frames (2026-10-02) showed that a
difference between two models carries about +-0.023 of noise in F1, +-0.018 in
the mask score and +-0.10 in crystal F1, while the gate's tolerance was 0.005.
- tools/gate_stats.py: per-frame counts and a paired bootstrap (pure Python).
- train_run.py: when the baseline's gold predictions exist, the gate requires
the candidate to beat it on F1 or the mask score in 90 % of the draws and
not to lose the other in more than 90 %; the point rule is the fallback and
is always reported (pass_point_rule).
- tools/compare_unseen.py replaces the Spark's ~/fair_gate_all.py: all models
on the frames none trained on, on the frozen core (data/gold/unseen_core.json,
149 frames) and on all unseen frames once the gold set grows, with the
interval and win share against the active model.
- measure_weights.py keeps a version's gate verdict when it re-measures it.
- tools/spark_cycle.sh: gold growth, export, re-measure, train, gate, compare,
push, with what cycles 3 and 4 taught built in.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
duan_j