454 of the 1 029 gold outlines are a proposal kept unchanged (loop 159 of 317, pin 140 of
224, face 140 of 295, crystal 14 of 149): the pre-labelling model's own masks. The
evaluation now ignores them object by object for every model, like a crowd region: they
leave the gold, and an unmatched prediction on one of them is not a false alarm.
tools/gold_kept.py finds them against the studio's batch proposals (mask IoU >= 0.98) and
writes data/gold/kept_machine.json; evaluate_image, evaluate_dirs and gate_stats.per_frame
read it. With it, the frame-level exclusion of 845bd87 is no longer applied (it stays as
the fallback without the file), so the gold set can grow with frames the active model
pre-labelled. Every registry model is measured again on this basis and the three v5 runs
are re-gated, without promotion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013uu3Bxsu5JqFhaWa9Ftu6z
Where a reviewer keeps an outline the model proposed, the gold holds the model's own mask,
which overlaps itself perfectly: on the 50 gold frames v4 had pre-labelled it scored a
matched IoU of 0.959 against 0.905 elsewhere, every other model 0.93, and that alone
produced the "mask dip" that failed all three v5 runs. tools/gold_prelabel.py records
which model proposed each gold frame (data/gold/prelabel.json, 265 of 335 frames);
gate_stats.without_own_proposals drops those frames from that model's measure; the unseen
comparison and the gate's paired bootstrap use it, so two models are compared on the
frames neither of them touched. The three v5 runs are re-gated on that basis, without
promotion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
231 frames unseen by every model (core 148), 82 crystals on them. v4-noflip-mask1 stays
the best: F1 0.855, mask score 0.798; v3-noflip-mask1 0.848 / 0.793, within noise;
the small cluster model now clearly loses on masks (-0.021, won in 0 % of draws).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A paired bootstrap over the 149 unseen gold frames (2026-10-02) showed that a
difference between two models carries about +-0.023 of noise in F1, +-0.018 in
the mask score and +-0.10 in crystal F1, while the gate's tolerance was 0.005.
- tools/gate_stats.py: per-frame counts and a paired bootstrap (pure Python).
- train_run.py: when the baseline's gold predictions exist, the gate requires
the candidate to beat it on F1 or the mask score in 90 % of the draws and
not to lose the other in more than 90 %; the point rule is the fallback and
is always reported (pass_point_rule).
- tools/compare_unseen.py replaces the Spark's ~/fair_gate_all.py: all models
on the frames none trained on, on the frozen core (data/gold/unseen_core.json,
149 frames) and on all unseen frames once the gold set grows, with the
interval and win share against the active model.
- measure_weights.py keeps a version's gate verdict when it re-measures it.
- tools/spark_cycle.sh: gold growth, export, re-measure, train, gate, compare,
push, with what cycles 3 and 4 taught built in.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
jungfrau
duan_j