4 Commits

Include renames

Author	SHA1	Message	Date
jungfrauandClaude Fable 5.1	63ee07fe61	a gold outline the reviewer kept from the model's proposal is not judged ... 454 of the 1 029 gold outlines are a proposal kept unchanged (loop 159 of 317, pin 140 of 224, face 140 of 295, crystal 14 of 149): the pre-labelling model's own masks. The evaluation now ignores them object by object for every model, like a crowd region: they leave the gold, and an unmatched prediction on one of them is not a false alarm. tools/gold_kept.py finds them against the studio's batch proposals (mask IoU >= 0.98) and writes data/gold/kept_machine.json; evaluate_image, evaluate_dirs and gate_stats.per_frame read it. With it, the frame-level exclusion of 845bd87 is no longer applied (it stays as the fallback without the file), so the gold set can grow with frames the active model pre-labelled. Every registry model is measured again on this basis and the three v5 runs are re-gated, without promotion. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013uu3Bxsu5JqFhaWa9Ftu6z	2026-10-25 09:48:20 +01:00
jungfrauandClaude Fable 5.1	845bd87bb3	a model is not measured on the gold frames it pre-labelled itself ... Where a reviewer keeps an outline the model proposed, the gold holds the model's own mask, which overlaps itself perfectly: on the 50 gold frames v4 had pre-labelled it scored a matched IoU of 0.959 against 0.905 elsewhere, every other model 0.93, and that alone produced the "mask dip" that failed all three v5 runs. tools/gold_prelabel.py records which model proposed each gold frame (data/gold/prelabel.json, 265 of 335 frames); gate_stats.without_own_proposals drops those frames from that model's measure; the unseen comparison and the gate's paired bootstrap use it, so two models are compared on the frames neither of them touched. The three v5 runs are re-gated on that basis, without promotion. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-25 02:48:09 +02:00
GotthardGandClaude Fable 5.1	82f0d9099b	gate: bootstrap by mount, light states and excluded frames of the gold set ... Frames of one mount are the same loop seen again, so the paired bootstrap now draws mounts (data/gold/groups.json, built by tools/gold_groups.py from the mount ids the studio writes in the gold manifest). The comparison also reports the unseen core without the saturated frames, and frames the studio dropped from the gold set are pruned. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-02 15:46:31 +02:00
GotthardGandClaude Fable 5.1	831367d35d	noise-aware gate, frozen core of unseen frames, and a cycle script for the Spark ... A paired bootstrap over the 149 unseen gold frames (2026-10-02) showed that a difference between two models carries about +-0.023 of noise in F1, +-0.018 in the mask score and +-0.10 in crystal F1, while the gate's tolerance was 0.005. - tools/gate_stats.py: per-frame counts and a paired bootstrap (pure Python). - train_run.py: when the baseline's gold predictions exist, the gate requires the candidate to beat it on F1 or the mask score in 90 % of the draws and not to lose the other in more than 90 %; the point rule is the fallback and is always reported (pass_point_rule). - tools/compare_unseen.py replaces the Spark's ~/fair_gate_all.py: all models on the frames none trained on, on the frozen core (data/gold/unseen_core.json, 149 frames) and on all unseen frames once the gold set grows, with the interval and win share against the active model. - measure_weights.py keeps a version's gate verdict when it re-measures it. - tools/spark_cycle.sh: gold growth, export, re-measure, train, gate, compare, push, with what cycles 3 and 4 taught built in. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-02 13:25:45 +02:00