Containment (intersection over the smaller area) between human-drawn crystal pairs never
exceeds 0.42, while the model-drawn duplicates sit at 0.96-1.0, so a rule at 0.9 cannot
lose a real pair. On 231 unseen frames, v4: precision 0.831 -> 0.876, recall 0.881 -> 0.869,
F1 0.855 -> 0.873, 46 outlines removed, 8 true positives lost where the less confident of
two outlines was the one matching the label.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
231 frames unseen by every model (core 148), 82 crystals on them. v4-noflip-mask1 stays
the best: F1 0.855, mask score 0.798; v3-noflip-mask1 0.848 / 0.793, within noise;
the small cluster model now clearly loses on masks (-0.021, won in 0 % of draws).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Frames of one mount are the same loop seen again, so the paired bootstrap now draws
mounts (data/gold/groups.json, built by tools/gold_groups.py from the mount ids the
studio writes in the gold manifest). The comparison also reports the unseen core without
the saturated frames, and frames the studio dropped from the gold set are pruned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Written to runs/_unseen_per_frame.json (not tracked): overall and crystal
tp/fp/fn and matched IoU per unseen gold frame and model, which lets the
monitor page bootstrap the difference between any pair.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The cluster's first run was gated against a baseline measured on the Spark.
The same weights score lower there (F1 0.840 vs 0.861, mask score 0.706 vs
0.757 on the 203 gold frames), so the gate failed a model that beats the
active one on F1 in 98.7 % of bootstrap draws when both are measured on one
machine.
Gold reports now record the environment (ultralytics, torch, host). When the
active model's report comes from other versions, train_run.py measures its
weights again in the job and gates against that; the predictions feed the
paired bootstrap. Without the weights there is no verdict and no promotion.
The SLURM scripts pull the active model's weights.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A paired bootstrap over the 149 unseen gold frames (2026-10-02) showed that a
difference between two models carries about +-0.023 of noise in F1, +-0.018 in
the mask score and +-0.10 in crystal F1, while the gate's tolerance was 0.005.
- tools/gate_stats.py: per-frame counts and a paired bootstrap (pure Python).
- train_run.py: when the baseline's gold predictions exist, the gate requires
the candidate to beat it on F1 or the mask score in 90 % of the draws and
not to lose the other in more than 90 %; the point rule is the fallback and
is always reported (pass_point_rule).
- tools/compare_unseen.py replaces the Spark's ~/fair_gate_all.py: all models
on the frames none trained on, on the frozen core (data/gold/unseen_core.json,
149 frames) and on all unseen frames once the gold set grows, with the
interval and win share against the active model.
- measure_weights.py keeps a version's gate verdict when it re-measures it.
- tools/spark_cycle.sh: gold growth, export, re-measure, train, gate, compare,
push, with what cycles 3 and 4 taught built in.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The exit code 2 of a failed gate made the cycle script retry and train
2026-10-01_v4-recipeA twice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three ledger runs for the cluster, exported on dataset/2026-09-29_v3 with the
promoted recipe: nano at 1024 px, small at 640 px, small at 1024 px. A run can
now carry its own training arguments (train_args in run.json, set with
ledger new --kv), which train_run.py applies under any --set; train.sbatch
forwards extra arguments after the run id (e.g. --batch 8). yolo26s-seg.pt
joins the base weights under LFS. slurm/README lists the runs, the questions
they answer and the commands.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
duan_j