--skip-train registers the version with the day it runs, so the nano and the yolo26s of
cycle 5 existed twice in the registry (2026-10-23 and 2026-10-25); the runs point at the
10-25 ones, the others are removed. A run whose base weights are a registry version (the
warm start) was named after the file weights.pt; it is now named after that version's family.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013uu3Bxsu5JqFhaWa9Ftu6z
454 of the 1 029 gold outlines are a proposal kept unchanged (loop 159 of 317, pin 140 of
224, face 140 of 295, crystal 14 of 149): the pre-labelling model's own masks. The
evaluation now ignores them object by object for every model, like a crowd region: they
leave the gold, and an unmatched prediction on one of them is not a false alarm.
tools/gold_kept.py finds them against the studio's batch proposals (mask IoU >= 0.98) and
writes data/gold/kept_machine.json; evaluate_image, evaluate_dirs and gate_stats.per_frame
read it. With it, the frame-level exclusion of 845bd87 is no longer applied (it stays as
the fallback without the file), so the gold set can grow with frames the active model
pre-labelled. Every registry model is measured again on this basis and the three v5 runs
are re-gated, without promotion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013uu3Bxsu5JqFhaWa9Ftu6z
Where a reviewer keeps an outline the model proposed, the gold holds the model's own mask,
which overlaps itself perfectly: on the 50 gold frames v4 had pre-labelled it scored a
matched IoU of 0.959 against 0.905 elsewhere, every other model 0.93, and that alone
produced the "mask dip" that failed all three v5 runs. tools/gold_prelabel.py records
which model proposed each gold frame (data/gold/prelabel.json, 265 of 335 frames);
gate_stats.without_own_proposals drops those frames from that model's measure; the unseen
comparison and the gate's paired bootstrap use it, so two models are compared on the
frames neither of them touched. The three v5 runs are re-gated on that basis, without
promotion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MODELS="n s m l" runs the promoted recipe with yolo26n, s, m and l one after the other;
only the nano run may promote itself, the others are candidates until their speed at the
beamline is known. Each training waits for MEM_FLOOR_GB of free memory, an out-of-memory
run is tried again at batch 8, and vLLM is restarted at the end only if it was running.
tools/bench_latency.py measures the answer time per frame as the live server asks for it
(frame scaled by infer_scale, letterboxed to 640, PyTorch fp16) and writes runs/<id>/latency.json.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Containment (intersection over the smaller area) between human-drawn crystal pairs never
exceeds 0.42, while the model-drawn duplicates sit at 0.96-1.0, so a rule at 0.9 cannot
lose a real pair. On 231 unseen frames, v4: precision 0.831 -> 0.876, recall 0.881 -> 0.869,
F1 0.855 -> 0.873, 46 outlines removed, 8 true positives lost where the less confident of
two outlines was the one matching the label.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
231 frames unseen by every model (core 148), 82 crystals on them. v4-noflip-mask1 stays
the best: F1 0.855, mask score 0.798; v3-noflip-mask1 0.848 / 0.793, within noise;
the small cluster model now clearly loses on masks (-0.021, won in 0 % of draws).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Frames of one mount are the same loop seen again, so the paired bootstrap now draws
mounts (data/gold/groups.json, built by tools/gold_groups.py from the mount ids the
studio writes in the gold manifest). The comparison also reports the unseen core without
the saturated frames, and frames the studio dropped from the gold set are pruned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Written to runs/_unseen_per_frame.json (not tracked): overall and crystal
tp/fp/fn and matched IoU per unseen gold frame and model, which lets the
monitor page bootstrap the difference between any pair.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The cluster's first run was gated against a baseline measured on the Spark.
The same weights score lower there (F1 0.840 vs 0.861, mask score 0.706 vs
0.757 on the 203 gold frames), so the gate failed a model that beats the
active one on F1 in 98.7 % of bootstrap draws when both are measured on one
machine.
Gold reports now record the environment (ultralytics, torch, host). When the
active model's report comes from other versions, train_run.py measures its
weights again in the job and gates against that; the predictions feed the
paired bootstrap. Without the weights there is no verdict and no promotion.
The SLURM scripts pull the active model's weights.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A paired bootstrap over the 149 unseen gold frames (2026-10-02) showed that a
difference between two models carries about +-0.023 of noise in F1, +-0.018 in
the mask score and +-0.10 in crystal F1, while the gate's tolerance was 0.005.
- tools/gate_stats.py: per-frame counts and a paired bootstrap (pure Python).
- train_run.py: when the baseline's gold predictions exist, the gate requires
the candidate to beat it on F1 or the mask score in 90 % of the draws and
not to lose the other in more than 90 %; the point rule is the fallback and
is always reported (pass_point_rule).
- tools/compare_unseen.py replaces the Spark's ~/fair_gate_all.py: all models
on the frames none trained on, on the frozen core (data/gold/unseen_core.json,
149 frames) and on all unseen frames once the gold set grows, with the
interval and win share against the active model.
- measure_weights.py keeps a version's gate verdict when it re-measures it.
- tools/spark_cycle.sh: gold growth, export, re-measure, train, gate, compare,
push, with what cycles 3 and 4 taught built in.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The exit code 2 of a failed gate made the cycle script retry and train
2026-10-01_v4-recipeA twice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three ledger runs for the cluster, exported on dataset/2026-09-29_v3 with the
promoted recipe: nano at 1024 px, small at 640 px, small at 1024 px. A run can
now carry its own training arguments (train_args in run.json, set with
ledger new --kv), which train_run.py applies under any --set; train.sbatch
forwards extra arguments after the run id (e.g. --batch 8). yolo26s-seg.pt
joins the base weights under LFS. slurm/README lists the runs, the questions
they answer and the commands.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
duan_j