From 8c6670a3f41f06881465b6f0752cddd8e5877cfe Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 9 Oct 2026 14:10:30 +0200 Subject: [PATCH 1/5] battery: a ci tier, a closing table per set and a failing exit on it `battery.py run` now ends with one line per set, each column rugnux / reference: the space group with a tick or cross by the scorer's own verdict, d_min, ISa, CC1/2 overall and in the highest shell, R_meas of the lowest shell, the model's R-free and rugnux's wall time. `battery.py table RUN` prints it for any finished run (rows are rescored, so older runs get the new columns). The highest-shell CC1/2 is new in results.json (cc_half_high, read off rugnux's own shell table) with XDS's beside it (cc_half_high_ref, from CORRECT.LP's finest shell; the in-house manifest's refs regenerated with `refs --write`, which added only that key). A new "ci" tier: 25 sets that passed with the same space group and steady d_min / ISa in all four rc174/rc175 full runs (and in the rc175-next verdict runs where they ran), with no warning that questions the lattice or the group. About 10 minutes of rugnux (616 s measured) on the RTX 5080. A run that includes ci-tier sets exits 1 when one of them fails (lattice, symmetry, crash, timeout, missing input); nothing else fails a run. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi --- tools/battery/README.md | 37 ++++++++++++++-- tools/battery/battery.py | 30 ++++++++++--- tools/battery/inhouse.json | 86 +++++++++++++++++++------------------- tools/battery/inputs.py | 4 +- tools/battery/open.json | 32 +++++++------- tools/battery/report.py | 39 +++++++++++++++++ tools/battery/score.py | 31 ++++++++++---- 7 files changed, 182 insertions(+), 77 deletions(-) diff --git a/tools/battery/README.md b/tools/battery/README.md index d7763accb..0a61f0e5f 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -134,6 +134,9 @@ the reference: **Timing.** Each set runs once, so every time includes reading the images from disk (unless they were still in the page cache from before the run). +The in-house standard proteins run with `--model` since rc174, so their times include placing +the model; an earlier run (rc173 and before) did not, and a time delta on that arm against one is +partly that, not a change of speed. Compare in-house times between runs on the same side of it. The private arm is never mixed with the others. A private run gets its own run directory (suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a @@ -225,7 +228,8 @@ B=tools/battery/battery.py R=build/rugnux/rugnux $B run --rugnux $R # full battery: open + inhouse -$B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases +$B run --rugnux $R --tier smoke # 21 sets covering readers, symmetry, known hard cases +$B run --rugnux $R --tier ci # 25 sets every recent release passed (CI, below) $B run --rugnux $R --only 5reo,lyso_ref # named sets $B run --rugnux $R --arm private # the private arm, on its own ``` @@ -239,8 +243,34 @@ Useful options: - `--baseline RUN|none` sets the run the report is compared against. - `--model-check` adds the REFMAC check to the open arm's rows (off by default, see below). -A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the -manifests (`"tiers": {"smoke": ""}`). +A run prints one line per set as it goes and ends by writing its report and printing a table, one +line per set, each column rugnux / reference: the space group with ✓ where the set passed by the +scorer's rule and ✗ where it failed (lattice, symmetry, crash, no input), d_min, ISa, CC1/2 overall +and in the highest shell, R_meas of the lowest shell, R-free of the model (rugnux's placement / +the model's published) and rugnux's wall time. On the open arm only the space group, d_min and +R-free have a reference. `$B table RUN` prints it again for any finished run. + +**Tiers** are marked in the manifests (`"tiers": {"": ""}`): + +- **smoke** covers the readers, the symmetry classes and some known hard cases; several of its + sets fail on purpose. +- **ci** is the routine regression check: 25 sets that passed with the same space group and steady + d_min and ISa in every full run of rc174 and rc175, with no warning that questions the lattice or + the group (supercell, harmonic, twinning, untested centring). It spans HDF5, miniCBF, cbf.gz, + marCCD (both formats) and SMV; every crystal system, centred and high-symmetry lattices, small + molecules, anomalous and long-wavelength data, pink beam, a twinnable P31 and one large set + (8qaw, about 2 minutes) - about 10 minutes of rugnux in all on the RTX 5080. A set that starts + failing for a known, accepted reason leaves the tier; one that is merely slow stays. + +**Exit status.** `run` exits with status 1 when a ci-tier set in the run failed - a wrong lattice +or space group, a crash, a timeout or a missing input - and names those sets. This holds in any +run that includes them (`--tier ci`, a full run, `--only` naming one), so a red CI job means the +same thing whichever run it started. Nothing else fails a run: other sets' failures and every +metric are reported, not judged. + +**CI.** `.gitea/workflows/battery.yml` builds rugnux with the CI flags and runs the ci tier on every +push, or by hand any subset (empty: the full open + inhouse battery), on the machine that holds the +data. The job keeps report.md and the log as its artifact; the private arm is never mounted there. **Full runs.** A run without `--only` or `--tier` takes the site lock (`flock`, non-blocking), so two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and @@ -433,6 +463,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. | `refres_range`, `refres_shells_past_limit`, `refres_unique_reflections`, `refres_completeness`, `refres_multiplicity`, `refres_i_over_sigma`, `refres_r_meas`, `refres_cc_half`, `refres_isa` | XDS arms: the REFRES_* keys, the same merge over the reference range | | `isa_ratio`, `r_meas_ratio` | refres_isa / isa_ref and refres_r_meas / r_meas_ref | | `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), reported, not scored | +| `cc_half_high`, `cc_half_high_ref` | CC1/2 of the highest-resolution shell: rugnux's own table, XDS's (`cc_half_high` in the manifest's `ref`), each at its own limit | | `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref | | `model`, `model_note` | open arm, and in-house sets with a `model`: the coordinates given to `--model`, or why there were none | | `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | any row with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited | diff --git a/tools/battery/battery.py b/tools/battery/battery.py index 98511e6c3..8760b81cc 100644 --- a/tools/battery/battery.py +++ b/tools/battery/battery.py @@ -11,12 +11,13 @@ Arms and where their truth comes from: site config) and a private run gets its own run directory and report. Usage (README.md in this directory is the full guide): - battery.py run --rugnux BIN [--label L] [--arm open,inhouse|private] [--only a,b | --tier smoke] + battery.py run --rugnux BIN [--label L] [--arm open,inhouse|private] [--only a,b | --tier smoke|ci] [--build-dir DIR] [--gpulock] [--threads N] [--model-check] [--baseline RUN|none] battery.py list runs under the runs root, with their state battery.py compare RUN_A RUN_B [--all] [--rerun-changed] battery.py report RUN --out DIR [--baseline RUN] re-render a run's report into DIR/.{md,html} + battery.py table RUN the one-line-per-set table a run ends with battery.py abort RUN [--reason TEXT] mark an unfinished run as aborted battery.py recheck RUN [--only a,b] [--jobs N] rerun the model checks of a finished run's open arm on its p.mtz files; recorded in the manifest @@ -37,6 +38,7 @@ A run writes /__