diff --git a/CLAUDE.md b/CLAUDE.md
index 407dfb80b..ef6f72df0 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -106,9 +106,16 @@ scripts.
```
tools/battery/battery.py run --rugnux build/rugnux/rugnux --only 5reo,lyso_ref # named sets
tools/battery/battery.py run --rugnux build/rugnux/rugnux --tier smoke # smoke tier
+tools/battery/battery.py run --rugnux build/rugnux/rugnux --variants bare # one variant only
tools/battery/battery.py compare RUN_A RUN_B [--rerun-changed]
```
+Each set runs in all its arm's **variants** back to back, so its images are read from disk once:
+`bare` (plain `rugnux `, what a user gets) first everywhere, then `model` (`--model` with the
+deposited coordinates, R-free vs the published one) on the open arm, or `xds` (XDS's resolution
+range forced, `-A` where XDS was anomalous) on the XDS arms. Only the first variant's time includes
+the disk read; compare times within one variant. The REFMAC check is opt-in (`--model-check`).
+
- **Never start a full battery run (no `--only`/`--tier`) without the owner's go-ahead.**
- Judge a change against a **persisted baseline run** (the site config's `baseline`) with
`compare`, never against numbers remembered from an earlier session.
diff --git a/docs/TESTS.md b/docs/TESTS.md
index e9372bc61..5c560c0c7 100644
--- a/docs/TESTS.md
+++ b/docs/TESTS.md
@@ -40,9 +40,9 @@ pipeline is exercised on, and the DOI to cite for each, are listed in
* `tools/battery/battery.py` - the rotation battery, the one canonical way to judge a rotation
change: public PDB depositions scored against the deposition, in-house standard crystals and
no-crystal controls scored against XDS, and a local-only private arm. Its manifests, the run
- and compare protocol and the report are described in `tools/battery/README.md`.
-* `tools/battery/model_sweep.py` - the battery's open arm run with `--model`: each merge validated
- against the depositor's own model and compared with the published resolution and R-work/R-free.
+ and compare protocol and the report are described in `tools/battery/README.md`. Each set runs
+ in several variants back to back: plain, with the deposited model (`--model`, R-free against the
+ published one) on the open arm, and with XDS's settings on the XDS arms.
* `rugnux_stills_ab.py` - the stills analogue of the battery: scores a change on a serial
dataset by what it does to the merge. Takes its dataset list from outside the repository.
* `rugnux_anomalous.py` - the anomalous-peak-height arbiter, below.
diff --git a/tools/battery/README.md b/tools/battery/README.md
index 628c9d785..bb4c5f12d 100644
--- a/tools/battery/README.md
+++ b/tools/battery/README.md
@@ -23,6 +23,45 @@ scored with `sgequiv`. On the XDS arms only the point group is scored, because X
screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported.
`score.py` has the details.
+## Variants
+
+Every set is run more than once, with different settings, and each run is a **variant**. The
+variants of one set run straight after each other, before the next set starts. The battery is
+limited by reading images from disk, and this way only the first variant reads them; the others
+find them in the page cache.
+
+| arm (reference) | variants, in the order they run |
+|---|---|
+| open (deposition) | `bare`, then `model` |
+| inhouse, private (XDS) | `bare`, then `xds` |
+
+- **`bare`**: plain `rugnux ` and nothing else (no `-A`, no resolution range). This is
+ what a user gets. It is scored against the reference like the others, and its d_min ratio
+ against the reference is plotted.
+- **`model`** (open arm): `rugnux --model `. The coordinates (`.cif.gz`)
+ and the published R-free come from the RCSB and are cached in the site's `pdb_cache` (default
+ `/data/battery/pdb_cache`), so the first run of a new entry needs network access. The entry is the
+ set id up to the first `_`, so `6h2p_native` uses `6h2p`. A set with no PDB entry (small
+ molecules) or whose coordinates cannot be downloaded gets a `not_run` row with cause `no_model`.
+ The row records the R-free and R-work that rugnux reports for the model, the published R-free
+ and their ratio, which the report plots. rugnux's R-free is that of a rigid-body placement against
+ its own merge, and the published one comes from full refinement against the depositor's own
+ data, so the ratio is not a like-for-like score; watch how it moves between runs.
+- **`xds`** (XDS arms): XDS's reference settings. XDS's resolution range is forced (see "The XDS
+ reference d_min" below), and where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true`
+ in the manifest) rugnux gets `-A`. Both programs then merge the same reflections the same way.
+
+The variants are defined in one place, `VARIANTS` in `battery.py`. `--variants` runs a subset,
+for example `--variants bare`, and they always run in the table's order. `--unforced` is gone:
+`--variants bare` gives rugnux its own resolution cut (and, unlike `--unforced`, no `-A`).
+
+**Order and timing.** `bare` goes first on every arm. So its time always includes reading the
+data from disk (unless the images were still cached from before the run), whichever variants a run
+selects, and it is the time a user sees. The time of a later variant is taken on cached images.
+Each row records whether it was its set's `first_read`, and the report's timing table says which
+variant carries the disk reads. Compare times of one variant between runs only, never across
+variants.
+
The private arm is never mixed with the others. A private run gets its own run directory
(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a
public report or be written into the repository. Its names, paths and values never go into a
@@ -86,6 +125,7 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.
"gpulock": "/data/tmp/gpulock",
"baseline": "/data/battery/runs/",
"baseline_private": null,
+ "pdb_cache": "/data/battery/pdb_cache",
"arms": {
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/open"},
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/home/data/inhouse"},
@@ -97,7 +137,8 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.
A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that
`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of
-the shared GPU slots. Set it to `null` if you have no such wrapper.
+the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where
+the deposited models and their metadata are cached for the `model` variant and the REFMAC check.
## Running
@@ -109,7 +150,7 @@ $B run --rugnux $R # full battery: open + inhou
$B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --only 5reo,lyso_ref # named sets
$B run --rugnux $R --arm private # the private arm, on its own
-$B run --rugnux $R --arm inhouse --unforced # XDS arms without XDS's resolution forced
+$B run --rugnux $R --variants bare # only what a user gets (no model, no XDS settings)
```
Useful options:
@@ -119,7 +160,8 @@ Useful options:
- `--gpulock` queues every rugnux behind the site's GPU wrapper.
- `--extra ""` adds rugnux arguments to every set.
- `--baseline RUN|none` sets the run the report is compared against.
-- `--model-check` / `--no-model-check` turns the open-arm R-free check on or off (see below).
+- `--variants bare,model,xds` runs a subset of the variants (default: all of them).
+- `--model-check` adds the REFMAC check to the open arm's `bare` rows (off by default, see below).
A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the
manifests (`"tiers": {"smoke": ""}`).
@@ -128,23 +170,23 @@ manifests (`"tiers": {"smoke": ""}`).
two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and
everything else on the machine competes with it.
-### The model check (open arm)
+### The REFMAC model check (open arm, opt-in)
-`tools/battery/model_check.py` scores each open-arm merge (`p.mtz`) against the deposited model,
+The `model` variant is the battery's R-free check. `--model-check` adds a second, independent one
+that needs CCP4: `tools/battery/model_check.py` scores each open-arm `bare` merge (`p.mtz`) against
+the deposited model,
unmodified (ligands, waters and hydrogens kept): REFMAC's rigid-body mode, R-factors taken before
any shift, the data reindexed into the model's setting and reduced to the deposited space group.
-Our own free set was mostly work reflections for the depositor, so `rfree` reads low. The fair
-comparison is `rfree_depflags` (our data on the depositor's free set) against `rfree_depdata` (the
-depositor's structure factors, same free set, same protocol); `rfree_ratio` is that ratio, and the
-report plots it. It is null when the entry deposited no structure factors with a free set.
-`rfree_deposited` is the value in the entry, often from twin or full refinement. The runner records
-`rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`,
-`model_status` and `model_reason`. The check is on by default when the module exists. The PDB entry
-is the set id up to the first `_`, so `6h2p_native` is checked as `6h2p`. Small-molecule sets are skipped.
+Our own free set was mostly work reflections for the depositor, so `refmac_rfree` reads low. The
+fair comparison is `refmac_rfree_depflags` (our data on the depositor's free set) against
+`refmac_rfree_depdata` (the depositor's structure factors, same free set, same protocol);
+`refmac_rfree_ratio` is that ratio, and the report plots it. It is null when the entry deposited no
+structure factors with a free set. The runner records `refmac_rfree`, `refmac_rwork`,
+`refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status` and
+`refmac_reason`. Small-molecule sets are skipped.
-`model_sweep.py` is a different tool: it runs `rugnux --model` over the open arm (same inputs, same
-site config) and compares the R-free that rugnux itself reports with the published value. It needs
-network access to the RCSB and runs separately from `battery.py`.
+`model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The
+`model` variant does the same inside the battery.
## A run directory
@@ -155,9 +197,9 @@ read-only when the run completes.
|---|---|
| `bin/rugnux` | the exact binary that ran (sha256 in the manifest) |
| `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags |
-| `results.json` | one row per set (the schema is below) |
+| `results.json` | one row per set and variant (the schema is below) |
| `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) |
-| `work///` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |
+| `work////` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |
**Run states** (`battery.py list`): `complete`; `running` (its runner process is alive);
`aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`);
@@ -165,16 +207,24 @@ read-only when the run completes.
covers an arbitrary part of its sets. `compare` and `report` refuse it unless given
`--allow-incomplete`, and the report then says so at the top.
-### results.json (schema 1)
+### results.json (schema 2)
-This is a flat list with one object per set, and every arm uses the same keys. Missing values are
-`null`. `manifest.json` records the schema version as `results_schema`.
+This is a flat list with one object per set and variant, and every arm uses the same keys. Missing
+values are `null`. `manifest.json` records the schema version as `results_schema`, and the variants
+each arm ran as `variants`.
+
+Schema 2 added `variant` and `first_read`. It also changed the meaning of `rfree`, `rwork` and
+`rfree_ratio`: they are now rugnux's own `--model` values (model variant), and the REFMAC check's
+moved to `refmac_*`. A schema-1 run is still read, and compared, as if each set had one variant:
+`bare` on the open arm; on the XDS arms `xds`, or `bare` if it was run `--unforced` (which still
+passed `-A`).
| key | meaning |
|---|---|
-| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran |
-| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) |
-| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` |
+| `set`, `arm`, `variant`, `tags`, `input`, `cmd` | the set's id, arm, variant (`bare`, `model`, `xds`), population tags, input file, and the exact command that ran |
+| `first_read` | true for the first variant of its set that ran: the one that read the images from disk |
+| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing, or no model for the `model` variant) |
+| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` / `no_model` |
| `reason` | one line for a human |
| `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours / the reference's |
| `sg_relation` | open arm: how the two groups relate (`sgequiv`) |
@@ -183,7 +233,8 @@ This is a flat list with one object per set, and every arm uses the same keys. M
| `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged |
| `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt` |
| `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref` | XDS's overall values, over XDS's own range |
-| `rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`, `model_status`, `model_reason` | the model check (open arm) |
+| `rfree`, `rwork`, `model_fit`, `rfree_deposited`, `rfree_ratio` | `model` variant: R-free, R-work and MODEL_FIT as rugnux reports them for the deposited model, the published R-free, and rfree / rfree_deposited |
+| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm, `bare` rows) |
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |
## The report
@@ -191,16 +242,19 @@ This is a flat list with one object per set, and every arm uses the same keys. M
Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and
SVG, no external files):
-- a header with the binary, build flags, host, subset, resolution mode and whether timing counts;
-- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain,
- ISa, R_meas, time) with a verdict bar chart;
-- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
- the main metrics;
-- plots per set: d_min(rugnux) / d_min(reference) for each arm whose resolution is not forced, and
- R_free / deposited R_free for the open arm (HTML only; hovering a point shows the set);
-- the **failures**, and, if there is a baseline, the **delta against it**: pass rates on the
- common sets and every set that moved beyond noise;
-- **one row per set** with all the numbers.
+- a header with the binary, build flags, host, subset, variants and whether timing counts;
+- a **timing** table per variant, saying which variant's times include reading the data;
+- **a section per variant**, each with:
+ - a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain,
+ ISa, R_meas, time) with a verdict bar chart;
+ - a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
+ the main metrics;
+ - plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference)
+ for each arm (not for `xds`, whose resolution is forced), and, for `model`, R_free / published
+ R_free;
+ - the **failures**, and **one row per set** with all the numbers;
+- if there is a baseline, the **delta against it**: pass rates per arm and variant on the common
+ rows, and every row that moved beyond noise.
Re-render a report, for example against another baseline (run directories are read-only, so the
output goes elsewhere and is named after the run):
@@ -221,8 +275,9 @@ $B compare RUN_A RUN_B [--all] # per-set table of what move
$B compare RUN_A RUN_B --rerun-changed # rerun the changed sets with A's binary
```
-`compare` pairs sets by arm and id, following the manifests' `aliases` across renames, and lists
-every set whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell,
+`compare` pairs rows by arm, set id and variant, following the manifests' `aliases` across
+renames. Only the variants both runs ran on an arm are compared, so a `--variants bare` run against a full
+one compares `bare`. It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell,
R-free or time moved by more than the noise thresholds in `report.py` (`NOISE`). Those thresholds
are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with
A's saved binary and options. A set that moves again under the same binary is noise. A set that
@@ -240,13 +295,11 @@ reproduces A was really changed by B.
fix. For one fix, run the sets it touches with `--only`, plus the smoke tier.
4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a
reference". Only a run with the machine to itself can be used to judge speed.
-5. **Resolution on the XDS arms.** By default XDS's resolution range is forced on rugnux, so both
- programs merge the same reflections and the statistics are like for like. This answers "is our
- processing better than XDS's, reflection for reflection". `--unforced` leaves rugnux its own
- cut, which answers "what does a user get". A change can win one and lose the other, so at
- important moments run both. Never quote the forced run as what rugnux delivers.
- Where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the manifest), rugnux gets
- `-A` in both modes, so Friedel mates are merged separately on both sides.
+5. **Resolution on the XDS arms.** The `xds` variant forces XDS's resolution range (and `-A`
+ where XDS kept Friedel mates apart), so both programs merge the same reflections and the
+ statistics are like for like. This answers "is our processing better than XDS's, reflection for
+ reflection". The `bare` variant answers "what does a user get". A change can win one and lose
+ the other, and every run has both. Never quote the `xds` variant as what rugnux delivers.
### The XDS reference d_min
@@ -262,7 +315,7 @@ is significant, and that decides which case applies (`inputs.reference_dmin`):
point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in
1/d² between shell centres.
-The derived value is both the score reference and the forced range. The manifest keeps XDS's own
+The derived value is both the score reference and the range the `xds` variant forces. The manifest keeps XDS's own
limit as `dmin_xds` and the rule used as `dmin_rule`. XDS's pooled R_meas, CC1/2 and completeness
still cover its whole range.
diff --git a/tools/battery/battery.py b/tools/battery/battery.py
index 7fd9f48d3..4d31afa80 100644
--- a/tools/battery/battery.py
+++ b/tools/battery/battery.py
@@ -12,7 +12,8 @@ Arms and where their truth comes from:
Usage (README.md in this directory is the full guide):
battery.py run --rugnux BIN [--label L] [--arm open,inhouse|private] [--only a,b | --tier smoke]
- [--build-dir DIR] [--gpulock] [--threads N] [--unforced] [--baseline RUN|none]
+ [--variants bare,model,xds] [--build-dir DIR] [--gpulock] [--threads N]
+ [--model-check] [--baseline RUN|none]
battery.py list runs under the runs root, with their state
battery.py compare RUN_A RUN_B [--all] [--rerun-changed]
battery.py report RUN --out DIR [--baseline RUN] re-render a run's report into DIR/.{md,html}
@@ -24,9 +25,13 @@ Usage (README.md in this directory is the full guide):
The site config (data roots, runs root, lock, gpulock, private manifest, baselines) is --site, else
$JFJOCH_BATTERY_SITE, else site.json beside this script (not committed; site.example.json is).
+Every set runs in each of its arm's VARIANTS (bare + model on the open arm, bare + xds on the XDS
+arms), one straight after the other, so only the first reads the images from disk.
+
A run writes /__