diff --git a/CLAUDE.md b/CLAUDE.md index 407dfb80b..ef6f72df0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,9 +106,16 @@ scripts. ``` tools/battery/battery.py run --rugnux build/rugnux/rugnux --only 5reo,lyso_ref # named sets tools/battery/battery.py run --rugnux build/rugnux/rugnux --tier smoke # smoke tier +tools/battery/battery.py run --rugnux build/rugnux/rugnux --variants bare # one variant only tools/battery/battery.py compare RUN_A RUN_B [--rerun-changed] ``` +Each set runs in all its arm's **variants** back to back, so its images are read from disk once: +`bare` (plain `rugnux `, what a user gets) first everywhere, then `model` (`--model` with the +deposited coordinates, R-free vs the published one) on the open arm, or `xds` (XDS's resolution +range forced, `-A` where XDS was anomalous) on the XDS arms. Only the first variant's time includes +the disk read; compare times within one variant. The REFMAC check is opt-in (`--model-check`). + - **Never start a full battery run (no `--only`/`--tier`) without the owner's go-ahead.** - Judge a change against a **persisted baseline run** (the site config's `baseline`) with `compare`, never against numbers remembered from an earlier session. diff --git a/docs/TESTS.md b/docs/TESTS.md index e9372bc61..5c560c0c7 100644 --- a/docs/TESTS.md +++ b/docs/TESTS.md @@ -40,9 +40,9 @@ pipeline is exercised on, and the DOI to cite for each, are listed in * `tools/battery/battery.py` - the rotation battery, the one canonical way to judge a rotation change: public PDB depositions scored against the deposition, in-house standard crystals and no-crystal controls scored against XDS, and a local-only private arm. Its manifests, the run - and compare protocol and the report are described in `tools/battery/README.md`. -* `tools/battery/model_sweep.py` - the battery's open arm run with `--model`: each merge validated - against the depositor's own model and compared with the published resolution and R-work/R-free. + and compare protocol and the report are described in `tools/battery/README.md`. Each set runs + in several variants back to back: plain, with the deposited model (`--model`, R-free against the + published one) on the open arm, and with XDS's settings on the XDS arms. * `rugnux_stills_ab.py` - the stills analogue of the battery: scores a change on a serial dataset by what it does to the merge. Takes its dataset list from outside the repository. * `rugnux_anomalous.py` - the anomalous-peak-height arbiter, below. diff --git a/tools/battery/README.md b/tools/battery/README.md index 628c9d785..bb4c5f12d 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -23,6 +23,45 @@ scored with `sgequiv`. On the XDS arms only the point group is scored, because X screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported. `score.py` has the details. +## Variants + +Every set is run more than once, with different settings, and each run is a **variant**. The +variants of one set run straight after each other, before the next set starts. The battery is +limited by reading images from disk, and this way only the first variant reads them; the others +find them in the page cache. + +| arm (reference) | variants, in the order they run | +|---|---| +| open (deposition) | `bare`, then `model` | +| inhouse, private (XDS) | `bare`, then `xds` | + +- **`bare`**: plain `rugnux ` and nothing else (no `-A`, no resolution range). This is + what a user gets. It is scored against the reference like the others, and its d_min ratio + against the reference is plotted. +- **`model`** (open arm): `rugnux --model `. The coordinates (`.cif.gz`) + and the published R-free come from the RCSB and are cached in the site's `pdb_cache` (default + `/data/battery/pdb_cache`), so the first run of a new entry needs network access. The entry is the + set id up to the first `_`, so `6h2p_native` uses `6h2p`. A set with no PDB entry (small + molecules) or whose coordinates cannot be downloaded gets a `not_run` row with cause `no_model`. + The row records the R-free and R-work that rugnux reports for the model, the published R-free + and their ratio, which the report plots. rugnux's R-free is that of a rigid-body placement against + its own merge, and the published one comes from full refinement against the depositor's own + data, so the ratio is not a like-for-like score; watch how it moves between runs. +- **`xds`** (XDS arms): XDS's reference settings. XDS's resolution range is forced (see "The XDS + reference d_min" below), and where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` + in the manifest) rugnux gets `-A`. Both programs then merge the same reflections the same way. + +The variants are defined in one place, `VARIANTS` in `battery.py`. `--variants` runs a subset, +for example `--variants bare`, and they always run in the table's order. `--unforced` is gone: +`--variants bare` gives rugnux its own resolution cut (and, unlike `--unforced`, no `-A`). + +**Order and timing.** `bare` goes first on every arm. So its time always includes reading the +data from disk (unless the images were still cached from before the run), whichever variants a run +selects, and it is the time a user sees. The time of a later variant is taken on cached images. +Each row records whether it was its set's `first_read`, and the report's timing table says which +variant carries the disk reads. Compare times of one variant between runs only, never across +variants. + The private arm is never mixed with the others. A private run gets its own run directory (suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a public report or be written into the repository. Its names, paths and values never go into a @@ -86,6 +125,7 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`. "gpulock": "/data/tmp/gpulock", "baseline": "/data/battery/runs/", "baseline_private": null, + "pdb_cache": "/data/battery/pdb_cache", "arms": { "open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/open"}, "inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/home/data/inhouse"}, @@ -97,7 +137,8 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`. A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that `rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of -the shared GPU slots. Set it to `null` if you have no such wrapper. +the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where +the deposited models and their metadata are cached for the `model` variant and the REFMAC check. ## Running @@ -109,7 +150,7 @@ $B run --rugnux $R # full battery: open + inhou $B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases $B run --rugnux $R --only 5reo,lyso_ref # named sets $B run --rugnux $R --arm private # the private arm, on its own -$B run --rugnux $R --arm inhouse --unforced # XDS arms without XDS's resolution forced +$B run --rugnux $R --variants bare # only what a user gets (no model, no XDS settings) ``` Useful options: @@ -119,7 +160,8 @@ Useful options: - `--gpulock` queues every rugnux behind the site's GPU wrapper. - `--extra ""` adds rugnux arguments to every set. - `--baseline RUN|none` sets the run the report is compared against. -- `--model-check` / `--no-model-check` turns the open-arm R-free check on or off (see below). +- `--variants bare,model,xds` runs a subset of the variants (default: all of them). +- `--model-check` adds the REFMAC check to the open arm's `bare` rows (off by default, see below). A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the manifests (`"tiers": {"smoke": ""}`). @@ -128,23 +170,23 @@ manifests (`"tiers": {"smoke": ""}`). two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and everything else on the machine competes with it. -### The model check (open arm) +### The REFMAC model check (open arm, opt-in) -`tools/battery/model_check.py` scores each open-arm merge (`p.mtz`) against the deposited model, +The `model` variant is the battery's R-free check. `--model-check` adds a second, independent one +that needs CCP4: `tools/battery/model_check.py` scores each open-arm `bare` merge (`p.mtz`) against +the deposited model, unmodified (ligands, waters and hydrogens kept): REFMAC's rigid-body mode, R-factors taken before any shift, the data reindexed into the model's setting and reduced to the deposited space group. -Our own free set was mostly work reflections for the depositor, so `rfree` reads low. The fair -comparison is `rfree_depflags` (our data on the depositor's free set) against `rfree_depdata` (the -depositor's structure factors, same free set, same protocol); `rfree_ratio` is that ratio, and the -report plots it. It is null when the entry deposited no structure factors with a free set. -`rfree_deposited` is the value in the entry, often from twin or full refinement. The runner records -`rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`, -`model_status` and `model_reason`. The check is on by default when the module exists. The PDB entry -is the set id up to the first `_`, so `6h2p_native` is checked as `6h2p`. Small-molecule sets are skipped. +Our own free set was mostly work reflections for the depositor, so `refmac_rfree` reads low. The +fair comparison is `refmac_rfree_depflags` (our data on the depositor's free set) against +`refmac_rfree_depdata` (the depositor's structure factors, same free set, same protocol); +`refmac_rfree_ratio` is that ratio, and the report plots it. It is null when the entry deposited no +structure factors with a free set. The runner records `refmac_rfree`, `refmac_rwork`, +`refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status` and +`refmac_reason`. Small-molecule sets are skipped. -`model_sweep.py` is a different tool: it runs `rugnux --model` over the open arm (same inputs, same -site config) and compares the R-free that rugnux itself reports with the published value. It needs -network access to the RCSB and runs separately from `battery.py`. +`model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The +`model` variant does the same inside the battery. ## A run directory @@ -155,9 +197,9 @@ read-only when the run completes. |---|---| | `bin/rugnux` | the exact binary that ran (sha256 in the manifest) | | `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags | -| `results.json` | one row per set (the schema is below) | +| `results.json` | one row per set and variant (the schema is below) | | `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) | -| `work///` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` | +| `work////` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` | **Run states** (`battery.py list`): `complete`; `running` (its runner process is alive); `aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`); @@ -165,16 +207,24 @@ read-only when the run completes. covers an arbitrary part of its sets. `compare` and `report` refuse it unless given `--allow-incomplete`, and the report then says so at the top. -### results.json (schema 1) +### results.json (schema 2) -This is a flat list with one object per set, and every arm uses the same keys. Missing values are -`null`. `manifest.json` records the schema version as `results_schema`. +This is a flat list with one object per set and variant, and every arm uses the same keys. Missing +values are `null`. `manifest.json` records the schema version as `results_schema`, and the variants +each arm ran as `variants`. + +Schema 2 added `variant` and `first_read`. It also changed the meaning of `rfree`, `rwork` and +`rfree_ratio`: they are now rugnux's own `--model` values (model variant), and the REFMAC check's +moved to `refmac_*`. A schema-1 run is still read, and compared, as if each set had one variant: +`bare` on the open arm; on the XDS arms `xds`, or `bare` if it was run `--unforced` (which still +passed `-A`). | key | meaning | |---|---| -| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran | -| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) | -| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` | +| `set`, `arm`, `variant`, `tags`, `input`, `cmd` | the set's id, arm, variant (`bare`, `model`, `xds`), population tags, input file, and the exact command that ran | +| `first_read` | true for the first variant of its set that ran: the one that read the images from disk | +| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing, or no model for the `model` variant) | +| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` / `no_model` | | `reason` | one line for a human | | `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours / the reference's | | `sg_relation` | open arm: how the two groups relate (`sgequiv`) | @@ -183,7 +233,8 @@ This is a flat list with one object per set, and every arm uses the same keys. M | `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged | | `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt` | | `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref` | XDS's overall values, over XDS's own range | -| `rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`, `model_status`, `model_reason` | the model check (open arm) | +| `rfree`, `rwork`, `model_fit`, `rfree_deposited`, `rfree_ratio` | `model` variant: R-free, R-work and MODEL_FIT as rugnux reports them for the deposited model, the published R-free, and rfree / rfree_deposited | +| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm, `bare` rows) | | `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line | ## The report @@ -191,16 +242,19 @@ This is a flat list with one object per set, and every arm uses the same keys. M Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and SVG, no external files): -- a header with the binary, build flags, host, subset, resolution mode and whether timing counts; -- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain, - ISa, R_meas, time) with a verdict bar chart; -- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of - the main metrics; -- plots per set: d_min(rugnux) / d_min(reference) for each arm whose resolution is not forced, and - R_free / deposited R_free for the open arm (HTML only; hovering a point shows the set); -- the **failures**, and, if there is a baseline, the **delta against it**: pass rates on the - common sets and every set that moved beyond noise; -- **one row per set** with all the numbers. +- a header with the binary, build flags, host, subset, variants and whether timing counts; +- a **timing** table per variant, saying which variant's times include reading the data; +- **a section per variant**, each with: + - a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain, + ISa, R_meas, time) with a verdict bar chart; + - a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of + the main metrics; + - plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference) + for each arm (not for `xds`, whose resolution is forced), and, for `model`, R_free / published + R_free; + - the **failures**, and **one row per set** with all the numbers; +- if there is a baseline, the **delta against it**: pass rates per arm and variant on the common + rows, and every row that moved beyond noise. Re-render a report, for example against another baseline (run directories are read-only, so the output goes elsewhere and is named after the run): @@ -221,8 +275,9 @@ $B compare RUN_A RUN_B [--all] # per-set table of what move $B compare RUN_A RUN_B --rerun-changed # rerun the changed sets with A's binary ``` -`compare` pairs sets by arm and id, following the manifests' `aliases` across renames, and lists -every set whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, +`compare` pairs rows by arm, set id and variant, following the manifests' `aliases` across +renames. Only the variants both runs ran on an arm are compared, so a `--variants bare` run against a full +one compares `bare`. It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free or time moved by more than the noise thresholds in `report.py` (`NOISE`). Those thresholds are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with A's saved binary and options. A set that moves again under the same binary is noise. A set that @@ -240,13 +295,11 @@ reproduces A was really changed by B. fix. For one fix, run the sets it touches with `--only`, plus the smoke tier. 4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a reference". Only a run with the machine to itself can be used to judge speed. -5. **Resolution on the XDS arms.** By default XDS's resolution range is forced on rugnux, so both - programs merge the same reflections and the statistics are like for like. This answers "is our - processing better than XDS's, reflection for reflection". `--unforced` leaves rugnux its own - cut, which answers "what does a user get". A change can win one and lose the other, so at - important moments run both. Never quote the forced run as what rugnux delivers. - Where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the manifest), rugnux gets - `-A` in both modes, so Friedel mates are merged separately on both sides. +5. **Resolution on the XDS arms.** The `xds` variant forces XDS's resolution range (and `-A` + where XDS kept Friedel mates apart), so both programs merge the same reflections and the + statistics are like for like. This answers "is our processing better than XDS's, reflection for + reflection". The `bare` variant answers "what does a user get". A change can win one and lose + the other, and every run has both. Never quote the `xds` variant as what rugnux delivers. ### The XDS reference d_min @@ -262,7 +315,7 @@ is significant, and that decides which case applies (`inputs.reference_dmin`): point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in 1/d² between shell centres. -The derived value is both the score reference and the forced range. The manifest keeps XDS's own +The derived value is both the score reference and the range the `xds` variant forces. The manifest keeps XDS's own limit as `dmin_xds` and the rule used as `dmin_rule`. XDS's pooled R_meas, CC1/2 and completeness still cover its whole range. diff --git a/tools/battery/battery.py b/tools/battery/battery.py index 7fd9f48d3..4d31afa80 100644 --- a/tools/battery/battery.py +++ b/tools/battery/battery.py @@ -12,7 +12,8 @@ Arms and where their truth comes from: Usage (README.md in this directory is the full guide): battery.py run --rugnux BIN [--label L] [--arm open,inhouse|private] [--only a,b | --tier smoke] - [--build-dir DIR] [--gpulock] [--threads N] [--unforced] [--baseline RUN|none] + [--variants bare,model,xds] [--build-dir DIR] [--gpulock] [--threads N] + [--model-check] [--baseline RUN|none] battery.py list runs under the runs root, with their state battery.py compare RUN_A RUN_B [--all] [--rerun-changed] battery.py report RUN --out DIR [--baseline RUN] re-render a run's report into DIR/.{md,html} @@ -24,9 +25,13 @@ Usage (README.md in this directory is the full guide): The site config (data roots, runs root, lock, gpulock, private manifest, baselines) is --site, else $JFJOCH_BATTERY_SITE, else site.json beside this script (not committed; site.example.json is). +Every set runs in each of its arm's VARIANTS (bare + model on the open arm, bare + xds on the XDS +arms), one straight after the other, so only the first reads the images from disk. + A run writes /__