# Rugnux validation battery

`battery.py` is the one way to run the Rugnux validation battery: run `rugnux` over a fixed list of
rotation datasets, score each result against a reference, and write a standard report. Two runs
of the same manifests with the same options give results you can compare set by set. Do not use
ad-hoc runner scripts instead. The old ones (`rugnux_vs_xds.py`, `battery_run.py`,
`battery2_serial.py`, `batstats.py`, ...) are retired because their numbers could not be compared
with each other.

[Measuring Rugnux](../../docs/RUGNUX_BENCHMARKING.md) is the overview for someone measuring on a new
machine: the protocol, the REFMAC check, the timing benchmark, how to take timings that compare, and
what to send back.

`test_score.py` covers the scoring rules that decide a verdict (stdlib `unittest`, no data and no
run needed): `cd tools/battery && python3 test_score.py`. `doc_check.py` checks the manifests
against the pages that document their datasets - `open.json` against
[External test data](../../docs/EXTERNAL_TEST_DATA.md), `still_manifest.json` against its stills
page, `inhouse.json` against the Arms table below - and exits non-zero on a mismatch (stdlib, no
data): `cd tools/battery && python3 doc_check.py`.

## Arms

| arm | datasets | reference | manifest |
|---|---|---|---|
| **open** | public PDB depositions of raw diffraction data, plus a few published small-molecule sets | the deposited space group, cell and resolution | `open.json` (committed) |
| **inhouse** | standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), small-molecule standards (aspirin, citric acid, HEPES, YAG, KDP, L-cystine), plus no-crystal controls | XDS, from the `CORRECT.LP` beside each dataset (the standard proteins also carry a reference model for R-free, reported only: `--model` below); for the small molecules the space group is the literature's (`ref_override`), as XDS reports only Sohncke groups; L-cystine, which XDS does not index, has only the literature reference | `inhouse.json` (committed) |
| **private** | user data | XDS, like inhouse | outside the repository; the local site config gives its path |

Scoring checks these things in order, and the first one that fails decides the verdict: did it
run, is the lattice right, is the symmetry right, and on the XDS arms, is the merge as good as
XDS's. The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio
must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is
scored with `sgequiv`. On the XDS arms only the point group is scored, because XDS never tests a
screw axis. On both, a group with the reference's Laue class and absences passes (below). A
no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported. `score.py`
has the details.

**A question the data could not answer is `unscored`, not a failure.** Two of these. The hand of an
enantiomorphic pair (P4_1 vs P4_3) is not in the intensities at all, and `sgequiv` already counts it
as a match. The other is a **screw axis on a row the sweep never recorded** - the axial row lies in
the spindle's blind cone, or outside the resolution range, so nothing was measured that could
confirm or refuse the screw. rugnux then offers the whole set in `SPACE_GROUP_ALTERNATIVES`, names
the open axis in `SPACE_GROUP_SCREW_UNDETERMINED`, and writes the member claiming no screw, because
a reflection file must carry one group. Such a set scores `unscored` / `screw_undetermined` rather
than `sym_screw`, on three conditions, all necessary: rugnux named the axis, the reference group is
among the ones it offered, and the only axis the two groups differ on is one it named. A screw
called wrongly where the row WAS measured meets none of them and stays a `sym_screw` failure - this
rule excuses an unanswerable question, never a wrong answer.

**A group the intensities cannot separate from the reference passes, as
`indistinguishable_alternative`.** Two space groups of one Laue class with exactly the same
systematic absences give the same merged intensities: acentric vs centric (P2 vs P2/m, Pc vs P2/c,
Pna21 vs Pnma) and the two acentric groups of one class (I41md vs I-42d). Only a model, or an
anomalous signal strong enough to break Friedel's law, tells them apart. rugnux must still write
one, and lists the others in `SPACE_GROUP_ALTERNATIVES`. `sgequiv` keys each group, in every
setting, on its Laue rotations (the operations and their negatives), its centring and its absences;
the row passes when some setting of the reference agrees with our group in our own setting, on
both arms, and its reason says whether rugnux listed the reference. This is a pass rather than
`unscored` because, unlike an unmeasured screw, everything was measured and the answer is as right
as the data allow; the separate cause keeps it countable. Being listed in
`SPACE_GROUP_ALTERNATIVES` is not by itself enough, because that list also carries the screw
alternatives of an undetermined axis, whose absences differ - those stay under the rule above.

**A row may accept more than one reference.** Where the deposition and our reduction disagree,
the disagreement is real, and no test available to us settles it - a knife-edge - the row lists
the other acceptable answer in `ref_alternatives` and passes if our answer matches any of its
references. This is not a statement that our answer is right; it says both descriptions are
defensible and that picking either one is acceptable. `ref` keeps the deposition verbatim, each
alternative must carry a `why` a reader can check (an alternative without one is a schema error,
not a silent pass), and the report counts these rows on their own - summary column `of which
alt`, its own bar-chart segment, and an **Accepted alternatives** section printing each reason -
so they are never folded into the ordinary passes. Prefer this to `unscored`, which drops the
row from scoring altogether and loses a test that still works, and to `ref_override`, which
asserts that the reference is wrong.

**Data quality is not scored.** A verdict is about the lattice and the symmetry only. Resolution,
CC1/2, R_meas (overall and lowest shell), completeness, multiplicity and ISa are reported beside
XDS's or the deposition's values, in the per-set tables and plots, for a human to judge. On the XDS
arms `cc_half_noise_ratio` = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) over XDS's own
range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as
XDS's would be with half of its observations.

**R_free does not compare between runs; `rmodel_shell_scaled` does.** rugnux scales the model to
the data with an overall factor and a symmetry-constrained anisotropic B, which can only bend one
way with resolution - deliberately, so that the maps of a batch stay comparable. Whatever the
amplitudes' own radial profile does that this shape cannot follow is reported as R. So two runs
whose merged amplitudes have different radial profiles have R_free values that differ for a reason
that has nothing to do with the data fitting the model: measured on one such pair, R_free moved a
median +0.042 over eleven sets where `phenix.model_vs_data`, which scales per shell, moved +0.002.
`rmodel_shell_scaled` is the same R with one free scale per resolution shell, over every reflection
rather than the free 5% (nothing is refined here, so the free set buys no cross-validation and only
costs precision): on that same pair its median was +0.001, and it still flagged the one set that
was a real regression. `radial_misfit` is the size of that per-shell rescale, the RMS of its
logarithm; when it moves, R_free beside it cannot be read. All three are reported for a human,
never scored.

**Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own
table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's
`CORRECT.LP` (`lowres_r_meas_ref`, stored in the manifest as `r_meas_low`), with the shells'
high-resolution limits. The reference-range table uses XDS's shell limits, so its lowest shell and
XDS's cover the same reflections; `lowres_r_meas_ratio` is the one over the other, plotted per set.

## One command per arm

Every set runs **once**, with one command per arm:

| arm (reference) | command |
|---|---|
| open (deposition) | `rugnux -o p --no-export-unmerged --model <deposited coordinates> <input>`; sets without a deposited model (small molecules, unpublished sets, or coordinates that cannot be downloaded) run the same command without `--model` |
| inhouse, private (XDS) | `rugnux -o p --no-export-unmerged [-A] --report-resolution <XDS d_min>,<XDS d_max> [--model <reference model>] <input>`; `--model` where the set names one (the in-house standard proteins, below) |

Neither `--model` nor `--report-resolution` changes the processing: the merged reflections, the
resolution cut and every decision are the same with and without them (verified bit for bit). So
the one run is both what a user gets and, through the extra report sections, the comparison with
the reference:

- **Own result** (every arm): rugnux's own cut and statistics, scored against the reference and
  plotted as d_min(rugnux) / d_min(reference). This is what a user gets.
- **`--model`** (open arm): the deposited coordinates (`<entry>.cif.gz`) and the published R-free
  come from the RCSB and are cached in the site's `pdb_cache` (default `/data/battery/pdb_cache`),
  so the first run of a new entry needs network access. The entry is the set id up to the first
  `_`, so `6h2p_native` uses `6h2p`. The row records MODEL_FIT, R_FREE, R_WORK and CC_MODEL as
  **trend fields**: rugnux's R-free is a rigid-body placement scored on rugnux's own free set, not
  a refinement against the depositor's, so the ratio to the published R-free is not a like-for-like
  score; watch how it moves between runs. The comparable number is the REFMAC check (below).
  A model that fits may put its enantiomorph on the space-group label; the scorer reads the
  data's own determination instead (`SOHNCKE_SPACE_GROUP` where `SPACE_GROUP_ENANTIOMORPH` is
  `ASSUMED_FROM_MODEL`) and keeps the label in `sg_label`.
- **`--model`** (in-house standard proteins): a set may name the PDB entry of its crystal form in
  its `model` field - or an mmCIF file beside its data, for a crystal form with no deposited entry
  (`rfree_deposited` is then the file's own `_refine` R-free); it is fetched and cached like an
  open-arm entry and passed to `--model`, so
  the row carries the same model fields (`rfree`, `rmodel_shell_scaled`, `rfree_deposited`,
  `rfree_ratio`, `anom_sigma`, ...) - quality on an external yardstick beside the XDS statistics.
  They are reported, never scored: the arm is scored against XDS exactly as without them. One
  entry per crystal form, chosen by cell and space group and confirmed by `rugnux --model` on our
  sets (R-free of the placed model, not refined):

  | sets | model | why |
  |---|---|---|
  | `lyso*` (P4₃2₁2) | 4AXT | hen egg-white lysozyme, 1.1 Å, 100 K, a 78.2 c 37.5 - the middle of our sets' cells (76.9-78.4 / 37.0-37.9); rugnux re-fractionalises the model into the data cell, so the R-free of the placed model rises by a few points per % of cell difference (6G8A, a 78.9 c 36.9: R-free +0.04 on `lyso_x06da_ref`). No ions in the model, so `anom_sigma` is the sulfurs' |
  | `thau*` (P4₁2₁2) | 5A47 | thaumatin from *T. daniellii*, 1.2 Å, a 57.9 c 150.6 (6G89, 2.4 Å: R-free +0.06) |
  | `insu_I*` (I2₁3) | 2BN3 | bovine insulin, cubic, 1.4 Å, 100 K, a 77.9 (9INS, pig, a 78.9, no R-free deposited: R-free +0.07) |
  | `insu_H*` (H3) | 4E7T | bovine T6 2Zn insulin, 1.4 Å, a 81.0 c 33.5 (1MSO, human: R-free +0.08 - our insulin is bovine) |
  | `cytc*` (P3₂21) | `reference_model.cif` beside each set's data | bovine heart cytochrome c in a form with no deposited entry: P3₂21, three molecules in the asymmetric unit (two related by a pseudo-translation 0.30 0.70 0.33), c 86.0-88.6 between crystals. Our own model, placed by molecular replacement with 6FF5 and refined against each set's XDS data with rugnux's free flags - one per set, because the 3% spread in c costs a single model ~5 points of R-free in the other crystals. The data cannot tell the hand: the model's settles it. `rfree_deposited` is the R-free of that refinement. `cytc_x10sa_mr` (c 86.5) started from `cytc_x10sa`'s model instead of a new molecular replacement; XDS indexed it in the other setting of the 321 ambiguity, so its data were reindexed -h,-k,l for the refinement |
  | `myob*` (P2₁) | 4DC8 | horse heart myoglobin (b 28.8 Å; sperm whale's P2₁ form has b 30.8), met form with sulfate, 1.5 Å, 100 K, a 35.4 b 28.8 c 63.1 β 106.1 - our setting; the unexposed control of its deposition, so no ligand or modified heme (5ZZE, same form, R-free 0.235 vs 0.233) |

- **`--report-resolution`** (XDS arms): a second statistics table from the same merge, binned over
  XDS's own range (`dmin_xds` and XDS's d_max as given, 999 included, so it covers exactly what
  XDS's CORRECT.LP totals cover), and `-A` where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the
  manifest), so the two tables count the same way. Its REFRES_* numbers are shown beside XDS's
  CORRECT.LP totals: completeness, multiplicity, R_meas, the lowest shell's R_meas, CC1/2 and ISa
  (I/sigma is shown; the manifest holds no XDS value for it). Only CC1/2 is scored (above). Where rugnux's own cut is coarser than the reference
  (`REFRES_SHELLS_PAST_LIMIT` > 0), the shells past it are not merged, so the completeness is read
  as **coverage** of the reference range, not as a quality regression, and the other numbers are
  over the shells rugnux reached.

**Timing.** Each set runs once, so every time includes reading the images from disk (unless they
were still in the page cache from before the run).
The in-house standard proteins run with `--model` since rc174, so their times include placing
the model; an earlier run (rc173 and before) did not, and a time delta on that arm against one is
partly that, not a change of speed. Compare in-house times between runs on the same side of it.

The private arm is never mixed with the others. A private run gets its own run directory
(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a
public report or be written into the repository. Its names, paths and values never go into a
commit, an issue or a public report.

## Getting the data

Each manifest row gives an `input` path relative to that arm's **data root**. The data root is
set in the site config, and the first directory of each input is the dataset's directory:

```
<open data root>/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5
<inhouse data root>/lyso_ref/hewl_35_001_master.h5
```

To move the battery to another machine, copy the two data roots and point the new site config at
the copies. The paths inside the manifests do not change.

- **open**: every dataset is listed with its source repository and dataset DOI in
  [`docs/EXTERNAL_TEST_DATA.md`](../../docs/EXTERNAL_TEST_DATA.md). The sources are IRRMC
  (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what
  each archive holds. Please cite those DOIs. At PSI the data root is `/home/data/open`, a
  symlink to `/home/data/nonsls/raw`. 42 of the dataset directories in it are themselves symlinks
  into `/data/scout_staging2`, so **copy with symlinks followed**: `rsync -aL` or `cp -rL`.
  A plain `rsync -a` copies dangling links. The open arm is about 1.7 TB.
- **inhouse**: these are our own measurements and cannot be downloaded publicly. Ask the
  maintainers for a copy. Each dataset directory holds the images and the XDS output
  (`CORRECT.LP`, `XDS_ASCII.HKL`, ...) that the reference comes from. At PSI the data root is
  `/home/data/inhouse` (about 130 GB). **Leave out `.master_backups_kek`** when copying it
  (`rsync -a --exclude .master_backups_kek`): it holds backups of private masters, not battery data.
- **private**: its data root and manifest are named only in the local site config.

## Prerequisites

- Python 3.9 or newer, with `gemmi` (scoring). `h5py` is needed only by `discover`.
- A `rugnux` binary. Build it with the flags CI uses (`-march=x86-64-v3`, see `CLAUDE.md`),
  because CPU-bound results and timings differ between builds. The runner records the build
  flags from the `CMakeCache.txt` above the binary, or from the tree given with `--build-dir`.
- XDS is not needed. The XDS references are already in the manifests (`refs` re-reads them from
  `CORRECT.LP`).

## Site config

Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an
optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it
in this order:

1. `--site PATH`
2. `$JFJOCH_BATTERY_SITE`
3. `tools/battery/site.json` (git-ignored)

`site.example.json` is the committed template. Copy it and fill in your paths. The local file is
kept out of git because it names the private manifest. At PSI the config is
`/data/battery/site.json`: link it (`ln -s /data/battery/site.json tools/battery/site.json`) or
export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.

```json
{
 "runs_root": "/data/battery/runs",
 "lock_file": "/data/battery/full_run.lock",
 "gpulock": "/data/tmp/gpulock",
 "baseline": "/data/battery/runs/<a blessed full run>",
 "baseline_private": null,
 "pdb_cache": "/data/battery/pdb_cache",
 "cod_cache": "/data/battery/cod_cache",
 "shelxl": "/opt/xtal/ccp4-9/bin/shelxl",
 "arms": {
  "open":    {"manifest": "open.json",    "reference": "deposition", "data_root": "/home/data/open"},
  "inhouse": {"manifest": "inhouse.json", "reference": "xds",        "data_root": "/home/data/inhouse"},
  "private": {"manifest": "/path/to/private.json", "reference": "xds",
              "data_root": "/path/to/private/data", "private": true}
 }
}
```

A relative `manifest` path is resolved against `tools/battery/`; every other relative path (runs root,
lock file, caches, data roots) against the site config's own directory. `gpulock` is a command that
`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of
the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where
the deposited models and their metadata are cached for `--model` and the REFMAC check; an entry
committed in `models/pdb/` (the timing benchmark's) is read from there instead. `baseline_timing`
(optional) names, per benchmark, the timing run (`battery.py timing`, below) a new timing run is
reported against: `{"standard": RUN, "inhouse": RUN}`. For the open and in-house arms `manifest` and
`reference` may be left out (they default to the values above), so an arm can be just its `data_root`.
`cod_cache` (optional, default `/data/battery/cod_cache`) caches the small molecules' COD structures,
and `shelxl` (optional; else `shelxl` on `PATH`, e.g. from CCP4) is the SHELXL the small-molecule
check calls - without one the check is skipped and says so.

## Running

```
B=tools/battery/battery.py
R=build/rugnux/rugnux

$B run --rugnux $R                                  # full battery: open + inhouse
$B run --rugnux $R --tier smoke                     # 21 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --tier ci                        # 29 sets every recent release passed (CI, below), plus the wrong-model null
$B nullmodel --rugnux $R                            # the wrong-model null on its own (CI's second step)
$B run --rugnux $R --only 5reo,lyso_ref             # named sets
$B run --rugnux $R --arm private                    # the private arm, on its own
$B timing --rugnux $R                               # the timing benchmark: 3 sets placed by MR (below)
```

Useful options:

- `--label` names the run. By default the label is the rugnux version.
- `--threads N` passes `-N` to rugnux.
- `--gpulock` queues every rugnux behind the site's GPU wrapper.
- `--extra "<args>"` adds rugnux arguments to every set.
- `--baseline RUN|none` sets the run the report is compared against.
- `--model-check` adds the REFMAC check to the open arm's rows (off by default, see below).

A run prints one line per set as it goes and ends by writing its report and printing a table, one
line per set, each column rugnux / reference: the space group with ✓ where the set passed by the
scorer's rule and ✗ where it failed (lattice, symmetry, crash, no input), d_min, ISa, CC1/2 overall
and in the highest shell, R_meas of the lowest shell, R-free of the model (rugnux's placement /
the model's published) and rugnux's wall time. On the open arm only the space group, d_min and
R-free have a reference. Every crashed set is listed again under the table (`CRASHED  <set>: why`).
`$B table RUN` prints it again for any finished run.

**Crashes.** A run is a `crash` - its own verdict, cause `crash` - when rugnux was killed by a
signal, or its log or terminal output has `Processing failed`, a device fault (`unspecified launch
failure`, `illegal memory access` and the like - also one rugnux handled and went on from: that is a
defect, not a property of the data; a planned fallback such as a buffer that did not fit is not one),
`terminate called`, `Segmentation fault`, or a compute-sanitizer `ERROR SUMMARY` that is not zero
(`score.crash_note`). It is decided before anything else: a crash fails the set whatever its
report says, a no-crystal control included (their normal `No crystal lattice found`, exit 1, is not
a crash). A crash counts as a failure in every pass rate, gets its own `CRASHED` section at the top
of the report, and is read again from the log when an older run is reported or compared, so an old
run's crash shows as one too.

**Tiers** are marked in the manifests (`"tiers": {"<tier>": "<why this set>"}`):

- **smoke** covers the readers, the symmetry classes and some known hard cases; several of its
  sets fail on purpose.
- **ci** is the routine regression check: 29 sets that passed with the same space group and steady
  d_min and ISa in every full run of rc174 and rc175, with no warning that questions the lattice or
  the group (supercell, harmonic, twinning, untested centring). It spans HDF5, miniCBF, cbf.gz,
  marCCD (both formats) and SMV; every crystal system, centred and high-symmetry lattices, small
  molecules, anomalous and long-wavelength data, pink beam, a twinnable P31 and two large sets
  (8qaw, about 2 minutes; the 4nwv virus capsid, about 1 minute) - about 11 minutes of rugnux in
  all on the RTX 5080. A set that starts failing for a known, accepted reason leaves the tier; one that is merely slow stays. KDP joined
  later than the rest: it passes as an `indistinguishable_alternative` (I41md for I-42d), and its
  twinning warning is most likely a false alarm from four heavy atoms in the cell. The two no-crystal
  controls (nothing_1, nothing_2) joined once the no-crystal test got its own significance (1e-9):
  they pass by reporting no lattice, about seven decades short of that level.

**Exit status.** `run` exits with status 1 when any set crashed, in or out of the ci tier, or when
a ci-tier set in the run failed - a wrong lattice or space group, a timeout or a missing input - or
when a pair of the wrong-model null (below) failed, and names those sets. This holds in any run that includes them (`--tier ci`, a full run, `--only` naming
one), so a red CI job means the same thing whichever run it started. Nothing else fails a run: other
sets' failures and every metric are reported, not judged. CI runs the two as separate steps -
`run --tier ci --no-nullmodel`, then `nullmodel` - so the job page shows which of them failed.

### The wrong-model null (with every ci-tier run)

`--model` must not change the processing, and a model that does not belong to the crystal must be
recognised as not belonging. Every `--tier ci` run (and any run given `--nullmodel`; not one given
`--no-nullmodel`) therefore also runs the sets of `nullmodel.json` once without `--model` and once with each of five WRONG models,
and scores every wrong-model run against the no-model run of the same set (`nullmodel.py`):

- **(a) data**: `p.hkl` byte for byte; the `p.mtz` reflections value for value (space group, cell,
  columns, every number - the file differs by the command line in its history); `p.cif` without its
  command line.
- **(b) decisions**: every `KEY=` line of `p_report.txt` outside MODEL VALIDATION - space group,
  `SOHNCKE_SPACE_GROUP` and `SPACE_GROUP_ENANTIOMORPH`, cell, resolution cut, twinning, warnings - with
  only `DATE`, `COMMAND_LINE` and `WALL_TIME` left out.
- **(c) a proper null**: `MODEL_FIT` is not `ACCEPTED` (`REJECTED`: the fit is no better than the same
  model in random orientations), molecular replacement - which runs wherever the rigid body does not
  place a model - does not place it either (`MODEL_PLACEMENT` is not `MR`),
  `MODEL_DECISIONS_TAKEN` is `NONE` with no enantiomorph adopted,
  `CC_MODEL_CONFIRMED_TO_D_MIN` is `NONE` (measured against its empirical null, see
  docs/RUGNUX_REPORT.md), and `R_FREE` sits where a random structure's does -
  0.586 for acentric reflections, 0.828 for centric (Wilson 1950); the scale, anisotropic B, bulk
  solvent and rigid body fitted on the working set can take only a few points off that, so the
  test is `R_FREE >= 0.50`.

The sets are five of the ci tier: 8u0i (P4₃2₁2, all-beta; rugnux writes the other enantiomorph
without a model and its own model's hand with it), 6iu5 (P3₁, eight helices; enantiomorphic and with
a merohedral twin law, so a model can claim both the hand and the indexing), 9khr (P2₁2₁2₁), 9b22
(P2₁, two chains) and 6pxc (I222). The wrong models are the deposited coordinates of OTHER open-arm
entries, so they are already in the site's `pdb_cache` and the step needs no data of its own. Each is
a real protein with its own B-factors in its own deposited cell, of similar size (residues of the
asymmetric unit within about 30% of the set's own model), unrelated in sequence (identities over the
shorter chain under 25%, local BLOSUM62 alignment of chains of 30 residues or more) and in fold
(gesamt Q-score at most 0.20 between the longest chains; a homologue sits above 0.5). Each set has
one hard-ish model (the same secondary-structure class and size), one in the same space group or a
similar cell where the open arm has one, and the two enantiomorphic sets a model in the other
enantiomorph - the one decision a model may write into the reflection files. 7l84 on 8u0i is the only
model outside the size window (+50%): no smaller open-arm entry is in P4₃2₁2. `nullmodel.json` gives
every model's size, identity, Q-score and reason. 9jq9 is NOT usable as a wrong model for 9khr: it
is the same protein.

The pairs go to `nullmodel_results.json` in the run directory and to a **Wrong-model null** section
at the top of the report; the sets' own rows in `results.json` are those of the run without it.
`battery.py nullmodel` runs the step alone, in a run directory of its own whose report is that section,
and exits with status 1 when a pair failed. It
adds 30 rugnux runs, about 6 minutes of rugnux on the RTX 5080 (measured: 6.1 min of WALL_TIME).

### The timing benchmark

`battery.py timing` is the run to repeat on every machine rugnux is measured on (the workstation, the
4x T4 server, a GB10, a Windows laptop, a Mac): a few sets, each started from a model it has to place
by molecular replacement, so one run times the whole pipeline - reading, indexing, integration,
scaling, the space-group search, MR and model validation - and says whether its answers are right.

```
$B timing --rugnux $R [--benchmark standard|inhouse] [--gpulock] [--threads N] [--label L] [--baseline RUN|none]
```

`timing.json` holds two benchmarks. **`standard`** (the default) needs only open data: three EIGER
16M rotation sets of ordinary size and quality, the kind of crystallography rugnux is mostly run on,
and the 4NWV virus capsid as the large case.

| set | start model (`--model`) | reference model it is checked against | expected MR group |
|---|---|---|---|
| `9zlo` | AlphaFold DB AF-P17090-F1 v6 (UreE, *Proteus mirabilis*, UniProt P17090), `models/`; 2 copies to place | 9ZLO as deposited, `models/pdb/` | P 21 21 21 |
| `9s02` | AlphaFold DB AF-P32322-F1 v6 (PYCR1, human, UniProt P32322; the entry deposits isoform 3), `models/`; 5 copies | 9S02, `models/pdb/` | P 21 21 2 |
| `9gjx` | AlphaFold DB AF-Q65MG6-F1 v6 (nitroreductase, *Bacillus licheniformis*, UniProt Q65MG6), `models/`; 6 copies | 9GJX, `models/pdb/` | P 1 21 1 |
| `4nwv` | the deposited 4NWV (`models/pdb/`), strict NCS expanded (45 chains), turned 30 deg about (0.36, 0.48, 0.80) through its centroid and shifted (12, -18, 25) A | 4NWV, `models/pdb/` | I 2 2 2 |

**`inhouse`** is the benchmark as it was first set up, kept so that runs made with it stay comparable:
two in-house standard proteins and the same 4NWV.

| set | start model (`--model`) | reference model it is checked against | expected MR group |
|---|---|---|---|
| `myob_x10sa` | AlphaFold DB AF-P68082-F1 v6 (horse myoglobin, UniProt P68082), `models/` | 4DC8, `models/pdb/` | P 1 21 1 |
| `cytc_x10sa` | AlphaFold DB AF-P62894-F1 v6 (bovine cytochrome c, UniProt P62894), `models/` | `reference_model.cif` beside the data | P 32 2 1 |
| `4nwv` | as above | as above | I 2 2 2 |

**Everything but the raw data is in the repository.** The predictions are committed in `models/`
exactly as the AlphaFold DB serves them, each with its predicted aligned error under its own name,
which rugnux finds beside the model and cuts the prediction into rigid domains by (CC BY 4.0,
`models/LICENSE.txt`). The deposited entries the sets are checked against, and 4NWV's start model, are
committed in `models/pdb/` as the compressed mmCIF the RCSB serves (PDB data are CC0; both in
`THIRD_PARTY_NOTICES.md`). `model_check.deposition` reads an entry from `models/pdb/` before it looks
at the site's `pdb_cache`, and its refinement record (R-free, d_min) comes from that file itself, so a
timing run needs no `pdb_cache` and no network. The moved 4NWV (10 MB) is not committed: the runner
makes it from `models/pdb/4nwv.cif.gz` into the set's work directory as `4nwv_moved.cif.gz`, with the
transform in `timing.json`. That is far outside what the rigid body corrects, so the model is rejected
as read and MR places it. Each set otherwise runs the battery's own command for its arm (above).

**On another machine** only the dataset directories have to be copied, under an open data root:

| directory | size | source (please cite) | master file / first image |
|---|---:|---|---|
| `open/9zlo` | 11 GB | Zenodo [10.5281/zenodo.18652652](https://doi.org/10.5281/zenodo.18652652) | `9zlo/ASP0887_09_0114_master.h5` |
| `open/9s02` | 11 GB | MX Raw Data Repository [10.60884/NRNGS4](https://doi.org/10.60884/NRNGS4) | `9s02/PYCR1-D11_1_master.h5` |
| `open/9gjx` | 37 GB | IRRMC [10.18430/M39GJX](https://doi.org/10.18430/M39GJX) | `9gjx/BlNTR_9gjx/data/SC6_1_master.h5` |
| `open/4nwv` | 16 GB | SBGrid Data Bank [10.15785/sbgrid/9](https://doi.org/10.15785/sbgrid/9) | `4nwv/sbgrid9/formal.001` |

75 GB in all (`docs/EXTERNAL_TEST_DATA.md` says what each archive holds; the paths are `open.json`'s
`input`s). For `--benchmark inhouse` add `inhouse/myob_x10sa` and `inhouse/cytc_x10sa` (from the
maintainers) and keep `open/4nwv`. The site config then needs only where runs go and the data roots;
`site.timing.example.json` is that config, and every other key is optional (the two committed arms
default their `manifest` and `reference`):

```json
{
 "runs_root": "/path/to/runs",
 "arms": {
  "open": {"data_root": "/path/to/open"}
 }
}
```

```
python3 tools/battery/battery.py --site /path/to/site.json timing --rugnux /path/to/rugnux
```

Add `"gpulock"` (a command rugnux is prefixed with) only to use `--gpulock`, and `"inhouse":
{"data_root": ...}` for the in-house benchmark. Python 3.9+ with `gemmi` and `numpy` (`pip install
gemmi numpy`) is all the runner needs. macOS runs it as Linux does. Windows has no process groups and
no `flock`, which a timing run does not need; run it as `python -X utf8 tools\battery\battery.py ...`
(the report and the console table are UTF-8), with `--rugnux` naming `rugnux.exe` (the run directory
keeps its own copy, so any DLL it needs must be found on `PATH`). Relative paths in the site config are
taken from the config's own directory ([Measuring Rugnux](../../docs/RUGNUX_BENCHMARKING.md#setting-up-a-machine)).

A set passes when every check holds (`timing.py`; thresholds in `timing.json`):

| check | passes when |
|---|---|
| `space_group` | the battery's own verdict passes: lattice and space group against the reference, by the scorer's rules (the hand excepted) |
| `placed` | `MODEL_PLACEMENT` is `MR`: molecular replacement placed the model, and the R-factors, maps and model written are of that placement |
| `mr_group` | `MR_SPACE_GROUP` is the expected group (tables above), hand included: a model decides the hand |
| `r_free` | `R_FREE` at most 0.50 for the predictions and 0.35 for 4NWV. Measured: 0.468, 0.371 and 0.384 (9zlo, 9s02, 9gjx), 0.44 (myoglobin), 0.315 (4NWV after MR); a random structure 0.586 |
| `confirmed` | `CC_MODEL_CONFIRMED_TO_D_MIN` is not `NONE`: some shell's CC(model, data) clears rugnux's empirical null |
| `placement_cc` | the placed model `p_model.pdb` has the reference model's intensities: the correlation of the two models' \|F\|^2, each shell normalised to its mean, 10-4 A, the best over the changes of basis the two lattices allow, is at least 0.30. \|F\| is the same for every origin and every symmetry copy, so this needs no superposition. Measured: 0.61, 0.78 and 0.67 (9zlo, 9s02, 9gjx), 0.55 (myoglobin, AlphaFold against 4DC8: 0.9 A apart as models), 0.90 (cytochrome c), 0.96 (4NWV); a wrong placement 0.01-0.04, the cytochrome c placement relabelled to the other hand 0.18 |

`placement_cc` is independent of rugnux's own scores and is what catches a placement rugnux reports
as right: the myoglobin placement turned by 30 deg in a copy of its work directory, with rugnux's
report unchanged, fails it at 0.035, and the moved 4NWV start model at 0.011. A wrong model
(the cytochrome c prediction on the myoglobin data) is not placed by rugnux and fails `placed`.
The three standard sets were chosen from the 2026-10-16 AlphaFold MR scan of the open arm, which placed
all three with every one of these checks passing.

The run is an ordinary run directory (`manifest.json`, `results.json`, `report.md`), so `list`,
`table`, `compare` and `keep` work on it; its default label is `timing-<benchmark>`. Each row carries
`bench_verdict`, `bench_reason`, `bench_checks` (`{check: [ok, value, limit]}`), `mr_seconds`,
`mr_decision` and `steps` (seconds per rugnux step, from the `TIMING step` lines of `p_log.txt`); the
report opens with a **Timing benchmark** section (the checks per set, then the step times), and the
console prints both. `wall_s` is rugnux's own WALL_TIME. The run exits with status 1 when a set failed
a check or crashed. Its report is compared against the site's `baseline_timing` for that benchmark
(`{"standard": RUN, "inhouse": RUN}`; none by default). As everywhere, `--gpulock` marks the timing
"NOT a reference": run it with the machine to itself when the numbers are the point.

Measured on the workstation (RTX 5080, rc176 bc191bb80, 2026-10-19, machine to itself): for `standard`
9zlo 14 s, 9s02 25 s, 9gjx 17 s and 4NWV 81 s (of which MR 17 s and model validation 7 s); with
`--no-gpu` 125, 143, 140 and 516 s; for `inhouse` myoglobin 9 s, cytochrome c 13 s, 4NWV 77 s. 4NWV
is the largest by far in memory too (about 16 GB peak resident on rc175). Every platform measured on
rc176 - workstation, the 4x T4 server, H200/L4/A100 cluster nodes, a DGX Spark, a Windows laptop and a
Mac mini - is in `results/timing_platforms/rugnux_timing_platforms_bc191b.md`. Cytochrome c is the hard one: the prediction is a monomer and the
crystal holds three copies related by a pseudo-translation, and MR places three copies in both P 32 2 1
(R-free 0.427) and P 31 2 1 (0.437), under the 0.03 R-free margin. It was `MR_DECISION= UNDECIDED`
when the benchmark was added; with as many copies in both groups, MR now compares them on every
reflection (paired residuals, [RUGNUX_REPORT ▸ Molecular replacement](../../docs/RUGNUX_REPORT.md#molecular-replacement))
and decides P 32 2 1.

### Stress: hunting crashes that do not reproduce

One rugnux process runs several GPU users at once (image workers on every card, the space-group
arms on a cloned scaling engine, a speculative probe pass beside the merge, model validation), so a
race between them shows up once in a few hundred runs - and a set that crashed once and passed on
three reruns is a defect, not noise. `stress` runs sets again and again and asks only whether any
run crashed (same rule as above); it exits 1 if one did, and prints per set how many runs crashed,
timed out and how many distinct `p.hkl` they wrote (more than one is non-determinism, reported, not
failed). It writes `<runs_root>/stress/<stamp>_<git>_<label>/` (binary copy, `stress.json`, one work
directory per run) and never touches the run list, a baseline or a report.

```
G=/data/battery/gpulock.sh   # take the GPU once for the whole stress run, not per rugnux
$G $B stress --rugnux build/rugnux/rugnux --tier ci --repeat 3                  # ci tier, 3 times
$G $B stress --rugnux build/rugnux/rugnux --tier ci --concurrent 2              # two sets side by side on the card
$G $B stress --rugnux build/rugnux/rugnux --only 5reo,11if --sanitizer memcheck --extra "-e 60 -N 8"
$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --extra "-e 30 -N 4" --timeout 36000
```

- `--repeat N` runs every set N times; `--concurrent K` keeps K rugnux processes running at once,
  interleaving the sets so different sets share the card. Two processes do not share a CUDA context,
  so this does not create a race between them - it changes every process's timing, which is what
  makes a race inside one process show. A GPU out-of-memory failure under `--concurrent` means the
  card is too small for K of those sets, not a defect.
- `--sanitizer memcheck|racecheck|synccheck|initcheck` runs every rugnux under NVIDIA's
  compute-sanitizer (API errors rugnux handles itself are not reported). It is slow - the whole of
  5reo under memcheck was still in its first pass after ten minutes - so cut the sweep and the
  threads with `--extra "-e 60 -N 8"` (fewer images still runs every GPU engine) on the smoke
  tier's fastest sets, give a long `--timeout`, and remember it holds the GPU lock all that time.
- `--launch-blocking` sets `CUDA_LAUNCH_BLOCKING=1`, so a kernel's fault is reported by that kernel's
  own launch and not by whichever thread next touched the device - the message then names the culprit.

Before a release: the ci tier with `--repeat 3 --concurrent 2`, and memcheck on two small sets.
After a device fault, `journalctl -k | grep Xid` says which engine of the card faulted (a copy
engine `CE`, or the graphics/compute engine for a kernel) and at what address - evidence the
rugnux log, which names only the first thread to notice, does not have.

**CI.** The `battery` job of `.gitea/workflows/build_and_test.yml` builds rugnux with the CI flags
and runs the ci tier on every push, after that run's unit tests (they share the GPU), on the machine
that holds the data; by hand (Run workflow, `battery_subset`; empty: the full open + inhouse battery)
any subset. A release run skips it. The job keeps report.md and the log as its artifact; its site
config names no private arm.

**Full runs.** A run without `--only` or `--tier` takes the site lock (`flock`, non-blocking), so
two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and
everything else on the machine competes with it.

### The REFMAC model check (open arm, opt-in)

rugnux's own `--model` R-free is a trend number. `--model-check` adds an independent one, comparable
with the deposition, that needs CCP4: `tools/battery/model_check.py` scores each open-arm merge
(`p.mtz`) against the deposited model, unmodified (ligands, waters and hydrogens kept; only atoms of
unknown element, UNX, are left out because REFMAC stops on them): REFMAC's rigid-body mode,
R-factors taken before any shift, the data reindexed into the model's setting and reduced to the
deposited space group. The model is never refined. The depositor's own structure factors go through
the same setting choice (they can sit in the other branch of a merohedral ambiguity) and are scored
by the same protocol.

Both data sets are scored on the same reflections: the hkl present in both, at `d_min_used` =
max(our d_min, the deposited d_min), i.e. the lower of the two resolutions, so neither data set is
scored past what the other reaches and a cut either one made (low-resolution limit, anisotropic
cut) applies to both. Bulk solvent and the overall anisotropic scale are REFMAC's defaults, fitted
to each data set. The four R-free fields, all of the same deposited model:

| field | data | free set | reads as |
|---|---|---|---|
| `refmac_rfree` | ours | ours | close to a work R: most of our free reflections were work reflections to the depositor's refinement |
| `refmac_rfree_depflags` | ours | the depositor's | **the fair number for our data** |
| `refmac_rfree_depdata` | the depositor's | the depositor's | the baseline: same model, same protocol, the data the model was refined against |
| `refmac_rfree_ratio` | | | `refmac_rfree_depflags / refmac_rfree_depdata`: **the headline**; below 1 = our data fit the model better; the report plots it |

`refmac_rwork` is R on our own work reflections. The depositor-data numbers carry a home
advantage (the model was refined against those data), and `refmac_rfree_depdata` is not the
published R-free either: different program (REFMAC vs e.g. PHENIX, whose resolution-binned scaling
absorbs a non-Wilson falloff that REFMAC's overall scale cannot) and no refinement.
The depositor's amplitudes are `F_meas`, else the mean of F(+)/F(-); where only intensities were
deposited they are converted by French-Wilson (ctruncate), as our own amplitudes are.
An entry that declares twinning (`_pdbx_reflns_twin`) is scored with REFMAC's twin refinement for
both data sets, since its published R-free is a twinned one; the untwinned numbers are kept in
`refmac_untwinned`. The ratio is null when the entry deposited no structure factors with a free
set (`refmac_reason` says why). The runner records `refmac_rfree`, `refmac_rwork`,
`refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_twin`,
`refmac_untwinned`, `refmac_status` and `refmac_reason`. Small-molecule sets are skipped.
`model_check.py <p.mtz> <pdb id> --workdir DIR` runs it on one merge and prints every field
(setting, reindexing operators of both data sets, what the depositor's amplitudes were made from).
A `--model-check` run records the REFMAC version in its manifest (`refmac`), as does every recheck,
and the report lists it.

### Rechecking an existing run

`battery.py recheck RUN [--only a,b] [--jobs N]` reruns the model check and the depositor-data
comparison on every open-arm row of a finished run, from the run's own `p.mtz` (CPU only; rugnux is
not run again), so numbers made by an older `model_check.py` / `depdata_check.py` can be brought up
to date without reprocessing. It updates the `refmac_*` and `dep_*` fields of `results.json` in
place and nothing else (verdicts stay as scored). The previous files stay beside the new ones:
`results.json.pre-recheck-<stamp>`, `report.{md,html}.pre-recheck-<stamp>` and each set's
`model_check.pre-recheck-<stamp>/`. The report is re-rendered, and `manifest.json` gains an entry
in its `rechecks` list - `date`, `end`, `runner_git`, `runner_dirty` (the tools version that made
the new numbers), `sets`, `fields`, `previous_results` and `command` - so which version of the
checks produced which figures stays auditable. The run directory is made read-only again.

### The depositor-data comparison (open arm, always on)

`tools/battery/depdata_check.py` compares our merge with the depositor's own data without CCP4:
per resolution shell, the rank correlation of our IMEAN with |Fc|^2 minus that of the deposited
intensities (or amplitudes squared), on the reflections both carry, with |Fc| from the deposited
model as it is (gemmi, no bulk solvent, no refinement), the data carried into the model's setting
as the REFMAC check does it. The model was refined against the depositor's data, so the comparison
has a home advantage for the deposition; around -0.01 is typical, and a set far below that merged
worse than the deposition at those resolutions. It runs on every open-arm set with a model
(seconds to a minute, gemmi only; the `-sf.cif` is cached in `pdb_cache` beside the coordinates),
and `report` computes it for an older run that lacks it. It is reported, never scored. Row keys:
`dep_cc_delta_all` (all common reflections at d < 4 A), `dep_cc_delta_outer` (mean of the two
outermost of eight equal-count shells), `dep_beyond_cc` and `dep_beyond_d` (the correlation of our
reflections with |Fc|^2 in the outermost of up to three shells past the deposited data's limit,
and that shell's limit: the model never saw these, so clearly above zero is signal), plus
`dep_kind` (`I` or `F`), `dep_d_min` (the deposited data's limit), `dep_n_common`, `dep_status`
and `dep_reason`.

### The SHELXL check (small molecules, always on)

`tools/battery/shelx_check.py` refines a small-molecule set's published structure against our
`p.hkl` with SHELXL, the ground truth a chemical crystallographer would use. The structure is the
row's `cod` entry of the Crystallography Open Database (downloaded once into `cod_cache`). One fixed
recipe, so two runs compare: the data reindexed into the COD setting (every integer matrix with
entries -1..1 and det +1 that maps our cell onto the COD cell within 3% is tried with 6 cycles, the
lowest R1 wins), non-H anisotropic, H fixed at the COD positions with the COD U, EXTI refined,
`MERG 2` (SHELXL merges equivalents itself, so a merged or an unmerged HKLF 4 file both work),
weights from SHELXL's own suggestion over three rounds of 12 cycles. It also scores the data
against |Fc| of the COD model as published (gemmi, one scale, nothing refined), which no refinement
can absorb. `report` and `compare` compute it for an older run that lacks it (temporary directory;
about 5-40 s a set). Reported, never scored. Row keys: `shx_r1` (R1 for Fo > 4 sigma, `shx_n4` of
them), `shx_r1_all`, `shx_wr2`, `shx_goof`, `shx_exti`, `shx_wght_a`/`shx_wght_b` (the suggested
weights; a near SHELXL's 0.2 ceiling means sigmas or strong reflections are off), `shx_peak`/
`shx_hole` (residual density, e/A^3), `shx_rint`/`shx_rsigma` (as SHELXL prints them; R(int) only
means something for unmerged data), `shx_k_top` (<Fo2>/<Fc2> of the strongest analysis-of-variance
bin; below 1: strong reflections read low), `shx_fixed_r1`, `shx_cod`, `shx_status`, `shx_reason`.
`python3 shelx_check.py <set work dir> <COD id>` runs it on one finished set.

### Symmetry evidence from external programs (open arm, by hand)

Where our space group and the deposition's differ, an independent assessment by established
programs is kept beside the manifest in `symmetry/<id>/`, and the row names it in
`symmetry_evidence`. Nothing reads it for scoring; it is what a reader of the battery, or of a
`ref_alternatives` `why`, can check. Its purpose is to decide which answers the battery accepts,
not to judge the depositions: an exact pseudo-symmetry, a twin or another special effect can
produce any of these results, so a summary reports what the programs found and keeps the
question open. Each directory holds a hand-written `summary.md` and the logs
`symmetry_evidence.py` wrote:

- `pointless_rugnux.log` - POINTLESS on our unmerged observations (Laue group, space group and
  their probabilities);
- `pointless_reindex.log` - how they were carried into the deposited model's setting;
- `zanuda_rugnux_free{0,1,2}.log` - ZANUDA with the deposited model against our observations
  merged by AIMLESS in the deposited group, once for each of three free sets, each chosen per
  orbit of the lattice's holohedry so that no added or twin operator relates a free reflection to a
  work one; its verdict can move with the free set, and the three show by how much;
- `pointless_deposited.log`, `zanuda_deposited.log` - the same programs on the depositor's
  structure factors, with their own free set (`deposited_data.txt` names the columns used);
- `axial_reflections.txt` - the reflections along each axis in both data sets, all of them and
  those a 2-, 3-, 4- or 6-fold screw would extinguish, in the deposited setting: the screw-axis
  evidence, which ZANUDA cannot weigh.

ZANUDA only offers a higher group the deposited coordinates nearly obey (C-alpha r.m.s.d. under
3 A); where the model obeys no such operator - one molecule in the asymmetric unit, or a twin law
that is not a pseudo-symmetry of the content - it can keep the deposited group but says nothing for
or against the higher group. To make the evidence for a new set (CCP4 set up; the work directory
empty; several minutes to a few hours, mostly ZANUDA):

```
python3 symmetry_evidence.py <battery run dir> <id> /scratch/<id> symmetry/<id> --lock /data/tmp/gpulock
```

It re-runs the battery run's own binary on the set with the unmerged export on (under the lock),
checks that `p.hkl` is the run's, and then runs the programs above. Write `summary.md` from the
logs, add `"symmetry_evidence": "symmetry/<id>"` to the row, and check the directory for
private-arm names before committing.

`model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The
open arm's command does the same inside the battery.

## A run directory

`<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>[_private]/` is never reused, and it is made
read-only when the run completes.

| path | content |
|---|---|
| `bin/rugnux` | the exact binary that ran (sha256 in the manifest) |
| `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, hardware (CPU model and nominal frequency, physical/thread/socket counts, RAM, GPUs - empty on runs from before this was recorded), pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags |
| `results.json` | one row per set (the schema is below) |
| `nullmodel_results.json` | ci-tier runs: one row per wrong-model pair (The wrong-model null, above) |
| `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) |
| `work/<arm>/<set>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |

**Run states** (`battery.py list`): `complete`; `running` (its runner process is alive);
`aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`);
`unfinished` (it died without a trace, which is treated the same as aborted). An unfinished run
covers an arbitrary part of its sets. `compare` and `report` refuse it unless given
`--allow-incomplete`, and the report then says so at the top.

### results.json (schema 3)

This is a flat list with one object per set, and every arm uses the same keys. Missing values are
`null`. `manifest.json` records the schema version as `results_schema`, and each arm's command as
`command_doc`.

**Verdicts are scored again whenever a run is read** (`report`, `compare`, the baseline delta):
every row whose `p_report.txt` has a lattice (or that is a control) goes through today's
`score.judge` with today's manifest row (reference, `unscored`, `expect`), so the two sides of a
comparison are always scored by the same rules against the same references. A row whose report
has no lattice keeps its stored verdict, because why it failed came from the run. `results.json`
keeps the verdicts as they were scored when the run finished.

Schema 3 is one row per set again: `variant` and `first_read` are gone, and `refres_*`,
`isa_ratio`, `r_meas_ratio`, `multiplicity_ref`, `cc_model`, `sg_label`, `model` and `model_note`
are new. `sgno`/`sg` are now the data's own space group where `--model` relabelled the hand.
Older runs are still read and compared: a schema-2 run as its `bare` rows, with the open arm's
`model` rows' R-factors folded in. On the open arm that is the same processing as today's
command; on the XDS arms today's command adds `-A` where XDS was anomalous, which changes how the
merging statistics count (unique reflections, multiplicity, I/sigma, ISa), so expect those to
move there against an old run; its `xds` rows, which
forced XDS's range on the processing, have no counterpart and are left out. A schema-1 run is read
as schema 2 with one variant first (`bare` on the open arm; on the XDS arms `xds`, or `bare` if it
was run `--unforced`), so its forced XDS-arm rows are left out the same way.
`compare --rerun-changed` needs A to be a schema-3 run.

| key | meaning |
|---|---|
| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran |
| `verdict` | `pass`, `fail`, `crash` (see Crashes above), `unscored` (no reference, or a question the data could not answer) or `not_run` (input missing) |
| `cause` | why it failed: `crash` (verdict `crash`), `error` (an exit without a lattice that is none of the others), `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `false_lattice`; or `no_reference` / `no_input` / `reference_problem` / `screw_undetermined`; or, on a pass, `accepted_alternative` or `indistinguishable_alternative` |
| `accepted_alt`, `accepted_alt_why` | the alternative reference the answer matched and the reason the row accepts it, on a row that passed that way; null otherwise |
| `reason` | one line for a human |
| `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours (the data's own determination) / the reference's |
| `sg_alternatives` | rugnux's `SPACE_GROUP_ALTERNATIVES` as written (`\|`-separated), else null |
| `sg_label` | the space group rugnux reported where `--model` put the model's enantiomorph on it, else null |
| `sg_relation` | open arm: how the two groups relate (`sgequiv`) |
| `cell`, `cell_ref`, `cell_dev_pct`, `volume_ratio` | cells, the largest edge deviation in % (over axis orders), and the primitive volume ratio |
| `d_min`, `d_min_ref`, `res_gain_pct` | our high-resolution limit, the reference's, and (ref - ours) / ref in % |
| `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged |
| `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt`, over its own range |
| `mosaicity_deg`, `extra_lattice_intensity_pct` | from the same report: `MOSAICITY_DEG=` (sigma_M as XDS) and `EXTRA_LATTICE_INTENSITY_PCT=` (spot intensity on further lattices); null in reports without them |
| `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref`, `multiplicity_ref` | XDS's overall values, over XDS's own range |
| `refres_range`, `refres_shells_past_limit`, `refres_unique_reflections`, `refres_completeness`, `refres_multiplicity`, `refres_i_over_sigma`, `refres_r_meas`, `refres_cc_half`, `refres_isa` | XDS arms: the REFRES_* keys, the same merge over the reference range |
| `isa_ratio`, `r_meas_ratio` | refres_isa / isa_ref and refres_r_meas / r_meas_ref |
| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), reported, not scored |
| `cc_half_high`, `cc_half_high_ref` | CC1/2 of the highest-resolution shell: rugnux's own table, XDS's (`cc_half_high` in the manifest's `ref`), each at its own limit |
| `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref |
| `model`, `model_note` | open arm, and in-house sets with a `model`: the coordinates given to `--model`, or why there were none |
| `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | any row with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited |
| `anom_sigma` | any row with a model: ANOMALOUS_SCATTERER_MEAN_SIGMA, the anomalous difference map (F+ - F- on the model's phases) read at every model atom from phosphorus up (S of Met/Cys, metals, Cl, I, ...) and averaged, in map sigma - how much anomalous signal the merge carries; null where the merge kept no Bijvoet split or the model has no such atom |
| `rmodel`, `rmodel_shell_scaled`, `radial_misfit` | any row with a model: R_MODEL, R_MODEL_SHELL_SCALED and MODEL_RADIAL_MISFIT - R over all the reflections, with and without one free scale per resolution shell, and how big that rescale was. `rmodel_shell_scaled` is the R that compares between runs (below) |
| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_twin`, `refmac_untwinned`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm; see its section for which R-free is which) |
| `dep_status`, `dep_reason`, `dep_kind`, `dep_d_min`, `dep_n_common`, `dep_cc_delta_all`, `dep_cc_delta_outer`, `dep_beyond_cc`, `dep_beyond_d` | open arm with a model: our merge against the depositor's data per shell (`depdata_check.py`, above) |
| `shx_*` | small molecule with a `cod` structure: the SHELXL check (`shelx_check.py`, above) |
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |

## The report

Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and
SVG, no external files):

- a header with the binary, build flags, host, hardware (CPU, cores/threads/sockets, RAM, GPUs -
  `not recorded` on an older run), subset, each arm's command and whether timing counts;
- a **CRASHED** section first when any set crashed, then a **summary per arm** (pass, of which
  accepted on an alternative reference / fail / crash / unscored /
  not run, pass rate, median resolution gain, ISa, R_meas, time) with a verdict bar chart, all
  from rugnux's own run, and an **Accepted alternatives** section naming those rows and why;
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
  the main metrics;
- plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference)
  for each arm; on the XDS arms the reference-range ISa, R_meas, lowest-shell R_meas and CC1/2
  noise over XDS's; on every row with a model R_free / published R_free;
- on the open arm, **against the depositor's data**: the per-shell CC difference and the CC past
  the deposited limit, one row per set, sorted by the outer-shell difference;
- on the XDS arms the **like-for-like table**: the reference-range numbers beside XDS's, with the
  rows where rugnux's cut is coarser marked as coverage;
- for the small molecules, the **SHELXL table**: R1, wR2, GooF, EXTI, weights, residual density,
  R(int)/R(sigma), the strongest bin's K and the fixed-model R1(F) of the COD structure refined
  against our data;
- the **failures**, and **one row per set** with all the numbers;
- if there is a baseline, the **delta against it**: pass rates per arm on the common sets, and
  every set that moved beyond noise.

Re-render a report, for example against another baseline (run directories are read-only, so the
output goes elsewhere and is named after the run):

```
$B report RUN --out DIR [--baseline OTHER_RUN]      # writes DIR/<run>.md and DIR/<run>.html
```

**Keeping a report in the repository.** Public reports of blessed runs go to
`tools/battery/reports/<run>.md` (Markdown only; the HTML stays with the run). `report` refuses
to write a private run's report anywhere inside the repository.

## Comparing runs

```
$B list                                             # runs, their state, and the baselines
$B compare RUN_A RUN_B [--all]                      # per-set table of what moved beyond noise
$B compare RUN_A RUN_B --rerun-changed              # rerun the changed sets with A's binary
```

`compare` pairs rows by arm and set id, following the manifests' `aliases` across renames. A run
from before schema 3 is compared through its `bare` rows (see the schema above). It lists every
row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free,
reference-range R_meas, lowest-shell R_meas or ISa, SHELXL R1, wR2, GooF or EXTI (where both runs have them), or time moved by more than the noise
thresholds in `report.py` (`NOISE`). Those thresholds
are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with
A's saved binary and options. A set that moves again under the same binary is noise. A set that
reproduces A was really changed by B.

## Protocol

1. **Compare against a persisted baseline, never against a number from memory.** The site
   config's `baseline` (and `baseline_private`) names a complete run. Every new run is reported
   against it by default. Set a new baseline only when the owner blesses a run.
2. **Keep the manifests unchanged between the two runs you compare.** A change to a manifest
   (new sets, a new reference, a different input) is committed on its own, and the baseline is
   re-run with it.
3. **Pool fixes.** Run the full battery once on a branch that carries several fixes, not once per
   fix. For one fix, run the sets it touches with `--only`, plus the smoke tier.
4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a
   reference". Only a run with the machine to itself can be used to judge speed.
5. **Resolution on the XDS arms.** Nothing forces XDS's range on the processing any more: rugnux
   keeps its own cut, and the reference-range table re-bins the same merge over XDS's range.
   The own numbers answer "what does a user get"; the reference-range numbers answer "is our
   merge better than XDS's over the same range". Where rugnux's cut is coarser than XDS's, the
   second question is only answered over the shells rugnux reached, and the missing shells are
   coverage.

### The XDS reference d_min

`CORRECT.LP` gives the range XDS merged. That range is a resolution limit only if someone chose
it. With `INCLUDE_RESOLUTION_RANGE` left at `0.0`, or set at the detector edge, it is the detector's
reach, not the crystal's. It then gives a meaningless reference, and forcing it on rugnux makes
rugnux merge shells of pure noise. On one insulin set, forcing XDS's 1.08 A (where the data end
near 1.8 A) turned rugnux's correct I23 into I222. XDS marks each shell's CC1/2 with `*` when it
is significant, and that decides which case applies (`inputs.reference_dmin`):

- `xds_range`: the finest shell's CC1/2 is significant, so XDS's limit stands.
- `cc_half_0.30`: it is not, meaning XDS merged past its own signal. The reference is then the
  point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in
  1/d² between shell centres.

The derived value is the reference rugnux's own cut is scored against. The reference-range table
uses XDS's own limit, `dmin_xds`, because XDS's pooled R_meas, CC1/2 and completeness cover its
whole range; the manifest keeps the rule used as `dmin_rule`.

## Adding or changing datasets

- **Open arm**: put the dataset directory, named by its PDB code, under the open data root, then
  run `battery.py discover`. It proposes an input for every directory the manifest does not have
  (the largest sweep, chosen by frame count), and with `--refresh` it shows where discovery now
  disagrees with an unpinned manifest input. Add the row to `open.json` with the deposited
  reference (`sg`, `sgno`, `cell`, `dmin`) and tags. Set `"pinned": true` when the choice between
  sweeps was deliberate. Record the source and DOI in `docs/EXTERNAL_TEST_DATA.md`, then run
  `python3 doc_check.py`, which fails until the page and the manifests agree (below).
  A set whose frames are not the crystal the deposited data came from (checked independently,
  e.g. with XDS on the same frames) gets `"reference_mismatch": "<why>"`. It is an annotation only:
  every check still runs and every number still counts; the report prints the note beside the set's
  row and under the REFMAC plot, so an outlier that has an explanation stays visible with it.
- **Small molecules** carry `"cod": "<COD id>"`, the published structure the SHELXL check refines
  (pick an entry whose Niggli-reduced cell and space group match the row's reference; prefer a
  modern, ordered, low-R one).
- **Every row** carries `wavelength` (A), the value rugnux reads from the image headers
  (`WAVELENGTH` in `p_report.txt`). It is descriptive only; nothing scores it.
- **Inhouse / private**: add the row (`id`, `input`, tags, and `"expect": "no_lattice"` for a
  control), then run `battery.py refs --arm inhouse --write` to fill in `ref` from the
  `CORRECT.LP` beside the input. Without `--write`, `refs` only shows what would change.
  A reference measured by hand goes in `ref_override`, which `refs` leaves alone.
  A set whose reference is known to be wrong gets `"unscored": "<why>"`: it still runs, but
  scores `unscored` instead of pass or fail.
  A set where two answers are both defensible gets
  `"ref_alternatives": [{"sg": ..., "sgno": ..., "cell": [...], "why": "<reason>"}]` - the fields
  given replace the reference's, the row passes on any of its references, and `why` is mandatory.
- **Renamed data directories**: `battery.py remap --arm inhouse MAP.json --write`, where
  `MAP.json` is `{"old": "new"}` (renames the set and its directory) or
  `{"old id": {"id": "...", "input": "..."}}`. Each old id is recorded under `aliases` in the
  manifest, so runs made before the rename still compare set by set. Without `--write`, the
  command only shows the result and flags inputs that do not exist.

## Kept results

Run directories are scratch: they live under the site's `runs_root` (on CI, deleted after the job).
A run worth citing - a release's full battery, the runs a paper figure is drawn from - is kept in the
repository under `results/<role>/<run>/`: its `report.md`, `results.json` and `manifest.json` exactly as
the battery wrote them, nothing regenerated. `<role>` names the machine: `workstation` (1 socket, RTX
5080) or `server` (4 sockets, 4x T4); the files themselves record its host and hardware.

```
$B keep 20261011-1440_57fef1_rc175-57fef1c-full --as workstation   # a run under runs_root, or a path to one
```

`keep` copies what the run directory has, reading the run name from its report. It keeps open and
in-house runs only: a run whose manifest marks it private, whose report is `report_PRIVATE.md` or
says PRIVATE ARM, or with any arm or row outside `open` and `inhouse`, is refused. Check the files for
private-arm names before committing all the same. The CI job leaves the three files of every run in
`/data/battery/ci_reports/<run>/` and in its artifact, so a run on either machine can be kept from
there. `paper_plots.py` finds a kept run by its name, after `runs_root`.

Kept so far: the rc175 full battery on the workstation, with the REFMAC check (`20261016-0057_e25876_rc175-e258760-full-refmac`), and
on the server (`20261011-1236_549939_1.0.0-rc.175`, and `20261014-0141_6e9b89_1.0.0-rc.175` at 6e9b891eb).

## Figures for publication

`paper_plots.py RUN [RUN-REDO ...] --out DIR [--time-runs RUN ...] [--uncontended RUN]`
draws three open-arm figures from a run's `results.json`: R-free of the deposited model by the REFMAC
check (`--model-check`) on the depositor's data against Rugnux's data, the deposited high-resolution limit against Rugnux's cut, and a
histogram of the processing time per dataset with the median marked. Later runs fill in sets an
earlier one did not run (a `-redo`). `--time-runs` takes the time histogram from other runs - one
whose sets had the GPU to themselves - and `summary.txt` lists the provenance of both. Beside the PDF/PNG figures it writes the
values (`rfree.dat`, `dmin.dat`, with each set's beamline; `time.dat` with set, arm, seconds,
images and `gpu_others`), every excluded set with its reason (`excluded.txt`), the gnuplot script
(`figures.gp`) and `summary.txt` with the runs' binary and runner provenance, the structure,
facility and beamline counts and the timing statistics. Facility and beamline are read from each
entry's cached mmCIF (`_diffrn_source`). Needs gnuplot, pdftoppm and gemmi.

The time is `rugnux_wall_s`, rugnux's own WALL_TIME over the whole invocation, not the runner's
`elapsed_s`, which also counts the wait for a GPU slot under `--gpulock`. Every set that ran to a
report is counted, whatever its verdict; crashed sets and the no-crystal controls (rugnux exits
without a report) are listed in `excluded.txt`. Two caveats travel with the figure: a time includes
reading the images from disk unless they were still in the page cache (see **Timing** above), and
sets that ran with other processes on the GPU (`gpu_others` > 0) are drawn as the lighter top
segment of their bar rather than dropped; `--uncontended RUN` names a run of the same sets with the
GPU to itself and `summary.txt` then reports their median slowdown against it.


## The still battery

`still_battery.py` is the serial (XFEL stills) counterpart of the rotation battery, for the public
datasets on [External test data: serial crystallography](../../docs/EXTERNAL_TEST_DATA_STILLS.md)
(manifest `still_manifest.json`). It shares the site config, the verbs and the run-directory layout
with `battery.py`, and reports something different: for stills the cell and space group are given, the
way a serial user runs (`-C`/`-S`), so there is no lattice or symmetry verdict to score. Each set runs
once and its row says what the merge is worth and what it cost:

| column | meaning |
|---|---|
| facility, beamline, detector, dataset, PDB | from the manifest |
| images | frames in the input files (rugnux's `IMAGES_PROCESSED`); images (deposition) the deposition's own count, where it is known |
| indexed %, d_min, R_split | rugnux's report (`INDEXING_RATE`, `INCLUDE_RESOLUTION_RANGE`, `R_SPLIT`: whole frames go to one half, as in CrystFEL) |
| R_free rugnux | the deposited model against our merge, not refined: the rotation battery's REFMAC check (`model_check.py`, see above), i.e. `refmac_rfree_depflags` - REFMAC's first cycle, the deposition's own free set, the common reflections at the lower of the two resolutions |
| R_free deposition | the same protocol on the deposited data (`refmac_rfree_depdata`): the R-free to beat |
| R_free published | the entry's own R-free (dark and ground-state entries only), from its own refinement program; for information |
| difference, quality | ours minus the deposition's; win where ours is lower by more than 0.01, draw within 0.01, loss otherwise |
| time, ms/image | wall clock of the rugnux command, which ends with the model validation |
| I/O floor | the input's bytes over `--io-gbps` (default 5 GB/s): the time reading alone would take from an SSD |
| page cache | share of the input already in memory when the run started (`fincore`): cold, warm or partial |

The I/O floor and the page-cache state are there because several of these formats are uncompressed
(XTC, float32 SwissFEL HDF5): a run whose time is at its floor measured the disk, not rugnux, and a
warm run of a set smaller than the RAM measured neither. For honest timings run with `--cold`, which
evicts each set's input from the page cache just before its run (`posix_fadvise`, no root needed), and
check that the column says cold.

```
still_battery.py run --rugnux build/rugnux/rugnux [--only pal90,sacla236 | --tier smoke] [--threads N] [--cold]
still_battery.py compare RUN_A RUN_B
still_battery.py report RUN [--out DIR]
still_battery.py list
still_battery.py keep RUN --as workstation|server
```

Both R-free values come from one `model_check.py` call per set (seconds), in `work/<set>/model_check/`.
`report RUN --out DIR` of a run scored by an older protocol rescores its merges into `DIR` (rugnux is
not run again).

A run writes `<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_still-<label>/` with `manifest.json`,
`results.json`, `report.md` and `work/<set>/` (the rugnux outputs and the REFMAC check), made
read-only when done; `keep` copies the three files into `results/<role>/<run>/`, like a rotation run.
`--tier smoke` runs the sets marked so in the manifest on their first few thousand frames. Data
paths come from the `"still"` section of the site config:

```json
"still": {"manifest": "still_manifest.json", "data_root": "/ssd/data/still_test", "private_manifest": null}
```

`private_manifest` names a local-only manifest of user data (`--private`); such a run gets its own
run directory and `report_PRIVATE.md`, and `keep` refuses it. Needs `gemmi` (Python), CCP4's
`refmac5` and `ctruncate`, and `fincore` (util-linux) for the page-cache column.

Data preparation for the `_int16` sets is `still/float32_to_int16.py` (see the stills data page).
`still/rugnux_stills_ab.py` is the older A/B harness: it compares two argument sets on a fixed
subset of one dataset, shell by shell, and takes its dataset list from outside the repository.