Jungfraujoch/tools/battery/README.md

# Rugnux validation battery

`battery.py` is the one way to run the Rugnux validation battery: run `rugnux` over a fixed list of
rotation datasets, score each result against a reference, and write a standard report. Two runs
of the same manifests with the same options give results you can compare set by set. Do not use
ad-hoc runner scripts instead. The old ones (`rugnux_vs_xds.py`, `battery_run.py`,
`battery2_serial.py`, `batstats.py`, ...) are retired because their numbers could not be compared
with each other.

`test_score.py` covers the scoring rules that decide a verdict (stdlib `unittest`, no data and no
run needed): `cd tools/battery && python3 test_score.py`.

## Arms

| arm | datasets | reference | manifest |
|---|---|---|---|
| **open** | public PDB depositions of raw diffraction data, plus a few published small-molecule sets | the deposited space group, cell and resolution | `open.json` (committed) |
| **inhouse** | standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), small-molecule standards (aspirin, citric acid, HEPES, YAG, L-cystine), plus no-crystal controls | XDS, from the `CORRECT.LP` beside each dataset; for the small molecules the space group is the literature's (`ref_override`), as XDS reports only Sohncke groups; L-cystine, which XDS does not index, has only the literature reference | `inhouse.json` (committed) |
| **private** | user data | XDS, like inhouse | outside the repository; the local site config gives its path |

Scoring checks these things in order, and the first one that fails decides the verdict: did it
run, is the lattice right, is the symmetry right, and on the XDS arms, is the merge as good as
XDS's. The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio
must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is
scored with `sgequiv`. On the XDS arms only the point group is scored, because XDS never tests a
screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported.
`score.py` has the details.

**A question the data could not answer is `unscored`, not a failure.** Two of these. The hand of an
enantiomorphic pair (P4_1 vs P4_3) is not in the intensities at all, and `sgequiv` already counts it
as a match. The other is a **screw axis on a row the sweep never recorded** - the axial row lies in
the spindle's blind cone, or outside the resolution range, so nothing was measured that could
confirm or refuse the screw. rugnux then offers the whole set in `SPACE_GROUP_ALTERNATIVES`, names
the open axis in `SPACE_GROUP_SCREW_UNDETERMINED`, and writes the member claiming no screw, because
a reflection file must carry one group. Such a set scores `unscored` / `screw_undetermined` rather
than `sym_screw`, on three conditions, all necessary: rugnux named the axis, the reference group is
among the ones it offered, and the only axis the two groups differ on is one it named. A screw
called wrongly where the row WAS measured meets none of them and stays a `sym_screw` failure - this
rule excuses an unanswerable question, never a wrong answer.

**A row may accept more than one reference.** Where the deposition and our reduction disagree,
the disagreement is real, and no test available to us settles it - a knife-edge - the row lists
the other acceptable answer in `ref_alternatives` and passes if our answer matches any of its
references. This is not a statement that our answer is right; it says both descriptions are
defensible and that picking either one is acceptable. `ref` keeps the deposition verbatim, each
alternative must carry a `why` a reader can check (an alternative without one is a schema error,
not a silent pass), and the report counts these rows on their own - summary column `of which
alt`, its own bar-chart segment, and an **Accepted alternatives** section printing each reason -
so they are never folded into the ordinary passes. Prefer this to `unscored`, which drops the
row from scoring altogether and loses a test that still works, and to `ref_override`, which
asserts that the reference is wrong.

**Data quality is not scored.** A verdict is about the lattice and the symmetry only. Resolution,
CC1/2, R_meas (overall and lowest shell), completeness, multiplicity and ISa are reported beside
XDS's or the deposition's values, in the per-set tables and plots, for a human to judge. On the XDS
arms `cc_half_noise_ratio` = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) over XDS's own
range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as
XDS's would be with half of its observations.

**R_free does not compare between runs; `rmodel_shell_scaled` does.** rugnux scales the model to
the data with an overall factor and a symmetry-constrained anisotropic B, which can only bend one
way with resolution - deliberately, so that the maps of a batch stay comparable. Whatever the
amplitudes' own radial profile does that this shape cannot follow is reported as R. So two runs
whose merged amplitudes have different radial profiles have R_free values that differ for a reason
that has nothing to do with the data fitting the model: measured on one such pair, R_free moved a
median +0.042 over eleven sets where `phenix.model_vs_data`, which scales per shell, moved +0.002.
`rmodel_shell_scaled` is the same R with one free scale per resolution shell, over every reflection
rather than the free 5% (nothing is refined here, so the free set buys no cross-validation and only
costs precision): on that same pair its median was +0.001, and it still flagged the one set that
was a real regression. `radial_misfit` is the size of that per-shell rescale, the RMS of its
logarithm; when it moves, R_free beside it cannot be read. All three are reported for a human,
never scored.

**Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own
table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's
`CORRECT.LP` (`lowres_r_meas_ref`, stored in the manifest as `r_meas_low`), with the shells'
high-resolution limits. The reference-range table uses XDS's shell limits, so its lowest shell and
XDS's cover the same reflections; `lowres_r_meas_ratio` is the one over the other, plotted per set.

## One command per arm

Every set runs **once**, with one command per arm:

| arm (reference) | command |
|---|---|
| open (deposition) | `rugnux -o p --no-export-unmerged --model <deposited coordinates> <input>`; sets without a deposited model (small molecules, unpublished sets, or coordinates that cannot be downloaded) run the same command without `--model` |
| inhouse, private (XDS) | `rugnux -o p --no-export-unmerged [-A] --report-resolution <XDS d_min>,<XDS d_max> <input>` |

Neither `--model` nor `--report-resolution` changes the processing: the merged reflections, the
resolution cut and every decision are the same with and without them (verified bit for bit). So
the one run is both what a user gets and, through the extra report sections, the comparison with
the reference:

- **Own result** (every arm): rugnux's own cut and statistics, scored against the reference and
  plotted as d_min(rugnux) / d_min(reference). This is what a user gets.
- **`--model`** (open arm): the deposited coordinates (`<entry>.cif.gz`) and the published R-free
  come from the RCSB and are cached in the site's `pdb_cache` (default `/data/battery/pdb_cache`),
  so the first run of a new entry needs network access. The entry is the set id up to the first
  `_`, so `6h2p_native` uses `6h2p`. The row records MODEL_FIT, R_FREE, R_WORK and CC_MODEL as
  **trend fields**: rugnux's R-free is a rigid-body placement scored on rugnux's own free set, not
  a refinement against the depositor's, so the ratio to the published R-free is not a like-for-like
  score; watch how it moves between runs. The comparable number is the REFMAC check (below).
  A model that fits may put its enantiomorph on the space-group label; the scorer reads the
  data's own determination instead (`SOHNCKE_SPACE_GROUP` where `SPACE_GROUP_ENANTIOMORPH` is
  `ASSUMED_FROM_MODEL`) and keeps the label in `sg_label`.
- **`--report-resolution`** (XDS arms): a second statistics table from the same merge, binned over
  XDS's own range (`dmin_xds` and XDS's d_max as given, 999 included, so it covers exactly what
  XDS's CORRECT.LP totals cover), and `-A` where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the
  manifest), so the two tables count the same way. Its REFRES_* numbers are shown beside XDS's
  CORRECT.LP totals: completeness, multiplicity, R_meas, the lowest shell's R_meas, CC1/2 and ISa
  (I/sigma is shown; the manifest holds no XDS value for it). Only CC1/2 is scored (above). Where rugnux's own cut is coarser than the reference
  (`REFRES_SHELLS_PAST_LIMIT` > 0), the shells past it are not merged, so the completeness is read
  as **coverage** of the reference range, not as a quality regression, and the other numbers are
  over the shells rugnux reached.

**Timing.** Each set runs once, so every time includes reading the images from disk (unless they
were still in the page cache from before the run).

The private arm is never mixed with the others. A private run gets its own run directory
(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a
public report or be written into the repository. Its names, paths and values never go into a
commit, an issue or a public report.

## Getting the data

Each manifest row gives an `input` path relative to that arm's **data root**. The data root is
set in the site config, and the first directory of each input is the dataset's directory:

```
<open data root>/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5
<inhouse data root>/lyso_ref/hewl_35_001_master.h5
```

To move the battery to another machine, copy the two data roots and point the new site config at
the copies. The paths inside the manifests do not change.

- **open**: every dataset is listed with its source repository and dataset DOI in
  [`docs/EXTERNAL_TEST_DATA.md`](../../docs/EXTERNAL_TEST_DATA.md). The sources are IRRMC
  (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what
  each archive holds. Please cite those DOIs. At PSI the data root is `/home/data/open`, a
  symlink to `/home/data/nonsls/raw`. 42 of the dataset directories in it are themselves symlinks
  into `/data/scout_staging2`, so **copy with symlinks followed**: `rsync -aL` or `cp -rL`.
  A plain `rsync -a` copies dangling links. The open arm is about 1.7 TB.
- **inhouse**: these are our own measurements and cannot be downloaded publicly. Ask the
  maintainers for a copy. Each dataset directory holds the images and the XDS output
  (`CORRECT.LP`, `XDS_ASCII.HKL`, ...) that the reference comes from. At PSI the data root is
  `/home/data/inhouse` (about 130 GB). **Leave out `.master_backups_kek`** when copying it
  (`rsync -a --exclude .master_backups_kek`): it holds backups of private masters, not battery data.
- **private**: its data root and manifest are named only in the local site config.

## Prerequisites

- Python 3.9 or newer, with `gemmi` (scoring). `h5py` is needed only by `discover`.
- A `rugnux` binary. Build it with the flags CI uses (`-march=x86-64-v3`, see `CLAUDE.md`),
  because CPU-bound results and timings differ between builds. The runner records the build
  flags from the `CMakeCache.txt` above the binary, or from the tree given with `--build-dir`.
- XDS is not needed. The XDS references are already in the manifests (`refs` re-reads them from
  `CORRECT.LP`).

## Site config

Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an
optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it
in this order:

1. `--site PATH`
2. `$JFJOCH_BATTERY_SITE`
3. `tools/battery/site.json` (git-ignored)

`site.example.json` is the committed template. Copy it and fill in your paths. The local file is
kept out of git because it names the private manifest. At PSI the config is
`/data/battery/site.json`: link it (`ln -s /data/battery/site.json tools/battery/site.json`) or
export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.

```json
{
 "runs_root": "/data/battery/runs",
 "lock_file": "/data/battery/full_run.lock",
 "gpulock": "/data/tmp/gpulock",
 "baseline": "/data/battery/runs/<a blessed full run>",
 "baseline_private": null,
 "pdb_cache": "/data/battery/pdb_cache",
 "arms": {
  "open":    {"manifest": "open.json",    "reference": "deposition", "data_root": "/home/data/open"},
  "inhouse": {"manifest": "inhouse.json", "reference": "xds",        "data_root": "/home/data/inhouse"},
  "private": {"manifest": "/path/to/private.json", "reference": "xds",
              "data_root": "/path/to/private/data", "private": true}
 }
}
```

A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that
`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of
the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where
the deposited models and their metadata are cached for `--model` and the REFMAC check.

## Running

```
B=tools/battery/battery.py
R=build/rugnux/rugnux

$B run --rugnux $R                                  # full battery: open + inhouse
$B run --rugnux $R --tier smoke                     # 22 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --only 5reo,lyso_ref             # named sets
$B run --rugnux $R --arm private                    # the private arm, on its own
```

Useful options:

- `--label` names the run. By default the label is the rugnux version.
- `--threads N` passes `-N` to rugnux.
- `--gpulock` queues every rugnux behind the site's GPU wrapper.
- `--extra "<args>"` adds rugnux arguments to every set.
- `--baseline RUN|none` sets the run the report is compared against.
- `--model-check` adds the REFMAC check to the open arm's rows (off by default, see below).

A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the
manifests (`"tiers": {"smoke": "<why this set>"}`).

**Full runs.** A run without `--only` or `--tier` takes the site lock (`flock`, non-blocking), so
two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and
everything else on the machine competes with it.

### The REFMAC model check (open arm, opt-in)

rugnux's own `--model` R-free is a trend number. `--model-check` adds an independent one, comparable
with the deposition, that needs CCP4: `tools/battery/model_check.py` scores each open-arm merge
(`p.mtz`) against the deposited model, unmodified (ligands, waters and hydrogens kept; only atoms of
unknown element, UNX, are left out because REFMAC stops on them): REFMAC's rigid-body mode,
R-factors taken before any shift, the data reindexed into the model's setting and reduced to the
deposited space group. The depositor's own structure factors go through the same setting choice
(they can sit in the other branch of a merohedral ambiguity) and are scored by the same protocol.

All REFMAC numbers use the same high-resolution limit, `d_min_used` = max(our d_min, the deposited
d_min), i.e. the lower of the two resolutions, so neither data set is scored past what the other
reaches. The four R-free fields, all of the same deposited model:

| field | data | free set | reads as |
|---|---|---|---|
| `refmac_rfree` | ours | ours | close to a work R: most of our free reflections were work reflections to the depositor's refinement |
| `refmac_rfree_depflags` | ours | the depositor's | **the fair number for our data** |
| `refmac_rfree_depdata` | the depositor's | the depositor's | the baseline: same model, same protocol, the data the model was refined against |
| `refmac_rfree_ratio` | | | `refmac_rfree_depflags / refmac_rfree_depdata`; below 1 = our data fit the model better; the report plots it |

`refmac_rwork` is R on our own work reflections. The depositor-data numbers carry a home
advantage (the model was refined against those data), and `refmac_rfree_depdata` is not the
published R-free either: different program (REFMAC vs e.g. PHENIX, whose resolution-binned scaling
absorbs a non-Wilson falloff that REFMAC's overall scale cannot) and no refinement.
The depositor's amplitudes are `F_meas`, else the mean of F(+)/F(-); where only intensities were
deposited they are converted by French-Wilson (ctruncate), as our own amplitudes are.
An entry that declares twinning (`_pdbx_reflns_twin`) is scored with REFMAC's twin refinement for
both data sets, since its published R-free is a twinned one; the untwinned numbers are kept in
`refmac_untwinned`. The ratio is null when the entry deposited no structure factors with a free
set (`refmac_reason` says why). The runner records `refmac_rfree`, `refmac_rwork`,
`refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_twin`,
`refmac_untwinned`, `refmac_status` and `refmac_reason`. Small-molecule sets are skipped.
`model_check.py <p.mtz> <pdb id> --workdir DIR` runs it on one merge and prints every field
(setting, reindexing operators of both data sets, what the depositor's amplitudes were made from).

**Refined, shared free set (second protocol).** The first-cycle numbers above score the model as
deposited at the common resolution limit. The second protocol refines the same deposited model
against each data set at that data set's OWN resolution and compares R-free on a free set both
share:

- same starting model, same REFMAC protocol for both data sets: restrained refinement, 10 cycles,
  automatic weight, isotropic B for every atom (a deposited ANISOU is often TLS-derived, and
  refining it atom by atom diverged), twin refinement where the entry declares twinning; a ligand
  whose code the monomer library describes with other atoms is renamed to a code the library
  lacks, so REFMAC makes its restraints from the model's own coordinates (`ligands_renamed` in
  the check's output);
- each data set over its whole resolution range; its work set is every reflection it has except
  the shared free set;
- shared free set = the depositor's free reflections present in BOTH data sets (after the change
  of basis), which takes the lower of the two limits in every direction, anisotropic cuts and
  rejected reflections included; depositor free reflections outside it are removed from both;
- R-free recomputed from each output MTZ over exactly the shared set (F against FC_ALL_LS, i.e.
  with REFMAC's own scale and bulk solvent, which reproduces REFMAC's R-free over its free set);
  for a twinned refinement the output MTZ holds no twinned Fc, so REFMAC's own R-free is used -
  its free set IS the shared set.

| field | meaning |
|---|---|
| `refmac_refined_rfree_shared` | our data, refined, R-free on the shared set - **the refined number for our data** |
| `refmac_refined_rfree_shared_depdata` | the depositor's data, same protocol, same shared set |
| `refmac_refined_ratio` | the ratio of the two; below 1 = our data give the better refined model |
| `refmac_refined_rwork`, `refmac_refined_rwork_depdata` | REFMAC's R-work after refinement, each data set (their own work sets and resolutions: not comparable with each other) |
| `refmac_refined_rfree_refmac`, `refmac_refined_rfree_refmac_depdata` | REFMAC's own R-free, for reference |
| `refmac_shared_free_n` | size of the shared free set |
| `refmac_refined_reason` | why the refined protocol gave nothing, when it did not (the first-cycle numbers stand) |

The refined numbers need the depositor's structure factors with a free set, like the ratio; they
take minutes to an hour per set. `model_check.py --no-refine` skips them.

### Rechecking an existing run

`battery.py recheck RUN [--only a,b] [--jobs N]` reruns the model check and the depositor-data
comparison on every open-arm row of a finished run, from the run's own `p.mtz` (CPU only; rugnux is
not run again), so numbers made by an older `model_check.py` / `depdata_check.py` can be brought up
to date without reprocessing. It updates the `refmac_*` and `dep_*` fields of `results.json` in
place and nothing else (verdicts stay as scored). The previous files stay beside the new ones:
`results.json.pre-recheck-<stamp>`, `report.{md,html}.pre-recheck-<stamp>` and each set's
`model_check.pre-recheck-<stamp>/`. The report is re-rendered, and `manifest.json` gains an entry
in its `rechecks` list - `date`, `end`, `runner_git`, `runner_dirty` (the tools version that made
the new numbers), `sets`, `fields`, `previous_results` and `command` - so which version of the
checks produced which figures stays auditable. The run directory is made read-only again.

### The depositor-data comparison (open arm, always on)

`tools/battery/depdata_check.py` compares our merge with the depositor's own data without CCP4:
per resolution shell, the rank correlation of our IMEAN with |Fc|^2 minus that of the deposited
intensities (or amplitudes squared), on the reflections both carry, with |Fc| from the deposited
model as it is (gemmi, no bulk solvent, no refinement), the data carried into the model's setting
as the REFMAC check does it. The model was refined against the depositor's data, so the comparison
has a home advantage for the deposition; around -0.01 is typical, and a set far below that merged
worse than the deposition at those resolutions. It runs on every open-arm set with a model
(seconds to a minute, gemmi only; the `-sf.cif` is cached in `pdb_cache` beside the coordinates),
and `report` computes it for an older run that lacks it. It is reported, never scored. Row keys:
`dep_cc_delta_all` (all common reflections at d < 4 A), `dep_cc_delta_outer` (mean of the two
outermost of eight equal-count shells), `dep_beyond_cc` and `dep_beyond_d` (the correlation of our
reflections with |Fc|^2 in the outermost of up to three shells past the deposited data's limit,
and that shell's limit: the model never saw these, so clearly above zero is signal), plus
`dep_kind` (`I` or `F`), `dep_d_min` (the deposited data's limit), `dep_n_common`, `dep_status`
and `dep_reason`.

`model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The
open arm's command does the same inside the battery.

## A run directory

`<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>[_private]/` is never reused, and it is made
read-only when the run completes.

| path | content |
|---|---|
| `bin/rugnux` | the exact binary that ran (sha256 in the manifest) |
| `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags |
| `results.json` | one row per set (the schema is below) |
| `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) |
| `work/<arm>/<set>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |

**Run states** (`battery.py list`): `complete`; `running` (its runner process is alive);
`aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`);
`unfinished` (it died without a trace, which is treated the same as aborted). An unfinished run
covers an arbitrary part of its sets. `compare` and `report` refuse it unless given
`--allow-incomplete`, and the report then says so at the top.

### results.json (schema 3)

This is a flat list with one object per set, and every arm uses the same keys. Missing values are
`null`. `manifest.json` records the schema version as `results_schema`, and each arm's command as
`command_doc`.

**Verdicts are scored again whenever a run is read** (`report`, `compare`, the baseline delta):
every row whose `p_report.txt` has a lattice (or that is a control) goes through today's
`score.judge` with today's manifest row (reference, `unscored`, `expect`), so the two sides of a
comparison are always scored by the same rules against the same references. A row whose report
has no lattice keeps its stored verdict, because why it failed came from the run. `results.json`
keeps the verdicts as they were scored when the run finished.

Schema 3 is one row per set again: `variant` and `first_read` are gone, and `refres_*`,
`isa_ratio`, `r_meas_ratio`, `multiplicity_ref`, `cc_model`, `sg_label`, `model` and `model_note`
are new. `sgno`/`sg` are now the data's own space group where `--model` relabelled the hand.
Older runs are still read and compared: a schema-2 run as its `bare` rows, with the open arm's
`model` rows' R-factors folded in. On the open arm that is the same processing as today's
command; on the XDS arms today's command adds `-A` where XDS was anomalous, which changes how the
merging statistics count (unique reflections, multiplicity, I/sigma, ISa), so expect those to
move there against an old run; its `xds` rows, which
forced XDS's range on the processing, have no counterpart and are left out. A schema-1 run is read
as schema 2 with one variant first (`bare` on the open arm; on the XDS arms `xds`, or `bare` if it
was run `--unforced`), so its forced XDS-arm rows are left out the same way.
`compare --rerun-changed` needs A to be a schema-3 run.

| key | meaning |
|---|---|
| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran |
| `verdict` | `pass`, `fail`, `unscored` (no reference, or a question the data could not answer) or `not_run` (input missing) |
| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `false_lattice`; or `no_reference` / `no_input` / `reference_problem` / `screw_undetermined`; or, on a pass, `accepted_alternative` |
| `accepted_alt`, `accepted_alt_why` | the alternative reference the answer matched and the reason the row accepts it, on a row that passed that way; null otherwise |
| `reason` | one line for a human |
| `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours (the data's own determination) / the reference's |
| `sg_label` | the space group rugnux reported where `--model` put the model's enantiomorph on it, else null |
| `sg_relation` | open arm: how the two groups relate (`sgequiv`) |
| `cell`, `cell_ref`, `cell_dev_pct`, `volume_ratio` | cells, the largest edge deviation in % (over axis orders), and the primitive volume ratio |
| `d_min`, `d_min_ref`, `res_gain_pct` | our high-resolution limit, the reference's, and (ref - ours) / ref in % |
| `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged |
| `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt`, over its own range |
| `mosaicity_deg`, `extra_lattice_intensity_pct` | from the same report: `MOSAICITY_DEG=` (sigma_M as XDS) and `EXTRA_LATTICE_INTENSITY_PCT=` (spot intensity on further lattices); null in reports without them |
| `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref`, `multiplicity_ref` | XDS's overall values, over XDS's own range |
| `refres_range`, `refres_shells_past_limit`, `refres_unique_reflections`, `refres_completeness`, `refres_multiplicity`, `refres_i_over_sigma`, `refres_r_meas`, `refres_cc_half`, `refres_isa` | XDS arms: the REFRES_* keys, the same merge over the reference range |
| `isa_ratio`, `r_meas_ratio` | refres_isa / isa_ref and refres_r_meas / r_meas_ref |
| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), reported, not scored |
| `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref |
| `model`, `model_note` | open arm: the deposited coordinates given to `--model`, or why there were none |
| `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited |
| `rmodel`, `rmodel_shell_scaled`, `radial_misfit` | open arm with a model: R_MODEL, R_MODEL_SHELL_SCALED and MODEL_RADIAL_MISFIT - R over all the reflections, with and without one free scale per resolution shell, and how big that rescale was. `rmodel_shell_scaled` is the R that compares between runs (below) |
| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_twin`, `refmac_untwinned`, `refmac_status`, `refmac_reason`, `refmac_refined_*`, `refmac_shared_free_n` | the REFMAC check (`--model-check`, open arm; see its section for which R-free is which) |
| `dep_status`, `dep_reason`, `dep_kind`, `dep_d_min`, `dep_n_common`, `dep_cc_delta_all`, `dep_cc_delta_outer`, `dep_beyond_cc`, `dep_beyond_d` | open arm with a model: our merge against the depositor's data per shell (`depdata_check.py`, above) |
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |

## The report

Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and
SVG, no external files):

- a header with the binary, build flags, host, subset, each arm's command and whether timing counts;
- a **summary per arm** (pass, of which accepted on an alternative reference / fail / unscored /
  not run, pass rate, median resolution gain, ISa, R_meas, time) with a verdict bar chart, all
  from rugnux's own run, and an **Accepted alternatives** section naming those rows and why;
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
  the main metrics;
- plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference)
  for each arm; on the XDS arms the reference-range ISa, R_meas, lowest-shell R_meas and CC1/2
  noise over XDS's; on the open arm R_free / published R_free;
- on the open arm, **against the depositor's data**: the per-shell CC difference and the CC past
  the deposited limit, one row per set, sorted by the outer-shell difference;
- on the XDS arms the **like-for-like table**: the reference-range numbers beside XDS's, with the
  rows where rugnux's cut is coarser marked as coverage;
- the **failures**, and **one row per set** with all the numbers;
- if there is a baseline, the **delta against it**: pass rates per arm on the common sets, and
  every set that moved beyond noise.

Re-render a report, for example against another baseline (run directories are read-only, so the
output goes elsewhere and is named after the run):

```
$B report RUN --out DIR [--baseline OTHER_RUN]      # writes DIR/<run>.md and DIR/<run>.html
```

**Keeping a report in the repository.** Public reports of blessed runs go to
`tools/battery/reports/<run>.md` (Markdown only; the HTML stays with the run). `report` refuses
to write a private run's report anywhere inside the repository.

## Comparing runs

```
$B list                                             # runs, their state, and the baselines
$B compare RUN_A RUN_B [--all]                      # per-set table of what moved beyond noise
$B compare RUN_A RUN_B --rerun-changed              # rerun the changed sets with A's binary
```

`compare` pairs rows by arm and set id, following the manifests' `aliases` across renames. A run
from before schema 3 is compared through its `bare` rows (see the schema above). It lists every
row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free,
reference-range R_meas, lowest-shell R_meas or ISa (where both runs have them), or time moved by more than the noise
thresholds in `report.py` (`NOISE`). Those thresholds
are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with
A's saved binary and options. A set that moves again under the same binary is noise. A set that
reproduces A was really changed by B.

## Protocol

1. **Compare against a persisted baseline, never against a number from memory.** The site
   config's `baseline` (and `baseline_private`) names a complete run. Every new run is reported
   against it by default. Set a new baseline only when the owner blesses a run.
2. **Keep the manifests unchanged between the two runs you compare.** A change to a manifest
   (new sets, a new reference, a different input) is committed on its own, and the baseline is
   re-run with it.
3. **Pool fixes.** Run the full battery once on a branch that carries several fixes, not once per
   fix. For one fix, run the sets it touches with `--only`, plus the smoke tier.
4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a
   reference". Only a run with the machine to itself can be used to judge speed.
5. **Resolution on the XDS arms.** Nothing forces XDS's range on the processing any more: rugnux
   keeps its own cut, and the reference-range table re-bins the same merge over XDS's range.
   The own numbers answer "what does a user get"; the reference-range numbers answer "is our
   merge better than XDS's over the same range". Where rugnux's cut is coarser than XDS's, the
   second question is only answered over the shells rugnux reached, and the missing shells are
   coverage.

### The XDS reference d_min

`CORRECT.LP` gives the range XDS merged. That range is a resolution limit only if someone chose
it. With `INCLUDE_RESOLUTION_RANGE` left at `0.0`, or set at the detector edge, it is the detector's
reach, not the crystal's. It then gives a meaningless reference, and forcing it on rugnux makes
rugnux merge shells of pure noise. On one insulin set, forcing XDS's 1.08 A (where the data end
near 1.8 A) turned rugnux's correct I23 into I222. XDS marks each shell's CC1/2 with `*` when it
is significant, and that decides which case applies (`inputs.reference_dmin`):

- `xds_range`: the finest shell's CC1/2 is significant, so XDS's limit stands.
- `cc_half_0.30`: it is not, meaning XDS merged past its own signal. The reference is then the
  point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in
  1/d² between shell centres.

The derived value is the reference rugnux's own cut is scored against. The reference-range table
uses XDS's own limit, `dmin_xds`, because XDS's pooled R_meas, CC1/2 and completeness cover its
whole range; the manifest keeps the rule used as `dmin_rule`.

## Adding or changing datasets

- **Open arm**: put the dataset directory, named by its PDB code, under the open data root, then
  run `battery.py discover`. It proposes an input for every directory the manifest does not have
  (the largest sweep, chosen by frame count), and with `--refresh` it shows where discovery now
  disagrees with an unpinned manifest input. Add the row to `open.json` with the deposited
  reference (`sg`, `sgno`, `cell`, `dmin`) and tags. Set `"pinned": true` when the choice between
  sweeps was deliberate. Record the source and DOI in `docs/EXTERNAL_TEST_DATA.md`.
- **Every row** carries `wavelength` (A), the value rugnux reads from the image headers
  (`WAVELENGTH` in `p_report.txt`). It is descriptive only; nothing scores it.
- **Inhouse / private**: add the row (`id`, `input`, tags, and `"expect": "no_lattice"` for a
  control), then run `battery.py refs --arm inhouse --write` to fill in `ref` from the
  `CORRECT.LP` beside the input. Without `--write`, `refs` only shows what would change.
  A reference measured by hand goes in `ref_override`, which `refs` leaves alone.
  A set whose reference is known to be wrong gets `"unscored": "<why>"`: it still runs, but
  scores `unscored` instead of pass or fail.
  A set where two answers are both defensible gets
  `"ref_alternatives": [{"sg": ..., "sgno": ..., "cell": [...], "why": "<reason>"}]` - the fields
  given replace the reference's, the row passes on any of its references, and `why` is mandatory.
- **Renamed data directories**: `battery.py remap --arm inhouse MAP.json --write`, where
  `MAP.json` is `{"old": "new"}` (renames the set and its directory) or
  `{"old id": {"id": "...", "input": "..."}}`. Each old id is recorded under `aliases` in the
  manifest, so runs made before the rename still compare set by set. Without `--write`, the
  command only shows the result and flags inputs that do not exist.

## Figures for publication

`paper_plots.py RUN [RUN-REDO ...] --out DIR [--time-runs RUN ...] [--uncontended RUN] [--rfree refined|first-cycle]`
draws three open-arm figures from a run's `results.json`: R-free of the deposited model after the REFMAC
check (`--model-check`) on the depositor's data against Rugnux's data (by default the refined protocol:
each data set at its own resolution, R-free on the shared free set; `--rfree first-cycle` for the
first-cycle numbers at the common resolution), the deposited high-resolution limit against Rugnux's cut, and a
histogram of the processing time per dataset with the median marked. Later runs fill in sets an
earlier one did not run (a `-redo`). `--time-runs` takes the time histogram from other runs - one
whose sets had the GPU to themselves - and `summary.txt` lists the provenance of both. Beside the PDF/PNG figures it writes the
values (`rfree.dat`, `dmin.dat`, with each set's beamline; `time.dat` with set, arm, seconds,
images and `gpu_others`), every excluded set with its reason (`excluded.txt`), the gnuplot script
(`figures.gp`) and `summary.txt` with the runs' binary and runner provenance, the structure,
facility and beamline counts and the timing statistics. Facility and beamline are read from each
entry's cached mmCIF (`_diffrn_source`). Needs gnuplot, pdftoppm and gemmi.

The time is `rugnux_wall_s`, rugnux's own WALL_TIME over the whole invocation, not the runner's
`elapsed_s`, which also counts the wait for a GPU slot under `--gpulock`. Every set that ran to a
report is counted, whatever its verdict; crashed sets and the no-crystal controls (rugnux exits
without a report) are listed in `excluded.txt`. Two caveats travel with the figure: a time includes
reading the images from disk unless they were still in the page cache (see **Timing** above), and
sets that ran with other processes on the GPU (`gpu_others` > 0) are drawn as the lighter top
segment of their bar rather than dropped; `--uncontended RUN` names a run of the same sets with the
GPU to itself and `summary.txt` then reports their median slowdown against it.