Jungfraujoch/tools/battery/README.md

# Rugnux validation battery

`battery.py` is the one way to run the Rugnux validation battery: run `rugnux` over a fixed list of
rotation datasets, score each result against a reference, and write a standard report. Two runs
of the same manifests with the same options give results you can compare set by set. Do not use
ad-hoc runner scripts instead. The old ones (`rugnux_vs_xds.py`, `battery_run.py`,
`battery2_serial.py`, `batstats.py`, ...) are retired because their numbers could not be compared
with each other.

## Arms

| arm | datasets | reference | manifest |
|---|---|---|---|
| **open** | public PDB depositions of raw diffraction data, plus a few published small-molecule sets | the deposited space group, cell and resolution | `open.json` (committed) |
| **inhouse** | standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), plus no-crystal controls | XDS, from the `CORRECT.LP` beside each dataset | `inhouse.json` (committed) |
| **private** | user data | XDS, like inhouse | outside the repository; the local site config gives its path |

Scoring checks these things in order, and the first one that fails decides the verdict: did it
run, is the lattice right, is the symmetry right, and on the XDS arms, is the merge as good as
XDS's. The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio
must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is
scored with `sgequiv`. On the XDS arms only the point group is scored, because XDS never tests a
screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported.
`score.py` has the details.

**Merge quality is judged only against XDS, and only by CC1/2.** No R_meas is a pass/fail
criterion, and no absolute CC1/2 either. On the XDS arms the set fails `merge` when

    (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) > 2

with rugnux's CC1/2 from the reference-range table (`REFRES_CC_HALF`, XDS's own range) and XDS's
from the `CORRECT.LP` total. CC1/2 = S / (S + E), with S the signal variance and E the half-set
error variance, so 1/CC1/2 - 1 = E/S at any CC1/2, and E goes as 1/(observations): a ratio above 2
is a merge noisier than XDS's would be with half of its observations. XDS prints CC1/2 to 0.1%, so
its value is taken at the bottom of that rounding (-0.0005), and a printed 100.0 does not demand a
perfect CC1/2. The ratio is stored as `cc_half_noise_ratio`. A set without the reference-range
table or without an XDS CC1/2 (the controls, a hand-measured reference) has no merge criterion.
**The open arm has no merge criterion at all**: it is scored on lattice and symmetry, and its
resolution, CC1/2 and R_meas are reported, not judged.

**Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own
table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's
`CORRECT.LP` (`lowres_r_meas_ref`, stored in the manifest as `r_meas_low`), with the shells'
high-resolution limits. The reference-range table uses XDS's shell limits, so its lowest shell and
XDS's cover the same reflections; `lowres_r_meas_ratio` is the one over the other, plotted per set.

## One command per arm

Every set runs **once**, with one command per arm:

| arm (reference) | command |
|---|---|
| open (deposition) | `rugnux -o p --no-export-unmerged --model <deposited coordinates> <input>`; sets without a deposited model (small molecules, unpublished sets, or coordinates that cannot be downloaded) run the same command without `--model` |
| inhouse, private (XDS) | `rugnux -o p --no-export-unmerged [-A] --report-resolution <XDS d_min>,<XDS d_max> <input>` |

Neither `--model` nor `--report-resolution` changes the processing: the merged reflections, the
resolution cut and every decision are the same with and without them (verified bit for bit). So
the one run is both what a user gets and, through the extra report sections, the comparison with
the reference:

- **Own result** (every arm): rugnux's own cut and statistics, scored against the reference and
  plotted as d_min(rugnux) / d_min(reference). This is what a user gets.
- **`--model`** (open arm): the deposited coordinates (`<entry>.cif.gz`) and the published R-free
  come from the RCSB and are cached in the site's `pdb_cache` (default `/data/battery/pdb_cache`),
  so the first run of a new entry needs network access. The entry is the set id up to the first
  `_`, so `6h2p_native` uses `6h2p`. The row records MODEL_FIT, R_FREE, R_WORK and CC_MODEL as
  **trend fields**: rugnux's R-free is a rigid-body placement scored on rugnux's own free set, not
  a refinement against the depositor's, so the ratio to the published R-free is not a like-for-like
  score; watch how it moves between runs. The comparable number is the REFMAC check (below).
  A model that fits may put its enantiomorph on the space-group label; the scorer reads the
  data's own determination instead (`SOHNCKE_SPACE_GROUP` where `SPACE_GROUP_ENANTIOMORPH` is
  `ASSUMED_FROM_MODEL`) and keeps the label in `sg_label`.
- **`--report-resolution`** (XDS arms): a second statistics table from the same merge, binned over
  XDS's own range (`dmin_xds` and XDS's d_max as given, 999 included, so it covers exactly what
  XDS's CORRECT.LP totals cover), and `-A` where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the
  manifest), so the two tables count the same way. Its REFRES_* numbers are shown beside XDS's
  CORRECT.LP totals: completeness, multiplicity, R_meas, the lowest shell's R_meas, CC1/2 and ISa
  (I/sigma is shown; the manifest holds no XDS value for it). Only CC1/2 is scored (above). Where rugnux's own cut is coarser than the reference
  (`REFRES_SHELLS_PAST_LIMIT` > 0), the shells past it are not merged, so the completeness is read
  as **coverage** of the reference range, not as a quality regression, and the other numbers are
  over the shells rugnux reached.

**Timing.** Each set runs once, so every time includes reading the images from disk (unless they
were still in the page cache from before the run).

The private arm is never mixed with the others. A private run gets its own run directory
(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a
public report or be written into the repository. Its names, paths and values never go into a
commit, an issue or a public report.

## Getting the data

Each manifest row gives an `input` path relative to that arm's **data root**. The data root is
set in the site config, and the first directory of each input is the dataset's directory:

```
<open data root>/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5
<inhouse data root>/lyso_ref/hewl_35_001_master.h5
```

To move the battery to another machine, copy the two data roots and point the new site config at
the copies. The paths inside the manifests do not change.

- **open**: every dataset is listed with its source repository and dataset DOI in
  [`docs/EXTERNAL_TEST_DATA.md`](../../docs/EXTERNAL_TEST_DATA.md). The sources are IRRMC
  (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what
  each archive holds. Please cite those DOIs. At PSI the data root is `/home/data/open`, a
  symlink to `/home/data/nonsls/raw`. 51 of the dataset directories in it are themselves symlinks
  into `/data/scout_staging2`, so **copy with symlinks followed**: `rsync -aL` or `cp -rL`.
  A plain `rsync -a` copies dangling links. The open arm is about 1.7 TB.
- **inhouse**: these are our own measurements and cannot be downloaded publicly. Ask the
  maintainers for a copy. Each dataset directory holds the images and the XDS output
  (`CORRECT.LP`, `XDS_ASCII.HKL`, ...) that the reference comes from. At PSI the data root is
  `/home/data/inhouse` (about 130 GB). **Leave out `.master_backups_kek`** when copying it
  (`rsync -a --exclude .master_backups_kek`): it holds backups of private masters, not battery data.
- **private**: its data root and manifest are named only in the local site config.

## Prerequisites

- Python 3.9 or newer, with `gemmi` (scoring). `h5py` is needed only by `discover`.
- A `rugnux` binary. Build it with the flags CI uses (`-march=x86-64-v3`, see `CLAUDE.md`),
  because CPU-bound results and timings differ between builds. The runner records the build
  flags from the `CMakeCache.txt` above the binary, or from the tree given with `--build-dir`.
- XDS is not needed. The XDS references are already in the manifests (`refs` re-reads them from
  `CORRECT.LP`).

## Site config

Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an
optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it
in this order:

1. `--site PATH`
2. `$JFJOCH_BATTERY_SITE`
3. `tools/battery/site.json` (git-ignored)

`site.example.json` is the committed template. Copy it and fill in your paths. The local file is
kept out of git because it names the private manifest. At PSI the config is
`/data/battery/site.json`: link it (`ln -s /data/battery/site.json tools/battery/site.json`) or
export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.

```json
{
 "runs_root": "/data/battery/runs",
 "lock_file": "/data/battery/full_run.lock",
 "gpulock": "/data/tmp/gpulock",
 "baseline": "/data/battery/runs/<a blessed full run>",
 "baseline_private": null,
 "pdb_cache": "/data/battery/pdb_cache",
 "arms": {
  "open":    {"manifest": "open.json",    "reference": "deposition", "data_root": "/home/data/open"},
  "inhouse": {"manifest": "inhouse.json", "reference": "xds",        "data_root": "/home/data/inhouse"},
  "private": {"manifest": "/path/to/private.json", "reference": "xds",
              "data_root": "/path/to/private/data", "private": true}
 }
}
```

A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that
`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of
the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where
the deposited models and their metadata are cached for `--model` and the REFMAC check.

## Running

```
B=tools/battery/battery.py
R=build/rugnux/rugnux

$B run --rugnux $R                                  # full battery: open + inhouse
$B run --rugnux $R --tier smoke                     # 22 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --only 5reo,lyso_ref             # named sets
$B run --rugnux $R --arm private                    # the private arm, on its own
```

Useful options:

- `--label` names the run. By default the label is the rugnux version.
- `--threads N` passes `-N` to rugnux.
- `--gpulock` queues every rugnux behind the site's GPU wrapper.
- `--extra "<args>"` adds rugnux arguments to every set.
- `--baseline RUN|none` sets the run the report is compared against.
- `--model-check` adds the REFMAC check to the open arm's rows (off by default, see below).

A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the
manifests (`"tiers": {"smoke": "<why this set>"}`).

**Full runs.** A run without `--only` or `--tier` takes the site lock (`flock`, non-blocking), so
two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and
everything else on the machine competes with it.

### The REFMAC model check (open arm, opt-in)

rugnux's own `--model` R-free is a trend number. `--model-check` adds an independent one, comparable
with the deposition, that needs CCP4: `tools/battery/model_check.py` scores each open-arm merge
(`p.mtz`) against the deposited model,
unmodified (ligands, waters and hydrogens kept): REFMAC's rigid-body mode, R-factors taken before
any shift, the data reindexed into the model's setting and reduced to the deposited space group.
Our own free set was mostly work reflections for the depositor, so `refmac_rfree` reads low. The
fair comparison is `refmac_rfree_depflags` (our data on the depositor's free set) against
`refmac_rfree_depdata` (the depositor's structure factors, same free set, same protocol);
`refmac_rfree_ratio` is that ratio, and the report plots it. It is null when the entry deposited no
structure factors with a free set. The runner records `refmac_rfree`, `refmac_rwork`,
`refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status` and
`refmac_reason`. Small-molecule sets are skipped.

`model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The
open arm's command does the same inside the battery.

## A run directory

`<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>[_private]/` is never reused, and it is made
read-only when the run completes.

| path | content |
|---|---|
| `bin/rugnux` | the exact binary that ran (sha256 in the manifest) |
| `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags |
| `results.json` | one row per set (the schema is below) |
| `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) |
| `work/<arm>/<set>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |

**Run states** (`battery.py list`): `complete`; `running` (its runner process is alive);
`aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`);
`unfinished` (it died without a trace, which is treated the same as aborted). An unfinished run
covers an arbitrary part of its sets. `compare` and `report` refuse it unless given
`--allow-incomplete`, and the report then says so at the top.

### results.json (schema 3)

This is a flat list with one object per set, and every arm uses the same keys. Missing values are
`null`. `manifest.json` records the schema version as `results_schema`, and each arm's command as
`command_doc`.

**Verdicts are scored again whenever a run is read** (`report`, `compare`, the baseline delta):
every row whose `p_report.txt` has a lattice (or that is a control) goes through today's
`score.judge` with today's manifest row (reference, `unscored`, `expect`), so the two sides of a
comparison are always scored by the same rules against the same references. A row whose report
has no lattice keeps its stored verdict, because why it failed came from the run. `results.json`
keeps the verdicts as they were scored when the run finished.

Schema 3 is one row per set again: `variant` and `first_read` are gone, and `refres_*`,
`isa_ratio`, `r_meas_ratio`, `multiplicity_ref`, `cc_model`, `sg_label`, `model` and `model_note`
are new. `sgno`/`sg` are now the data's own space group where `--model` relabelled the hand.
Older runs are still read and compared: a schema-2 run as its `bare` rows, with the open arm's
`model` rows' R-factors folded in. On the open arm that is the same processing as today's
command; on the XDS arms today's command adds `-A` where XDS was anomalous, which changes how the
merging statistics count (unique reflections, multiplicity, I/sigma, ISa), so expect those to
move there against an old run; its `xds` rows, which
forced XDS's range on the processing, have no counterpart and are left out. A schema-1 run is read
as schema 2 with one variant first (`bare` on the open arm; on the XDS arms `xds`, or `bare` if it
was run `--unforced`), so its forced XDS-arm rows are left out the same way.
`compare --rerun-changed` needs A to be a schema-3 run.

| key | meaning |
|---|---|
| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran |
| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) |
| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge` (XDS arms: CC1/2 noise ratio above 2), `false_lattice`; or `no_reference` / `no_input` / `reference_problem` |
| `reason` | one line for a human |
| `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours (the data's own determination) / the reference's |
| `sg_label` | the space group rugnux reported where `--model` put the model's enantiomorph on it, else null |
| `sg_relation` | open arm: how the two groups relate (`sgequiv`) |
| `cell`, `cell_ref`, `cell_dev_pct`, `volume_ratio` | cells, the largest edge deviation in % (over axis orders), and the primitive volume ratio |
| `d_min`, `d_min_ref`, `res_gain_pct` | our high-resolution limit, the reference's, and (ref - ours) / ref in % |
| `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged |
| `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt`, over its own range |
| `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref`, `multiplicity_ref` | XDS's overall values, over XDS's own range |
| `refres_range`, `refres_shells_past_limit`, `refres_unique_reflections`, `refres_completeness`, `refres_multiplicity`, `refres_i_over_sigma`, `refres_r_meas`, `refres_cc_half`, `refres_isa` | XDS arms: the REFRES_* keys, the same merge over the reference range |
| `isa_ratio`, `r_meas_ratio` | refres_isa / isa_ref and refres_r_meas / r_meas_ref |
| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), the merge criterion |
| `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref |
| `model`, `model_note` | open arm: the deposited coordinates given to `--model`, or why there were none |
| `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited |
| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm) |
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |

## The report

Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and
SVG, no external files):

- a header with the binary, build flags, host, subset, each arm's command and whether timing counts;
- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain, ISa,
  R_meas, time) with a verdict bar chart, all from rugnux's own run;
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
  the main metrics;
- plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference)
  for each arm; on the XDS arms the reference-range ISa, R_meas, lowest-shell R_meas and CC1/2
  noise over XDS's; on the open arm R_free / published R_free;
- on the XDS arms the **like-for-like table**: the reference-range numbers beside XDS's, with the
  rows where rugnux's cut is coarser marked as coverage;
- the **failures**, and **one row per set** with all the numbers;
- if there is a baseline, the **delta against it**: pass rates per arm on the common sets, and
  every set that moved beyond noise.

Re-render a report, for example against another baseline (run directories are read-only, so the
output goes elsewhere and is named after the run):

```
$B report RUN --out DIR [--baseline OTHER_RUN]      # writes DIR/<run>.md and DIR/<run>.html
```

**Keeping a report in the repository.** Public reports of blessed runs go to
`tools/battery/reports/<run>.md` (Markdown only; the HTML stays with the run). `report` refuses
to write a private run's report anywhere inside the repository.

## Comparing runs

```
$B list                                             # runs, their state, and the baselines
$B compare RUN_A RUN_B [--all]                      # per-set table of what moved beyond noise
$B compare RUN_A RUN_B --rerun-changed              # rerun the changed sets with A's binary
```

`compare` pairs rows by arm and set id, following the manifests' `aliases` across renames. A run
from before schema 3 is compared through its `bare` rows (see the schema above). It lists every
row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free,
reference-range R_meas, lowest-shell R_meas or ISa (where both runs have them), or time moved by more than the noise
thresholds in `report.py` (`NOISE`). Those thresholds
are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with
A's saved binary and options. A set that moves again under the same binary is noise. A set that
reproduces A was really changed by B.

## Protocol

1. **Compare against a persisted baseline, never against a number from memory.** The site
   config's `baseline` (and `baseline_private`) names a complete run. Every new run is reported
   against it by default. Set a new baseline only when the owner blesses a run.
2. **Keep the manifests unchanged between the two runs you compare.** A change to a manifest
   (new sets, a new reference, a different input) is committed on its own, and the baseline is
   re-run with it.
3. **Pool fixes.** Run the full battery once on a branch that carries several fixes, not once per
   fix. For one fix, run the sets it touches with `--only`, plus the smoke tier.
4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a
   reference". Only a run with the machine to itself can be used to judge speed.
5. **Resolution on the XDS arms.** Nothing forces XDS's range on the processing any more: rugnux
   keeps its own cut, and the reference-range table re-bins the same merge over XDS's range.
   The own numbers answer "what does a user get"; the reference-range numbers answer "is our
   merge better than XDS's over the same range". Where rugnux's cut is coarser than XDS's, the
   second question is only answered over the shells rugnux reached, and the missing shells are
   coverage.

### The XDS reference d_min

`CORRECT.LP` gives the range XDS merged. That range is a resolution limit only if someone chose
it. With `INCLUDE_RESOLUTION_RANGE` left at `0.0`, or set at the detector edge, it is the detector's
reach, not the crystal's. It then gives a meaningless reference, and forcing it on rugnux makes
rugnux merge shells of pure noise. On one insulin set, forcing XDS's 1.08 A (where the data end
near 1.8 A) turned rugnux's correct I23 into I222. XDS marks each shell's CC1/2 with `*` when it
is significant, and that decides which case applies (`inputs.reference_dmin`):

- `xds_range`: the finest shell's CC1/2 is significant, so XDS's limit stands.
- `cc_half_0.30`: it is not, meaning XDS merged past its own signal. The reference is then the
  point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in
  1/d² between shell centres.

The derived value is the reference rugnux's own cut is scored against. The reference-range table
uses XDS's own limit, `dmin_xds`, because XDS's pooled R_meas, CC1/2 and completeness cover its
whole range; the manifest keeps the rule used as `dmin_rule`.

## Adding or changing datasets

- **Open arm**: put the dataset directory, named by its PDB code, under the open data root, then
  run `battery.py discover`. It proposes an input for every directory the manifest does not have
  (the largest sweep, chosen by frame count), and with `--refresh` it shows where discovery now
  disagrees with an unpinned manifest input. Add the row to `open.json` with the deposited
  reference (`sg`, `sgno`, `cell`, `dmin`) and tags. Set `"pinned": true` when the choice between
  sweeps was deliberate. Record the source and DOI in `docs/EXTERNAL_TEST_DATA.md`.
- **Inhouse / private**: add the row (`id`, `input`, tags, and `"expect": "no_lattice"` for a
  control), then run `battery.py refs --arm inhouse --write` to fill in `ref` from the
  `CORRECT.LP` beside the input. Without `--write`, `refs` only shows what would change.
  A reference measured by hand goes in `ref_override`, which `refs` leaves alone.
  A set whose reference is known to be wrong gets `"unscored": "<why>"`: it still runs, but
  scores `unscored` instead of pass or fail.
- **Renamed data directories**: `battery.py remap --arm inhouse MAP.json --write`, where
  `MAP.json` is `{"old": "new"}` (renames the set and its directory) or
  `{"old id": {"id": "...", "input": "..."}}`. Each old id is recorded under `aliases` in the
  manifest, so runs made before the rename still compare set by set. Without `--write`, the
  command only shows the result and flags inputs that do not exist.