Jungfraujoch/tools/battery/README.md

Rugnux validation battery

battery.py is the one way to run the Rugnux validation battery: run rugnux over a fixed list of rotation datasets, score each result against a reference, and write a standard report. Two runs of the same manifests with the same options give results you can compare set by set. Do not use ad-hoc runner scripts instead. The old ones (rugnux_vs_xds.py, battery_run.py, battery2_serial.py, batstats.py, ...) are retired because their numbers could not be compared with each other.

Arms

arm	datasets	reference	manifest
open	public PDB depositions of raw diffraction data, plus a few published small-molecule sets	the deposited space group, cell and resolution	open.json (committed)
inhouse	standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), plus no-crystal controls	XDS, from the CORRECT.LP beside each dataset	inhouse.json (committed)
private	user data	XDS, like inhouse	outside the repository; the local site config gives its path

Scoring checks four things in order, and the first one that fails decides the verdict: did it run, is the lattice right, is the symmetry right, is the merge usable (R_meas at most 60%, CC1/2 at least 0.5). The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is scored with sgequiv. On the XDS arms only the point group is scored, because XDS never tests a screw axis. A no-crystal control ("expect": "no_lattice") passes only if no lattice is reported. score.py has the details.

Variants

Every set is run more than once, with different settings, and each run is a variant. The variants of one set run straight after each other, before the next set starts. The battery is limited by reading images from disk, and this way only the first variant reads them; the others find them in the page cache.

arm (reference)	variants, in the order they run
open (deposition)	bare, then model
inhouse, private (XDS)	bare, then xds

• bare: plain rugnux <input> and nothing else (no -A, no resolution range). This is what a user gets. It is scored against the reference like the others, and its d_min ratio against the reference is plotted.
• model (open arm): rugnux --model <deposited coordinates>. The coordinates (<entry>.cif.gz) and the published R-free come from the RCSB and are cached in the site's pdb_cache (default /data/battery/pdb_cache), so the first run of a new entry needs network access. The entry is the set id up to the first _, so 6h2p_native uses 6h2p. A set with no PDB entry (small molecules) or whose coordinates cannot be downloaded gets a not_run row with cause no_model. The row records the R-free and R-work that rugnux reports for the model, the published R-free and their ratio, which the report plots. rugnux's R-free is that of a rigid-body placement against its own merge, and the published one comes from full refinement against the depositor's own data, so the ratio is not a like-for-like score; watch how it moves between runs.
• xds (XDS arms): XDS's reference settings. XDS's resolution range is forced (see "The XDS reference d_min" below), and where XDS merged with FRIEDEL'S_LAW=FALSE ("anomalous": true in the manifest) rugnux gets -A. Both programs then merge the same reflections the same way.

The variants are defined in one place, VARIANTS in battery.py. --variants runs a subset, for example --variants bare, and they always run in the table's order. --unforced is gone: --variants bare gives rugnux its own resolution cut (and, unlike --unforced, no -A).

Order and timing. bare goes first on every arm. So its time always includes reading the data from disk (unless the images were still cached from before the run), whichever variants a run selects, and it is the time a user sees. The time of a later variant is taken on cached images. Each row records whether it was its set's first_read, and the report's timing table says which variant carries the disk reads. Compare times of one variant between runs only, never across variants.

The private arm is never mixed with the others. A private run gets its own run directory (suffix _private) and its own report (report_PRIVATE.*), and it cannot be the baseline of a public report or be written into the repository. Its names, paths and values never go into a commit, an issue or a public report.

Getting the data

Each manifest row gives an input path relative to that arm's data root. The data root is set in the site config, and the first directory of each input is the dataset's directory:

<open data root>/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5
<inhouse data root>/lyso_ref/hewl_35_001_master.h5

To move the battery to another machine, copy the two data roots and point the new site config at the copies. The paths inside the manifests do not change.

• open: every dataset is listed with its source repository and dataset DOI in docs/EXTERNAL_TEST_DATA.md. The sources are IRRMC (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what each archive holds. Please cite those DOIs. At PSI the data root is /home/data/open, a symlink to /home/data/nonsls/raw. 51 of the dataset directories in it are themselves symlinks into /data/scout_staging2, so copy with symlinks followed: rsync -aL or cp -rL. A plain rsync -a copies dangling links. The open arm is about 1.7 TB.
• inhouse: these are our own measurements and cannot be downloaded publicly. Ask the maintainers for a copy. Each dataset directory holds the images and the XDS output (CORRECT.LP, XDS_ASCII.HKL, ...) that the reference comes from. At PSI the data root is /home/data/inhouse (about 130 GB). Leave out .master_backups_kek when copying it (rsync -a --exclude .master_backups_kek): it holds backups of private masters, not battery data.
• private: its data root and manifest are named only in the local site config.

Prerequisites

• Python 3.9 or newer, with gemmi (scoring). h5py is needed only by discover.
• A rugnux binary. Build it with the flags CI uses (-march=x86-64-v3, see CLAUDE.md), because CPU-bound results and timings differ between builds. The runner records the build flags from the CMakeCache.txt above the binary, or from the tree given with --build-dir.
• XDS is not needed. The XDS references are already in the manifests (refs re-reads them from CORRECT.LP).

Site config

Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it in this order:

1. --site PATH
2. $JFJOCH_BATTERY_SITE
3. tools/battery/site.json (git-ignored)

site.example.json is the committed template. Copy it and fill in your paths. The local file is kept out of git because it names the private manifest. At PSI the config is /data/battery/site.json: link it (ln -s /data/battery/site.json tools/battery/site.json) or export JFJOCH_BATTERY_SITE=/data/battery/site.json.

{
 "runs_root": "/data/battery/runs",
 "lock_file": "/data/battery/full_run.lock",
 "gpulock": "/data/tmp/gpulock",
 "baseline": "/data/battery/runs/<a blessed full run>",
 "baseline_private": null,
 "pdb_cache": "/data/battery/pdb_cache",
 "arms": {
  "open":    {"manifest": "open.json",    "reference": "deposition", "data_root": "/home/data/open"},
  "inhouse": {"manifest": "inhouse.json", "reference": "xds",        "data_root": "/home/data/inhouse"},
  "private": {"manifest": "/path/to/private.json", "reference": "xds",
              "data_root": "/path/to/private/data", "private": true}
 }
}

A relative manifest path is resolved against tools/battery/. gpulock is a command that rugnux is prefixed with under --gpulock. At PSI it is a wrapper script that queues for one of the shared GPU slots. Set it to null if you have no such wrapper. pdb_cache (optional) is where the deposited models and their metadata are cached for the model variant and the REFMAC check.

Running

B=tools/battery/battery.py
R=build/rugnux/rugnux

$B run --rugnux $R                                  # full battery: open + inhouse
$B run --rugnux $R --tier smoke                     # 22 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --only 5reo,lyso_ref             # named sets
$B run --rugnux $R --arm private                    # the private arm, on its own
$B run --rugnux $R --variants bare                  # only what a user gets (no model, no XDS settings)

Useful options:

• --label names the run. By default the label is the rugnux version.
• --threads N passes -N to rugnux.
• --gpulock queues every rugnux behind the site's GPU wrapper.
• --extra "<args>" adds rugnux arguments to every set.
• --baseline RUN|none sets the run the report is compared against.
• --variants bare,model,xds runs a subset of the variants (default: all of them).
• --model-check adds the REFMAC check to the open arm's bare rows (off by default, see below).

A run prints one line per set and ends by writing its report. The smoke tier is marked in the manifests ("tiers": {"smoke": "<why this set>"}).

Full runs. A run without --only or --tier takes the site lock (flock, non-blocking), so two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and everything else on the machine competes with it.

The REFMAC model check (open arm, opt-in)

The model variant is the battery's R-free check. --model-check adds a second, independent one that needs CCP4: tools/battery/model_check.py scores each open-arm bare merge (p.mtz) against the deposited model, unmodified (ligands, waters and hydrogens kept): REFMAC's rigid-body mode, R-factors taken before any shift, the data reindexed into the model's setting and reduced to the deposited space group. Our own free set was mostly work reflections for the depositor, so refmac_rfree reads low. The fair comparison is refmac_rfree_depflags (our data on the depositor's free set) against refmac_rfree_depdata (the depositor's structure factors, same free set, same protocol); refmac_rfree_ratio is that ratio, and the report plots it. It is null when the entry deposited no structure factors with a free set. The runner records refmac_rfree, refmac_rwork, refmac_rfree_depflags, refmac_rfree_depdata, refmac_rfree_ratio, refmac_status and refmac_reason. Small-molecule sets are skipped.

model_sweep.py, which ran rugnux --model over the open arm as a separate sweep, is retired. The model variant does the same inside the battery.

A run directory

<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>[_private]/ is never reused, and it is made read-only when the run completes.

path	content
bin/rugnux	the exact binary that ran (sha256 in the manifest)
manifest.json	the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, complete / aborted, gpu_shared, and every set with its resolved input, reference and tags
results.json	one row per set and variant (the schema is below)
report.md, report.html	the report (report_PRIVATE.* for a private run)
work/<arm>/<set>/<variant>/	run.log, p_report.txt, p.mtz, p.cif, ... and model_check/

Run states (battery.py list): complete; running (its runner process is alive); aborted (interrupted by Ctrl-C or SIGTERM, or marked with battery.py abort RUN --reason ...); unfinished (it died without a trace, which is treated the same as aborted). An unfinished run covers an arbitrary part of its sets. compare and report refuse it unless given --allow-incomplete, and the report then says so at the top.

results.json (schema 2)

This is a flat list with one object per set and variant, and every arm uses the same keys. Missing values are null. manifest.json records the schema version as results_schema, and the variants each arm ran as variants.

Schema 2 added variant and first_read. It also changed the meaning of rfree, rwork and rfree_ratio: they are now rugnux's own --model values (model variant), and the REFMAC check's moved to refmac_*. A schema-1 run is still read, and compared, as if each set had one variant: bare on the open arm; on the XDS arms xds, or bare if it was run --unforced (which still passed -A).

key	meaning
set, arm, variant, tags, input, cmd	the set's id, arm, variant (bare, model, xds), population tags, input file, and the exact command that ran
first_read	true for the first variant of its set that ran: the one that read the images from disk
verdict	pass, fail, unscored (no reference) or not_run (input missing, or no model for the model variant)
cause	why it failed: crash, reader, indexing, timeout, lattice_halved, lattice_doubled, lattice_other, sym_under, sym_over, sym_screw, sym_other, merge, false_lattice; or no_reference / no_input / no_model
reason	one line for a human
sgno, sg, pg / sgno_ref, sg_ref, pg_ref	space group number and name, and point group: ours / the reference's
sg_relation	open arm: how the two groups relate (sgequiv)
cell, cell_ref, cell_dev_pct, volume_ratio	cells, the largest edge deviation in % (over axis orders), and the primitive volume ratio
d_min, d_min_ref, res_gain_pct	our high-resolution limit, the reference's, and (ref - ours) / ref in %
d_min_ref_rule, d_min_xds	XDS arms: how d_min_ref was derived (below), and the range XDS merged
r_meas, cc_half, isa, completeness, multiplicity, i_over_sigma, indexing_rate, images	from rugnux's _report.txt
r_meas_ref, cc_half_ref, isa_ref, completeness_ref	XDS's overall values, over XDS's own range
rfree, rwork, model_fit, rfree_deposited, rfree_ratio	model variant: R-free, R-work and MODEL_FIT as rugnux reports them for the deposited model, the published R-free, and rfree / rfree_deposited
refmac_rfree, refmac_rwork, refmac_rfree_depflags, refmac_rfree_depdata, refmac_rfree_ratio, refmac_status, refmac_reason	the REFMAC check (--model-check, open arm, bare rows)
wall_s, rugnux_wall_s, elapsed_s, exit_code, gpu_others, rugnux_verdict	timing (wall_s is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line

The report

Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and SVG, no external files):

• a header with the binary, build flags, host, subset, variants and whether timing counts;
• a timing table per variant, saying which variant's times include reading the data;
• a section per variant, each with:
  • a summary per arm (pass / fail / unscored / not run, pass rate, median resolution gain, ISa, R_meas, time) with a verdict bar chart;
  • a table per population (tags such as cubic, cbf, lysozyme) and the distributions of the main metrics;
  • plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference) for each arm (not for xds, whose resolution is forced), and, for model, R_free / published R_free;
  • the failures, and one row per set with all the numbers;
• if there is a baseline, the delta against it: pass rates per arm and variant on the common rows, and every row that moved beyond noise.

Re-render a report, for example against another baseline (run directories are read-only, so the output goes elsewhere and is named after the run):

$B report RUN --out DIR [--baseline OTHER_RUN]      # writes DIR/<run>.md and DIR/<run>.html

Keeping a report in the repository. Public reports of blessed runs go to tools/battery/reports/<run>.md (Markdown only; the HTML stays with the run). report refuses to write a private run's report anywhere inside the repository.

Comparing runs

$B list                                             # runs, their state, and the baselines
$B compare RUN_A RUN_B [--all]                      # per-set table of what moved beyond noise
$B compare RUN_A RUN_B --rerun-changed              # rerun the changed sets with A's binary

compare pairs rows by arm, set id and variant, following the manifests' aliases across renames. Only the variants both runs ran on an arm are compared, so a --variants bare run against a full one compares bare. It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free or time moved by more than the noise thresholds in report.py (NOISE). Those thresholds are a first guess. --rerun-changed measures the noise directly: it reruns the changed sets with A's saved binary and options. A set that moves again under the same binary is noise. A set that reproduces A was really changed by B.

Protocol

1. Compare against a persisted baseline, never against a number from memory. The site config's baseline (and baseline_private) names a complete run. Every new run is reported against it by default. Set a new baseline only when the owner blesses a run.
2. Keep the manifests unchanged between the two runs you compare. A change to a manifest (new sets, a new reference, a different input) is committed on its own, and the baseline is re-run with it.
3. Pool fixes. Run the full battery once on a branch that carries several fixes, not once per fix. For one fix, run the sets it touches with --only, plus the smoke tier.
4. Timing. --gpulock, or another process seen on the GPU, marks the run "timing NOT a reference". Only a run with the machine to itself can be used to judge speed.
5. Resolution on the XDS arms. The xds variant forces XDS's resolution range (and -A where XDS kept Friedel mates apart), so both programs merge the same reflections and the statistics are like for like. This answers "is our processing better than XDS's, reflection for reflection". The bare variant answers "what does a user get". A change can win one and lose the other, and every run has both. Never quote the xds variant as what rugnux delivers.

The XDS reference d_min

CORRECT.LP gives the range XDS merged. That range is a resolution limit only if someone chose it. With INCLUDE_RESOLUTION_RANGE left at 0.0, or set at the detector edge, it is the detector's reach, not the crystal's. It then gives a meaningless reference, and forcing it on rugnux makes rugnux merge shells of pure noise. On one insulin set, forcing XDS's 1.08 A (where the data end near 1.8 A) turned rugnux's correct I23 into I222. XDS marks each shell's CC1/2 with * when it is significant, and that decides which case applies (inputs.reference_dmin):

• xds_range: the finest shell's CC1/2 is significant, so XDS's limit stands.
• cc_half_0.30: it is not, meaning XDS merged past its own signal. The reference is then the point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in 1/d² between shell centres.

The derived value is both the score reference and the range the xds variant forces. The manifest keeps XDS's own limit as dmin_xds and the rule used as dmin_rule. XDS's pooled R_meas, CC1/2 and completeness still cover its whole range.

Adding or changing datasets

• Open arm: put the dataset directory, named by its PDB code, under the open data root, then run battery.py discover. It proposes an input for every directory the manifest does not have (the largest sweep, chosen by frame count), and with --refresh it shows where discovery now disagrees with an unpinned manifest input. Add the row to open.json with the deposited reference (sg, sgno, cell, dmin) and tags. Set "pinned": true when the choice between sweeps was deliberate. Record the source and DOI in docs/EXTERNAL_TEST_DATA.md.
• Inhouse / private: add the row (id, input, tags, and "expect": "no_lattice" for a control), then run battery.py refs --arm inhouse --write to fill in ref from the CORRECT.LP beside the input. Without --write, refs only shows what would change. A reference measured by hand goes in ref_override, which refs leaves alone.
• Renamed data directories: battery.py remap --arm inhouse MAP.json --write, where MAP.json is {"old": "new"} (renames the set and its directory) or {"old id": {"id": "...", "input": "..."}}. Each old id is recorded under aliases in the manifest, so runs made before the rename still compare set by set. Without --write, the command only shows the result and flags inputs that do not exist.