Jungfraujoch/tools/battery

Rugnux validation battery

battery.py is the one way to run the Rugnux validation battery: run rugnux over a fixed list of rotation datasets, score each result against a reference, and write a standard report. Two runs of the same manifests with the same options give results you can compare set by set. Do not use ad-hoc runner scripts instead. The old ones (rugnux_vs_xds.py, battery_run.py, battery2_serial.py, batstats.py, ...) are retired because their numbers could not be compared with each other.

test_score.py covers the scoring rules that decide a verdict (stdlib unittest, no data and no run needed): cd tools/battery && python3 test_score.py.

Arms

arm	datasets	reference	manifest
open	public PDB depositions of raw diffraction data, plus a few published small-molecule sets	the deposited space group, cell and resolution	open.json (committed)
inhouse	standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), plus no-crystal controls	XDS, from the CORRECT.LP beside each dataset	inhouse.json (committed)
private	user data	XDS, like inhouse	outside the repository; the local site config gives its path

Scoring checks these things in order, and the first one that fails decides the verdict: did it run, is the lattice right, is the symmetry right, and on the XDS arms, is the merge as good as XDS's. The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is scored with sgequiv. On the XDS arms only the point group is scored, because XDS never tests a screw axis. A no-crystal control ("expect": "no_lattice") passes only if no lattice is reported. score.py has the details.

A question the data could not answer is unscored, not a failure. Two of these. The hand of an enantiomorphic pair (P4_1 vs P4_3) is not in the intensities at all, and sgequiv already counts it as a match. The other is a screw axis on a row the sweep never recorded - the axial row lies in the spindle's blind cone, or outside the resolution range, so nothing was measured that could confirm or refuse the screw. rugnux then offers the whole set in SPACE_GROUP_ALTERNATIVES, names the open axis in SPACE_GROUP_SCREW_UNDETERMINED, and writes the member claiming no screw, because a reflection file must carry one group. Such a set scores unscored / screw_undetermined rather than sym_screw, on three conditions, all necessary: rugnux named the axis, the reference group is among the ones it offered, and the only axis the two groups differ on is one it named. A screw called wrongly where the row WAS measured meets none of them and stays a sym_screw failure - this rule excuses an unanswerable question, never a wrong answer.

A row may accept more than one reference. Where the deposition and our reduction disagree, the disagreement is real, and no test available to us settles it - a knife-edge - the row lists the other acceptable answer in ref_alternatives and passes if our answer matches any of its references. This is not a statement that our answer is right; it says both descriptions are defensible and that picking either one is acceptable. ref keeps the deposition verbatim, each alternative must carry a why a reader can check (an alternative without one is a schema error, not a silent pass), and the report counts these rows on their own - summary column of which alt, its own bar-chart segment, and an Accepted alternatives section printing each reason - so they are never folded into the ordinary passes. Prefer this to unscored, which drops the row from scoring altogether and loses a test that still works, and to ref_override, which asserts that the reference is wrong.

Data quality is not scored. A verdict is about the lattice and the symmetry only. Resolution, CC1/2, R_meas (overall and lowest shell), completeness, multiplicity and ISa are reported beside XDS's or the deposition's values, in the per-set tables and plots, for a human to judge. On the XDS arms cc_half_noise_ratio = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) over XDS's own range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as XDS's would be with half of its observations.

Low-resolution R_meas is reported, not scored: the lowest-resolution shell of rugnux's own table (lowres_r_meas), of the reference-range table (refres_lowres_r_meas) and of XDS's CORRECT.LP (lowres_r_meas_ref, stored in the manifest as r_meas_low), with the shells' high-resolution limits. The reference-range table uses XDS's shell limits, so its lowest shell and XDS's cover the same reflections; lowres_r_meas_ratio is the one over the other, plotted per set.

One command per arm

Every set runs once, with one command per arm:

arm (reference)	command
open (deposition)	rugnux -o p --no-export-unmerged --model <deposited coordinates> <input>; sets without a deposited model (small molecules, unpublished sets, or coordinates that cannot be downloaded) run the same command without --model
inhouse, private (XDS)	rugnux -o p --no-export-unmerged [-A] --report-resolution <XDS d_min>,<XDS d_max> <input>

Neither --model nor --report-resolution changes the processing: the merged reflections, the resolution cut and every decision are the same with and without them (verified bit for bit). So the one run is both what a user gets and, through the extra report sections, the comparison with the reference:

• Own result (every arm): rugnux's own cut and statistics, scored against the reference and plotted as d_min(rugnux) / d_min(reference). This is what a user gets.
• --model (open arm): the deposited coordinates (<entry>.cif.gz) and the published R-free come from the RCSB and are cached in the site's pdb_cache (default /data/battery/pdb_cache), so the first run of a new entry needs network access. The entry is the set id up to the first _, so 6h2p_native uses 6h2p. The row records MODEL_FIT, R_FREE, R_WORK and CC_MODEL as trend fields: rugnux's R-free is a rigid-body placement scored on rugnux's own free set, not a refinement against the depositor's, so the ratio to the published R-free is not a like-for-like score; watch how it moves between runs. The comparable number is the REFMAC check (below). A model that fits may put its enantiomorph on the space-group label; the scorer reads the data's own determination instead (SOHNCKE_SPACE_GROUP where SPACE_GROUP_ENANTIOMORPH is ASSUMED_FROM_MODEL) and keeps the label in sg_label.
• --report-resolution (XDS arms): a second statistics table from the same merge, binned over XDS's own range (dmin_xds and XDS's d_max as given, 999 included, so it covers exactly what XDS's CORRECT.LP totals cover), and -A where XDS merged with FRIEDEL'S_LAW=FALSE ("anomalous": true in the manifest), so the two tables count the same way. Its REFRES_* numbers are shown beside XDS's CORRECT.LP totals: completeness, multiplicity, R_meas, the lowest shell's R_meas, CC1/2 and ISa (I/sigma is shown; the manifest holds no XDS value for it). Only CC1/2 is scored (above). Where rugnux's own cut is coarser than the reference (REFRES_SHELLS_PAST_LIMIT > 0), the shells past it are not merged, so the completeness is read as coverage of the reference range, not as a quality regression, and the other numbers are over the shells rugnux reached.

Timing. Each set runs once, so every time includes reading the images from disk (unless they were still in the page cache from before the run).

The private arm is never mixed with the others. A private run gets its own run directory (suffix _private) and its own report (report_PRIVATE.*), and it cannot be the baseline of a public report or be written into the repository. Its names, paths and values never go into a commit, an issue or a public report.

Getting the data

Each manifest row gives an input path relative to that arm's data root. The data root is set in the site config, and the first directory of each input is the dataset's directory:

<open data root>/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5
<inhouse data root>/lyso_ref/hewl_35_001_master.h5

To move the battery to another machine, copy the two data roots and point the new site config at the copies. The paths inside the manifests do not change.

• open: every dataset is listed with its source repository and dataset DOI in docs/EXTERNAL_TEST_DATA.md. The sources are IRRMC (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what each archive holds. Please cite those DOIs. At PSI the data root is /home/data/open, a symlink to /home/data/nonsls/raw. 51 of the dataset directories in it are themselves symlinks into /data/scout_staging2, so copy with symlinks followed: rsync -aL or cp -rL. A plain rsync -a copies dangling links. The open arm is about 1.7 TB.
• inhouse: these are our own measurements and cannot be downloaded publicly. Ask the maintainers for a copy. Each dataset directory holds the images and the XDS output (CORRECT.LP, XDS_ASCII.HKL, ...) that the reference comes from. At PSI the data root is /home/data/inhouse (about 130 GB). Leave out .master_backups_kek when copying it (rsync -a --exclude .master_backups_kek): it holds backups of private masters, not battery data.
• private: its data root and manifest are named only in the local site config.

Prerequisites

• Python 3.9 or newer, with gemmi (scoring). h5py is needed only by discover.
• A rugnux binary. Build it with the flags CI uses (-march=x86-64-v3, see CLAUDE.md), because CPU-bound results and timings differ between builds. The runner records the build flags from the CMakeCache.txt above the binary, or from the tree given with --build-dir.
• XDS is not needed. The XDS references are already in the manifests (refs re-reads them from CORRECT.LP).

Site config

Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it in this order:

1. --site PATH
2. $JFJOCH_BATTERY_SITE
3. tools/battery/site.json (git-ignored)

site.example.json is the committed template. Copy it and fill in your paths. The local file is kept out of git because it names the private manifest. At PSI the config is /data/battery/site.json: link it (ln -s /data/battery/site.json tools/battery/site.json) or export JFJOCH_BATTERY_SITE=/data/battery/site.json.

{
 "runs_root": "/data/battery/runs",
 "lock_file": "/data/battery/full_run.lock",
 "gpulock": "/data/tmp/gpulock",
 "baseline": "/data/battery/runs/<a blessed full run>",
 "baseline_private": null,
 "pdb_cache": "/data/battery/pdb_cache",
 "arms": {
  "open":    {"manifest": "open.json",    "reference": "deposition", "data_root": "/home/data/open"},
  "inhouse": {"manifest": "inhouse.json", "reference": "xds",        "data_root": "/home/data/inhouse"},
  "private": {"manifest": "/path/to/private.json", "reference": "xds",
              "data_root": "/path/to/private/data", "private": true}
 }
}

A relative manifest path is resolved against tools/battery/. gpulock is a command that rugnux is prefixed with under --gpulock. At PSI it is a wrapper script that queues for one of the shared GPU slots. Set it to null if you have no such wrapper. pdb_cache (optional) is where the deposited models and their metadata are cached for --model and the REFMAC check.

Running

B=tools/battery/battery.py
R=build/rugnux/rugnux

$B run --rugnux $R                                  # full battery: open + inhouse
$B run --rugnux $R --tier smoke                     # 22 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --only 5reo,lyso_ref             # named sets
$B run --rugnux $R --arm private                    # the private arm, on its own

Useful options:

• --label names the run. By default the label is the rugnux version.
• --threads N passes -N to rugnux.
• --gpulock queues every rugnux behind the site's GPU wrapper.
• --extra "<args>" adds rugnux arguments to every set.
• --baseline RUN|none sets the run the report is compared against.
• --model-check adds the REFMAC check to the open arm's rows (off by default, see below).

A run prints one line per set and ends by writing its report. The smoke tier is marked in the manifests ("tiers": {"smoke": "<why this set>"}).

Full runs. A run without --only or --tier takes the site lock (flock, non-blocking), so two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and everything else on the machine competes with it.

The REFMAC model check (open arm, opt-in)

rugnux's own --model R-free is a trend number. --model-check adds an independent one, comparable with the deposition, that needs CCP4: tools/battery/model_check.py scores each open-arm merge (p.mtz) against the deposited model, unmodified (ligands, waters and hydrogens kept): REFMAC's rigid-body mode, R-factors taken before any shift, the data reindexed into the model's setting and reduced to the deposited space group. Our own free set was mostly work reflections for the depositor, so refmac_rfree reads low. The fair comparison is refmac_rfree_depflags (our data on the depositor's free set) against refmac_rfree_depdata (the depositor's structure factors, same free set, same protocol); refmac_rfree_ratio is that ratio, and the report plots it. It is null when the entry deposited no structure factors with a free set. The runner records refmac_rfree, refmac_rwork, refmac_rfree_depflags, refmac_rfree_depdata, refmac_rfree_ratio, refmac_status and refmac_reason. Small-molecule sets are skipped.

model_sweep.py, which ran rugnux --model over the open arm as a separate sweep, is retired. The open arm's command does the same inside the battery.

A run directory

<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>[_private]/ is never reused, and it is made read-only when the run completes.

path	content
bin/rugnux	the exact binary that ran (sha256 in the manifest)
manifest.json	the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, complete / aborted, gpu_shared, and every set with its resolved input, reference and tags
results.json	one row per set (the schema is below)
report.md, report.html	the report (report_PRIVATE.* for a private run)
work/<arm>/<set>/	run.log, p_report.txt, p.mtz, p.cif, ... and model_check/

Run states (battery.py list): complete; running (its runner process is alive); aborted (interrupted by Ctrl-C or SIGTERM, or marked with battery.py abort RUN --reason ...); unfinished (it died without a trace, which is treated the same as aborted). An unfinished run covers an arbitrary part of its sets. compare and report refuse it unless given --allow-incomplete, and the report then says so at the top.

results.json (schema 3)

This is a flat list with one object per set, and every arm uses the same keys. Missing values are null. manifest.json records the schema version as results_schema, and each arm's command as command_doc.

Verdicts are scored again whenever a run is read (report, compare, the baseline delta): every row whose p_report.txt has a lattice (or that is a control) goes through today's score.judge with today's manifest row (reference, unscored, expect), so the two sides of a comparison are always scored by the same rules against the same references. A row whose report has no lattice keeps its stored verdict, because why it failed came from the run. results.json keeps the verdicts as they were scored when the run finished.

Schema 3 is one row per set again: variant and first_read are gone, and refres_*, isa_ratio, r_meas_ratio, multiplicity_ref, cc_model, sg_label, model and model_note are new. sgno/sg are now the data's own space group where --model relabelled the hand. Older runs are still read and compared: a schema-2 run as its bare rows, with the open arm's model rows' R-factors folded in. On the open arm that is the same processing as today's command; on the XDS arms today's command adds -A where XDS was anomalous, which changes how the merging statistics count (unique reflections, multiplicity, I/sigma, ISa), so expect those to move there against an old run; its xds rows, which forced XDS's range on the processing, have no counterpart and are left out. A schema-1 run is read as schema 2 with one variant first (bare on the open arm; on the XDS arms xds, or bare if it was run --unforced), so its forced XDS-arm rows are left out the same way. compare --rerun-changed needs A to be a schema-3 run.

key	meaning
set, arm, tags, input, cmd	the set's id, arm, population tags, input file, and the exact command that ran
verdict	pass, fail, unscored (no reference, or a question the data could not answer) or not_run (input missing)
cause	why it failed: crash, reader, indexing, timeout, lattice_halved, lattice_doubled, lattice_other, sym_under, sym_over, sym_screw, sym_other, false_lattice; or no_reference / no_input / reference_problem / screw_undetermined; or, on a pass, accepted_alternative
accepted_alt, accepted_alt_why	the alternative reference the answer matched and the reason the row accepts it, on a row that passed that way; null otherwise
reason	one line for a human
sgno, sg, pg / sgno_ref, sg_ref, pg_ref	space group number and name, and point group: ours (the data's own determination) / the reference's
sg_label	the space group rugnux reported where --model put the model's enantiomorph on it, else null
sg_relation	open arm: how the two groups relate (sgequiv)
cell, cell_ref, cell_dev_pct, volume_ratio	cells, the largest edge deviation in % (over axis orders), and the primitive volume ratio
d_min, d_min_ref, res_gain_pct	our high-resolution limit, the reference's, and (ref - ours) / ref in %
d_min_ref_rule, d_min_xds	XDS arms: how d_min_ref was derived (below), and the range XDS merged
r_meas, cc_half, isa, completeness, multiplicity, i_over_sigma, indexing_rate, images	from rugnux's _report.txt, over its own range
r_meas_ref, cc_half_ref, isa_ref, completeness_ref, multiplicity_ref	XDS's overall values, over XDS's own range
refres_range, refres_shells_past_limit, refres_unique_reflections, refres_completeness, refres_multiplicity, refres_i_over_sigma, refres_r_meas, refres_cc_half, refres_isa	XDS arms: the REFRES_* keys, the same merge over the reference range
isa_ratio, r_meas_ratio	refres_isa / isa_ref and refres_r_meas / r_meas_ref
cc_half_noise_ratio	XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), reported, not scored
lowres_d, lowres_r_meas, refres_lowres_d, refres_lowres_r_meas, lowres_d_ref, lowres_r_meas_ref, lowres_r_meas_ratio	the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref
model, model_note	open arm: the deposited coordinates given to --model, or why there were none
rfree, rwork, cc_model, model_fit, rfree_deposited, rfree_ratio	open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited
refmac_rfree, refmac_rwork, refmac_rfree_depflags, refmac_rfree_depdata, refmac_rfree_ratio, refmac_status, refmac_reason	the REFMAC check (--model-check, open arm)
wall_s, rugnux_wall_s, elapsed_s, exit_code, gpu_others, rugnux_verdict	timing (wall_s is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line

The report

Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and SVG, no external files):

• a header with the binary, build flags, host, subset, each arm's command and whether timing counts;
• a summary per arm (pass, of which accepted on an alternative reference / fail / unscored / not run, pass rate, median resolution gain, ISa, R_meas, time) with a verdict bar chart, all from rugnux's own run, and an Accepted alternatives section naming those rows and why;
• a table per population (tags such as cubic, cbf, lysozyme) and the distributions of the main metrics;
• plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference) for each arm; on the XDS arms the reference-range ISa, R_meas, lowest-shell R_meas and CC1/2 noise over XDS's; on the open arm R_free / published R_free;
• on the XDS arms the like-for-like table: the reference-range numbers beside XDS's, with the rows where rugnux's cut is coarser marked as coverage;
• the failures, and one row per set with all the numbers;
• if there is a baseline, the delta against it: pass rates per arm on the common sets, and every set that moved beyond noise.

Re-render a report, for example against another baseline (run directories are read-only, so the output goes elsewhere and is named after the run):

$B report RUN --out DIR [--baseline OTHER_RUN]      # writes DIR/<run>.md and DIR/<run>.html

Keeping a report in the repository. Public reports of blessed runs go to tools/battery/reports/<run>.md (Markdown only; the HTML stays with the run). report refuses to write a private run's report anywhere inside the repository.

Comparing runs

$B list                                             # runs, their state, and the baselines
$B compare RUN_A RUN_B [--all]                      # per-set table of what moved beyond noise
$B compare RUN_A RUN_B --rerun-changed              # rerun the changed sets with A's binary

compare pairs rows by arm and set id, following the manifests' aliases across renames. A run from before schema 3 is compared through its bare rows (see the schema above). It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free, reference-range R_meas, lowest-shell R_meas or ISa (where both runs have them), or time moved by more than the noise thresholds in report.py (NOISE). Those thresholds are a first guess. --rerun-changed measures the noise directly: it reruns the changed sets with A's saved binary and options. A set that moves again under the same binary is noise. A set that reproduces A was really changed by B.

Protocol

1. Compare against a persisted baseline, never against a number from memory. The site config's baseline (and baseline_private) names a complete run. Every new run is reported against it by default. Set a new baseline only when the owner blesses a run.
2. Keep the manifests unchanged between the two runs you compare. A change to a manifest (new sets, a new reference, a different input) is committed on its own, and the baseline is re-run with it.
3. Pool fixes. Run the full battery once on a branch that carries several fixes, not once per fix. For one fix, run the sets it touches with --only, plus the smoke tier.
4. Timing. --gpulock, or another process seen on the GPU, marks the run "timing NOT a reference". Only a run with the machine to itself can be used to judge speed.
5. Resolution on the XDS arms. Nothing forces XDS's range on the processing any more: rugnux keeps its own cut, and the reference-range table re-bins the same merge over XDS's range. The own numbers answer "what does a user get"; the reference-range numbers answer "is our merge better than XDS's over the same range". Where rugnux's cut is coarser than XDS's, the second question is only answered over the shells rugnux reached, and the missing shells are coverage.

The XDS reference d_min

CORRECT.LP gives the range XDS merged. That range is a resolution limit only if someone chose it. With INCLUDE_RESOLUTION_RANGE left at 0.0, or set at the detector edge, it is the detector's reach, not the crystal's. It then gives a meaningless reference, and forcing it on rugnux makes rugnux merge shells of pure noise. On one insulin set, forcing XDS's 1.08 A (where the data end near 1.8 A) turned rugnux's correct I23 into I222. XDS marks each shell's CC1/2 with * when it is significant, and that decides which case applies (inputs.reference_dmin):

• xds_range: the finest shell's CC1/2 is significant, so XDS's limit stands.
• cc_half_0.30: it is not, meaning XDS merged past its own signal. The reference is then the point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in 1/d² between shell centres.

The derived value is the reference rugnux's own cut is scored against. The reference-range table uses XDS's own limit, dmin_xds, because XDS's pooled R_meas, CC1/2 and completeness cover its whole range; the manifest keeps the rule used as dmin_rule.

Adding or changing datasets

• Open arm: put the dataset directory, named by its PDB code, under the open data root, then run battery.py discover. It proposes an input for every directory the manifest does not have (the largest sweep, chosen by frame count), and with --refresh it shows where discovery now disagrees with an unpinned manifest input. Add the row to open.json with the deposited reference (sg, sgno, cell, dmin) and tags. Set "pinned": true when the choice between sweeps was deliberate. Record the source and DOI in docs/EXTERNAL_TEST_DATA.md.
• Inhouse / private: add the row (id, input, tags, and "expect": "no_lattice" for a control), then run battery.py refs --arm inhouse --write to fill in ref from the CORRECT.LP beside the input. Without --write, refs only shows what would change. A reference measured by hand goes in ref_override, which refs leaves alone. A set whose reference is known to be wrong gets "unscored": "<why>": it still runs, but scores unscored instead of pass or fail. A set where two answers are both defensible gets "ref_alternatives": [{"sg": ..., "sgno": ..., "cell": [...], "why": "<reason>"}] - the fields given replace the reference's, the row passes on any of its references, and why is mandatory.
• Renamed data directories: battery.py remap --arm inhouse MAP.json --write, where MAP.json is {"old": "new"} (renames the set and its directory) or {"old id": {"id": "...", "input": "..."}}. Each old id is recorded under aliases in the manifest, so runs made before the rename still compare set by set. Without --write, the command only shows the result and flags inputs that do not exist.