Jungfraujoch/tools/battery

Rugnux validation battery

battery.py is the one way to run the Rugnux validation battery: run rugnux over a fixed list of rotation datasets, score each result against a reference, and write a standard report. Two runs of the same manifests with the same options give results you can compare set by set. Do not use ad-hoc runner scripts instead. The old ones (rugnux_vs_xds.py, battery_run.py, battery2_serial.py, batstats.py, ...) are retired because their numbers could not be compared with each other.

test_score.py covers the scoring rules that decide a verdict (stdlib unittest, no data and no run needed): cd tools/battery && python3 test_score.py.

Arms

arm	datasets	reference	manifest
open	public PDB depositions of raw diffraction data, plus a few published small-molecule sets	the deposited space group, cell and resolution	open.json (committed)
inhouse	standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), plus no-crystal controls	XDS, from the CORRECT.LP beside each dataset	inhouse.json (committed)
private	user data	XDS, like inhouse	outside the repository; the local site config gives its path

Scoring checks these things in order, and the first one that fails decides the verdict: did it run, is the lattice right, is the symmetry right, and on the XDS arms, is the merge as good as XDS's. The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is scored with sgequiv. On the XDS arms only the point group is scored, because XDS never tests a screw axis. A no-crystal control ("expect": "no_lattice") passes only if no lattice is reported. score.py has the details.

A question the data could not answer is unscored, not a failure. Two of these. The hand of an enantiomorphic pair (P4_1 vs P4_3) is not in the intensities at all, and sgequiv already counts it as a match. The other is a screw axis on a row the sweep never recorded - the axial row lies in the spindle's blind cone, or outside the resolution range, so nothing was measured that could confirm or refuse the screw. rugnux then offers the whole set in SPACE_GROUP_ALTERNATIVES, names the open axis in SPACE_GROUP_SCREW_UNDETERMINED, and writes the member claiming no screw, because a reflection file must carry one group. Such a set scores unscored / screw_undetermined rather than sym_screw, on three conditions, all necessary: rugnux named the axis, the reference group is among the ones it offered, and the only axis the two groups differ on is one it named. A screw called wrongly where the row WAS measured meets none of them and stays a sym_screw failure - this rule excuses an unanswerable question, never a wrong answer.

A row may accept more than one reference. Where the deposition and our reduction disagree, the disagreement is real, and no test available to us settles it - a knife-edge - the row lists the other acceptable answer in ref_alternatives and passes if our answer matches any of its references. This is not a statement that our answer is right; it says both descriptions are defensible and that picking either one is acceptable. ref keeps the deposition verbatim, each alternative must carry a why a reader can check (an alternative without one is a schema error, not a silent pass), and the report counts these rows on their own - summary column of which alt, its own bar-chart segment, and an Accepted alternatives section printing each reason - so they are never folded into the ordinary passes. Prefer this to unscored, which drops the row from scoring altogether and loses a test that still works, and to ref_override, which asserts that the reference is wrong.

Data quality is not scored. A verdict is about the lattice and the symmetry only. Resolution, CC1/2, R_meas (overall and lowest shell), completeness, multiplicity and ISa are reported beside XDS's or the deposition's values, in the per-set tables and plots, for a human to judge. On the XDS arms cc_half_noise_ratio = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) over XDS's own range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as XDS's would be with half of its observations.

R_free does not compare between runs; rmodel_shell_scaled does. rugnux scales the model to the data with an overall factor and a symmetry-constrained anisotropic B, which can only bend one way with resolution - deliberately, so that the maps of a batch stay comparable. Whatever the amplitudes' own radial profile does that this shape cannot follow is reported as R. So two runs whose merged amplitudes have different radial profiles have R_free values that differ for a reason that has nothing to do with the data fitting the model: measured on one such pair, R_free moved a median +0.042 over eleven sets where phenix.model_vs_data, which scales per shell, moved +0.002. rmodel_shell_scaled is the same R with one free scale per resolution shell, over every reflection rather than the free 5% (nothing is refined here, so the free set buys no cross-validation and only costs precision): on that same pair its median was +0.001, and it still flagged the one set that was a real regression. radial_misfit is the size of that per-shell rescale, the RMS of its logarithm; when it moves, R_free beside it cannot be read. All three are reported for a human, never scored.

Low-resolution R_meas is reported, not scored: the lowest-resolution shell of rugnux's own table (lowres_r_meas), of the reference-range table (refres_lowres_r_meas) and of XDS's CORRECT.LP (lowres_r_meas_ref, stored in the manifest as r_meas_low), with the shells' high-resolution limits. The reference-range table uses XDS's shell limits, so its lowest shell and XDS's cover the same reflections; lowres_r_meas_ratio is the one over the other, plotted per set.

One command per arm

Every set runs once, with one command per arm:

arm (reference)	command
open (deposition)	rugnux -o p --no-export-unmerged --model <deposited coordinates> <input>; sets without a deposited model (small molecules, unpublished sets, or coordinates that cannot be downloaded) run the same command without --model
inhouse, private (XDS)	rugnux -o p --no-export-unmerged [-A] --report-resolution <XDS d_min>,<XDS d_max> <input>

Neither --model nor --report-resolution changes the processing: the merged reflections, the resolution cut and every decision are the same with and without them (verified bit for bit). So the one run is both what a user gets and, through the extra report sections, the comparison with the reference:

• Own result (every arm): rugnux's own cut and statistics, scored against the reference and plotted as d_min(rugnux) / d_min(reference). This is what a user gets.
• --model (open arm): the deposited coordinates (<entry>.cif.gz) and the published R-free come from the RCSB and are cached in the site's pdb_cache (default /data/battery/pdb_cache), so the first run of a new entry needs network access. The entry is the set id up to the first _, so 6h2p_native uses 6h2p. The row records MODEL_FIT, R_FREE, R_WORK and CC_MODEL as trend fields: rugnux's R-free is a rigid-body placement scored on rugnux's own free set, not a refinement against the depositor's, so the ratio to the published R-free is not a like-for-like score; watch how it moves between runs. The comparable number is the REFMAC check (below). A model that fits may put its enantiomorph on the space-group label; the scorer reads the data's own determination instead (SOHNCKE_SPACE_GROUP where SPACE_GROUP_ENANTIOMORPH is ASSUMED_FROM_MODEL) and keeps the label in sg_label.
• --report-resolution (XDS arms): a second statistics table from the same merge, binned over XDS's own range (dmin_xds and XDS's d_max as given, 999 included, so it covers exactly what XDS's CORRECT.LP totals cover), and -A where XDS merged with FRIEDEL'S_LAW=FALSE ("anomalous": true in the manifest), so the two tables count the same way. Its REFRES_* numbers are shown beside XDS's CORRECT.LP totals: completeness, multiplicity, R_meas, the lowest shell's R_meas, CC1/2 and ISa (I/sigma is shown; the manifest holds no XDS value for it). Only CC1/2 is scored (above). Where rugnux's own cut is coarser than the reference (REFRES_SHELLS_PAST_LIMIT > 0), the shells past it are not merged, so the completeness is read as coverage of the reference range, not as a quality regression, and the other numbers are over the shells rugnux reached.

Timing. Each set runs once, so every time includes reading the images from disk (unless they were still in the page cache from before the run).

The private arm is never mixed with the others. A private run gets its own run directory (suffix _private) and its own report (report_PRIVATE.*), and it cannot be the baseline of a public report or be written into the repository. Its names, paths and values never go into a commit, an issue or a public report.

Getting the data

Each manifest row gives an input path relative to that arm's data root. The data root is set in the site config, and the first directory of each input is the dataset's directory:

<open data root>/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5
<inhouse data root>/lyso_ref/hewl_35_001_master.h5

To move the battery to another machine, copy the two data roots and point the new site config at the copies. The paths inside the manifests do not change.

• open: every dataset is listed with its source repository and dataset DOI in docs/EXTERNAL_TEST_DATA.md. The sources are IRRMC (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what each archive holds. Please cite those DOIs. At PSI the data root is /home/data/open, a symlink to /home/data/nonsls/raw. 51 of the dataset directories in it are themselves symlinks into /data/scout_staging2, so copy with symlinks followed: rsync -aL or cp -rL. A plain rsync -a copies dangling links. The open arm is about 1.7 TB.
• inhouse: these are our own measurements and cannot be downloaded publicly. Ask the maintainers for a copy. Each dataset directory holds the images and the XDS output (CORRECT.LP, XDS_ASCII.HKL, ...) that the reference comes from. At PSI the data root is /home/data/inhouse (about 130 GB). Leave out .master_backups_kek when copying it (rsync -a --exclude .master_backups_kek): it holds backups of private masters, not battery data.
• private: its data root and manifest are named only in the local site config.

Prerequisites

• Python 3.9 or newer, with gemmi (scoring). h5py is needed only by discover.
• A rugnux binary. Build it with the flags CI uses (-march=x86-64-v3, see CLAUDE.md), because CPU-bound results and timings differ between builds. The runner records the build flags from the CMakeCache.txt above the binary, or from the tree given with --build-dir.
• XDS is not needed. The XDS references are already in the manifests (refs re-reads them from CORRECT.LP).

Site config

Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it in this order:

1. --site PATH
2. $JFJOCH_BATTERY_SITE
3. tools/battery/site.json (git-ignored)

site.example.json is the committed template. Copy it and fill in your paths. The local file is kept out of git because it names the private manifest. At PSI the config is /data/battery/site.json: link it (ln -s /data/battery/site.json tools/battery/site.json) or export JFJOCH_BATTERY_SITE=/data/battery/site.json.

{
 "runs_root": "/data/battery/runs",
 "lock_file": "/data/battery/full_run.lock",
 "gpulock": "/data/tmp/gpulock",
 "baseline": "/data/battery/runs/<a blessed full run>",
 "baseline_private": null,
 "pdb_cache": "/data/battery/pdb_cache",
 "arms": {
  "open":    {"manifest": "open.json",    "reference": "deposition", "data_root": "/home/data/open"},
  "inhouse": {"manifest": "inhouse.json", "reference": "xds",        "data_root": "/home/data/inhouse"},
  "private": {"manifest": "/path/to/private.json", "reference": "xds",
              "data_root": "/path/to/private/data", "private": true}
 }
}

A relative manifest path is resolved against tools/battery/. gpulock is a command that rugnux is prefixed with under --gpulock. At PSI it is a wrapper script that queues for one of the shared GPU slots. Set it to null if you have no such wrapper. pdb_cache (optional) is where the deposited models and their metadata are cached for --model and the REFMAC check.

Running

B=tools/battery/battery.py
R=build/rugnux/rugnux

$B run --rugnux $R                                  # full battery: open + inhouse
$B run --rugnux $R --tier smoke                     # 22 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --only 5reo,lyso_ref             # named sets
$B run --rugnux $R --arm private                    # the private arm, on its own

Useful options:

• --label names the run. By default the label is the rugnux version.
• --threads N passes -N to rugnux.
• --gpulock queues every rugnux behind the site's GPU wrapper.
• --extra "<args>" adds rugnux arguments to every set.
• --baseline RUN|none sets the run the report is compared against.
• --model-check adds the REFMAC check to the open arm's rows (off by default, see below).

A run prints one line per set and ends by writing its report. The smoke tier is marked in the manifests ("tiers": {"smoke": "<why this set>"}).

Full runs. A run without --only or --tier takes the site lock (flock, non-blocking), so two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and everything else on the machine competes with it.

The REFMAC model check (open arm, opt-in)

rugnux's own --model R-free is a trend number. --model-check adds an independent one, comparable with the deposition, that needs CCP4: tools/battery/model_check.py scores each open-arm merge (p.mtz) against the deposited model, unmodified (ligands, waters and hydrogens kept; only atoms of unknown element, UNX, are left out because REFMAC stops on them): REFMAC's rigid-body mode, R-factors taken before any shift, the data reindexed into the model's setting and reduced to the deposited space group. The depositor's own structure factors go through the same setting choice (they can sit in the other branch of a merohedral ambiguity) and are scored by the same protocol.

All REFMAC numbers use the same high-resolution limit, d_min_used = max(our d_min, the deposited d_min), i.e. the lower of the two resolutions, so neither data set is scored past what the other reaches. The four R-free fields, all of the same deposited model:

field	data	free set	reads as
refmac_rfree	ours	ours	close to a work R: most of our free reflections were work reflections to the depositor's refinement
refmac_rfree_depflags	ours	the depositor's	the fair number for our data
refmac_rfree_depdata	the depositor's	the depositor's	the baseline: same model, same protocol, the data the model was refined against
refmac_rfree_ratio			refmac_rfree_depflags / refmac_rfree_depdata; below 1 = our data fit the model better; the report plots it

refmac_rwork is R on our own work reflections. The depositor-data numbers carry a home advantage (the model was refined against those data), and refmac_rfree_depdata is not the published R-free either: different program (REFMAC vs e.g. PHENIX, whose resolution-binned scaling absorbs a non-Wilson falloff that REFMAC's overall scale cannot) and no refinement. The depositor's amplitudes are F_meas, else the mean of F(+)/F(-); where only intensities were deposited they are converted by French-Wilson (ctruncate), as our own amplitudes are. An entry that declares twinning (_pdbx_reflns_twin) is scored with REFMAC's twin refinement for both data sets, since its published R-free is a twinned one; the untwinned numbers are kept in refmac_untwinned. The ratio is null when the entry deposited no structure factors with a free set (refmac_reason says why). The runner records refmac_rfree, refmac_rwork, refmac_rfree_depflags, refmac_rfree_depdata, refmac_rfree_ratio, refmac_twin, refmac_untwinned, refmac_status and refmac_reason. Small-molecule sets are skipped. model_check.py <p.mtz> <pdb id> --workdir DIR runs it on one merge and prints every field (setting, reindexing operators of both data sets, what the depositor's amplitudes were made from).

Refined, shared free set (second protocol). The first-cycle numbers above score the model as deposited at the common resolution limit. The second protocol refines the same deposited model against each data set at that data set's OWN resolution and compares R-free on a free set both share:

• same starting model, same REFMAC protocol for both data sets: restrained refinement, 10 cycles, automatic weight, isotropic B for every atom (a deposited ANISOU is often TLS-derived, and refining it atom by atom diverged), twin refinement where the entry declares twinning; a ligand whose code the monomer library describes with other atoms is renamed to a code the library lacks, so REFMAC makes its restraints from the model's own coordinates (ligands_renamed in the check's output);
• each data set over its whole resolution range; its work set is every reflection it has except the shared free set;
• shared free set = the depositor's free reflections present in BOTH data sets (after the change of basis), which takes the lower of the two limits in every direction, anisotropic cuts and rejected reflections included; depositor free reflections outside it are removed from both;
• R-free recomputed from each output MTZ over exactly the shared set (F against FC_ALL_LS, i.e. with REFMAC's own scale and bulk solvent, which reproduces REFMAC's R-free over its free set); for a twinned refinement the output MTZ holds no twinned Fc, so REFMAC's own R-free is used - its free set IS the shared set.

field	meaning
refmac_refined_rfree_shared	our data, refined, R-free on the shared set - the refined number for our data
refmac_refined_rfree_shared_depdata	the depositor's data, same protocol, same shared set
refmac_refined_ratio	the ratio of the two; below 1 = our data give the better refined model
refmac_refined_rwork, refmac_refined_rwork_depdata	REFMAC's R-work after refinement, each data set (their own work sets and resolutions: not comparable with each other)
refmac_refined_rfree_refmac, refmac_refined_rfree_refmac_depdata	REFMAC's own R-free, for reference
refmac_shared_free_n	size of the shared free set
refmac_refined_reason	why the refined protocol gave nothing, when it did not (the first-cycle numbers stand)

The refined numbers need the depositor's structure factors with a free set, like the ratio; they take minutes to an hour per set. model_check.py --no-refine skips them.

Rechecking an existing run

battery.py recheck RUN [--only a,b] [--jobs N] reruns the model check and the depositor-data comparison on every open-arm row of a finished run, from the run's own p.mtz (CPU only; rugnux is not run again), so numbers made by an older model_check.py / depdata_check.py can be brought up to date without reprocessing. It updates the refmac_* and dep_* fields of results.json in place and nothing else (verdicts stay as scored). The previous files stay beside the new ones: results.json.pre-recheck-<stamp>, report.{md,html}.pre-recheck-<stamp> and each set's model_check.pre-recheck-<stamp>/. The report is re-rendered, and manifest.json gains an entry in its rechecks list - date, end, runner_git, runner_dirty (the tools version that made the new numbers), sets, fields, previous_results and command - so which version of the checks produced which figures stays auditable. The run directory is made read-only again.

The depositor-data comparison (open arm, always on)

tools/battery/depdata_check.py compares our merge with the depositor's own data without CCP4: per resolution shell, the rank correlation of our IMEAN with |Fc|^2 minus that of the deposited intensities (or amplitudes squared), on the reflections both carry, with |Fc| from the deposited model as it is (gemmi, no bulk solvent, no refinement), the data carried into the model's setting as the REFMAC check does it. The model was refined against the depositor's data, so the comparison has a home advantage for the deposition; around -0.01 is typical, and a set far below that merged worse than the deposition at those resolutions. It runs on every open-arm set with a model (seconds to a minute, gemmi only; the -sf.cif is cached in pdb_cache beside the coordinates), and report computes it for an older run that lacks it. It is reported, never scored. Row keys: dep_cc_delta_all (all common reflections at d < 4 A), dep_cc_delta_outer (mean of the two outermost of eight equal-count shells), dep_beyond_cc and dep_beyond_d (the correlation of our reflections with |Fc|^2 in the outermost of up to three shells past the deposited data's limit, and that shell's limit: the model never saw these, so clearly above zero is signal), plus dep_kind (I or F), dep_d_min (the deposited data's limit), dep_n_common, dep_status and dep_reason.

model_sweep.py, which ran rugnux --model over the open arm as a separate sweep, is retired. The open arm's command does the same inside the battery.

A run directory

<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>[_private]/ is never reused, and it is made read-only when the run completes.

path	content
bin/rugnux	the exact binary that ran (sha256 in the manifest)
manifest.json	the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, complete / aborted, gpu_shared, and every set with its resolved input, reference and tags
results.json	one row per set (the schema is below)
report.md, report.html	the report (report_PRIVATE.* for a private run)
work/<arm>/<set>/	run.log, p_report.txt, p.mtz, p.cif, ... and model_check/

Run states (battery.py list): complete; running (its runner process is alive); aborted (interrupted by Ctrl-C or SIGTERM, or marked with battery.py abort RUN --reason ...); unfinished (it died without a trace, which is treated the same as aborted). An unfinished run covers an arbitrary part of its sets. compare and report refuse it unless given --allow-incomplete, and the report then says so at the top.

results.json (schema 3)

This is a flat list with one object per set, and every arm uses the same keys. Missing values are null. manifest.json records the schema version as results_schema, and each arm's command as command_doc.

Verdicts are scored again whenever a run is read (report, compare, the baseline delta): every row whose p_report.txt has a lattice (or that is a control) goes through today's score.judge with today's manifest row (reference, unscored, expect), so the two sides of a comparison are always scored by the same rules against the same references. A row whose report has no lattice keeps its stored verdict, because why it failed came from the run. results.json keeps the verdicts as they were scored when the run finished.

Schema 3 is one row per set again: variant and first_read are gone, and refres_*, isa_ratio, r_meas_ratio, multiplicity_ref, cc_model, sg_label, model and model_note are new. sgno/sg are now the data's own space group where --model relabelled the hand. Older runs are still read and compared: a schema-2 run as its bare rows, with the open arm's model rows' R-factors folded in. On the open arm that is the same processing as today's command; on the XDS arms today's command adds -A where XDS was anomalous, which changes how the merging statistics count (unique reflections, multiplicity, I/sigma, ISa), so expect those to move there against an old run; its xds rows, which forced XDS's range on the processing, have no counterpart and are left out. A schema-1 run is read as schema 2 with one variant first (bare on the open arm; on the XDS arms xds, or bare if it was run --unforced), so its forced XDS-arm rows are left out the same way. compare --rerun-changed needs A to be a schema-3 run.

key	meaning
set, arm, tags, input, cmd	the set's id, arm, population tags, input file, and the exact command that ran
verdict	pass, fail, unscored (no reference, or a question the data could not answer) or not_run (input missing)
cause	why it failed: crash, reader, indexing, timeout, lattice_halved, lattice_doubled, lattice_other, sym_under, sym_over, sym_screw, sym_other, false_lattice; or no_reference / no_input / reference_problem / screw_undetermined; or, on a pass, accepted_alternative
accepted_alt, accepted_alt_why	the alternative reference the answer matched and the reason the row accepts it, on a row that passed that way; null otherwise
reason	one line for a human
sgno, sg, pg / sgno_ref, sg_ref, pg_ref	space group number and name, and point group: ours (the data's own determination) / the reference's
sg_label	the space group rugnux reported where --model put the model's enantiomorph on it, else null
sg_relation	open arm: how the two groups relate (sgequiv)
cell, cell_ref, cell_dev_pct, volume_ratio	cells, the largest edge deviation in % (over axis orders), and the primitive volume ratio
d_min, d_min_ref, res_gain_pct	our high-resolution limit, the reference's, and (ref - ours) / ref in %
d_min_ref_rule, d_min_xds	XDS arms: how d_min_ref was derived (below), and the range XDS merged
r_meas, cc_half, isa, completeness, multiplicity, i_over_sigma, indexing_rate, images	from rugnux's _report.txt, over its own range
mosaicity_deg, extra_lattice_intensity_pct	from the same report: MOSAICITY_DEG= (sigma_M as XDS) and EXTRA_LATTICE_INTENSITY_PCT= (spot intensity on further lattices); null in reports without them
r_meas_ref, cc_half_ref, isa_ref, completeness_ref, multiplicity_ref	XDS's overall values, over XDS's own range
refres_range, refres_shells_past_limit, refres_unique_reflections, refres_completeness, refres_multiplicity, refres_i_over_sigma, refres_r_meas, refres_cc_half, refres_isa	XDS arms: the REFRES_* keys, the same merge over the reference range
isa_ratio, r_meas_ratio	refres_isa / isa_ref and refres_r_meas / r_meas_ref
cc_half_noise_ratio	XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), reported, not scored
lowres_d, lowres_r_meas, refres_lowres_d, refres_lowres_r_meas, lowres_d_ref, lowres_r_meas_ref, lowres_r_meas_ratio	the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref
model, model_note	open arm: the deposited coordinates given to --model, or why there were none
rfree, rwork, cc_model, model_fit, rfree_deposited, rfree_ratio	open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited
rmodel, rmodel_shell_scaled, radial_misfit	open arm with a model: R_MODEL, R_MODEL_SHELL_SCALED and MODEL_RADIAL_MISFIT - R over all the reflections, with and without one free scale per resolution shell, and how big that rescale was. rmodel_shell_scaled is the R that compares between runs (below)
refmac_rfree, refmac_rwork, refmac_rfree_depflags, refmac_rfree_depdata, refmac_rfree_ratio, refmac_twin, refmac_untwinned, refmac_status, refmac_reason, refmac_refined_*, refmac_shared_free_n	the REFMAC check (--model-check, open arm; see its section for which R-free is which)
dep_status, dep_reason, dep_kind, dep_d_min, dep_n_common, dep_cc_delta_all, dep_cc_delta_outer, dep_beyond_cc, dep_beyond_d	open arm with a model: our merge against the depositor's data per shell (depdata_check.py, above)
wall_s, rugnux_wall_s, elapsed_s, exit_code, gpu_others, rugnux_verdict	timing (wall_s is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line

The report

Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and SVG, no external files):

• a header with the binary, build flags, host, subset, each arm's command and whether timing counts;
• a summary per arm (pass, of which accepted on an alternative reference / fail / unscored / not run, pass rate, median resolution gain, ISa, R_meas, time) with a verdict bar chart, all from rugnux's own run, and an Accepted alternatives section naming those rows and why;
• a table per population (tags such as cubic, cbf, lysozyme) and the distributions of the main metrics;
• plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference) for each arm; on the XDS arms the reference-range ISa, R_meas, lowest-shell R_meas and CC1/2 noise over XDS's; on the open arm R_free / published R_free;
• on the open arm, against the depositor's data: the per-shell CC difference and the CC past the deposited limit, one row per set, sorted by the outer-shell difference;
• on the XDS arms the like-for-like table: the reference-range numbers beside XDS's, with the rows where rugnux's cut is coarser marked as coverage;
• the failures, and one row per set with all the numbers;
• if there is a baseline, the delta against it: pass rates per arm on the common sets, and every set that moved beyond noise.

Re-render a report, for example against another baseline (run directories are read-only, so the output goes elsewhere and is named after the run):

$B report RUN --out DIR [--baseline OTHER_RUN]      # writes DIR/<run>.md and DIR/<run>.html

Keeping a report in the repository. Public reports of blessed runs go to tools/battery/reports/<run>.md (Markdown only; the HTML stays with the run). report refuses to write a private run's report anywhere inside the repository.

Comparing runs

$B list                                             # runs, their state, and the baselines
$B compare RUN_A RUN_B [--all]                      # per-set table of what moved beyond noise
$B compare RUN_A RUN_B --rerun-changed              # rerun the changed sets with A's binary

compare pairs rows by arm and set id, following the manifests' aliases across renames. A run from before schema 3 is compared through its bare rows (see the schema above). It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free, reference-range R_meas, lowest-shell R_meas or ISa (where both runs have them), or time moved by more than the noise thresholds in report.py (NOISE). Those thresholds are a first guess. --rerun-changed measures the noise directly: it reruns the changed sets with A's saved binary and options. A set that moves again under the same binary is noise. A set that reproduces A was really changed by B.

Protocol

1. Compare against a persisted baseline, never against a number from memory. The site config's baseline (and baseline_private) names a complete run. Every new run is reported against it by default. Set a new baseline only when the owner blesses a run.
2. Keep the manifests unchanged between the two runs you compare. A change to a manifest (new sets, a new reference, a different input) is committed on its own, and the baseline is re-run with it.
3. Pool fixes. Run the full battery once on a branch that carries several fixes, not once per fix. For one fix, run the sets it touches with --only, plus the smoke tier.
4. Timing. --gpulock, or another process seen on the GPU, marks the run "timing NOT a reference". Only a run with the machine to itself can be used to judge speed.
5. Resolution on the XDS arms. Nothing forces XDS's range on the processing any more: rugnux keeps its own cut, and the reference-range table re-bins the same merge over XDS's range. The own numbers answer "what does a user get"; the reference-range numbers answer "is our merge better than XDS's over the same range". Where rugnux's cut is coarser than XDS's, the second question is only answered over the shells rugnux reached, and the missing shells are coverage.

The XDS reference d_min

CORRECT.LP gives the range XDS merged. That range is a resolution limit only if someone chose it. With INCLUDE_RESOLUTION_RANGE left at 0.0, or set at the detector edge, it is the detector's reach, not the crystal's. It then gives a meaningless reference, and forcing it on rugnux makes rugnux merge shells of pure noise. On one insulin set, forcing XDS's 1.08 A (where the data end near 1.8 A) turned rugnux's correct I23 into I222. XDS marks each shell's CC1/2 with * when it is significant, and that decides which case applies (inputs.reference_dmin):

• xds_range: the finest shell's CC1/2 is significant, so XDS's limit stands.
• cc_half_0.30: it is not, meaning XDS merged past its own signal. The reference is then the point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in 1/d² between shell centres.

The derived value is the reference rugnux's own cut is scored against. The reference-range table uses XDS's own limit, dmin_xds, because XDS's pooled R_meas, CC1/2 and completeness cover its whole range; the manifest keeps the rule used as dmin_rule.

Adding or changing datasets

• Open arm: put the dataset directory, named by its PDB code, under the open data root, then run battery.py discover. It proposes an input for every directory the manifest does not have (the largest sweep, chosen by frame count), and with --refresh it shows where discovery now disagrees with an unpinned manifest input. Add the row to open.json with the deposited reference (sg, sgno, cell, dmin) and tags. Set "pinned": true when the choice between sweeps was deliberate. Record the source and DOI in docs/EXTERNAL_TEST_DATA.md.
• Inhouse / private: add the row (id, input, tags, and "expect": "no_lattice" for a control), then run battery.py refs --arm inhouse --write to fill in ref from the CORRECT.LP beside the input. Without --write, refs only shows what would change. A reference measured by hand goes in ref_override, which refs leaves alone. A set whose reference is known to be wrong gets "unscored": "<why>": it still runs, but scores unscored instead of pass or fail. A set where two answers are both defensible gets "ref_alternatives": [{"sg": ..., "sgno": ..., "cell": [...], "why": "<reason>"}] - the fields given replace the reference's, the row passes on any of its references, and why is mandatory.
• Renamed data directories: battery.py remap --arm inhouse MAP.json --write, where MAP.json is {"old": "new"} (renames the set and its directory) or {"old id": {"id": "...", "input": "..."}}. Each old id is recorded under aliases in the manifest, so runs made before the rename still compare set by set. Without --write, the command only shows the result and flags inputs that do not exist.

Figures for publication

paper_plots.py RUN [RUN-REDO ...] --out DIR [--time-runs RUN ...] [--uncontended RUN] [--rfree refined|first-cycle] draws three open-arm figures from a run's results.json: R-free of the deposited model after the REFMAC check (--model-check) on the depositor's data against Rugnux's data (by default the refined protocol: each data set at its own resolution, R-free on the shared free set; --rfree first-cycle for the first-cycle numbers at the common resolution), the deposited high-resolution limit against Rugnux's cut, and a histogram of the processing time per dataset with the median marked. Later runs fill in sets an earlier one did not run (a -redo). --time-runs takes the time histogram from other runs - one whose sets had the GPU to themselves - and summary.txt lists the provenance of both. Beside the PDF/PNG figures it writes the values (rfree.dat, dmin.dat, with each set's beamline; time.dat with set, arm, seconds, images and gpu_others), every excluded set with its reason (excluded.txt), the gnuplot script (figures.gp) and summary.txt with the runs' binary and runner provenance, the structure, facility and beamline counts and the timing statistics. Facility and beamline are read from each entry's cached mmCIF (_diffrn_source). Needs gnuplot, pdftoppm and gemmi.

The time is rugnux_wall_s, rugnux's own WALL_TIME over the whole invocation, not the runner's elapsed_s, which also counts the wait for a GPU slot under --gpulock. Every set that ran to a report is counted, whatever its verdict; crashed sets and the no-crystal controls (rugnux exits without a report) are listed in excluded.txt. Two caveats travel with the figure: a time includes reading the images from disk unless they were still in the page cache (see Timing above), and sets that ran with other processes on the GPU (gpu_others > 0) are drawn as the lighter top segment of their bar rather than dropped; --uncontended RUN names a run of the same sets with the GPU to itself and summary.txt then reports their median slowdown against it.