# Rugnux validation battery `battery.py` is the one way to run the Rugnux validation battery: run `rugnux` over a fixed list of rotation datasets, score each result against a reference, and write a standard report. Two runs of the same manifests with the same options give results you can compare set by set. Do not use ad-hoc runner scripts instead. The old ones (`rugnux_vs_xds.py`, `battery_run.py`, `battery2_serial.py`, `batstats.py`, ...) are retired because their numbers could not be compared with each other. `test_score.py` covers the scoring rules that decide a verdict (stdlib `unittest`, no data and no run needed): `cd tools/battery && python3 test_score.py`. ## Arms | arm | datasets | reference | manifest | |---|---|---|---| | **open** | public PDB depositions of raw diffraction data, plus a few published small-molecule sets | the deposited space group, cell and resolution | `open.json` (committed) | | **inhouse** | standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), small-molecule standards (aspirin, citric acid, HEPES, YAG, L-cystine), plus no-crystal controls | XDS, from the `CORRECT.LP` beside each dataset; for the small molecules the space group is the literature's (`ref_override`), as XDS reports only Sohncke groups; L-cystine, which XDS does not index, has only the literature reference | `inhouse.json` (committed) | | **private** | user data | XDS, like inhouse | outside the repository; the local site config gives its path | Scoring checks these things in order, and the first one that fails decides the verdict: did it run, is the lattice right, is the symmetry right, and on the XDS arms, is the merge as good as XDS's. The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is scored with `sgequiv`. On the XDS arms only the point group is scored, because XDS never tests a screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported. `score.py` has the details. **A question the data could not answer is `unscored`, not a failure.** Two of these. The hand of an enantiomorphic pair (P4_1 vs P4_3) is not in the intensities at all, and `sgequiv` already counts it as a match. The other is a **screw axis on a row the sweep never recorded** - the axial row lies in the spindle's blind cone, or outside the resolution range, so nothing was measured that could confirm or refuse the screw. rugnux then offers the whole set in `SPACE_GROUP_ALTERNATIVES`, names the open axis in `SPACE_GROUP_SCREW_UNDETERMINED`, and writes the member claiming no screw, because a reflection file must carry one group. Such a set scores `unscored` / `screw_undetermined` rather than `sym_screw`, on three conditions, all necessary: rugnux named the axis, the reference group is among the ones it offered, and the only axis the two groups differ on is one it named. A screw called wrongly where the row WAS measured meets none of them and stays a `sym_screw` failure - this rule excuses an unanswerable question, never a wrong answer. **A row may accept more than one reference.** Where the deposition and our reduction disagree, the disagreement is real, and no test available to us settles it - a knife-edge - the row lists the other acceptable answer in `ref_alternatives` and passes if our answer matches any of its references. This is not a statement that our answer is right; it says both descriptions are defensible and that picking either one is acceptable. `ref` keeps the deposition verbatim, each alternative must carry a `why` a reader can check (an alternative without one is a schema error, not a silent pass), and the report counts these rows on their own - summary column `of which alt`, its own bar-chart segment, and an **Accepted alternatives** section printing each reason - so they are never folded into the ordinary passes. Prefer this to `unscored`, which drops the row from scoring altogether and loses a test that still works, and to `ref_override`, which asserts that the reference is wrong. **Data quality is not scored.** A verdict is about the lattice and the symmetry only. Resolution, CC1/2, R_meas (overall and lowest shell), completeness, multiplicity and ISa are reported beside XDS's or the deposition's values, in the per-set tables and plots, for a human to judge. On the XDS arms `cc_half_noise_ratio` = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) over XDS's own range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as XDS's would be with half of its observations. **R_free does not compare between runs; `rmodel_shell_scaled` does.** rugnux scales the model to the data with an overall factor and a symmetry-constrained anisotropic B, which can only bend one way with resolution - deliberately, so that the maps of a batch stay comparable. Whatever the amplitudes' own radial profile does that this shape cannot follow is reported as R. So two runs whose merged amplitudes have different radial profiles have R_free values that differ for a reason that has nothing to do with the data fitting the model: measured on one such pair, R_free moved a median +0.042 over eleven sets where `phenix.model_vs_data`, which scales per shell, moved +0.002. `rmodel_shell_scaled` is the same R with one free scale per resolution shell, over every reflection rather than the free 5% (nothing is refined here, so the free set buys no cross-validation and only costs precision): on that same pair its median was +0.001, and it still flagged the one set that was a real regression. `radial_misfit` is the size of that per-shell rescale, the RMS of its logarithm; when it moves, R_free beside it cannot be read. All three are reported for a human, never scored. **Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's `CORRECT.LP` (`lowres_r_meas_ref`, stored in the manifest as `r_meas_low`), with the shells' high-resolution limits. The reference-range table uses XDS's shell limits, so its lowest shell and XDS's cover the same reflections; `lowres_r_meas_ratio` is the one over the other, plotted per set. ## One command per arm Every set runs **once**, with one command per arm: | arm (reference) | command | |---|---| | open (deposition) | `rugnux -o p --no-export-unmerged --model `; sets without a deposited model (small molecules, unpublished sets, or coordinates that cannot be downloaded) run the same command without `--model` | | inhouse, private (XDS) | `rugnux -o p --no-export-unmerged [-A] --report-resolution , ` | Neither `--model` nor `--report-resolution` changes the processing: the merged reflections, the resolution cut and every decision are the same with and without them (verified bit for bit). So the one run is both what a user gets and, through the extra report sections, the comparison with the reference: - **Own result** (every arm): rugnux's own cut and statistics, scored against the reference and plotted as d_min(rugnux) / d_min(reference). This is what a user gets. - **`--model`** (open arm): the deposited coordinates (`.cif.gz`) and the published R-free come from the RCSB and are cached in the site's `pdb_cache` (default `/data/battery/pdb_cache`), so the first run of a new entry needs network access. The entry is the set id up to the first `_`, so `6h2p_native` uses `6h2p`. The row records MODEL_FIT, R_FREE, R_WORK and CC_MODEL as **trend fields**: rugnux's R-free is a rigid-body placement scored on rugnux's own free set, not a refinement against the depositor's, so the ratio to the published R-free is not a like-for-like score; watch how it moves between runs. The comparable number is the REFMAC check (below). A model that fits may put its enantiomorph on the space-group label; the scorer reads the data's own determination instead (`SOHNCKE_SPACE_GROUP` where `SPACE_GROUP_ENANTIOMORPH` is `ASSUMED_FROM_MODEL`) and keeps the label in `sg_label`. - **`--report-resolution`** (XDS arms): a second statistics table from the same merge, binned over XDS's own range (`dmin_xds` and XDS's d_max as given, 999 included, so it covers exactly what XDS's CORRECT.LP totals cover), and `-A` where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the manifest), so the two tables count the same way. Its REFRES_* numbers are shown beside XDS's CORRECT.LP totals: completeness, multiplicity, R_meas, the lowest shell's R_meas, CC1/2 and ISa (I/sigma is shown; the manifest holds no XDS value for it). Only CC1/2 is scored (above). Where rugnux's own cut is coarser than the reference (`REFRES_SHELLS_PAST_LIMIT` > 0), the shells past it are not merged, so the completeness is read as **coverage** of the reference range, not as a quality regression, and the other numbers are over the shells rugnux reached. **Timing.** Each set runs once, so every time includes reading the images from disk (unless they were still in the page cache from before the run). The private arm is never mixed with the others. A private run gets its own run directory (suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a public report or be written into the repository. Its names, paths and values never go into a commit, an issue or a public report. ## Getting the data Each manifest row gives an `input` path relative to that arm's **data root**. The data root is set in the site config, and the first directory of each input is the dataset's directory: ``` /11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5 /lyso_ref/hewl_35_001_master.h5 ``` To move the battery to another machine, copy the two data roots and point the new site config at the copies. The paths inside the manifests do not change. - **open**: every dataset is listed with its source repository and dataset DOI in [`docs/EXTERNAL_TEST_DATA.md`](../../docs/EXTERNAL_TEST_DATA.md). The sources are IRRMC (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what each archive holds. Please cite those DOIs. At PSI the data root is `/home/data/open`, a symlink to `/home/data/nonsls/raw`. 42 of the dataset directories in it are themselves symlinks into `/data/scout_staging2`, so **copy with symlinks followed**: `rsync -aL` or `cp -rL`. A plain `rsync -a` copies dangling links. The open arm is about 1.7 TB. - **inhouse**: these are our own measurements and cannot be downloaded publicly. Ask the maintainers for a copy. Each dataset directory holds the images and the XDS output (`CORRECT.LP`, `XDS_ASCII.HKL`, ...) that the reference comes from. At PSI the data root is `/home/data/inhouse` (about 130 GB). **Leave out `.master_backups_kek`** when copying it (`rsync -a --exclude .master_backups_kek`): it holds backups of private masters, not battery data. - **private**: its data root and manifest are named only in the local site config. ## Prerequisites - Python 3.9 or newer, with `gemmi` (scoring). `h5py` is needed only by `discover`. - A `rugnux` binary. Build it with the flags CI uses (`-march=x86-64-v3`, see `CLAUDE.md`), because CPU-bound results and timings differ between builds. The runner records the build flags from the `CMakeCache.txt` above the binary, or from the tree given with `--build-dir`. - XDS is not needed. The XDS references are already in the manifests (`refs` re-reads them from `CORRECT.LP`). ## Site config Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it in this order: 1. `--site PATH` 2. `$JFJOCH_BATTERY_SITE` 3. `tools/battery/site.json` (git-ignored) `site.example.json` is the committed template. Copy it and fill in your paths. The local file is kept out of git because it names the private manifest. At PSI the config is `/data/battery/site.json`: link it (`ln -s /data/battery/site.json tools/battery/site.json`) or export `JFJOCH_BATTERY_SITE=/data/battery/site.json`. ```json { "runs_root": "/data/battery/runs", "lock_file": "/data/battery/full_run.lock", "gpulock": "/data/tmp/gpulock", "baseline": "/data/battery/runs/", "baseline_private": null, "pdb_cache": "/data/battery/pdb_cache", "arms": { "open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/open"}, "inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/home/data/inhouse"}, "private": {"manifest": "/path/to/private.json", "reference": "xds", "data_root": "/path/to/private/data", "private": true} } } ``` A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that `rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where the deposited models and their metadata are cached for `--model` and the REFMAC check. ## Running ``` B=tools/battery/battery.py R=build/rugnux/rugnux $B run --rugnux $R # full battery: open + inhouse $B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases $B run --rugnux $R --only 5reo,lyso_ref # named sets $B run --rugnux $R --arm private # the private arm, on its own ``` Useful options: - `--label` names the run. By default the label is the rugnux version. - `--threads N` passes `-N` to rugnux. - `--gpulock` queues every rugnux behind the site's GPU wrapper. - `--extra ""` adds rugnux arguments to every set. - `--baseline RUN|none` sets the run the report is compared against. - `--model-check` adds the REFMAC check to the open arm's rows (off by default, see below). A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the manifests (`"tiers": {"smoke": ""}`). **Full runs.** A run without `--only` or `--tier` takes the site lock (`flock`, non-blocking), so two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and everything else on the machine competes with it. ### The REFMAC model check (open arm, opt-in) rugnux's own `--model` R-free is a trend number. `--model-check` adds an independent one, comparable with the deposition, that needs CCP4: `tools/battery/model_check.py` scores each open-arm merge (`p.mtz`) against the deposited model, unmodified (ligands, waters and hydrogens kept; only atoms of unknown element, UNX, are left out because REFMAC stops on them): REFMAC's rigid-body mode, R-factors taken before any shift, the data reindexed into the model's setting and reduced to the deposited space group. The depositor's own structure factors go through the same setting choice (they can sit in the other branch of a merohedral ambiguity) and are scored by the same protocol. All REFMAC numbers use the same high-resolution limit, `d_min_used` = max(our d_min, the deposited d_min), i.e. the lower of the two resolutions, so neither data set is scored past what the other reaches. The four R-free fields, all of the same deposited model: | field | data | free set | reads as | |---|---|---|---| | `refmac_rfree` | ours | ours | close to a work R: most of our free reflections were work reflections to the depositor's refinement | | `refmac_rfree_depflags` | ours | the depositor's | **the fair number for our data** | | `refmac_rfree_depdata` | the depositor's | the depositor's | the baseline: same model, same protocol, the data the model was refined against | | `refmac_rfree_ratio` | | | `refmac_rfree_depflags / refmac_rfree_depdata`; below 1 = our data fit the model better; the report plots it | `refmac_rwork` is R on our own work reflections. The depositor-data numbers carry a home advantage (the model was refined against those data), and `refmac_rfree_depdata` is not the published R-free either: different program (REFMAC vs e.g. PHENIX, whose resolution-binned scaling absorbs a non-Wilson falloff that REFMAC's overall scale cannot) and no refinement. The depositor's amplitudes are `F_meas`, else the mean of F(+)/F(-); where only intensities were deposited they are converted by French-Wilson (ctruncate), as our own amplitudes are. An entry that declares twinning (`_pdbx_reflns_twin`) is scored with REFMAC's twin refinement for both data sets, since its published R-free is a twinned one; the untwinned numbers are kept in `refmac_untwinned`. The ratio is null when the entry deposited no structure factors with a free set (`refmac_reason` says why). The runner records `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_twin`, `refmac_untwinned`, `refmac_status` and `refmac_reason`. Small-molecule sets are skipped. `model_check.py --workdir DIR` runs it on one merge and prints every field (setting, reindexing operators of both data sets, what the depositor's amplitudes were made from). **Refined, shared free set (second protocol).** The first-cycle numbers above score the model as deposited at the common resolution limit. The second protocol refines the same deposited model against each data set at that data set's OWN resolution and compares R-free on a free set both share: - same starting model, same REFMAC protocol for both data sets: restrained refinement, 10 cycles, automatic weight, isotropic B for every atom (a deposited ANISOU is often TLS-derived, and refining it atom by atom diverged), twin refinement where the entry declares twinning; a ligand whose code the monomer library describes with other atoms is renamed to a code the library lacks, so REFMAC makes its restraints from the model's own coordinates (`ligands_renamed` in the check's output); - each data set over its whole resolution range; its work set is every reflection it has except the shared free set; - shared free set = the depositor's free reflections present in BOTH data sets (after the change of basis), which takes the lower of the two limits in every direction, anisotropic cuts and rejected reflections included; depositor free reflections outside it are removed from both; - R-free recomputed from each output MTZ over exactly the shared set (F against FC_ALL_LS, i.e. with REFMAC's own scale and bulk solvent, which reproduces REFMAC's R-free over its free set); for a twinned refinement the output MTZ holds no twinned Fc, so REFMAC's own R-free is used - its free set IS the shared set. | field | meaning | |---|---| | `refmac_refined_rfree_shared` | our data, refined, R-free on the shared set - **the refined number for our data** | | `refmac_refined_rfree_shared_depdata` | the depositor's data, same protocol, same shared set | | `refmac_refined_ratio` | the ratio of the two; below 1 = our data give the better refined model | | `refmac_refined_rwork`, `refmac_refined_rwork_depdata` | REFMAC's R-work after refinement, each data set (their own work sets and resolutions: not comparable with each other) | | `refmac_refined_rfree_refmac`, `refmac_refined_rfree_refmac_depdata` | REFMAC's own R-free, for reference | | `refmac_shared_free_n` | size of the shared free set | | `refmac_refined_reason` | why the refined protocol gave nothing, when it did not (the first-cycle numbers stand) | The refined numbers need the depositor's structure factors with a free set, like the ratio; they take minutes to an hour per set. `model_check.py --no-refine` skips them. ### Rechecking an existing run `battery.py recheck RUN [--only a,b] [--jobs N]` reruns the model check and the depositor-data comparison on every open-arm row of a finished run, from the run's own `p.mtz` (CPU only; rugnux is not run again), so numbers made by an older `model_check.py` / `depdata_check.py` can be brought up to date without reprocessing. It updates the `refmac_*` and `dep_*` fields of `results.json` in place and nothing else (verdicts stay as scored). The previous files stay beside the new ones: `results.json.pre-recheck-`, `report.{md,html}.pre-recheck-` and each set's `model_check.pre-recheck-/`. The report is re-rendered, and `manifest.json` gains an entry in its `rechecks` list - `date`, `end`, `runner_git`, `runner_dirty` (the tools version that made the new numbers), `sets`, `fields`, `previous_results` and `command` - so which version of the checks produced which figures stays auditable. The run directory is made read-only again. ### The depositor-data comparison (open arm, always on) `tools/battery/depdata_check.py` compares our merge with the depositor's own data without CCP4: per resolution shell, the rank correlation of our IMEAN with |Fc|^2 minus that of the deposited intensities (or amplitudes squared), on the reflections both carry, with |Fc| from the deposited model as it is (gemmi, no bulk solvent, no refinement), the data carried into the model's setting as the REFMAC check does it. The model was refined against the depositor's data, so the comparison has a home advantage for the deposition; around -0.01 is typical, and a set far below that merged worse than the deposition at those resolutions. It runs on every open-arm set with a model (seconds to a minute, gemmi only; the `-sf.cif` is cached in `pdb_cache` beside the coordinates), and `report` computes it for an older run that lacks it. It is reported, never scored. Row keys: `dep_cc_delta_all` (all common reflections at d < 4 A), `dep_cc_delta_outer` (mean of the two outermost of eight equal-count shells), `dep_beyond_cc` and `dep_beyond_d` (the correlation of our reflections with |Fc|^2 in the outermost of up to three shells past the deposited data's limit, and that shell's limit: the model never saw these, so clearly above zero is signal), plus `dep_kind` (`I` or `F`), `dep_d_min` (the deposited data's limit), `dep_n_common`, `dep_status` and `dep_reason`. `model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The open arm's command does the same inside the battery. ## A run directory `/__