From a7a3c2a00c31cc765f2d778ba97f0ca673b24357 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 09:15:37 +0200 Subject: [PATCH] tools/battery: one canonical runner, derived XDS d_min, clean inhouse layout - Site config: site.example.json is committed; the local site.json (git-ignored) or $JFJOCH_BATTERY_SITE / --site names this machine's paths and the private manifest. Data roots are the arms' own directories, so copying them and editing the site config moves the battery. - Run states: manifest records the runner's pid; a run is complete, running, aborted (Ctrl-C/SIGTERM, or `abort RUN`) or unfinished. compare and report refuse a run that did not finish unless --allow-incomplete, and the report says so. - `list` shows runs and their state; `run` defaults its label to the rugnux version and its baseline to the site's persisted one; a baseline must be complete and of the same privacy as the run; a private report cannot be written into the repo. - XDS reference d_min: when the finest shell of XDS's table has no significant CC1/2, XDS merged past its own signal (no resolution cut, or a cut at the detector edge) and its limit is the detector's reach. The reference is then where XDS's CC1/2 falls through 0.30 (rugnux's own target), and that is also the range forced on rugnux. The manifest keeps XDS's limit (dmin_xds) and the rule (dmin_rule). Forcing the old 1.08 A on the weak insulin set turned rugnux's I23 into I222 (ISa 12.5); forced at the derived 1.81 A it is I23 again (ISa 18.0). Changed in the inhouse arm: insu_I_weak 1.08 -> 1.81, lyso_half_image 0.80 -> 2.60, cytc_eiger 2.04 -> 2.27, lyso_strong 1.18 -> 1.24; every other inhouse limit stands. - inhouse.json follows the directory rename (ids and inputs, data root /home/data/inhouse); `remap` applies such a rename and keeps the old ids as aliases, so runs from before it still compare set by set. - Optional open-arm model check (model_check.py, when present): R-free of the deposited model against each merge, recorded and plotted. - results.json has a fixed, documented row schema (results_schema 1); the HTML report gains inline-SVG plots (verdicts per arm, d_min and R-free ratios per set). - README.md: arms, data layout and provenance, prerequisites, running, run directory, results schema, report, comparing, protocol, adding datasets. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 5 +- tools/battery/README.md | 279 ++++++++++++++++++++++++++++++++ tools/battery/battery.py | 263 +++++++++++++++++++++++++----- tools/battery/inhouse.json | 74 ++++----- tools/battery/inputs.py | 46 +++++- tools/battery/render.py | 84 +++++++++- tools/battery/report.py | 105 ++++++++++-- tools/battery/score.py | 5 +- tools/battery/site.example.json | 12 ++ tools/battery/site.json | 11 -- 10 files changed, 773 insertions(+), 111 deletions(-) create mode 100644 tools/battery/README.md create mode 100644 tools/battery/site.example.json delete mode 100644 tools/battery/site.json diff --git a/.gitignore b/.gitignore index 09815790f..8ecf683a6 100644 --- a/.gitignore +++ b/.gitignore @@ -12,7 +12,7 @@ tmp_venv/ # gen_python_client.sh: the built sdist/wheel. /dist/ -# rugnux_vs_xds.py output dirs (the default, and the ad-hoc ones used when comparing runs) +# output dirs of the retired rugnux_vs_xds.py, still lying around in older working trees rugnux_cmp*/ # rugnux_anomalous.py ANODE work dir, when the config does not point it at the data @@ -42,3 +42,6 @@ docs/review/ # Python bytecode (tools/battery and the root scripts) __pycache__/ + +# tools/battery: the local site config (paths of this machine, private manifest); site.example.json is the template +tools/battery/site.json diff --git a/tools/battery/README.md b/tools/battery/README.md new file mode 100644 index 000000000..18bec5d32 --- /dev/null +++ b/tools/battery/README.md @@ -0,0 +1,279 @@ +# Rugnux validation battery + +`battery.py` is the one way to run the Rugnux validation battery: run `rugnux` over a fixed list of +rotation datasets, score each result against a reference, and write a standard report. Two runs +of the same manifests with the same options give results you can compare set by set. Do not use +ad-hoc runner scripts instead. The old ones (`rugnux_vs_xds.py`, `battery_run.py`, +`battery2_serial.py`, `batstats.py`, ...) are retired because their numbers could not be compared +with each other. + +## Arms + +| arm | datasets | reference | manifest | +|---|---|---|---| +| **open** | public PDB depositions of raw diffraction data, plus a few published small-molecule sets | the deposited space group, cell and resolution | `open.json` (committed) | +| **inhouse** | standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), plus no-crystal controls | XDS, from the `CORRECT.LP` beside each dataset | `inhouse.json` (committed) | +| **private** | user data | XDS, like inhouse | outside the repository; the local site config gives its path | + +Scoring checks four things in order, and the first one that fails decides the verdict: did it +run, is the lattice right, is the symmetry right, is the merge usable (R_meas at most 60%, CC1/2 at +least 0.5). The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio +must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is +scored with `sgequiv`. On the XDS arms only the point group is scored, because XDS never tests a +screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported. +`score.py` has the details. + +The private arm is never mixed with the others. A private run gets its own run directory +(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a +public report or be written into the repository. Its names, paths and values never go into a +commit, an issue or a public report. + +## Getting the data + +Each manifest row gives an `input` path relative to that arm's **data root**. The data root is +set in the site config, and the first directory of each input is the dataset's directory: + +``` +/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5 +/lyso_ref/hewl_35_001_master.h5 +``` + +To move the battery to another machine, copy the two data roots and point the new site config at +the copies. The paths inside the manifests do not change. + +- **open**: every dataset is listed with its source repository and dataset DOI in + [`docs/EXTERNAL_TEST_DATA.md`](../../docs/EXTERNAL_TEST_DATA.md). The sources are IRRMC + (proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what + each archive holds. Please cite those DOIs. At PSI the data root is `/home/data/open`, a + symlink to `/home/data/nonsls/raw`. 51 of the dataset directories in it are themselves symlinks + into `/data/scout_staging2`, so **copy with symlinks followed**: `rsync -aL` or `cp -rL`. + A plain `rsync -a` copies dangling links. The open arm is about 1.7 TB. +- **inhouse**: these are our own measurements and cannot be downloaded publicly. Ask the + maintainers for a copy. Each dataset directory holds the images and the XDS output + (`CORRECT.LP`, `XDS_ASCII.HKL`, ...) that the reference comes from. At PSI the data root is + `/home/data/inhouse` (about 130 GB). **Leave out `.master_backups_kek`** when copying it + (`rsync -a --exclude .master_backups_kek`): it holds backups of private masters, not battery data. +- **private**: its data root and manifest are named only in the local site config. + +## Prerequisites + +- Python 3.9 or newer, with `gemmi` (scoring). `h5py` is needed only by `discover`. +- A `rugnux` binary. Build it with the flags CI uses (`-march=x86-64-v3`, see `CLAUDE.md`), + because CPU-bound results and timings differ between builds. The runner records the build + flags from the `CMakeCache.txt` above the binary, or from the tree given with `--build-dir`. +- XDS is not needed. The XDS references are already in the manifests (`refs` re-reads them from + `CORRECT.LP`). + +## Site config + +Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an +optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it +in this order: + +1. `--site PATH` +2. `$JFJOCH_BATTERY_SITE` +3. `tools/battery/site.json` (git-ignored) + +`site.example.json` is the committed template. Copy it and fill in your paths. The local file is +kept out of git because it names the private manifest. At PSI the config is +`/data/battery/site.json`: link it (`ln -s /data/battery/site.json tools/battery/site.json`) or +export `JFJOCH_BATTERY_SITE=/data/battery/site.json`. + +```json +{ + "runs_root": "/data/battery/runs", + "lock_file": "/data/battery/full_run.lock", + "gpulock": "/data/tmp/gpulock", + "baseline": "/data/battery/runs/", + "baseline_private": null, + "arms": { + "open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/open"}, + "inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/home/data/inhouse"}, + "private": {"manifest": "/path/to/private.json", "reference": "xds", + "data_root": "/path/to/private/data", "private": true} + } +} +``` + +A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that +`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of +the shared GPU slots. Set it to `null` if you have no such wrapper. + +## Running + +``` +B=tools/battery/battery.py +R=build/rugnux/rugnux + +$B run --rugnux $R # full battery: open + inhouse +$B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases +$B run --rugnux $R --only 5reo,lyso_ref # named sets +$B run --rugnux $R --arm private # the private arm, on its own +$B run --rugnux $R --arm inhouse --unforced # XDS arms without XDS's resolution forced +``` + +Useful options: + +- `--label` names the run. By default the label is the rugnux version. +- `--threads N` passes `-N` to rugnux. +- `--gpulock` queues every rugnux behind the site's GPU wrapper. +- `--extra ""` adds rugnux arguments to every set. +- `--baseline RUN|none` sets the run the report is compared against. +- `--model-check` / `--no-model-check` turns the open-arm R-free check on or off (see below). + +A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the +manifests (`"tiers": {"smoke": ""}`). + +**Full runs.** A run without `--only` or `--tier` takes the site lock (`flock`, non-blocking), so +two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and +everything else on the machine competes with it. + +### The model check (open arm) + +If `tools/battery/model_check.py` is present, each open-arm merge (`p.mtz`) is also scored +against the deposited model: `model_check.check(mtz, pdb_id, workdir)` returns R-free and R-work +of the deposited model against our data, plus the deposited R-free. The runner records these as +`rfree`, `rwork`, `rfree_deposited`, `rfree_ratio`, `model_status` and `model_reason`, and the +report plots their ratio. The check is on by default when the module exists. Without the module +the fields stay null. The PDB entry is the set id up to the first `_`, so `6h2p_native` is checked +as `6h2p`. Small-molecule sets are skipped. + +`model_sweep.py` is a different tool: it runs `rugnux --model` over the open arm (same inputs, same +site config) and compares the R-free that rugnux itself reports with the published value. It needs +network access to the RCSB and runs separately from `battery.py`. + +## A run directory + +`/__