leonarski_fandClaude Opus 5
bb0648ae87
The battery is bound by reading images from disk, so instead of one battery run per setting each set now runs in all its variants in a row while its images are still in the page cache: open arm bare, then model (rugnux --model <deposited coordinates>) XDS arms bare, then xds (XDS's resolution range forced, -A where XDS was anomalous) bare (plain rugnux <input>) runs first on every arm, so its time always carries the set's disk read whichever variants are selected; each row records first_read and the report's timing table says which variant's times are cold. --variants runs a subset; --unforced is gone (--variants bare). Rows, work dirs (work/<arm>/<set>/<variant>/), compare and the report are per variant; compare pairs (arm, set, variant) on the variants both runs have on an arm, and a schema-1 run is read as one variant. The model variant takes the coordinates and published R-free from model_check's RCSB cache (site key pdb_cache) and records rugnux's R-free/R-work and the ratio; the REFMAC check is now opt-in (--model-check) and its keys moved to refmac_*. results_schema 2. model_sweep.py is retired: the model variant replaces it (its --spot/--scaling-low-resolution 50 were rugnux's defaults, so the command is the same). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
126 lines
7.6 KiB
Markdown
126 lines
7.6 KiB
Markdown
# Tests
|
|
|
|
The unit and integration tests are written with [Catch2](https://github.com/catchorg/Catch2) and
|
|
collected into a single binary, `tests/jfjoch_test`. Build and run it with:
|
|
|
|
```
|
|
make -j$(nproc) jfjoch_test
|
|
cd tests
|
|
./jfjoch_test # everything
|
|
./jfjoch_test "<test name>" # one test case
|
|
./jfjoch_test "[tag]" # by tag
|
|
```
|
|
|
|
There are also benchmark and hardware routines, each printing its own usage:
|
|
|
|
* `jfjoch_hdf5_test` to measure HDF5 dataset writing speed (single threaded). It doubles as the
|
|
generator of the HDF5 files used by the external-software tests below.
|
|
* `jfjoch_lite_perf_test` to measure the CPU/GPU ("lite") analysis path - indexing, integration and
|
|
optional file writing.
|
|
* `jfjoch_fpga_test` to test quality/performance of FPGA card(s) and software routines. With `-H` it
|
|
runs the high-level-synthesis C model on the CPU, so no FPGA device is needed.
|
|
|
|
Out-of-space handling is covered separately by `jfjoch_hdf5_enospc_test`, run under the `enospc_shim`
|
|
`LD_PRELOAD` module that makes writes fail with `ENOSPC`.
|
|
|
|
In addition, tests are executed to verify that datasets written by Jungfraujoch are readable by
|
|
other MX software (see [Integration with MX data processing software](SOFTWARE_INTEGRATION.md)) -
|
|
XDS through the Jungfraujoch, Durin and Neggia plugins, and DIALS `xia2.ssx` - for each of the
|
|
NXmx layouts. Input files for these programs are placed in the `tests/xds`, `tests/xds_durin`,
|
|
`tests/xds_neggia` and `tests/crystfel` folders. See `.gitea/workflows/build_and_test.yml` for the
|
|
exact commands; the CrystFEL fixtures are run by hand rather than in the pipeline.
|
|
|
|
## Judging a change to the analysis itself
|
|
|
|
The harnesses below run `rugnux` over stored datasets and score the result. None is part of CI -
|
|
run them when a change plausibly moves merged results, not as a reflex. The public datasets the
|
|
pipeline is exercised on, and the DOI to cite for each, are listed in
|
|
[External test data](EXTERNAL_TEST_DATA.md).
|
|
|
|
* `tools/battery/battery.py` - the rotation battery, the one canonical way to judge a rotation
|
|
change: public PDB depositions scored against the deposition, in-house standard crystals and
|
|
no-crystal controls scored against XDS, and a local-only private arm. Its manifests, the run
|
|
and compare protocol and the report are described in `tools/battery/README.md`. Each set runs
|
|
in several variants back to back: plain, with the deposited model (`--model`, R-free against the
|
|
published one) on the open arm, and with XDS's settings on the XDS arms.
|
|
* `rugnux_stills_ab.py` - the stills analogue of the battery: scores a change on a serial
|
|
dataset by what it does to the merge. Takes its dataset list from outside the repository.
|
|
* `rugnux_anomalous.py` - the anomalous-peak-height arbiter, below.
|
|
|
|
### The anomalous-peak-height arbiter
|
|
|
|
A change that touches **partiality** - a mosaicity estimator, a rocking-curve model, a background
|
|
change, anything that alters how partial reflections are weighted - cannot be judged by the
|
|
statistics we normally reach for:
|
|
|
|
| statistic | why it fails for this class of change |
|
|
|---|---|
|
|
| ISa, R_meas, error-model `b` | one measurement, not three; dominated by the low-resolution shells; not invariant to the uniform intensity rescale a partiality change produces |
|
|
| last-shell R_meas | moves with its denominator, i.e. the wrong way by construction |
|
|
| `rugnux --model` R-free | tracks its own zero-information floor, which moves ~22x more than R-free itself over the same sweep |
|
|
| per-shell agreement with `XDS_ASCII.HKL` | XDS never divides by partiality, so "divide less" moves us toward it mechanically; measured to put the optimum ~1.4x too low |
|
|
|
|
**Anomalous difference density at known scatterer sites** has none of these problems. It is read in
|
|
units of the map's own sigma, so a uniform intensity rescale cancels exactly, and it is referenced
|
|
to the structure rather than to another program's partiality model.
|
|
|
|
`rugnux_anomalous.py` measures it: `shelxc` + `anode -a` (CCP4) on each arm's merged reflections,
|
|
against a model that is placed **once** and then held fixed. It reports, per dataset, the mean site
|
|
height and the off-site noise floor, and, between arms, the **paired per-site** change.
|
|
|
|
```
|
|
# compare two arms (each a directory of <id>/<id>.hkl + .mtz)
|
|
./rugnux_anomalous.py --config <table>.json base=<dir-A> test=<dir-B>
|
|
|
|
# a parameter scan: numeric labels turn the arms into a curve with a per-dataset optimum
|
|
./rugnux_anomalous.py --config <table>.json \
|
|
0.85='<scan>/{name}/s0p85.hkl' 1.00='<scan>/{name}/s1.hkl' 1.20='<scan>/{name}/s1p2.hkl'
|
|
```
|
|
|
|
An arm is a Rugnux output directory or a path template containing `{name}`. `--place` does the
|
|
one-off model placement, `--write-config-template` prints the config skeleton, and ANODE results
|
|
are cached under the config's `workdir` (a full 9-dataset x 11-arm scan takes under a minute).
|
|
|
|
**The gate.** A dataset counts only if its **reference arm** shows top peak > 1.5x the highest
|
|
off-site peak **and** at least 3 sites over 5 sigma. A dataset that fails is reported as
|
|
`EXCLUDED`, never as a zero - the difference between two noise measurements is not a measurement.
|
|
|
|
**Standing dataset set** (2026-08): 8 datasets from 7 crystals - 114 site-measurements across the
|
|
datasets, 108 distinct sulfur sites - all judged on native sulfur signal.
|
|
|
|
| crystals | space group | photon energy | sites each |
|
|
|---|---|---|---|
|
|
| 2 | P4<sub>1</sub>2<sub>1</sub>2 | 12.4, 16.0 keV | 18 |
|
|
| 2 (lysozyme) | P4<sub>3</sub>2<sub>1</sub>2 | 13.0, 5.0 keV | 27 |
|
|
| 3 (4 datasets - one crystal contributes two energies) | cubic, I-centred | 13.0, 6.0, 5.0, 5.0 keV | 6 |
|
|
|
|
Report `n` as crystals, not datasets: two energies of one crystal are not two independent votes,
|
|
and the tool prints both counts for that reason.
|
|
|
|
**Traps this tool exists to encapsulate.** Every one of them has already cost a working day:
|
|
|
|
1. **The phasing space group comes from the config, never from the merged file.** I23 and
|
|
I2<sub>1</sub>3 have identical systematic absences (I-centring already forces the screw
|
|
condition), so no data can separate them, and phaser's automatic space-group test only tries
|
|
the *enantiomorph* - which for I23 is itself. Phasing an I-centred cubic case in the I23 that
|
|
both Rugnux and XDS report gives TFZ 7-11 where the other member gives 30-50, and drops the mean
|
|
site height by a factor 3-10 - enough to make four good datasets look signal-free. Thirteen
|
|
classes of chiral space group are indistinguishable this way; `--place` tries every member of
|
|
the class and reports each one's LLG/TFZ.
|
|
2. **Place the model once, from a reference arm, and reuse it unchanged.** Re-phasing per arm lets
|
|
the model move and contaminates the comparison. Refining the placed model against the dataset's
|
|
own amplitudes is allowed (it lifts the peaks another 4-10%) as long as the *same* refined model
|
|
is then used for every arm.
|
|
3. **The gate and the measurement must use the same model.** Gating on one model and scoring the
|
|
curve with another silently changes which datasets are in the set.
|
|
4. **The off-site floor skips special positions.** A peak on the cell origin is a ripple of the
|
|
calculated phases, not a sample of the background; leaving it in inflates the floor by several
|
|
sigma and can turn a passing dataset into a failing one. Such peaks are reported in their own
|
|
`spec` column rather than dropped silently.
|
|
|
|
**Reading the result.** Judge the paired per-site change, with its standard error, pooled over
|
|
*crystals*. A per-dataset optimum whose arm does not beat the reference on the paired test is
|
|
flagged `not significant vs ref` and must not be quoted as a preference; so must one sitting on the
|
|
edge of the scanned grid (`grid edge`) - extend the grid instead.
|
|
|