Jungfraujoch/docs/TESTS.md

Tests

The unit and integration tests are written with Catch2 and collected into a single binary, tests/jfjoch_test. Build and run it with:

make -j$(nproc) jfjoch_test
cd tests
./jfjoch_test                 # everything
./jfjoch_test "<test name>"   # one test case
./jfjoch_test "[tag]"         # by tag

There are also benchmark and hardware routines, each printing its own usage:

• jfjoch_hdf5_test to measure HDF5 dataset writing speed (single threaded). It doubles as the generator of the HDF5 files used by the external-software tests below.
• jfjoch_lite_perf_test to measure the CPU/GPU ("lite") analysis path - indexing, integration and optional file writing.
• jfjoch_fpga_test to test quality/performance of FPGA card(s) and software routines. With -H it runs the high-level-synthesis C model on the CPU, so no FPGA device is needed.

Out-of-space handling is covered separately by jfjoch_hdf5_enospc_test, run under the enospc_shim LD_PRELOAD module that makes writes fail with ENOSPC.

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable by other MX software (see Integration with MX data processing software) - XDS through the Jungfraujoch, Durin and Neggia plugins, and DIALS xia2.ssx - for each of the NXmx layouts. Input files for these programs are placed in the tests/xds, tests/xds_durin, tests/xds_neggia and tests/crystfel folders. See .gitea/workflows/build_and_test.yml for the exact commands; the CrystFEL fixtures are run by hand rather than in the pipeline.

Judging a change to the analysis itself

Two harnesses in the repository root run rugnux over a directory of stored datasets and score the result. Neither is part of CI - run them when a change plausibly moves merged results, not as a reflex. Both take their dataset list from outside the repository, because dataset and sample identities are not committed. The public datasets the pipeline is exercised on, and the DOI to cite for each, are listed in External test data.

• rugnux_vs_xds.py - the rotation battery. Runs rugnux de novo over every crystal under a data root and tabulates reflections, observations, space group, R_meas, CC1/2, ISa and wall-clock time against the XDS CORRECT.LP beside each dataset.
• rugnux_anomalous.py - the anomalous-peak-height arbiter, below.

The anomalous-peak-height arbiter

A change that touches partiality - a mosaicity estimator, a rocking-curve model, a background change, anything that alters how partial reflections are weighted - cannot be judged by the statistics we normally reach for:

statistic	why it fails for this class of change
ISa, R_meas, error-model b	one measurement, not three; dominated by the low-resolution shells; not invariant to the uniform intensity rescale a partiality change produces
last-shell R_meas	moves with its denominator, i.e. the wrong way by construction
rugnux --model R-free	tracks its own zero-information floor, which moves ~22x more than R-free itself over the same sweep
per-shell agreement with XDS_ASCII.HKL	XDS never divides by partiality, so "divide less" moves us toward it mechanically; measured to put the optimum ~1.4x too low

Anomalous difference density at known scatterer sites has none of these problems. It is read in units of the map's own sigma, so a uniform intensity rescale cancels exactly, and it is referenced to the structure rather than to another program's partiality model.

rugnux_anomalous.py measures it: shelxc + anode -a (CCP4) on each arm's merged reflections, against a model that is placed once and then held fixed. It reports, per dataset, the mean site height and the off-site noise floor, and, between arms, the paired per-site change.

# compare two arms (each a directory of <id>/<id>.hkl + .mtz)
./rugnux_anomalous.py --config <table>.json  base=<dir-A>  test=<dir-B>

# a parameter scan: numeric labels turn the arms into a curve with a per-dataset optimum
./rugnux_anomalous.py --config <table>.json \
    0.85='<scan>/{name}/s0p85.hkl' 1.00='<scan>/{name}/s1.hkl' 1.20='<scan>/{name}/s1p2.hkl'

An arm is a rugnux output directory or a path template containing {name}. --place does the one-off model placement, --write-config-template prints the config skeleton, and ANODE results are cached under the config's workdir (a full 9-dataset x 11-arm scan takes under a minute).

The gate. A dataset counts only if its reference arm shows top peak > 1.5x the highest off-site peak and at least 3 sites over 5 sigma. A dataset that fails is reported as EXCLUDED, never as a zero - the difference between two noise measurements is not a measurement.

Standing dataset set (2026-08): 8 datasets from 7 crystals, 114 sulfur sites, all judged on native sulfur signal.

crystals	space group	photon energy	sites each
2	P41212	12.4, 16.0 keV	18
2 (lysozyme)	P43212	13.0, 5.0 keV	27
3 (4 datasets - one crystal contributes two energies)	cubic, I-centred	13.0, 6.0, 5.0, 5.0 keV	6

Report n as crystals, not datasets: two energies of one crystal are not two independent votes, and the tool prints both counts for that reason.

Traps this tool exists to encapsulate. Every one of them has already cost a working day:

1. The phasing space group comes from the config, never from the merged file. I23 and I2₁3 have identical systematic absences (I-centring already forces the screw condition), so no data can separate them, and phaser's automatic space-group test only tries the enantiomorph - which for I23 is itself. Phasing an I-centred cubic case in the I23 that both rugnux and XDS report gives TFZ 7-11 where the other member gives 30-50, and drops the mean site height by a factor 3-10 - enough to make four good datasets look signal-free. Thirteen classes of chiral space group are indistinguishable this way; --place tries every member of the class and reports each one's LLG/TFZ.
2. Place the model once, from a reference arm, and reuse it unchanged. Re-phasing per arm lets the model move and contaminates the comparison. Refining the placed model against the dataset's own amplitudes is allowed (it lifts the peaks another 4-10%) as long as the same refined model is then used for every arm.
3. The gate and the measurement must use the same model. Gating on one model and scoring the curve with another silently changes which datasets are in the set.
4. The off-site floor skips special positions. A peak on the cell origin is a ripple of the calculated phases, not a sample of the background; leaving it in inflates the floor by several sigma and can turn a passing dataset into a failing one. Such peaks are reported in their own spec column rather than dropped silently.

Reading the result. Judge the paired per-site change, with its standard error, pooled over crystals. A per-dataset optimum whose arm does not beat the reference on the paired test is flagged not significant vs ref and must not be quoted as a preference; so must one sitting on the edge of the scanned grid (grid edge) - extend the grid instead.