Files
Jungfraujoch/tests/ReindexAmbiguityTest.cpp
T
leonarski_f 749db470ca
Build Packages / build:rpm (rocky9) (push) Successful in 19m56s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m57s
Build Packages / build:windows:cuda (push) Successful in 19m18s
Build Packages / build:viewer-tgz:cpu (push) Successful in 14m48s
Build Packages / build:viewer-tgz:cuda (push) Successful in 16m18s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 14m19s
Build Packages / build:rugnux:windows (push) Successful in 10m34s
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 8m49s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 20m55s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 17m4s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 20m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 19m15s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 24m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 20m32s
Build Packages / build:rpm (rocky8) (push) Successful in 23m39s
Build Packages / Generate python client (push) Successful in 46s
Build Packages / Build documentation (push) Successful in 1m45s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 11m3s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 11m30s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m10s
Build Packages / XDS test (neggia plugin) (push) Successful in 10m17s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 23m12s
Build Packages / DIALS test (push) Successful in 20m12s
v1.0.0-rc.164 (#74)
* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
* rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
* rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
* rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
* rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
* rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
* rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present.
* rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file.
* rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
* Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere.
* Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
* A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
* A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires.
* `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
* The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged.
* The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count.
* `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible.
* `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so.
* The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents.

Reviewed-on: #74
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-26 22:47:00 +02:00

145 lines
6.3 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <catch2/catch_all.hpp>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "../image_analysis/scale_merge/HKLKey.h"
#include "../image_analysis/scale_merge/ReindexAmbiguity.h"
namespace {
UnitCell Tetragonal() { return UnitCell{78.0f, 78.0f, 37.0f, 90.0f, 90.0f, 90.0f}; }
// A reference set: one distinct intensity per Laue-ASU reflection, so that reflections related by
// a twin law (which are NOT Laue-equivalent in P4) carry different intensities.
std::vector<MergedReflection> DistinctReference(int space_group_number) {
const HKLKeyGenerator key(true, *gemmi::find_spacegroup_by_number(space_group_number));
std::vector<MergedReflection> v;
std::unordered_set<uint64_t> seen;
for (int h = -8; h <= 8; ++h)
for (int k = -8; k <= 8; ++k)
for (int l = 0; l <= 8; ++l) {
if (h == 0 && k == 0 && l == 0) continue;
const uint64_t kk = key(h, k, l).pack();
if (!seen.insert(kk).second) continue; // one row per ASU reflection
MergedReflection r;
r.h = h; r.k = k; r.l = l;
r.I = static_cast<float>(100 + kk % 9973); // distinct, spread
r.sigma = 1.0f;
r.d = 60.0f / (1 + h * h + k * k + l * l);
v.push_back(r);
}
return v;
}
}
TEST_CASE("Reindex: holohedral crystal has no indexing ambiguity", "[reindex]") {
// P4(3)2(1)2 (422, holohedral for the tetragonal lattice) -> no twin laws.
CHECK(ReindexAmbiguityOperators(Tetragonal(), 96).empty());
// P422 likewise.
CHECK(ReindexAmbiguityOperators(Tetragonal(), 89).empty());
}
TEST_CASE("Reindex: merohedral crystal exposes the ambiguity operators", "[reindex]") {
// P4 (point group 4) in a tetragonal lattice (422) -> a non-trivial reindexing coset.
CHECK_FALSE(ReindexAmbiguityOperators(Tetragonal(), 75).empty());
}
TEST_CASE("Reindex: reference agreement recovers a misindexed dataset", "[reindex]") {
const int sg = 75; // P4
const auto reference = DistinctReference(sg);
const auto laws = ReindexAmbiguityOperators(Tetragonal(), sg);
REQUIRE_FALSE(laws.empty());
// Deliberately mis-index the data by one twin law.
const auto data = ReindexReflections(reference, laws.front());
const auto choice = ChooseReindex(
data, Tetragonal(), sg,
[&](const std::vector<MergedReflection> &m) { return ReferenceIntensityCC(m, reference, sg); });
CHECK_FALSE(choice.is_identity); // a reindex was needed
CHECK(choice.score > 0.99); // the winner realigns with the reference
CHECK(choice.identity_score < 0.9); // leaving it mis-indexed correlates poorly
CHECK(choice.n_candidates >= 2);
}
TEST_CASE("Reindex: a correctly indexed dataset keeps identity", "[reindex]") {
const int sg = 75;
const auto reference = DistinctReference(sg);
const auto choice = ChooseReindex(
reference, Tetragonal(), sg,
[&](const std::vector<MergedReflection> &m) { return ReferenceIntensityCC(m, reference, sg); });
CHECK(choice.is_identity);
CHECK(choice.score > 0.99);
}
TEST_CASE("Reindex into the ASU: a twin law permutes the reflections without losing any", "[reindex]") {
const int sg = 75; // P4
const auto reference = DistinctReference(sg);
const auto laws = ReindexAmbiguityOperators(Tetragonal(), sg);
REQUIRE_FALSE(laws.empty());
// Mis-index by a twin law, then reindex back into the ASU: the labels must land where an export
// needs them, and the reflection the label carries must be the one the reference has there.
const auto misindexed = ReindexReflections(reference, laws.front());
const auto fixed = ReindexMergedIntoAsu(misindexed, laws.front(), sg, /*merge_friedel=*/true);
REQUIRE(fixed.size() == reference.size());
const HKLKeyGenerator key(true, *gemmi::find_spacegroup_by_number(sg));
std::unordered_map<uint64_t, float> ref_by_key;
for (const auto &r : reference)
ref_by_key[key(r).pack()] = r.I;
std::unordered_set<uint64_t> seen;
for (const auto &r : fixed) {
const HKLKey k = key(r);
CHECK(k.h == r.h); // already at its own ASU index
CHECK(k.k == r.k);
CHECK(k.l == r.l);
CHECK(seen.insert(k.pack()).second); // no two reflections land on one label
const auto it = ref_by_key.find(k.pack());
REQUIRE(it != ref_by_key.end());
CHECK(it->second == r.I);
}
}
TEST_CASE("Reindex into the ASU: the change of hand swaps the Bijvoet halves", "[reindex]") {
// The change of hand is the inversion, so it leaves the Laue-ASU label alone and moves the
// anomalous signal instead - which is the whole of what adopting a model's hand does to intensities.
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_number(96); // P4(3)2(1)2
REQUIRE(sg != nullptr);
const HKLKeyGenerator key(true, *sg);
const HKLKey asu = key(3, 1, 2);
MergedReflection r;
r.h = asu.h; r.k = asu.k; r.l = asu.l;
r.I = 100.0f; r.sigma = 1.0f; r.d = 10.0f;
r.I_plus = 110.0f; r.sigma_plus = 2.0f; r.I_minus = 90.0f; r.sigma_minus = 3.0f;
r.F_plus = 10.5f; r.F_minus = 9.5f;
const auto merged = ReindexMergedIntoAsu({r}, sg->change_of_hand_op(), 96, /*merge_friedel=*/true);
REQUIRE(merged.size() == 1);
CHECK(merged[0].h == asu.h);
CHECK(merged[0].k == asu.k);
CHECK(merged[0].l == asu.l);
CHECK(merged[0].I == 100.0f);
CHECK(merged[0].I_plus == 90.0f);
CHECK(merged[0].I_minus == 110.0f);
CHECK(merged[0].sigma_plus == 3.0f);
CHECK(merged[0].sigma_minus == 2.0f);
CHECK(merged[0].F_plus == 9.5f);
CHECK(merged[0].F_minus == 10.5f);
// With the mates kept apart the merge stores the minus hand at -hkl, so the row moves there.
const auto anom = ReindexMergedIntoAsu({r}, sg->change_of_hand_op(), 96, /*merge_friedel=*/false);
REQUIRE(anom.size() == 1);
CHECK(anom[0].h == -asu.h);
CHECK(anom[0].k == -asu.k);
CHECK(anom[0].l == -asu.l);
CHECK(anom[0].I == 100.0f);
}