Files
Jungfraujoch/tests/RugnuxJointSweepsTest.cpp
leonarski_f 4809c1622c
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 16m43s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 28m46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 32m5s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 39m44s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m7s
Build Packages / Create release (push) Successful in 8s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m30s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m29s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 29m38s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m6s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m44s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 33m22s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m6s
Build Packages / Unit tests (push) Successful in 1h2m47s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 35m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 34m8s
Build Packages / Validation Battery (push) Successful in 38m31s
v1.0.0-rc.187 (#98)
* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform.
* Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models.
* Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own.
* Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log.
* Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell.
* jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card.
* jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own.
* jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob.
* jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then.
* Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates.
* Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option.
* Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes.

Reviewed-on: #98
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-11-19 20:49:28 +01:00

259 lines
12 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <catch2/catch_all.hpp>
#include <cmath>
#include <algorithm>
#include <map>
#include <random>
#include "../rugnux/RugnuxJointSweeps.h"
namespace {
// A tetragonal P4 crystal: Laue class 4/m on a 4/mmm lattice, so (k, h, -l) indexes it a second way.
const gemmi::SpaceGroup &P4() { return *gemmi::find_spacegroup_by_name("P 4"); }
constexpr float A = 60.0f, C = 40.0f;
std::array<int, 3> P4Key(int h, int k, int l) {
// the 4/m ASU representative, enough to give equivalents one intensity
std::array<int, 3> best{h, k, l};
const std::array<std::array<int, 3>, 8> eq{{{h, k, l}, {-k, h, l}, {-h, -k, l}, {k, -h, l},
{-h, -k, -l}, {k, -h, -l}, {h, k, -l}, {-k, h, -l}}};
for (const auto &e: eq) best = std::max(best, e);
return best;
}
// Four observations of every reflection, with counting noise of `sigma`.
JointMemberSweep MakeSweep(const std::map<std::array<int, 3>, double> &truth, const gemmi::Op &indexed_as,
int n_frames, uint32_t seed, double sigma = 10.0) {
std::mt19937 rng(seed);
std::normal_distribution<double> noise(0.0, sigma);
JointMemberSweep s;
s.n_frames = n_frames;
s.cell = UnitCell{A, A, C, 90, 90, 90};
s.space_group = P4();
s.isa = 1000.0 / sigma; // what its error model would resolve: the noise level against the intensities
for (const auto &[hkl, I]: truth) {
const auto m = indexed_as.apply_to_hkl({{hkl[0], hkl[1], hkl[2]}});
const auto d = static_cast<float>(1.0 / std::sqrt((hkl[0] * hkl[0] + hkl[1] * hkl[1]) / (A * A)
+ hkl[2] * hkl[2] / (C * C)));
double sum = 0;
for (int i = 0; i < 4; ++i) {
ScaledFull f;
f.h = m[0]; f.k = m[1]; f.l = m[2];
f.I = static_cast<float>(I + noise(rng));
f.sigma = static_cast<float>(sigma);
f.image_number = static_cast<float>(rng() % n_frames);
f.var_bkg = 4.0f;
f.var_per_I = 0.5f;
f.capture = 0.0f;
f.d = d;
s.fulls.push_back(f);
sum += f.I;
}
MergedReflection r;
r.h = m[0]; r.k = m[1]; r.l = m[2];
r.I = static_cast<float>(sum / 4);
r.sigma = static_cast<float>(sigma / 2);
r.I_half[0] = (s.fulls[s.fulls.size() - 4].I + s.fulls[s.fulls.size() - 2].I) / 2;
r.I_half[1] = (s.fulls[s.fulls.size() - 3].I + s.fulls[s.fulls.size() - 1].I) / 2;
r.d = d;
s.merged.push_back(r);
}
s.d_min = s.merged.empty() ? NAN : std::min_element(s.merged.begin(), s.merged.end(), [](const auto &a, const auto &b) {
return a.d < b.d;
})->d;
return s;
}
// The reflections of `truth` finer than `d` dropped: a sweep that stops there.
std::map<std::array<int, 3>, double> CoarserThan(const std::map<std::array<int, 3>, double> &truth, double d) {
std::map<std::array<int, 3>, double> t;
for (const auto &[hkl, I]: truth)
if (1.0 / std::sqrt((hkl[0] * hkl[0] + hkl[1] * hkl[1]) / (A * A) + hkl[2] * hkl[2] / (C * C)) >= d)
t[hkl] = I;
return t;
}
std::map<std::array<int, 3>, double> Truth(uint32_t seed) {
std::mt19937 rng(seed);
std::exponential_distribution<double> wilson(1.0);
std::map<std::array<int, 3>, double> t;
for (int h = -12; h <= 12; ++h)
for (int k = -12; k <= 12; ++k)
for (int l = -8; l <= 8; ++l) {
if (!(h || k || l)) continue;
const auto key = P4Key(h, k, l);
if (!t.contains(key)) t[key] = 1000.0 * wilson(rng);
}
return t;
}
} // namespace
TEST_CASE("RelateJointSweeps_AmbiguityRecoveredAndAStrangerLeftOut", "[Rugnux]") {
const auto truth = Truth(1);
const gemmi::Op twin = gemmi::parse_triplet("y,x,-z"); // h,k,l -> k,h,-l
std::vector<JointMemberSweep> sweeps;
sweeps.push_back(MakeSweep(truth, gemmi::Op::identity(), 100, 2, 10.0)); // the reference: the best measured
sweeps.push_back(MakeSweep(truth, twin, 80, 3, 20.0));
sweeps.push_back(MakeSweep(Truth(99), gemmi::Op::identity(), 50, 4, 20.0)); // another crystal
const auto relation = RelateJointSweeps(sweeps);
REQUIRE(relation.reference == 0);
CHECK(relation.sweeps[0].included);
REQUIRE(relation.sweeps[1].included);
CHECK(relation.sweeps[1].cc > 0.99);
CHECK(relation.sweeps[1].identity_cc < 0.5);
// the operator found takes sweep 2's indices back onto the reference's (up to the Laue group)
const auto back = relation.sweeps[1].cob.apply_to_hkl(twin.apply_to_hkl({{3, 1, 2}}));
CHECK(P4Key(back[0], back[1], back[2]) == P4Key(3, 1, 2));
CHECK(relation.sweeps[1].decision == "isomorphous");
// another crystal's intensities correlate with the reference's on no axes
CHECK_FALSE(relation.sweeps[2].included);
CHECK(relation.sweeps[2].decision == "no axes");
const auto input = BuildJointMergeInput(sweeps, relation);
REQUIRE(input.segments.size() == 2);
CHECK(input.segments[0].first_frame == 0);
CHECK(input.segments[0].n_frames == 100);
CHECK(input.segments[1].first_frame == 100);
CHECK(input.segments[1].n_frames == 80);
CHECK(input.segment_sweep == std::vector<int>{0, 1});
REQUIRE(input.outcomes.size() == 180);
// a full as the combine takes it: count I/c at correction c = var_per_I, variance var_bkg / c^2
const Reflection *r = nullptr;
for (const auto &o: input.outcomes)
if (!o.reflections.empty()) { r = &o.reflections.front(); break; }
REQUIRE(r != nullptr);
CHECK(r->image_scale_corr == Catch::Approx(0.5));
CHECK(r->var_bkg == Catch::Approx(16.0));
CHECK(r->partiality == 1.0f);
}
TEST_CASE("RelateJointSweeps_DoubtfulSweepDecidedOnTheMerge", "[Rugnux]") {
// Sweep 1: the same crystal measured noisily, its ratio uncertain; sweep 2: a crystal whose
// intensities differ by a few percent from the reference's, measured well enough to show it. Both fall in the doubtful band, and
// the held-out residual keeps the first and leaves out the second.
const auto truth = Truth(1);
auto shifted = truth;
std::mt19937 rng(7);
std::normal_distribution<double> change(0.0, 0.14);
for (auto &[hkl, I]: shifted)
I *= std::exp(change(rng));
std::vector<JointMemberSweep> sweeps;
sweeps.push_back(MakeSweep(truth, gemmi::Op::identity(), 100, 2, 50.0));
sweeps.push_back(MakeSweep(truth, gemmi::Op::identity(), 100, 3, 1500.0));
sweeps.push_back(MakeSweep(shifted, gemmi::Op::identity(), 100, 4, 100.0));
const auto relation = RelateJointSweeps(sweeps);
REQUIRE(relation.reference == 0);
INFO("ratios " << relation.sweeps[1].iso_ratio << " +- " << relation.sweeps[1].iso_se << ", "
<< relation.sweeps[2].iso_ratio << " +- " << relation.sweeps[2].iso_se);
CHECK(relation.sweeps[1].decision == "doubtful: kept");
CHECK(relation.sweeps[1].included);
CHECK(relation.sweeps[1].held_out_change < 0.0);
CHECK(relation.sweeps[2].decision == "doubtful: left out");
CHECK_FALSE(relation.sweeps[2].included);
CHECK(relation.sweeps[2].held_out_change > 0.0);
}
TEST_CASE("RelateJointSweeps_ReferenceIsTheBestMeasuredSweepNotTheFinestResolution", "[Rugnux]") {
// Sweep 1: the crystal measured well to 5 A. Sweep 2: the same reflections, and some finer ones, all of
// them noise - what a lattice that integrated nothing but background to the edge of the detector merges,
// with no ISa its error model could resolve. Its own merge reaches the finest resolution; the sweep
// with a measured accuracy is the reference.
const auto truth = Truth(1);
std::vector<JointMemberSweep> sweeps;
sweeps.push_back(MakeSweep(CoarserThan(truth, 5.0), gemmi::Op::identity(), 100, 2, 10.0));
sweeps.push_back(MakeSweep(truth, gemmi::Op::identity(), 100, 3, 5000.0));
sweeps[1].isa = NAN;
REQUIRE(sweeps[1].d_min < sweeps[0].d_min);
const auto relation = RelateJointSweeps(sweeps);
INFO("sweep 2: cc " << relation.sweeps[1].cc << ", ratio " << relation.sweeps[1].iso_ratio << " +- "
<< relation.sweeps[1].iso_se << ", held-out " << relation.sweeps[1].held_out_change << " +- "
<< relation.sweeps[1].held_out_se << ", " << relation.sweeps[1].decision);
CHECK(relation.reference == 0);
CHECK(relation.sweeps[0].decision == "reference");
CHECK_FALSE(relation.sweeps[1].included);
CHECK(relation.sweeps[1].decision == "no axes");
}
TEST_CASE("RelateJointSweeps_SweepsAtTheSameEdgeAreTiedOnResolution", "[Rugnux]") {
// Both sweeps stop at the same detector edge; the second's d_min is finer in the third decimal only, by
// the reflections it happened to measure there. That is a tie, and the higher ISa (the first) decides it.
const auto truth = Truth(1);
std::vector<JointMemberSweep> sweeps;
sweeps.push_back(MakeSweep(truth, gemmi::Op::identity(), 100, 2, 10.0));
sweeps.push_back(MakeSweep(truth, gemmi::Op::identity(), 100, 3, 10.0));
sweeps[0].isa = 40.0;
sweeps[1].isa = 20.0;
sweeps[1].d_min = sweeps[0].d_min * 0.995;
CHECK(RelateJointSweeps(sweeps).reference == 0);
// ... but a resolution finer beyond that is a better sweep.
sweeps[1].d_min = sweeps[0].d_min * 0.95;
CHECK(RelateJointSweeps(sweeps).reference == 1);
}
TEST_CASE("RelateJointSweeps_ReferenceIsASweepTheOthersAgreeWith", "[Rugnux]") {
// Two sweeps of the crystal to 3 A, and a third with a resolved ISa reaching the finest resolution of
// all whose intensities are another crystal's - what the crystal integrated in a wrong setting merges.
// On its own it is the best sweep, but no other correlates with it: the reference is one of the two.
const auto truth = Truth(1);
std::vector<JointMemberSweep> sweeps;
sweeps.push_back(MakeSweep(Truth(99), gemmi::Op::identity(), 100, 2, 10.0));
sweeps.push_back(MakeSweep(CoarserThan(truth, 3.0), gemmi::Op::identity(), 100, 3, 10.0));
sweeps.push_back(MakeSweep(CoarserThan(truth, 3.0), gemmi::Op::identity(), 100, 4, 20.0));
REQUIRE(sweeps[0].d_min < 0.99 * sweeps[1].d_min);
const auto relation = RelateJointSweeps(sweeps);
CHECK(relation.reference == 1);
CHECK(relation.sweeps[2].included);
CHECK_FALSE(relation.sweeps[0].included);
}
TEST_CASE("RelateJointSweeps_ASweepThatKeptTheTwinOverlapIsLeftOut", "[Rugnux]") {
// The reference's integration left out the reflections the other twin domain overlaps (index 2); the
// third sweep's kept them. Its intensities agree elsewhere, so only the integration tells them apart.
const auto truth = Truth(1);
std::vector<JointMemberSweep> sweeps;
for (uint32_t seed = 2; seed <= 4; ++seed)
sweeps.push_back(MakeSweep(truth, gemmi::Op::identity(), 100, seed, 10.0));
sweeps[0].isa = 200.0;
sweeps[0].twin_overlap_index = sweeps[1].twin_overlap_index = 2;
const auto relation = RelateJointSweeps(sweeps);
REQUIRE(relation.reference == 0);
CHECK(relation.sweeps[1].included);
CHECK_FALSE(relation.sweeps[2].included);
CHECK(relation.sweeps[2].decision == "twin overlap");
}
TEST_CASE("RelateJointSweeps_ASweepTheMergeCannotMeasureJoins", "[Rugnux]") {
// A cell of a few dozen unique reflections: too few for the isomorphism ratio, and the held-out
// residual of the merge moves by less than its own noise. Nothing measured refuses the sweep, so it
// joins - the second measurement of the same reflections is what a small-molecule merge needs.
std::map<std::array<int, 3>, double> small;
int n = 0;
for (const auto &[hkl, I]: Truth(1))
if (n++ % 50 == 0)
small[hkl] = I;
REQUIRE(small.size() < 200);
std::vector<JointMemberSweep> sweeps;
sweeps.push_back(MakeSweep(small, gemmi::Op::identity(), 100, 2, 200.0));
sweeps.push_back(MakeSweep(small, gemmi::Op::identity(), 100, 3, 300.0));
const auto relation = RelateJointSweeps(sweeps);
REQUIRE(relation.reference == 0);
INFO("held-out change " << relation.sweeps[1].held_out_change << " +- " << relation.sweeps[1].held_out_se
<< " on " << relation.sweeps[1].held_out_n << " reflections");
CHECK(std::isnan(relation.sweeps[1].iso_ratio));
CHECK(std::isnan(relation.sweeps[1].held_out_change));
CHECK(relation.sweeps[1].decision == "doubtful: kept");
CHECK(relation.sweeps[1].included);
}