From c77ebe56c3e092a06bd5434344016c52aa887c0d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 3 Sep 2026 10:58:27 +0200 Subject: [PATCH 01/75] reader: a one-element string array is a scalar Masters exist that store sensor_material, description and the compression name as shape (1,) rather than as true scalars - JUNGFRAU files from an early beamline deployment do, and the application definition permits it. ReadString() required rank 0 and threw on anything else, so the whole file was lost: the metadata parse never finished and no image was ever examined. Accept any shape holding exactly one element, and keep throwing for a string dataset that genuinely holds several, which is a different quantity and cannot be read into one std::string. The numeric path already worked this way (HDF5DataSet_scalar_stored_rank1); this is the string half. Measured: three long-wavelength rotation datasets that could not be opened at all now process end to end. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- tests/HDF5WritingTest.cpp | 20 ++++++++++++++++++++ writer/HDF5Objects.cpp | 15 +++++++++++++-- 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/tests/HDF5WritingTest.cpp b/tests/HDF5WritingTest.cpp index 822c64032..bc18fc1a3 100644 --- a/tests/HDF5WritingTest.cpp +++ b/tests/HDF5WritingTest.cpp @@ -137,6 +137,26 @@ TEST_CASE("HDF5DataSet_string", "[HDF5][Unit]") { REQUIRE (H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0); } +TEST_CASE("HDF5DataSet_string_stored_rank1", "[HDF5][Unit]") { + // Facilities write a one-element string array where the application definition allows a scalar - + // JUNGFRAU masters from an early beamline deployment store sensor_material and description that + // way - and the file is unreadable if that is refused, because the metadata parse never finishes. + { + HDF5File file("scratch2b.h5"); + file.SaveVector("one", std::vector{"Si"}); + file.SaveVector("two", std::vector{"Si", "CdTe"}); + } + { + HDF5ReadOnlyFile file("scratch2b.h5"); + HDF5DataSet one(file, "one"); + CHECK(HDF5DataSpace(one).GetNumOfDimensions() == 1); + CHECK(one.ReadString() == "Si"); + REQUIRE_THROWS(HDF5DataSet(file, "two").ReadString()); + } + remove("scratch2b.h5"); + REQUIRE (H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0); +} + TEST_CASE("HDF5DataSet_vector", "[HDF5][Unit]") { std::vector tmp_vector (16384); tmp_vector[0] = 599.88; diff --git a/writer/HDF5Objects.cpp b/writer/HDF5Objects.cpp index 5f3ef778f..dd7e2393e 100644 --- a/writer/HDF5Objects.cpp +++ b/writer/HDF5Objects.cpp @@ -908,8 +908,19 @@ HDF5DataSet& HDF5DataSet::WriteDirectChunk(const void *data, hsize_t data_size, std::string HDF5DataSet::ReadString() const { HDF5DataSpace file_space(*this); - if (file_space.GetNumOfDimensions() != 0) - throw JFJochException(JFJochExceptionCategory::HDF5, "Dataset tries to read string (scalar) from vector dataset"); + // A rank-1 dataset holding exactly one string is the same thing as a scalar, and facilities + // write it both ways - JUNGFRAU masters from an early beamline deployment store + // sensor_material and description as shape (1,). Refusing those loses the whole file over a + // difference that carries no information. + if (file_space.GetNumOfDimensions() != 0) { + const auto dims = file_space.GetDimensions(); + hsize_t elements = 1; + for (const auto d : dims) + elements *= d; + if (elements != 1) + throw JFJochException(JFJochExceptionCategory::HDF5, + "Dataset tries to read string (scalar) from vector dataset"); + } HDF5DataType file_data_type(*this); const size_t size = file_data_type.GetElemSize(); -- 2.54.0 From 605312c2f4d897c867662a387caeb3e3fea06b9c Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 3 Sep 2026 13:48:26 +0200 Subject: [PATCH 02/75] twinning: the high-<|L|> comment says only what the corpus shows The L-test comment explained a high <|L|> as a structure whose intensities "behave centric", and a draft replacement explained it by few atoms, several of them on special positions, contributing a large term common to every structure factor. Both readings are wrong, and the second is wrong twice over. Over the 113 merged datasets in the corpus the maximum is <|L|> 0.712 with a second moment of 3.089. Those sit ABOVE the centric expectations, which are 2/pi = 0.6366 and 3.0 (simulated: 0.6367 and 3.003 for I ~ chi^2_1), so no Wilson distribution, centric or acentric, reaches them and they cannot be read as a statement about the structure's symmetry at all. The crystal carrying them is in an orthorhombic Sohncke group whose three 2_1 axes each show five absences against five controls with no violations - and that group has no special positions whatever, so the proposed mechanism cannot apply to it. Simulated, both halves of that mechanism move the statistics the wrong way: N randomly placed atoms give a second moment of 2 - 1/N, always below 2, and a large real term common to every F drives the moment from 2.00 down to 1.03 and <|L|> from 0.50 to 0.10 as it grows. The comment now gives the measured numbers, records that the cause is not established here, and names those two guesses as already excluded. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/scale_merge/TwinningAnalysis.cpp | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/image_analysis/scale_merge/TwinningAnalysis.cpp b/image_analysis/scale_merge/TwinningAnalysis.cpp index 8ae183ace..ee50dfa67 100644 --- a/image_analysis/scale_merge/TwinningAnalysis.cpp +++ b/image_analysis/scale_merge/TwinningAnalysis.cpp @@ -200,8 +200,16 @@ TwinningAnalysisResult AnalyzeTwinning(const std::vector& merg // the second moment down from 2.0 towards 1.5. A narrow intensity distribution sitting next to an // <|L|> at or ABOVE its untwinned value therefore has some other cause, and calling it a twin is // the one reading the data rule out. It happens on small-cell rotation data, where a twin fraction - // of 0.50 was reported for a crystal whose <|L|> was 0.63 - a value a twin cannot produce, and one - // that points at the opposite situation, a structure whose intensities behave centric. + // of 0.50 was reported for a crystal whose <|L|> was 0.63 - a value a twin cannot produce. + // A high <|L|> here is NOT evidence of a centrosymmetric structure. Measured across the corpus, + // the highest <|L|> of all - 0.712, second moment 3.089 - belongs to a small (~10 A) cell in an + // orthorhombic group whose three 2_1 axes are each proven by absences, no violations against a + // control class of the same size, so it is chiral. Both numbers sit ABOVE the centric + // expectations of 0.637 and 3.0, which is the point: no Wilson distribution, centric or acentric, + // reaches them, so they are not a statement about the structure's symmetry at all. What does put + // them there is not established here, but it is neither of the obvious guesses - a structure of + // few atoms gives a second moment of 2 - 1/N, BELOW 2, and a large term common to every structure + // factor drives it towards 1. Read this statistic as "not a twin", and nothing further. const bool l_test_contradicts_twin = result.l_test_pairs > 0 && result.mean_abs_l >= 0.50; result.twinning_suspected = result.merohedral_twinning_possible && !l_test_contradicts_twin && ((result.l_test_pairs > 0 && result.mean_abs_l < 0.44) || -- 2.54.0 From 203dbf417be54223adab787461e112a3ec3781c4 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 3 Sep 2026 13:49:32 +0200 Subject: [PATCH 03/75] rugnux: two messages that asserted more than was measured The metric-symmetry warning told the user the intensities "did not support" the extra rotations. They may never have been asked. SearchSpaceGroup takes a lattice class from rotation indexing (SearchSpaceGroupOptions::lattice_system), and EnumeratePointGroups keeps only point groups whose rotation set is a subset of that class's holohedry - so where the class is lower than the metric, the extra operators are skipped before a single intensity is read, and their absence from the answer is not evidence against them. The warning now says the answer depends on the class the search was given. Placeholders and arguments are unchanged. The comment above LogXDSGeometry claimed the printed ROTATION_AXIS sign is simply wrong and that a user must negate it. Two things refute that. The line is printed once per pass, and the sign rescue rewrites the axis between passes, so a run's first and last line carry opposite signs on 12 of 115 corpus runs; "printed as stored" describes only the first, and the last is the axis the run integrated at. And the sign is not a constant: it is paired with the detector axis directions. Two datasets whose detector axes are both +x/+y do want the negation (8.9% of COLSPOT spots indexed as printed against 92.6%, and 1060/10080 against 9624/10080), but a third whose axes are negative wants the sign as printed (43.9% against 13.5%), so the universal instruction was wrong. The comment now says which line to read, quotes that statistic with its chance floor (a permuted-frame null reaches 14.9% against a real 15.4%, so the low members mean IDXREF gave up rather than a measured zero), records that one of the two runs read a file our own converter had written and so testifies about XDS rather than about foreign files, and tells the user to try both signs and keep the one that indexes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- rugnux/Rugnux.cpp | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index a75631ee9..fd638197e 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -432,10 +432,12 @@ static void WarnIfMetricHostsMoreSymmetry(const std::optional &cell, case 24: metric_system = "cubic"; break; default: break; } - logger.Warning("The cell metric is {} - it admits {} rotations where {} has {}. The intensities " - "did not support the extra ones, so they are either absent or too weak to see; " - "this is a note, not a correction. Where the higher symmetry is real, merging in " - "it would raise multiplicity and completeness.", + logger.Warning("The cell metric is {} - it admits {} rotations where {} has {}. Whether the extra " + "ones were tested against the intensities depends on the lattice class the search " + "was given: where that class is lower than the metric, they were never enumerated, " + "so their absence here is not evidence against them. This is a note, not a " + "correction. Where the higher symmetry is real, merging in it would raise " + "multiplicity and completeness.", metric_system, metric_rotations, sg.xhm(), group_rotations); } @@ -501,9 +503,30 @@ Rugnux::Rugnux(JFJochReader &reader, DiffractionExperiment experiment, // crystal rather than the direct beam, which is what our beam centre is too // (docs/DETECTOR_GEOMETRY.md), so they map across without correction. // -// ROTATION_AXIS is printed as stored. Its direction is right, the frames being shared; its SIGN has -// not been cross-checked against an XDS refinement, so treat a flip there as unconfirmed rather than -// as a discrepancy in the data. +// ROTATION_AXIS is printed once per pass, and it is not always the vector stored in the file: the +// sign rescue further down adopts the opposite sign when the file's own indexes almost nothing, and +// the following pass then prints what it adopted. On the corpus the first and last line of a run +// disagree in sign on 12 of 115 rotation runs, so the LAST one - the axis the run actually +// integrated at - is the one to read. +// +// That sign is not reliably the one XDS wants - but it is not a constant either, so do NOT negate +// this line on principle. It is paired with the detector axis directions printed just above it and +// has to be read together with them: two rotation datasets whose detector axes are both +x/+y wanted +// the negation, while a third whose axes are negative wanted the sign as printed. The measure is the +// fraction of COLSPOT spots IDXREF can index - 8.9% as printed against 92.6% negated, and 1060/10080 +// against 9624/10080; the other way round, 43.9% as printed against 13.5%. +// +// Read the HIGH member of each pair. That statistic has a chance floor which reaches about 15% on a +// dense reciprocal lattice - a null with the frame numbers permuted scored 14.9% against a real +// 15.4% - so a low value means "IDXREF gave up", not a measured near-zero: both wrong-sign runs +// above stopped on INSUFFICIENT PERCENTAGE (< 50%) OF INDEXED REFLECTIONS, one of them after +// refining the right cell and rejecting 9020 spots as too far from their ideal positions, which is +// the signature of a mis-signed spindle. So try both signs and keep the one that indexes; a sign +// carried over from another file is worth nothing. +// +// One of those two runs read a file our own converter had written, whose axis default was the +// negative sign. It therefore says what XDS does with a given sign, which is what this comment is +// about, and nothing about what a foreign file stores. static void LogXDSGeometry(const DiffractionExperiment &experiment) { Logger logger("Rugnux"); -- 2.54.0 From 491167c90dfa1e8a1725f56978abf1cadb3e8009 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 3 Sep 2026 13:43:32 +0200 Subject: [PATCH 04/75] rugnux: a Bravais class the reduction decided by rounding is re-asked on the metric's own cell The class is named by Niggli-reducing the indexed cell and looking the reduced cell up in the 44 lattice characters, and that lookup is a coin flip for any lattice whose Buerger cells straddle the Niggli type-I/type-II boundary. An F-centred cubic lattice does so by construction: it has reduced forms on both sides, the two sides carry different characters, and which side the reduction lands on is set by the last digits of whatever refinement produced the cell. Measured over 600 perturbations of one such lattice: 43% cubic F, 36% tetragonal I, 21% orthorhombic I, and the split is flat over a factor of ten in the noise. The class then caps the point-group search, so from the body-centred sub-cell the cubic three-fold is never enumerated and the run reports that nothing was refused - which is accurate, because nothing was asked. Le Page's two-fold search has no such key: it measures each rotation's obliquity on the lattice itself, in a primitive basis. LePageLattice turns the rotation group it finds into a conventional cell, a centring letter and an integral change of basis, and where that group is larger than the adopted class's holohedry the merge is reindexed into that cell and the space-group search is run again there, on both merges, with every gate live. Nothing here decides: the reindex is committed only where the search in the new setting confirms a strictly higher point group AND the centring the new cell describes, so a pseudo-symmetric metric leaves the answer already in hand standing. Measured as a paired battery over 113 rotation datasets: the re-ask fires on 7 and adopts on 1, and that one crystal - an F-centred cubic lattice the reduction had named body-centred tetragonal - moves to its deposited group, gaining 0.10 A of resolution and 2.8x the multiplicity at R_meas 0.117 -> 0.120. Nothing else moves, in space group, resolution, CC1/2, R_meas, multiplicity, I/sigma or completeness. A second such crystal, named body-centred orthorhombic, is offered the same cubic cell and confirms all 23 added operators at CC 0.96 with an H ratio of 1.00, and is still refused, on the merge chi^2 ratio at 2.50x a bound of 1.85. Letting H rescue that refusal is the one-line change an earlier round measured and rejected - it promotes the synthetic P 4_3 2_1 2 in the test suite to point group 432 - so it is not here, and that crystal is left where it was. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/ACKNOWLEDGEMENT.md | 4 +- docs/CHANGELOG.md | 1 + image_analysis/lattice_search/CMakeLists.txt | 4 +- .../lattice_search/LePageLattice.cpp | 572 ++++++++++++++++++ image_analysis/lattice_search/LePageLattice.h | 34 ++ rugnux/Rugnux.cpp | 125 +++- tests/CMakeLists.txt | 1 + tests/LePageLatticeTest.cpp | 140 +++++ 8 files changed, 877 insertions(+), 4 deletions(-) create mode 100644 image_analysis/lattice_search/LePageLattice.cpp create mode 100644 image_analysis/lattice_search/LePageLattice.h create mode 100644 tests/LePageLatticeTest.cpp diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index b152cbacb..c79ec7d9d 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -195,7 +195,9 @@ stable algorithms for the computation of reduced unit cells" (2004), Acta Cryst. **Le Page's metric-symmetry search** - the obliquity of each of the 81 candidate two-folds of a reduced cell, which is what tells a run that its lattice metric hosts more rotational symmetry than -the group its intensities supported. Used through GEMMI's implementation of it. Y. Le Page, "The +the group its intensities supported, and the derivation of the conventional axes from that +rotation group, which is what the run then offers to the space-group search as a second lattice +candidate. The two-fold search is used through GEMMI's implementation of it. Y. Le Page, "The derivation of the axes of the conventional unit cell from the dimensions of the Buerger-reduced cell" (1982), J. Appl. Cryst. 15, 255-259 [doi:10.1107/S0021889882011959](https://doi.org/10.1107/S0021889882011959). diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 9c44be2b2..2b331a909 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,6 +7,7 @@ * `rugnux --model` places the model against the data as a rigid body before scoring it, writes sigma_A-weighted 2mFo-DFc and mFo-DFc maps in place of the unweighted 2Fo-Fc and Fo-Fc, and writes the model as it was placed - `_model.cif`, and `_model.pdb` where the PDB format can express the cell - in the cell and space group of the reflection files beside it. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, and open masters written at other facilities, including Eiger 1.x and third-party NXmx. * `rugnux` determines the lattice and the space group more reliably - the true cell where the first pass offers a whole-number multiple of it, so a pseudo-translated crystal keeps its full-length axis and a small molecule is indexed on its own cell rather than a protein-sized one, and the point group, the setting and the systematic absences - and `-S` refuses or re-seats a fixed space group whose symmetry axes the indexed cell does not carry. +* `rugnux` asks the space-group search again on the cell the lattice metric supports, whenever that metric carries more rotational symmetry than the Bravais class the indexer named, so a lattice whose reduction landed in a lower-symmetry sub-cell can still reach its true point group; the higher symmetry is adopted only where the intensities confirm it. * `rugnux` measures the beam centre on every run and indexes with it when the file's value indexes nothing, refines only the detector-tilt component the data determine - a beam-centre error is no longer reported as a tilt - and places a detector swung out on a 2theta arm where the file says it stands. * `rugnux` writes the unmerged MTZ by default, with a P1 merge beside it, a batch header for every image the observations span, and events kept to the same `--min-captured-fraction` as the merge, so a wrong space group can be re-merged in a scaling program without reprocessing. * `rugnux` writes reflection files in the conventions downstream programs read: `FreeR_flag` is 0 for the test set and 1 for the working set - it was the other way round - the merged and P1 MTZ carry the reserved `HKL_base` dataset so a CCP4 program reads the wavelength instead of falling back to 1.54187 A, and the merged mmCIF marks the free set as `_refln.status` = `f`. diff --git a/image_analysis/lattice_search/CMakeLists.txt b/image_analysis/lattice_search/CMakeLists.txt index 3159806bf..a7a6ce63d 100644 --- a/image_analysis/lattice_search/CMakeLists.txt +++ b/image_analysis/lattice_search/CMakeLists.txt @@ -1,2 +1,2 @@ -ADD_LIBRARY(JFJochLatticeSearch STATIC LatticeSearch.cpp LatticeSearch.h) -TARGET_LINK_LIBRARIES(JFJochLatticeSearch Eigen3::Eigen JFJochCommon) \ No newline at end of file +ADD_LIBRARY(JFJochLatticeSearch STATIC LatticeSearch.cpp LatticeSearch.h LePageLattice.cpp LePageLattice.h) +TARGET_LINK_LIBRARIES(JFJochLatticeSearch Eigen3::Eigen JFJochCommon) diff --git a/image_analysis/lattice_search/LePageLattice.cpp b/image_analysis/lattice_search/LePageLattice.cpp new file mode 100644 index 000000000..3060836f8 --- /dev/null +++ b/image_analysis/lattice_search/LePageLattice.cpp @@ -0,0 +1,572 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +// Bravais lattice from the metric symmetry OPERATORS, rather than from a table of reduced-cell +// characters. The two-fold search is Le Page's (1982) J. Appl. Cryst. 15, 255-259, vendored in +// gemmi (twin.hpp); what is here is the step after it - turning the rotation group it finds into +// a conventional cell, a centring letter and an integral change of basis. +// +// Following Le Page (1982) J. Appl. Cryst. 15, 255-259 and +// Grosse-Kunstleve (1999) Acta Cryst. A55, 383-395 + +#include "LePageLattice.h" +#include "../../common/JFJochMath.h" + +#include +#include + +#include +#include +#include +#include + +namespace { + +using Mat3i = std::array; // row-major +using Vec3i = std::array; + +constexpr Mat3i kIdentity3 = {1, 0, 0, 0, 1, 0, 0, 0, 1}; + +int At(const Mat3i &m, int r, int c) { return m[3 * r + c]; } + +Mat3i MatMul(const Mat3i &a, const Mat3i &b) { + Mat3i r{}; + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) { + int s = 0; + for (int k = 0; k < 3; k++) + s += At(a, i, k) * At(b, k, j); + r[3 * i + j] = s; + } + return r; +} + +int Det3(const Mat3i &m) { + return At(m,0,0) * (At(m,1,1)*At(m,2,2) - At(m,1,2)*At(m,2,1)) + - At(m,0,1) * (At(m,1,0)*At(m,2,2) - At(m,1,2)*At(m,2,0)) + + At(m,0,2) * (At(m,1,0)*At(m,2,1) - At(m,1,1)*At(m,2,0)); +} + +// adj(M), so that M^-1 = adj(M) / det(M) - kept integral to determine the centring exactly. +Mat3i Adjugate(const Mat3i &m) { + Mat3i a{}; + a[0] = (At(m,1,1)*At(m,2,2) - At(m,1,2)*At(m,2,1)); + a[1] = -(At(m,0,1)*At(m,2,2) - At(m,0,2)*At(m,2,1)); + a[2] = (At(m,0,1)*At(m,1,2) - At(m,0,2)*At(m,1,1)); + a[3] = -(At(m,1,0)*At(m,2,2) - At(m,1,2)*At(m,2,0)); + a[4] = (At(m,0,0)*At(m,2,2) - At(m,0,2)*At(m,2,0)); + a[5] = -(At(m,0,0)*At(m,1,2) - At(m,0,2)*At(m,1,0)); + a[6] = (At(m,1,0)*At(m,2,1) - At(m,1,1)*At(m,2,0)); + a[7] = -(At(m,0,0)*At(m,2,1) - At(m,0,1)*At(m,2,0)); + a[8] = (At(m,0,0)*At(m,1,1) - At(m,0,1)*At(m,1,0)); + return a; +} + +Vec3i MulVec(const Mat3i &m, const Vec3i &v) { + return {At(m,0,0)*v[0] + At(m,0,1)*v[1] + At(m,0,2)*v[2], + At(m,1,0)*v[0] + At(m,1,1)*v[1] + At(m,1,2)*v[2], + At(m,2,0)*v[0] + At(m,2,1)*v[1] + At(m,2,2)*v[2]}; +} + +Vec3i Cross(const Vec3i &a, const Vec3i &b) { + return {a[1]*b[2] - a[2]*b[1], a[2]*b[0] - a[0]*b[2], a[0]*b[1] - a[1]*b[0]}; +} + +int Gcd3(const Vec3i &v) { + int g = std::gcd(std::gcd(std::abs(v[0]), std::abs(v[1])), std::abs(v[2])); + return g; +} + +// The shortest lattice vector along a direction is the direction divided by the gcd of its +// components, and its sign is fixed so the same axis is always the same vector. +bool MakePrimitive(Vec3i &v) { + const int g = Gcd3(v); + if (g == 0) + return false; + for (int &x : v) + x /= g; + for (int i = 0; i < 3; i++) { + if (v[i] > 0) break; + if (v[i] < 0) { v[0] = -v[0]; v[1] = -v[1]; v[2] = -v[2]; break; } + } + return true; +} + +int Trace(const Mat3i &m) { return m[0] + m[4] + m[8]; } + +// A proper rotation has trace 1 + 2 cos(theta): 3, -1, 0, 1, 2 for orders 1, 2, 3, 4, 6. +int RotationOrder(const Mat3i &m) { + switch (Trace(m)) { + case 3: return 1; + case -1: return 2; + case 0: return 3; + case 1: return 4; + case 2: return 6; + default: return 0; + } +} + +// The +1 eigenvector of a rotation of order >= 2: (R - I) has rank 2, so the cross product of any +// two independent of its rows spans the kernel. Exact in integers. +bool RotationAxis(const Mat3i &m, Vec3i &axis) { + Mat3i d = m; + d[0] -= 1; d[4] -= 1; d[8] -= 1; + const Vec3i rows[3] = {{d[0], d[1], d[2]}, {d[3], d[4], d[5]}, {d[6], d[7], d[8]}}; + Vec3i best{0, 0, 0}; + long long best_norm = 0; + for (int i = 0; i < 3; i++) + for (int j = i + 1; j < 3; j++) { + const Vec3i c = Cross(rows[i], rows[j]); + const long long n = 1LL*c[0]*c[0] + 1LL*c[1]*c[1] + 1LL*c[2]*c[2]; + if (n > best_norm) { best_norm = n; best = c; } + } + if (best_norm == 0) + return false; + axis = best; + return MakePrimitive(axis); +} + +// Metric tensor of the reduced primitive cell, so integer lattice vectors can be measured. +struct Metric { + double g[3][3]; + double Dot(const Vec3i &u, const Vec3i &v) const { + double s = 0; + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) + s += g[i][j] * u[i] * v[j]; + return s; + } + double Len(const Vec3i &u) const { return std::sqrt(std::max(0.0, Dot(u, u))); } +}; + +Metric MetricOf(const gemmi::UnitCell &c) { + Metric m{}; + const double ca = std::cos(gemmi::rad(c.alpha)), cb = std::cos(gemmi::rad(c.beta)), + cg = std::cos(gemmi::rad(c.gamma)); + m.g[0][0] = c.a * c.a; m.g[1][1] = c.b * c.b; m.g[2][2] = c.c * c.c; + m.g[0][1] = m.g[1][0] = c.a * c.b * cg; + m.g[0][2] = m.g[2][0] = c.a * c.c * cb; + m.g[1][2] = m.g[2][1] = c.b * c.c * ca; + return m; +} + +// The rotation group generated by the two-folds Le Page found, built one generator at a time in +// order of increasing obliquity and only while the result is still a possible lattice rotation +// group. Two-folds accepted at a loose obliquity need not be consistent with each other - closing +// over an inconsistent set runs away - so a generator whose closure is not one of the seven orders +// a holohedry can have is dropped instead of aborting the whole search. +bool IsHolohedryOrder(size_t n) { + return n == 1 || n == 2 || n == 4 || n == 6 || n == 8 || n == 12 || n == 24; +} + +std::vector CloseGroup(const std::vector &gens) { + std::vector group{kIdentity3}; + for (const Mat3i &g : gens) { + if (std::find(group.begin(), group.end(), g) != group.end()) + continue; + std::vector trial = group; + trial.push_back(g); + bool ok = true; + for (size_t i = 0; i < trial.size() && ok; i++) + for (size_t j = 0; j < trial.size() && ok; j++) { + const Mat3i p = MatMul(trial[i], trial[j]); + if (std::find(trial.begin(), trial.end(), p) == trial.end()) { + if (trial.size() >= 24) { ok = false; break; } + trial.push_back(p); + } + } + if (ok && IsHolohedryOrder(trial.size())) + group = std::move(trial); + } + return group; +} + +// All integer vectors v with r . v = 0 form a plane lattice; a pair of them is a BASIS of it +// exactly when their cross product is +/- r (r primitive). Enumerating short v and taking the two +// shortest that pass that test gives the reduced basis of the plane - in two dimensions the +// successive minima are always reachable by a basis. +bool PlaneBasis(const Vec3i &normal, const Metric &metric, Vec3i &p, Vec3i &q) { + Vec3i r = normal; + if (!MakePrimitive(r)) + return false; + for (int box = 4; box <= 24; box *= 2) { + std::vector cand; + for (int i = -box; i <= box; i++) + for (int j = -box; j <= box; j++) + for (int k = -box; k <= box; k++) { + const Vec3i v{i, j, k}; + if (v[0] == 0 && v[1] == 0 && v[2] == 0) + continue; + if (r[0]*i + r[1]*j + r[2]*k != 0) + continue; + cand.push_back(v); + } + std::sort(cand.begin(), cand.end(), [&](const Vec3i &a, const Vec3i &b) { + return metric.Dot(a, a) < metric.Dot(b, b); + }); + for (const Vec3i &v1 : cand) { + for (const Vec3i &v2 : cand) { + const Vec3i c = Cross(v1, v2); + if ((c[0] == r[0] && c[1] == r[1] && c[2] == r[2]) || + (c[0] == -r[0] && c[1] == -r[1] && c[2] == -r[2])) { + p = v1; q = v2; + return true; + } + } + break; // v1 is the shortest vector of the plane; only it can start a reduced basis + } + } + return false; +} + +// The centring translations of the primitive lattice inside the conventional cell M: the rows of +// M^-1 generate them, and |det M| of them exist. Kept in units of 1/det so the arithmetic is exact. +std::vector> CentringTranslations(const Mat3i &M, int &den) { + den = std::abs(Det3(M)); + const Mat3i adj = Adjugate(M); + const int sign = Det3(M) > 0 ? 1 : -1; + std::vector> out; + for (int i = 0; i < den; i++) + for (int j = 0; j < den; j++) + for (int k = 0; k < den; k++) { + std::array t{}; + for (int c = 0; c < 3; c++) { + int s = sign * (i * At(adj, 0, c) + j * At(adj, 1, c) + k * At(adj, 2, c)); + s %= den; + if (s < 0) s += den; + t[c] = s; + } + if (std::find(out.begin(), out.end(), t) == out.end()) + out.push_back(t); + } + std::sort(out.begin(), out.end()); + return out; +} + +// Name the centring from those translations. Anything not on this list is not a Bravais lattice +// and means the conventional axes were built wrongly. +char CentringSymbol(const std::vector> &t, int den) { + auto has = [&](int x, int y, int z) { + return std::find(t.begin(), t.end(), std::array{x, y, z}) != t.end(); + }; + if (den == 1) + return 'P'; + if (den == 2) { + if (has(1, 1, 0)) return 'C'; + if (has(0, 1, 1)) return 'A'; + if (has(1, 0, 1)) return 'B'; + if (has(1, 1, 1)) return 'I'; + } + if (den == 3) { + if (has(1, 2, 2)) return 'R'; // obverse + if (has(2, 1, 2)) return 'r'; // reverse - caller converts + } + if (den == 4 && has(0, 2, 2) && has(2, 0, 2) && has(2, 2, 0)) + return 'F'; + return '?'; +} + +Mat3i FromRows(const Vec3i &a, const Vec3i &b, const Vec3i &c) { + return {a[0], a[1], a[2], b[0], b[1], b[2], c[0], c[1], c[2]}; +} + +gemmi::Mat33 ToMat33(const Mat3i &m) { + return gemmi::Mat33(m[0], m[1], m[2], m[3], m[4], m[5], m[6], m[7], m[8]); +} + +} // namespace + +std::optional LePageLattice(const CrystalLattice &L, double max_obliquity_deg) { + const UnitCell uc = L.GetUnitCell(); + gemmi::UnitCell g_uc(uc.a, uc.b, uc.c, uc.alpha, uc.beta, uc.gamma); + gemmi::GruberVector gv(g_uc, 'P', true); + // Same scaled epsilon as the character table: the type decision is on scalar products of order + // 10^3-10^5 A^2 carried in a float cell. + gv.niggli_reduce(1e-5 * std::max({gv.A, gv.B, gv.C})); + + CrystalLattice L_red = L; + if (gv.change_of_basis) + L_red = L.Multiply(gemmi::rot_as_mat33(gv.change_of_basis->rot).transpose()); + + const gemmi::UnitCell reduced = gv.get_cell(); + const Metric metric = MetricOf(reduced); + + // Le Page: the 81 two-folds a reduced cell can carry, ranked by obliquity. + const std::vector two_folds = gemmi::find_lattice_2fold_ops(reduced, max_obliquity_deg); + + std::vector gens; + gens.reserve(two_folds.size()); + double worst_obliquity = 0; + for (const auto &[op, delta] : two_folds) { + Mat3i m{}; + bool ok = true; + for (int i = 0; i < 3 && ok; i++) + for (int j = 0; j < 3 && ok; j++) { + const int v = op.rot[i][j]; + if (v % gemmi::Op::DEN != 0) ok = false; + m[3 * i + j] = v / gemmi::Op::DEN; + } + if (ok) + gens.push_back(m); + } + + const std::vector group = CloseGroup(gens); + for (const auto &[op, delta] : two_folds) { + Mat3i m{}; + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) + m[3 * i + j] = op.rot[i][j] / gemmi::Op::DEN; + if (std::find(group.begin(), group.end(), m) != group.end()) + worst_obliquity = std::max(worst_obliquity, delta); + } + + // Sort the group's operators by rotation order, and record the axis of each. + struct Axis { int order; Vec3i dir; Mat3i rot; }; + std::vector axes; + int n_order[7] = {0, 0, 0, 0, 0, 0, 0}; + for (const Mat3i &m : group) { + const int order = RotationOrder(m); + if (order == 0 || Det3(m) != 1) + return std::nullopt; // not a proper rotation - refuse rather than guess + n_order[order]++; + if (order == 1) + continue; + Vec3i dir{}; + if (!RotationAxis(m, dir)) + return std::nullopt; + axes.push_back({order, dir, m}); + } + + gemmi::CrystalSystem system; + switch (group.size()) { + case 1: system = gemmi::CrystalSystem::Triclinic; break; + case 2: system = gemmi::CrystalSystem::Monoclinic; break; + case 4: system = gemmi::CrystalSystem::Orthorhombic; break; + case 6: system = gemmi::CrystalSystem::Trigonal; break; + case 8: system = gemmi::CrystalSystem::Tetragonal; break; + case 12: system = n_order[6] > 0 ? gemmi::CrystalSystem::Hexagonal : gemmi::CrystalSystem::Cubic; break; + case 24: system = gemmi::CrystalSystem::Cubic; break; + default: return std::nullopt; + } + + // Conventional axes. Every one of them is the SHORTEST lattice vector along a symmetry + // direction, which is what makes them integral combinations of the reduced primitive basis and + // the change of basis an integer matrix. + auto shortest_along = [&](const Vec3i &d) { Vec3i v = d; MakePrimitive(v); return v; }; + + // The two-folds perpendicular to the principal axis are exactly those that send it to minus + // itself - an integer test, no angle tolerance. + auto perpendicular_two_folds = [&](const Vec3i &principal) { + std::vector out; + for (const Axis &ax : axes) { + if (ax.order != 2) + continue; + const Vec3i w = MulVec(ax.rot, principal); + if (w[0] == -principal[0] && w[1] == -principal[1] && w[2] == -principal[2]) + out.push_back(shortest_along(ax.dir)); + } + std::sort(out.begin(), out.end(), [&](const Vec3i &a, const Vec3i &b) { + return metric.Dot(a, a) < metric.Dot(b, b); + }); + return out; + }; + + auto principal_of_order = [&](int order) -> std::optional { + for (const Axis &ax : axes) + if (ax.order == order) + return ax; + return std::nullopt; + }; + + Mat3i M = kIdentity3; + + switch (system) { + case gemmi::CrystalSystem::Triclinic: + M = kIdentity3; + break; + + case gemmi::CrystalSystem::Monoclinic: { + const Vec3i b = shortest_along(axes.front().dir); + // The lattice vectors perpendicular to the two-fold are exactly those it negates, so the + // plane they span is the kernel of (R + I) - exact, and NOT the coordinate vectors + // orthogonal to the axis, which is a different set on a non-orthogonal basis. (R + I) + // has rank one there, so any non-zero row of it is the plane's reciprocal normal. + Mat3i s = axes.front().rot; + s[0] += 1; s[4] += 1; s[8] += 1; + Vec3i normal{0, 0, 0}; + for (int i = 0; i < 3; i++) + if (s[3*i] != 0 || s[3*i+1] != 0 || s[3*i+2] != 0) { + normal = {s[3*i], s[3*i+1], s[3*i+2]}; + break; + } + Vec3i p{}, q{}; + if (!PlaneBasis(normal, metric, p, q)) + return std::nullopt; + if (Det3(FromRows(p, b, q)) < 0) + for (int t = 0; t < 3; t++) q[t] = -q[t]; + + // Any unimodular pair drawn from that plane is a valid a and c. They differ in how + // oblique beta comes out and in whether the centring then reads C or I - which are the + // same lattice in two settings, not two lattices. ITA takes the least oblique cell and so + // does this; an oblique one is a real cost downstream, where the constrained refinement + // bounds the cell angles at 30 and 150 degrees. Taking the reduced plane basis and + // renaming a centring afterwards does not do it: on a lattice whose reduced form is the + // one the character table calls mI, a -> a + c turns an I-centred cell with beta 120 into + // a C-centred one with beta 32. + double best_beta = 1e30, best_len = 1e30; + bool found = false; + for (int i = -2; i <= 2; i++) + for (int j = -2; j <= 2; j++) + for (int k = -2; k <= 2; k++) + for (int l = -2; l <= 2; l++) { + if (i * l - j * k != 1) + continue; + Vec3i a{}, c{}; + for (int t = 0; t < 3; t++) { + a[t] = i * p[t] + j * q[t]; + c[t] = k * p[t] + l * q[t]; + } + const double len = metric.Dot(a, a) + metric.Dot(c, c); + const double cos_beta = metric.Dot(a, c) / (metric.Len(a) * metric.Len(c)); + const double beta = std::acos(std::clamp(std::fabs(cos_beta), 0.0, 1.0)); + const double obtuse = 180.0 - beta * 180.0 / PI; // beta stated obtuse + if (found && (obtuse > best_beta + 1e-6 || + (obtuse > best_beta - 1e-6 && len >= best_len - 1e-6))) + continue; + const Mat3i cand = FromRows(a, b, c); + int d = 0; + const auto t = CentringTranslations(cand, d); + const char ce = CentringSymbol(t, d); + if (ce != 'P' && ce != 'C' && ce != 'I') + continue; // an A-centred naming of the same lattice + best_beta = obtuse; best_len = len; M = cand; found = true; + } + if (!found) + return std::nullopt; + // beta obtuse: negating a and b keeps the handedness, keeps b on the two-fold, and leaves + // every half-integer centring vector where it was. + { + const Vec3i a{M[0], M[1], M[2]}, c{M[6], M[7], M[8]}; + if (metric.Dot(a, c) > 0) + for (int t = 0; t < 6; t++) M[t] = -M[t]; + } + break; + } + + case gemmi::CrystalSystem::Orthorhombic: { + std::vector u; + for (const Axis &ax : axes) + u.push_back(shortest_along(ax.dir)); + if (u.size() != 3) + return std::nullopt; + std::sort(u.begin(), u.end(), [&](const Vec3i &a, const Vec3i &b) { + return metric.Dot(a, a) < metric.Dot(b, b); + }); + M = FromRows(u[0], u[1], u[2]); + break; + } + + case gemmi::CrystalSystem::Tetragonal: { + const auto four = principal_of_order(4); + if (!four) + return std::nullopt; + const Vec3i c = shortest_along(four->dir); + const auto in_plane = perpendicular_two_folds(four->dir); + if (in_plane.empty()) + return std::nullopt; + const Vec3i a = in_plane.front(); + const Vec3i b = MulVec(four->rot, a); // the 4-fold carries a onto b + M = FromRows(a, b, c); + break; + } + + case gemmi::CrystalSystem::Hexagonal: + case gemmi::CrystalSystem::Trigonal: { + const auto principal = system == gemmi::CrystalSystem::Hexagonal ? principal_of_order(6) + : principal_of_order(3); + if (!principal) + return std::nullopt; + const Vec3i c = shortest_along(principal->dir); + const auto in_plane = perpendicular_two_folds(principal->dir); + if (in_plane.empty()) + return std::nullopt; + const Vec3i a = in_plane.front(); + // gamma must come out 120, so b is a turned by the THREE-fold, not by the six-fold. + const Mat3i three = principal->order == 6 ? MatMul(principal->rot, principal->rot) + : principal->rot; + const Vec3i b = MulVec(three, a); + M = FromRows(a, b, c); + break; + } + + case gemmi::CrystalSystem::Cubic: { + std::vector u; + for (const Axis &ax : axes) + if (ax.order == 4) + u.push_back(shortest_along(ax.dir)); + std::sort(u.begin(), u.end()); + u.erase(std::unique(u.begin(), u.end()), u.end()); + if (u.size() != 3) { + // 23 has no four-folds; its conventional axes are the three two-folds. + u.clear(); + for (const Axis &ax : axes) + if (ax.order == 2) + u.push_back(shortest_along(ax.dir)); + std::sort(u.begin(), u.end()); + u.erase(std::unique(u.begin(), u.end()), u.end()); + } + if (u.size() != 3) + return std::nullopt; + M = FromRows(u[0], u[1], u[2]); + break; + } + default: + return std::nullopt; + } + + if (Det3(M) == 0) + return std::nullopt; + if (Det3(M) < 0) { // keep the basis right-handed + for (int j = 0; j < 3; j++) + M[6 + j] = -M[6 + j]; + } + + int den = 0; + auto trans = CentringTranslations(M, den); + char centring = CentringSymbol(trans, den); + + // Bring the answer into the conventional setting of its Bravais class: monoclinic and + // orthorhombic centrings are named C, and a rhombohedral lattice is described obverse. Each is a + // relabelling of the same lattice by a unimodular matrix, so the cell it names is unchanged. + auto apply = [&](const Mat3i &u) { + M = MatMul(u, M); + trans = CentringTranslations(M, den); + centring = CentringSymbol(trans, den); + }; + if (system == gemmi::CrystalSystem::Orthorhombic) { + // All three axes are equivalent, so name the centred face ab by permuting them, then put the + // two axes the centring does not pin back in length order. + if (centring == 'A') + apply({0, 1, 0, 0, 0, 1, 1, 0, 0}); // a,b,c -> b,c,a + else if (centring == 'B') + apply({1, 0, 0, 0, 0, 1, 0, -1, 0}); // a,b,c -> a,c,-b + const Vec3i a{M[0], M[1], M[2]}, b{M[3], M[4], M[5]}; + if (metric.Dot(a, a) > metric.Dot(b, b)) + apply({0, 1, 0, 1, 0, 0, 0, 0, -1}); // a <-> b + } + if (centring == 'r') + apply({-1, 0, 0, 0, -1, 0, 0, 0, 1}); // reverse -> obverse + + if (centring == '?' || centring == 'r') + return std::nullopt; + + LePageResult r; + r.system = system; + r.centering = centring; + r.reindex = ToMat33(M); + r.primitive_reduced = L_red; + r.conventional = L_red.Multiply(r.reindex); + r.max_obliquity_deg = worst_obliquity; + r.n_operators = static_cast(group.size()); + return r; +} diff --git a/image_analysis/lattice_search/LePageLattice.h b/image_analysis/lattice_search/LePageLattice.h new file mode 100644 index 000000000..704363e23 --- /dev/null +++ b/image_analysis/lattice_search/LePageLattice.h @@ -0,0 +1,34 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include "../../common/CrystalLattice.h" +#include +#include + +struct LePageResult { + CrystalLattice primitive_reduced; + CrystalLattice conventional; + gemmi::CrystalSystem system = gemmi::CrystalSystem::Triclinic; + char centering = 'P'; + gemmi::Mat33 reindex = gemmi::Mat33(1, 0, 0, 0, 1, 0, 0, 0, 1); + double max_obliquity_deg = 0; // largest Le Page delta among the operators kept + int n_operators = 1; // order of the metric rotation group +}; + +// How far a two-fold may sit from exact and still count. Le Page's obliquity is the angle between a +// direct-space axis and its reciprocal partner, so it measures the same deviation whatever basis the +// lattice arrives in - unlike a relative length tolerance plus an angle tolerance, which measure +// different things in different presentations of the same lattice. One degree is the middle of the +// band over which the answer is stable, and it is the same bound the run's metric-symmetry note is +// stated at, so the program has one notion of "the metric carries more symmetry than the group": +// everything that note reports has been offered to the intensities. +constexpr double LATTICE_MAX_OBLIQUITY_DEG = 1.0; + +// The Bravais lattice of L, built from the metric symmetry OPERATORS: Le Page's two-fold search +// (gemmi/twin.hpp) gives the rotation group of the reduced primitive cell, and the conventional +// axes are then the shortest lattice vectors along that group's symmetry directions. Returns +// nothing when the operators do not close into a lattice rotation group - never a guess. +std::optional LePageLattice(const CrystalLattice &L, + double max_obliquity_deg = LATTICE_MAX_OBLIQUITY_DEG); diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index fd638197e..298157b90 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -53,6 +53,7 @@ #include "../image_analysis/scale_merge/SearchSpaceGroup.h" #include "../image_analysis/geom_refinement/PostRefine.h" #include "../image_analysis/lattice_search/LatticeSearch.h" +#include "../image_analysis/lattice_search/LePageLattice.h" #include "../image_analysis/scale_merge/AnisotropyAnalysis.h" #include "../image_analysis/scale_merge/TwinningAnalysis.h" #include "../image_analysis/scale_merge/HKLKey.h" @@ -401,6 +402,21 @@ namespace { // construction, all five of its searches agree, and the small sigma they report says nothing is // wrong. Only the background estimator recovers that case, and only a human reading this line knows // to look. +// Proper rotations of a crystal system's holohedry - what a lattice of that class carries, and the +// bound SearchSpaceGroup caps its point-group candidates at. +static size_t HolohedryRotations(gemmi::CrystalSystem system) { + switch (system) { + case gemmi::CrystalSystem::Triclinic: return 1; + case gemmi::CrystalSystem::Monoclinic: return 2; + case gemmi::CrystalSystem::Orthorhombic: return 4; + case gemmi::CrystalSystem::Trigonal: return 6; + case gemmi::CrystalSystem::Tetragonal: return 8; + case gemmi::CrystalSystem::Hexagonal: return 12; + case gemmi::CrystalSystem::Cubic: return 24; + } + return 1; +} + // The lattice METRIC can carry more rotational symmetry than the group the intensities supported. // That is not by itself a mistake - a pseudo-symmetric metric is ordinary, and only the intensities // can say whether the extra rotations are real - but it is where the corpus's symmetry under-calls @@ -417,7 +433,7 @@ static void WarnIfMetricHostsMoreSymmetry(const std::optional &cell, Logger logger("Rugnux"); const gemmi::UnitCell gc(cell->a, cell->b, cell->c, cell->alpha, cell->beta, cell->gamma); const size_t metric_rotations = - gemmi::find_lattice_symmetry(gc, sg.centring_type(), 1.0).sym_ops.size(); + gemmi::find_lattice_symmetry(gc, sg.centring_type(), LATTICE_MAX_OBLIQUITY_DEG).sym_ops.size(); const size_t group_rotations = sg.operations().sym_ops.size(); if (metric_rotations <= group_rotations) return; @@ -4087,6 +4103,12 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b sg_opts.merge_isa = result.error_model_isa; auto sg_search = SearchSpaceGroup(sm.merged, sg_opts); + // The Lorentz-filtered arm's merge, kept past the point where the all-observation one + // replaces it. The metric re-ask below is a search like any other and answers to the same + // two-arm rule: a promotion is refused by the most damning statistic it can be shown, and + // only one of the two merges has to be starved for that to happen. + std::vector merged_filtered; + // Second opinion from a merge that keeps only well-measured observations (see // RotationScaleMerge::search_min_zeta). Where the two disagree the all-observation merge // decides. The filter earns its place because a reflection that crosses the Ewald sphere @@ -4192,6 +4214,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // The all-observation merge is the one carried forward: it is what the // pre-promotion twinning numbers below are measured on, and the filtered merge is the // second opinion reported above. + merged_filtered = std::move(sm.merged); sm = std::move(sm_all); } @@ -4317,6 +4340,106 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b } } + // A CLASS DECIDED BY ROUNDING. LatticeSearch names the Bravais class by Niggli-reducing the + // indexed cell and looking the reduced cell up in the 44 lattice characters. For a lattice + // whose Buerger cells straddle the Niggli type-I/type-II boundary that lookup is a coin + // flip - an F-centred cubic lattice has reduced forms on both sides, the two sides carry + // different characters, and which side the reduction lands on is decided by the last digits + // of whatever refinement produced the cell rather than by the data. Measured on such a + // lattice: 43% cubic F, 36% tetragonal I, 21% orthorhombic I, and the split is flat over a + // factor of ten in the noise. The class then caps the point-group search + // (SearchSpaceGroupOptions::lattice_system), so from the I-centred sub-cell the cubic + // three-fold is never enumerated and the run reports that nothing was refused - which is + // accurate, because nothing was asked. + // + // Le Page's two-fold search has no such key: it measures each rotation's obliquity on the + // lattice itself, in a primitive basis, and the conventional cell is then built from the + // axes of the rotations it found. Where it carries more rotations than the adopted class's + // holohedry, offer its conventional cell as a SECOND lattice candidate - reindex the merge + // into it and run the same search there, with every gate live. This only ever ASKS. The + // reindex is committed if and only if the search in the new setting confirms a strictly + // higher point group, and the centring the new cell describes, on these intensities; a + // pseudo-symmetric metric fails that and the answer already in hand stands. + if (!commit_reindex && end_msg.rotation_lattice.has_value() + && end_msg.rotation_lattice_type.has_value()) { + const auto cand = LePageLattice( + end_msg.rotation_lattice->ToPrimitive(end_msg.rotation_lattice_type->centering)); + if (cand && static_cast(cand->n_operators) + > HolohedryRotations(end_msg.rotation_lattice_type->crystal_system)) { + // Exact integer reindex from the indexed setting to the candidate's conventional + // one, P[i][j] = conv_real[i] . rot_reciprocal[j], as in the centred-lattice test + // above. + const Coord cv[3] = {cand->conventional.Vec0(), cand->conventional.Vec1(), + cand->conventional.Vec2()}; + const Coord rs[3] = {end_msg.rotation_lattice->Astar(), end_msg.rotation_lattice->Bstar(), + end_msg.rotation_lattice->Cstar()}; + gemmi::Mat33 reindex; + double reindex_res = 0.0; + for (int i = 0; i < 3; ++i) + for (int j = 0; j < 3; ++j) { + const double v = cv[i] * rs[j]; + reindex.a[i][j] = std::round(v); + reindex_res = std::max(reindex_res, std::fabs(v - reindex.a[i][j])); + } + if (reindex_res < 0.1) { + SearchSpaceGroupOptions o2 = sg_opts; + o2.lattice_system = cand->system; + o2.cell = gemmi::UnitCell(cand->conventional.GetUnitCell()); + const auto reindexed = [&](const std::vector &in) { + std::vector out = in; + for (auto &m : out) + reindex_hkl(m, reindex); + return out; + }; + const auto merged_all = reindexed(sm.merged); + auto s2 = SearchSpaceGroup(merged_all, o2); + // The same two-arm rule the search above follows: the Lorentz-filtered merge can + // confirm an operator the all-observation one refuses, because the filter removes + // the near-tangential measurements that make a genuine operator look like a twin. + // Where it finds the larger point group, the absences are still judged on all + // observations, with the point group pinned. + if (!merged_filtered.empty()) { + const auto filtered = SearchSpaceGroup(reindexed(merged_filtered), o2); + if (filtered.point_group_order > s2.point_group_order) { + o2.fixed_point_group = filtered.point_group_representative; + s2 = SearchSpaceGroup(merged_all, o2); + o2.fixed_point_group.reset(); + } + } + const bool higher = s2.best_space_group.has_value() + && s2.point_group_order > sg_search.point_group_order + && s2.best_space_group->centring_type() == cand->centering; + const auto &uc = cand->conventional.GetUnitCell(); + logger.Info("The cell metric carries {} rotations where the {} lattice the " + "indexer named has {}, so the search was asked again on the metric's " + "own cell (a={:.3f} b={:.3f} c={:.3f} alpha={:.2f} beta={:.2f} " + "gamma={:.2f}, centring {}, worst obliquity {:.3f} deg): {} (order " + "{} against {}){}", + cand->n_operators, + gemmi::crystal_system_str(end_msg.rotation_lattice_type->crystal_system), + HolohedryRotations(end_msg.rotation_lattice_type->crystal_system), + uc.a, uc.b, uc.c, uc.alpha, uc.beta, uc.gamma, cand->centering, + cand->max_obliquity_deg, + s2.best_space_group.has_value() ? s2.best_space_group->xhm() : "nothing", + s2.point_group_order, sg_search.point_group_order, + higher ? " - adopted" + : (s2.refused_reason.empty() + ? " - the intensities do not support it, keeping the group " + "already found" + : " - keeping the group already found; " + s2.refused_reason)); + if (higher) { + sg_search = s2; + commit_reindex = reindex; + commit_cell = cand->conventional.GetUnitCell(); + commit_lattice = end_msg.rotation_lattice->Multiply(reindex); + commit_lattice_type = LatticeMessage{ .centering = cand->centering, + .niggli_class = end_msg.rotation_lattice_type->niggli_class, + .crystal_system = cand->system }; + } + } + } + } + // A DOUBLED DESCRIPTION of a primitive lattice. gemmi states every setting as a change of // basis from its reference one, and where that change has determinant < 1 the setting names // a cell of 1/det times the reference volume. In the monoclinic family exactly two settings diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index a4af137b6..27998fa53 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -79,6 +79,7 @@ ADD_EXECUTABLE(jfjoch_test CalcBraggPredictionTest.cpp SpotUtilsTest.cpp LatticeSearchTest.cpp + LePageLatticeTest.cpp TimeTest.cpp RotationIndexerTest.cpp TopPixelsTest.cpp diff --git a/tests/LePageLatticeTest.cpp b/tests/LePageLatticeTest.cpp new file mode 100644 index 000000000..feed46103 --- /dev/null +++ b/tests/LePageLatticeTest.cpp @@ -0,0 +1,140 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include "../common/CrystalLattice.h" +#include "../common/Coord.h" +#include "../common/UnitCell.h" +#include "../image_analysis/lattice_search/LePageLattice.h" +#include "gemmi/symmetry.hpp" +#include +#include + +namespace { +int CentringMultiplicity(char c) { + switch (c) { + case 'A': case 'B': case 'C': case 'I': return 2; + case 'R': return 3; + case 'F': return 4; + default: return 1; + } +} + +// The lattice is what has to come back, not the axes: a conventional cell of the right class whose +// PRIMITIVE volume is the one we started from. Comparing lengths element-wise would fail a correct +// answer given in another setting. +void CheckLattice(const std::optional &r, gemmi::CrystalSystem system, char centering, + float primitive_volume) { + REQUIRE(r.has_value()); + CHECK(r->system == system); + CHECK(r->centering == centering); + const float v = std::fabs(r->conventional.CalcVolume()) / CentringMultiplicity(r->centering); + CHECK(v == Catch::Approx(primitive_volume).epsilon(0.02)); +} + +// The lattice as an indexer hands it over: some basis of it, in some orientation, with noise. +CrystalLattice Present(const CrystalLattice &L, std::mt19937 &rng, float noise_A) { + std::uniform_int_distribution pick(0, 2), amount(-1, 1); + std::uniform_real_distribution uni(0, 1); + std::normal_distribution gauss(0, noise_A); + gemmi::Mat33 m(1, 0, 0, 0, 1, 0, 0, 0, 1); + for (int n = 0; n < 4; n++) { + const int i = pick(rng), j = pick(rng); + if (i == j) + continue; + gemmi::Mat33 shear(1, 0, 0, 0, 1, 0, 0, 0, 1); + shear.a[i][j] = amount(rng); + m = shear.multiply(m); + } + const float theta = 2 * (float)M_PI * uni(rng), phi = std::acos(2 * uni(rng) - 1); + const Coord axis(std::sin(phi) * std::cos(theta), std::sin(phi) * std::sin(theta), std::cos(phi)); + CrystalLattice out = L.Multiply(m).Multiply(RotMatrix(2 * (float)M_PI * uni(rng), axis)); + Coord v[3] = {out.Vec0(), out.Vec1(), out.Vec2()}; + for (auto &k : v) { k.x += gauss(rng); k.y += gauss(rng); k.z += gauss(rng); } + return CrystalLattice(v[0], v[1], v[2]); +} +} // namespace + +TEST_CASE("LePageLattice - the fourteen Bravais lattices") { + struct Case { gemmi::CrystalSystem system; char centering; float a, b, c, al, be, ga; }; + const Case cases[] = { + {gemmi::CrystalSystem::Triclinic, 'P', 23, 31, 41, 81, 95, 71}, + {gemmi::CrystalSystem::Monoclinic, 'P', 31, 43, 57, 90, 103, 90}, + {gemmi::CrystalSystem::Monoclinic, 'C', 91, 43, 57, 90, 103, 90}, + {gemmi::CrystalSystem::Orthorhombic, 'P', 31, 43, 57, 90, 90, 90}, + {gemmi::CrystalSystem::Orthorhombic, 'C', 31, 43, 57, 90, 90, 90}, + {gemmi::CrystalSystem::Orthorhombic, 'I', 31, 43, 57, 90, 90, 90}, + {gemmi::CrystalSystem::Orthorhombic, 'F', 31, 43, 57, 90, 90, 90}, + {gemmi::CrystalSystem::Tetragonal, 'P', 47, 47, 71, 90, 90, 90}, + {gemmi::CrystalSystem::Tetragonal, 'I', 47, 47, 71, 90, 90, 90}, + {gemmi::CrystalSystem::Trigonal, 'R', 61, 61, 133, 90, 90, 120}, + {gemmi::CrystalSystem::Hexagonal, 'P', 61, 61, 97, 90, 90, 120}, + {gemmi::CrystalSystem::Cubic, 'P', 71, 71, 71, 90, 90, 90}, + {gemmi::CrystalSystem::Cubic, 'I', 71, 71, 71, 90, 90, 90}, + {gemmi::CrystalSystem::Cubic, 'F', 71, 71, 71, 90, 90, 90}, + }; + for (const Case &c : cases) { + const CrystalLattice conventional(c.a, c.b, c.c, c.al, c.be, c.ga); + const CrystalLattice primitive = conventional.ToPrimitive(c.centering); + const float primitive_volume = std::fabs(primitive.CalcVolume()); + std::mt19937 rng(20260831); + for (int i = 0; i < 20; i++) { + INFO("class " << (int)c.system << c.centering << " presentation " << i); + CheckLattice(LePageLattice(Present(primitive, rng, 0.02f)), c.system, c.centering, + primitive_volume); + } + } +} + +TEST_CASE("LePageLattice - a cubic F lattice on the Niggli type boundary") { + // An fcc lattice has both a 60/60/60 and a ~120/90/120 shortest-vector basis, so it sits ON the + // boundary between the two Niggli types by construction and the reduction lands on either side + // according to the last bits of the cell it is given. Reading the symmetry off the metric has no + // forms to fall between, so the answer does not depend on which side it landed on. + const CrystalLattice conventional(121.0f * std::sqrt(2.0f), 121.0f * std::sqrt(2.0f), + 121.0f * std::sqrt(2.0f), 90, 90, 90); + const CrystalLattice primitive = conventional.ToPrimitive('F'); + const float primitive_volume = std::fabs(primitive.CalcVolume()); + std::mt19937 rng(7); + for (int i = 0; i < 40; i++) { + INFO("presentation " << i); + CheckLattice(LePageLattice(Present(primitive, rng, 0.1f)), gemmi::CrystalSystem::Cubic, 'F', + primitive_volume); + } +} + +TEST_CASE("LePageLattice - a tetragonal I description of a cubic F lattice") { + // Same lattice as above, handed over in the setting a, a, a*sqrt(2) that describes it as body- + // centred tetragonal. Both descriptions are the same lattice, and the answer has to be the same. + const float a = 120.5f; + const CrystalLattice tetragonal(a, a, a * std::sqrt(2.0f), 90, 90, 90); + const CrystalLattice primitive = tetragonal.ToPrimitive('I'); + auto r = LePageLattice(primitive); + REQUIRE(r.has_value()); + CHECK(r->system == gemmi::CrystalSystem::Cubic); + CHECK(r->centering == 'F'); + CHECK(std::fabs(r->conventional.CalcVolume()) / 4 == + Catch::Approx(std::fabs(primitive.CalcVolume())).epsilon(0.01)); +} + +TEST_CASE("LePageLattice - the change of basis is integral and right-handed") { + const CrystalLattice conventional(47, 47, 71, 90, 90, 90); + const CrystalLattice primitive = conventional.ToPrimitive('I'); + auto r = LePageLattice(primitive); + REQUIRE(r.has_value()); + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) + CHECK(r->reindex[i][j] == Catch::Approx(std::round(r->reindex[i][j])).margin(1e-9)); + CHECK(r->reindex.determinant() > 0); + CHECK(r->conventional.CalcVolume() > 0); +} + +TEST_CASE("LePageLattice - a pseudo-symmetric metric is not promoted") { + // A monoclinic cell whose beta sits a few degrees from 90 is not orthorhombic, however close the + // reduced form is to an orthorhombic character. + const CrystalLattice L(31, 43, 57, 90, 93, 90); + auto r = LePageLattice(L); + REQUIRE(r.has_value()); + CHECK(r->system == gemmi::CrystalSystem::Monoclinic); + CHECK(r->centering == 'P'); +} -- 2.54.0 From e5a471a5dee49e72fde85f1fbaafb268d1c473c5 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 3 Sep 2026 18:54:53 +0200 Subject: [PATCH 05/75] rotation indexing: drive the winning candidate to the fit's fixed point The candidate loop calls the geometry optimiser once per candidate. One round is enough to RANK candidates against each other; it does not MEASURE the geometry, and the first pass's fit is what the second pass integrates at. So the tilt the run integrates at is whatever one round of a shared starting point produced. Re-solve the winning candidate alone against a re-accumulated reciprocal cloud until the tilt stops moving - 20 rounds, 0.6 mdeg, offline only. Measured on a long-wavelength rotation sweep: R_meas 0.1369 -> 0.1193, ISa 7.81 -> 9.19, and anomalous density at the known sulphur positions of a standard test protein 7.01 -> 7.82 sigma at Met and 7.70 -> 8.18 at Cys. Converged by round 10; the extra rounds cost 50% of wall time on a 3605-image sweep. The point is not the size of the gain but that the answer stops depending on where it started. The tilt component parallel to the spindle is a gauge - it re-expresses the beam centre it was handed - and one round launders the starting beam position into it at 11% of the geometric one-for-one rate. Iterating drops that to 0.6%, which is what makes it a measurement rather than an echo. The existing soft restraint stays and is complementary: it bounds each step while iteration walks the pair to their joint optimum. Paired over 113 datasets. R_meas better/worse 36/16, 12/10 and 3/9 on the three arms; one crystal's space group is rescued to its deposited value, and one moves to the ortho-hexagonal setting of the same lattice - a known under-call whose merged data improved. Both movers sit where the tilt is ill-conditioned. Also: the report claimed the refined tilt "is never written back onto the geometry". It is, first pass to second. Two statements corrected. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- .../rotation_indexer/RotationIndexer.cpp | 37 +++++++++++++++++++ rugnux/ResultReport.cpp | 32 +++++++++------- 2 files changed, 55 insertions(+), 14 deletions(-) diff --git a/image_analysis/rotation_indexer/RotationIndexer.cpp b/image_analysis/rotation_indexer/RotationIndexer.cpp index c2ac868b8..046d30714 100644 --- a/image_analysis/rotation_indexer/RotationIndexer.cpp +++ b/image_analysis/rotation_indexer/RotationIndexer.cpp @@ -393,6 +393,43 @@ void RotationIndexer::RunIndexing() { } } + // Drive the selected candidate to the fit's FIXED POINT. Refinement here is a chain - solve, + // re-accumulate the reciprocal-space cloud under the refined geometry, solve again - and the + // selection above calls the solver ONCE per candidate, so what it returns is a point on the way + // to that fixed point rather than the fixed point itself. One round is enough to rank the + // candidates; it is not enough to have measured the geometry. + // + // What that costs, on a sweep whose 2theta reaches far enough for the detector tilt to be + // determined at all (long wavelength, short distance): the spindle-parallel tilt comes out at + // 16 % of an independently measured value after one round and 87 % of it after twenty, and the + // three sweeps of one crystal at three wavelengths then agree with that external value to + // 0.02 deg. The data follow - R_meas 0.137 -> 0.119, ISa 7.8 -> 9.2, and the anomalous peak + // height at the sulphur positions of a known model 7.0 -> 7.8 sigma. + // + // It also removes the fit's dependence on where the beam centre started, which is the thing + // the gauge prior below exists to guard: displacing the starting centre over 8 px moves the + // refined tilt by 0.0004 deg/px here against 0.0077 deg/px for a single round, on a geometric + // one-for-one of 0.0710. So the prior stays, and iterating turns it from a prior that pins the + // answer near the header into a per-step limit that lets the pair walk to their joint optimum - + // pinning its anchor across the rounds instead leaves the tilt at 0.014 deg and the data worse + // than doing nothing. The winner only: the other candidates are discarded, and these solves are + // the dominant first-pass cost. Not in real time, where the budget is wall-clock. + constexpr int ROT_REFINE_OUTER_ROUNDS = 20; + constexpr double ROT_REFINE_TILT_SETTLED_RAD = 1.0e-5; // ~0.6 mdeg + if (have_best && !real_time) { + for (int r = 0; r < ROT_REFINE_OUTER_ROUNDS; ++r) { + XtalOptimizerData d = best_data; + if (!XtalOptimizer(d, v_, kCeresThreads)) + break; + const double d1 = std::abs(d.geom.GetPoniRot1_rad() - best_data.geom.GetPoniRot1_rad()); + const double d2 = std::abs(d.geom.GetPoniRot2_rad() - best_data.geom.GetPoniRot2_rad()); + best_data = std::move(d); + if (std::max(d1, d2) < ROT_REFINE_TILT_SETTLED_RAD) + break; + } + best_frac = IndexedFraction(best_data.latt, accumulate(best_data.geom, best_data.axis), index_tol); + } + if (have_best) { search_result_ = best_sr; indexed_lattice = best_data.latt; diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 78065bf30..3d1e32947 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -135,24 +135,28 @@ std::string RenderResultReport(const std::string &output_prefix, os << "\n" << " The distance and beam centre above are the ones this result was integrated at, which on\n" << " a rotation run is the post-refined geometry rather than the values in the input file.\n" - << " DETECTOR_TILT is rot1/rot2/rot3 in degrees, as the run INTEGRATED at them. Rotation\n" - << " indexing does refine rot1/rot2, but the result is never written back onto the geometry,\n" - << " so what it measured is reported separately as REFINED_DETECTOR_TILT.\n" + << " DETECTOR_TILT is rot1/rot2/rot3 in degrees, as the run INTEGRATED at them - on a rotation\n" + << " run that is a rot1/rot2 rotation indexing fitted, carried over from the first pass, not the\n" + << " value in the input file. REFINED_DETECTOR_TILT below is what the pass named at the end of\n" + << " section 1 fitted for itself, which nothing consumes.\n" << " BEAM_CENTRE is the PONI and DIRECT_BEAM is where the beam actually lands; they differ by\n" << " distance*tan(tilt)/pixel and are identical only when the tilt is zero. Quote whichever the\n" << " program you are feeding expects, and check which one it means.\n"; if (result.refined_detector_tilt_deg) - os << " REFINED_DETECTOR_TILT is rot1/rot2 in degrees as rotation indexing MEASURED them, and is\n" - << " NOT what this run integrated at - nothing writes a refined tilt back onto the geometry.\n" - << " One crystal does not measure a detector tilt. The tilt is aliased with the beam centre,\n" - << " so what a single sweep determines is the DIRECT_BEAM above, not the split between the\n" - << " two, and this fit stays close to the tilt it was started from - it moves a fraction of\n" - << " the way to the true value, and the fraction varies from crystal to crystal. Comparing\n" - << " one run against a powder calibration will therefore disagree with it by far more than\n" - << " either number's uncertainty. What is worth comparing is the MEDIAN of this value over\n" - << " several crystals collected on the same detector, which does track the calibration well\n" - << " enough to show up a placeholder or a stale tilt in the file - it does not replace the\n" - << " calibration. Do not feed a single run's value back into the instrument.\n" + os << " REFINED_DETECTOR_TILT is rot1/rot2 in degrees as THIS pass's rotation indexing fitted\n" + << " them, and is not what this pass integrated at: on a two-pass run the tilt above is the\n" + << " FIRST pass's fit, and this is the second pass re-fitting it on its own better geometry.\n" + << " Where the two agree the fit has settled; where they do not, that component is still\n" + << " travelling and neither number is the end of it.\n" + << " How much one sweep determines depends on how far its 2theta reaches. The tilt moves the\n" + << " direct beam exactly as the beam centre does, so at small 2theta what a sweep determines\n" + << " is the DIRECT_BEAM above and not the split between the two; what survives that alias\n" + << " grows as the square of the scattering angle, and on a short-distance long-wavelength\n" + << " sweep it is large enough that the fit reproduces a separately measured tilt to a few\n" + << " hundredths of a degree. Where 2theta is ordinary, compare the MEDIAN of this value over\n" + << " several crystals collected on the same detector rather than one run's: that does track\n" + << " a calibration well enough to show up a placeholder or a stale tilt in the file, and it\n" + << " does not replace the calibration.\n" << " rot3 is omitted because a rotation about the beam is an exact null of this experiment\n" << " and the fit cannot move it.\n"; -- 2.54.0 From 28a98e1a14cb8d7d14bd5d45618c3af6b1378d4f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 3 Sep 2026 23:57:12 +0200 Subject: [PATCH 06/75] stills: a per-image number for how much of a sweep's blind cone this orientation loses A rotation sweep never records a double cone of half-angle asin(lambda/2d) about the spindle. That loss is normally repaired by the point group; it is not repaired when an operator's axis lies inside the cone, because the cone then maps onto itself. A still cannot know the point group, but it can see where the crystal's short lattice rows are, and a symmetry axis is always one of them - measured over 107 solved cells its length is 1.4x the shortest row at the median. The number is the fraction of the blind cone that a 2-fold about the nearest short row carries back into the blind cone: two equal caps of angular radius theta_max whose centres are 2*beta apart, which is the circle-lens area in beta/theta_max and within 0.035 of the spherical value even for a 55 deg cone. 0 means one sweep reaches everything symmetry could give; 1 means the row is on the spindle and the whole cone is lost coherently. No goniometer geometry enters, so the number describes the problem and leaves the remedy to the beamline. Free: it rides on the FilterFFTResults shortlist the indexer already builds, needing only the spindle, the wavelength and the frame's own resolution. FilterFFTResults gains an optional out-parameter for each row's peak prominence, which the length window is taken over. Measured on 384 stills of 22 solved crystals against the frames' own symmetry axes, with the same procedure re-run along five decoy directions per frame as the null: 0.93 of severe orientations reported severe, 0.010 of harmless ones reported severe, AUC 0.948, and the value tracks the true severity to 0.10 at the 90th percentile. Below 60 spots the misses triple, so that is where it stops answering. It is blind to a symmetry axis much longer than the crystal's shortest row: measured on synthetic stills the search grid finds a 150 A row every time, a 200 A row half the time and a 300 A row once in sixteen. Two cheap consistency tests refuse to answer for part of that regime and cost nothing on real frames, but they do not cover it - the reach is a documented property of the number, not something every frame can detect. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- common/JFJochMessages.h | 6 ++ image_analysis/IndexAndRefine.cpp | 4 + image_analysis/indexing/CMakeLists.txt | 4 +- image_analysis/indexing/FFTIndexer.cpp | 68 ++++++++++++- image_analysis/indexing/FFTIndexer.h | 10 +- image_analysis/indexing/Indexer.cpp | 9 ++ image_analysis/indexing/Indexer.h | 12 +++ .../indexing/SpindleBlindFraction.cpp | 87 +++++++++++++++++ .../indexing/SpindleBlindFraction.h | 44 +++++++++ tests/CMakeLists.txt | 1 + tests/SpindleBlindFractionTest.cpp | 95 +++++++++++++++++++ 11 files changed, 333 insertions(+), 7 deletions(-) create mode 100644 image_analysis/indexing/SpindleBlindFraction.cpp create mode 100644 image_analysis/indexing/SpindleBlindFraction.h create mode 100644 tests/SpindleBlindFractionTest.cpp diff --git a/common/JFJochMessages.h b/common/JFJochMessages.h index ef2cd0601..a811bca17 100644 --- a/common/JFJochMessages.h +++ b/common/JFJochMessages.h @@ -128,6 +128,12 @@ struct DataMessage { std::optional bkg_estimate; std::optional ice_ring_score; // strongest ice ring over the smooth radial background (1 = none) + // How much of a single sweep's blind cone this orientation makes unrecoverable: 0 = one sweep + // about the spindle reaches everything the point group can give, 1 = a short lattice row lies on + // the spindle and the whole cone (1 - cos theta_max of every shell) is lost coherently. Absent + // when the frame gives nothing to decide on - too few spots, no lattice rows, no spindle - which + // is a third state, not a value of one half. + std::optional spindle_blind_fraction; std::optional indexing_result; std::optional indexing_lattice; diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index 6b624cd0c..4d4d072f3 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -202,6 +202,10 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data if (any_executed) msg.indexing_result = false; + // The severity is a property of the frame's lattice rows, not of a cell having closed, so it is + // reported whether or not the frame indexed. + msg.spindle_blind_fraction = indexer_result.spindle_blind_fraction; + if (!indexer_result.lattice.empty()) { auto latt = indexer_result.lattice[0]; if (latt.CalcVolume() > 1.0) { diff --git a/image_analysis/indexing/CMakeLists.txt b/image_analysis/indexing/CMakeLists.txt index e79dc4561..dc8f716ab 100644 --- a/image_analysis/indexing/CMakeLists.txt +++ b/image_analysis/indexing/CMakeLists.txt @@ -15,7 +15,9 @@ ADD_LIBRARY(JFJochIndexing STATIC FFTIndexer.h PostIndexingRefinement.cpp MultiLatticeSearch.cpp - MultiLatticeSearch.h) + MultiLatticeSearch.h + SpindleBlindFraction.cpp + SpindleBlindFraction.h) TARGET_LINK_LIBRARIES(JFJochIndexing JFJochCommon JFJochLatticeSearch) IF (JFJOCH_CUDA_AVAILABLE) diff --git a/image_analysis/indexing/FFTIndexer.cpp b/image_analysis/indexing/FFTIndexer.cpp index 3405b8784..4cbbfd26b 100644 --- a/image_analysis/indexing/FFTIndexer.cpp +++ b/image_analysis/indexing/FFTIndexer.cpp @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute // SPDX-License-Identifier: GPL-3.0-only +#include +#include #include "../../common/JFJochMath.h" #include "FFTIndexer.h" #include @@ -179,7 +181,8 @@ std::vector FFTIndexer::ReduceResults(const std::vector & return candidates; } -std::vector FFTIndexer::FilterFFTResults(size_t max_vectors) const { +std::vector FFTIndexer::FilterFFTResults(size_t max_vectors, + std::vector *magnitudes) const { std::multimap fft_result_map; for (int i = 0; i < direction_vectors.size(); i++) @@ -248,12 +251,32 @@ std::vector FFTIndexer::FilterFFTResults(size_t max_vectors) const { } Coord best_dir = direction_vectors.at(fft_result_filtered[best_idx].direction); ret.push_back(best_dir * fft_result_filtered[best_idx].length); + if (magnitudes) + magnitudes->push_back(fft_result_filtered[best_idx].magnitude); } // Sort filtered vectors by magnitude - std::sort(ret.begin(), ret.end(), [](const Coord &A, const Coord &B) { - return A.Length() < B.Length(); - }); + if (magnitudes) { + std::vector order(ret.size()); + std::iota(order.begin(), order.end(), 0); + std::sort(order.begin(), order.end(), [&](size_t a, size_t b) { + return ret[a].Length() < ret[b].Length(); + }); + std::vector sorted_ret; + std::vector sorted_mag; + sorted_ret.reserve(order.size()); + sorted_mag.reserve(order.size()); + for (const auto i : order) { + sorted_ret.push_back(ret[i]); + sorted_mag.push_back((*magnitudes)[i]); + } + ret = std::move(sorted_ret); + *magnitudes = std::move(sorted_mag); + } else { + std::sort(ret.begin(), ret.end(), [](const Coord &A, const Coord &B) { + return A.Length() < B.Length(); + }); + } // max_vectors counts RAW search directions, but one lattice row is sampled by many neighbouring // directions of the 16k half-sphere, so nearly all the strongest entries belong to the same two or @@ -281,6 +304,8 @@ std::vector FFTIndexer::FilterFFTResults(size_t max_vectors) const { if (distinct) { ret.push_back(dir * it->second.length); + if (magnitudes) + magnitudes->push_back(it->second.magnitude); extra++; } } @@ -454,7 +479,40 @@ std::vector FFTIndexer::RunInternal(const std::vector &co ExecuteFFT(coord, nspots); // Standard reduction: 30 strongest peaks, shortest-vector triples. Unchanged for the common case. - auto lattices = ReduceAndRefine(coord, nspots, FilterFFTResults(30), false); + std::vector magnitudes; + const auto shortlist = FilterFFTResults(30, &magnitudes); + + // Free ride on that shortlist: how much of a single sweep's blind cone this orientation makes + // unrecoverable. Needs the spindle, the wavelength and the frame's own resolution, all of which + // are already here; costs one pass over 30-odd rows. + // Below this the row list is no longer trustworthy: measured over 22 solved crystals, the + // fraction of severe orientations reported as harmless falls from 0.22 at 30 spots to 0.05 at 60, + // and flat above. Fewer spots means no value, not a middling one. + constexpr size_t SPINDLE_MIN_SPOTS = 60; + spindle_severity = {}; + if (spindle_axis && wavelength_A > 0 && nspots >= SPINDLE_MIN_SPOTS) { + float d_min = 0; + { // the spots' own resolution: the 95th percentile of |q|, so a few stray outliers do not + // set the cone width for the whole frame + std::vector q; + q.reserve(nspots); + for (size_t i = 0; i < nspots; i++) + q.push_back(coord[i].Length()); + if (!q.empty()) { + const size_t k = std::min(q.size() - 1, static_cast(0.95 * q.size())); + std::nth_element(q.begin(), q.begin() + k, q.end()); + if (q[k] > 0) + d_min = 1.0f / q[k]; + } + } + if (d_min > 0) { + const float sin_theta = std::min(1.0f, wavelength_A / (2.0f * d_min)); + const float theta_max_deg = static_cast(std::asin(sin_theta) * 180.0 / M_PI); + spindle_severity = SpindleBlindFraction(shortlist, magnitudes, *spindle_axis, theta_max_deg); + } + } + + auto lattices = ReduceAndRefine(coord, nspots, shortlist, false); // If the best cell indexes few of the (un-refined) accumulated spots, the true cell may be large/ // elongated with a long axis beyond the standard triple window (a superstructure, or a satellite- diff --git a/image_analysis/indexing/FFTIndexer.h b/image_analysis/indexing/FFTIndexer.h index dd0ab531b..a483eb9ba 100644 --- a/image_analysis/indexing/FFTIndexer.h +++ b/image_analysis/indexing/FFTIndexer.h @@ -9,6 +9,7 @@ #include "../../common/CrystalLattice.h" #include "../../common/IndexingSettings.h" #include "FFTResult.h" +#include "SpindleBlindFraction.h" #define FFT_MAX_SPOTS (64*1024) @@ -37,7 +38,11 @@ protected: // widen=true: anchor the two short axes but let the third range over all vectors (reaches the long // axis of a large/elongated cell). Used only as a fallback when the standard reduction indexes poorly. std::vector ReduceResults(const std::vector &results, bool widen) const; - std::vector FilterFFTResults(size_t max_vectors) const; + // `magnitudes`, when given, receives the FFT peak prominence of each returned row, in the + // same order. The spindle severity needs it to tell a crystal's real rows from the short + // spurious ones a long cell produces; nothing else does, so it is optional. + std::vector FilterFFTResults(size_t max_vectors, + std::vector *magnitudes = nullptr) const; std::vector ReduceAndRefine(const std::vector &coord, size_t nspots, const std::vector &filtered, bool widen); float IndexedFraction(const CrystalLattice &latt, const std::vector &coord, size_t nspots) const; @@ -49,6 +54,9 @@ protected: // Direction perpendicular to every vector of `filtered`, when they all lie in one plane and so // cannot close a cell; empty otherwise. std::optional DegeneratePlaneNormal(const std::vector &filtered) const; + // Filled by RunInternal from the shortlist it already computes; read out by Indexer::Run. + std::optional spindle_severity; + std::optional GetSpindleSeverity() const override { return spindle_severity; } virtual void ExecuteFFT(const std::vector &coord, size_t nspots) = 0; // Called after direction_vectors is rewritten, for implementations that keep a copy of it. diff --git a/image_analysis/indexing/Indexer.cpp b/image_analysis/indexing/Indexer.cpp index dfc1b24f4..91d052676 100644 --- a/image_analysis/indexing/Indexer.cpp +++ b/image_analysis/indexing/Indexer.cpp @@ -9,6 +9,10 @@ void Indexer::Setup(const DiffractionExperiment& experiment) { dist_tolerance_vs_reference = experiment.GetIndexingSettings().GetUnitCellDistTolerance(); viable_cell_min_spots = experiment.GetIndexingSettings().GetViableCellMinSpots(); index_ice_rings = experiment.GetIndexingSettings().GetIndexIceRings(); + wavelength_A = experiment.GetWavelength_A(); + spindle_axis = {}; + if (const auto goniometer = experiment.GetGoniometer()) + spindle_axis = goniometer->GetAxis(); SetupUnitCell(experiment.GetUnitCell()); } @@ -21,5 +25,10 @@ IndexerResult Indexer::Run(const std::vector &coord) { std::chrono::duration duration = end - start; ret.indexing_time_s = duration.count(); ret.executed = true; + if (const auto severity = GetSpindleSeverity()) { + ret.spindle_blind_fraction = severity->score; + ret.spindle_row_length_A = severity->row_length_A; + ret.spindle_miss_angle_deg = severity->miss_angle_deg; + } return ret; } diff --git a/image_analysis/indexing/Indexer.h b/image_analysis/indexing/Indexer.h index 26b7766e3..bdbe915ed 100644 --- a/image_analysis/indexing/Indexer.h +++ b/image_analysis/indexing/Indexer.h @@ -10,6 +10,7 @@ #include "../../common/DiffractionExperiment.h" #include "../../common/JFJochMessages.h" #include "../../common/SpotToSave.h" +#include "SpindleBlindFraction.h" struct IndexerResult { std::vector lattice; @@ -20,6 +21,11 @@ struct IndexerResult { // and no lattice fits it - whereas an error says nothing about the frame at all, the indexer // never having got to look, and will recur on the next one. std::optional error; + // How much of a single sweep's blind cone this orientation makes unrecoverable, in [0,1]. + // Absent when the frame gives nothing to decide on (no spindle defined, no lattice rows). + std::optional spindle_blind_fraction; + std::optional spindle_row_length_A; + std::optional spindle_miss_angle_deg; }; class Indexer { @@ -31,9 +37,15 @@ protected: DiffractionGeometry geom; std::optional reference_unit_cell; + // The rotation axis a subsequent sweep would use. Present on a grid scan too, where the axis is + // defined but stationary - that is exactly the case the spindle severity is for. + std::optional spindle_axis; + float wavelength_A = 0; virtual void SetupUnitCell(const std::optional& cell) = 0; virtual std::vector RunInternal(const std::vector &coord, size_t nspots) = 0; + // Set by RunInternal when the implementation computes it; read out by Run. + virtual std::optional GetSpindleSeverity() const { return {}; } public: virtual ~Indexer() = default; void Setup(const DiffractionExperiment& experiment); diff --git a/image_analysis/indexing/SpindleBlindFraction.cpp b/image_analysis/indexing/SpindleBlindFraction.cpp new file mode 100644 index 000000000..f969674d1 --- /dev/null +++ b/image_analysis/indexing/SpindleBlindFraction.cpp @@ -0,0 +1,87 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include + +#include "SpindleBlindFraction.h" + +namespace { + // Only rows short enough to be a plausible symmetry axis count. The cut is relative to the + // crystal's own shortest row, not an absolute length, so it works the same for a 40 A cell and + // a 200 A one: measured over 107 solved cells, 2.5 x the shortest row covers 88% of all + // symmetry axes and 98% of crystals' shortest one. + constexpr float MAX_ROW_LENGTH_RATIO = 2.5f; + // The reference length is taken over the strong rows only. A long-cell still makes the pass + // invent short spurious rows - measured at 0.4-0.6 of the true row's peak - and taking the + // reference over every row would let one of those shrink the window until the real aligned row + // fell outside it, turning a severe orientation into a reported zero. + constexpr float MIN_MAGNITUDE_RATIO = 0.5f; + // Where the search grid can no longer resolve the crystal's rows, the pass stops returning them + // and starts returning short spurious ones instead, and the shortlist becomes internally + // inconsistent: its strong rows are many times longer than its shortest entry. Measured over 384 + // frames of 22 solved crystals the ratio never exceeded 2.34 and was 1.00 at the median; on a + // synthetic still whose cell is past the grid's reach it runs 4-13. Past this the score would be + // reporting a cone it cannot see into, so it reports nothing instead. + constexpr float MAX_REFERENCE_LENGTH_RATIO = 3.0f; +} + +float BlindConeSelfOverlap(float x) { + if (x >= 1.0f) + return 0.0f; + if (x <= 0.0f) + return 1.0f; + return static_cast(2.0 / M_PI) * + (std::acos(x) - x * std::sqrt(1.0f - x * x)); +} + +std::optional SpindleBlindFraction(const std::vector &rows, + const std::vector &magnitudes, + const Coord &spindle, + float theta_max_deg) { + if (rows.empty() || rows.size() != magnitudes.size() || theta_max_deg <= 0) + return {}; + + const float axis_length = spindle.Length(); + if (axis_length < 1e-6f) + return {}; + const Coord axis = spindle.Normalize(); + + float max_magnitude = 0; + for (const auto &m : magnitudes) + max_magnitude = std::max(max_magnitude, m); + + float reference_length = 0; + for (size_t i = 0; i < rows.size(); i++) + if (magnitudes[i] >= MIN_MAGNITUDE_RATIO * max_magnitude) { + const float l = rows[i].Length(); + if (reference_length == 0 || l < reference_length) + reference_length = l; + } + if (reference_length == 0) + return {}; + + float shortest_length = reference_length; + for (const auto &r : rows) + shortest_length = std::min(shortest_length, r.Length()); + if (reference_length > MAX_REFERENCE_LENGTH_RATIO * shortest_length) + return {}; + + SpindleSeverity ret; + for (size_t i = 0; i < rows.size(); i++) { + const float length = rows[i].Length(); + if (length <= 0 || length > MAX_ROW_LENGTH_RATIO * reference_length) + continue; + if (magnitudes[i] < MIN_MAGNITUDE_RATIO * max_magnitude) + continue; + const float cos_beta = std::min(1.0f, std::fabs(rows[i] * axis) / length); + const float beta_deg = static_cast(std::acos(cos_beta) * 180.0 / M_PI); + const float score = BlindConeSelfOverlap(beta_deg / theta_max_deg); + if (score > ret.score || ret.row_length_A == 0) { + ret.score = score; + ret.row_length_A = length; + ret.miss_angle_deg = beta_deg; + } + } + return ret; +} diff --git a/image_analysis/indexing/SpindleBlindFraction.h b/image_analysis/indexing/SpindleBlindFraction.h new file mode 100644 index 000000000..2d98d2690 --- /dev/null +++ b/image_analysis/indexing/SpindleBlindFraction.h @@ -0,0 +1,44 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include + +#include "../../common/Coord.h" + +// How much of a single sweep's blind cone this orientation makes unrecoverable. +// +// A sweep about the spindle never brings a reciprocal point closer than theta_max = +// asin(lambda / 2d) to the axis onto the Ewald sphere, so a double cone of half-angle theta_max +// is missing from every resolution shell whatever the crystal does. That loss is normally +// repaired by the point group, which maps the cone onto measured territory. It is not repaired +// when the operator's axis lies inside the cone, because then the cone maps onto itself - and a +// still cannot know the point group, but it can measure where the crystal's short lattice rows +// are, and a symmetry axis is always one of them. +// +// Score = the fraction of the blind cone that a 2-fold about the nearest short row carries back +// into the blind cone: two equal caps of angular radius theta_max whose centres are 2*beta apart. +// 0 means any operator about that row moves the cone entirely off itself and one sweep loses +// nothing symmetry could have given; 1 means the row is on the spindle and the whole cone is +// lost. Nothing about the goniometer enters, so the number describes the problem and leaves the +// remedy - a chi offset, a second sweep, accepting the loss - to the beamline. + +// Fraction of a cap of angular radius theta that its own image under a 2-fold at beta covers. +// x = beta / theta_max. Exact in the flat limit and within 0.035 of the spherical value even at +// theta_max = 55 deg (the widest cone a 3.3 A beam produces), so the closed form is used as is. +float BlindConeSelfOverlap(float x); + +struct SpindleSeverity { + float score = 0.0f; // [0,1] + float row_length_A = 0; // the row the score was taken on + float miss_angle_deg = 0; +}; + +// rows/magnitudes: the FilterFFTResults shortlist and each entry's FFT peak prominence. +// Returns nothing when there is no shortlist to decide on. +std::optional SpindleBlindFraction(const std::vector &rows, + const std::vector &magnitudes, + const Coord &spindle, + float theta_max_deg); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 27998fa53..299a6a503 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -82,6 +82,7 @@ ADD_EXECUTABLE(jfjoch_test LePageLatticeTest.cpp TimeTest.cpp RotationIndexerTest.cpp + SpindleBlindFractionTest.cpp TopPixelsTest.cpp HKLKeyTest.cpp TCPImagePusherTest.cpp diff --git a/tests/SpindleBlindFractionTest.cpp b/tests/SpindleBlindFractionTest.cpp new file mode 100644 index 000000000..d53d71aed --- /dev/null +++ b/tests/SpindleBlindFractionTest.cpp @@ -0,0 +1,95 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include + +#include "../image_analysis/indexing/SpindleBlindFraction.h" + +using Catch::Matchers::WithinAbs; + +TEST_CASE("SpindleBlindFraction_Overlap", "[Indexing][Spindle]") { + // A row on the spindle leaves the whole cone unrecoverable; a row on the cone edge leaves none. + CHECK_THAT(BlindConeSelfOverlap(0.0f), WithinAbs(1.0f, 1e-6)); + CHECK_THAT(BlindConeSelfOverlap(1.0f), WithinAbs(0.0f, 1e-6)); + CHECK_THAT(BlindConeSelfOverlap(2.0f), WithinAbs(0.0f, 1e-6)); + // Half way into the cone the two caps still share 39% of their area. + CHECK_THAT(BlindConeSelfOverlap(0.5f), WithinAbs(0.3910f, 1e-3)); + // Monotone decreasing + for (int i = 0; i < 20; i++) + CHECK(BlindConeSelfOverlap(i / 20.0f) >= BlindConeSelfOverlap((i + 1) / 20.0f)); +} + +TEST_CASE("SpindleBlindFraction_Rows", "[Indexing][Spindle]") { + const Coord spindle(0, 0, 1); + const float theta_max = 20.0f; + + SECTION("a short row on the spindle is the worst case") { + const std::vector rows = {Coord(0, 0, 50), Coord(60, 0, 0), Coord(0, 70, 0)}; + const std::vector mag = {100, 90, 80}; + const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + CHECK_THAT(s->miss_angle_deg, WithinAbs(0.0f, 1e-3)); + CHECK_THAT(s->row_length_A, WithinAbs(50.0f, 1e-3)); + } + + SECTION("no short row inside the cone scores zero") { + const std::vector rows = {Coord(50, 0, 0), Coord(0, 60, 0), Coord(0, 60, 25)}; + const std::vector mag = {100, 90, 80}; + const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(0.0f, 1e-6)); + } + + SECTION("a row too long to be a symmetry axis is ignored") { + // 300 A along the spindle, in a crystal whose own rows are 50-70 A: 6x the shortest row is + // not a plausible symmetry axis, and the cone it sits in is not the crystal's problem. + const std::vector rows = {Coord(50, 0, 0), Coord(0, 60, 0), Coord(0, 0, 300)}; + const std::vector mag = {100, 90, 80}; + const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(0.0f, 1e-6)); + } + + SECTION("the same row in a crystal that IS that big is not ignored") { + const std::vector rows = {Coord(250, 0, 0), Coord(0, 280, 0), Coord(0, 0, 300)}; + const std::vector mag = {100, 90, 80}; + const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + } + + SECTION("a weak spurious short row does not shrink the length window") { + // What a long-cell still produces: the real rows near 300 A plus a weaker short peak. Taking + // the window off that peak would hide the aligned row and report the orientation harmless. + const std::vector rows = {Coord(100, 5, 0), Coord(250, 0, 0), Coord(0, 0, 300)}; + const std::vector mag = {40, 100, 95}; + const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + } + + SECTION("a shortlist the pass could not resolve gives no answer at all") { + // Strong rows ten times longer than the shortest entry: the grid has lost the crystal's real + // rows and is returning spurious short ones. Reporting zero here would be a silent "safe". + const std::vector rows = {Coord(12, 5, 0), Coord(250, 0, 0), Coord(0, 0, 300)}; + const std::vector mag = {40, 100, 95}; + CHECK_FALSE(SpindleBlindFraction(rows, mag, spindle, theta_max).has_value()); + } + + SECTION("no rows, no answer") { + CHECK_FALSE(SpindleBlindFraction({}, {}, spindle, theta_max).has_value()); + CHECK_FALSE(SpindleBlindFraction({Coord(0, 0, 50)}, {1.0f}, Coord(0, 0, 0), theta_max).has_value()); + } + + SECTION("a wider cone at long wavelength makes the same miss-angle worse") { + const std::vector rows = {Coord(0, 20, 50), Coord(60, 0, 0)}; + const std::vector mag = {100, 90}; + const auto narrow = SpindleBlindFraction(rows, mag, spindle, 10.0f); + const auto wide = SpindleBlindFraction(rows, mag, spindle, 35.0f); + REQUIRE(narrow.has_value()); + REQUIRE(wide.has_value()); + CHECK(wide->score > narrow->score); + } +} -- 2.54.0 From 025ecf066e64c999149763af25007b743cd70c7d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 3 Sep 2026 23:57:21 +0200 Subject: [PATCH 07/75] report: the spindle-to-symmetry-axis angle reaches p_report.txt, not only stdout Every rotation run already computes the angle between the spindle and the nearest symmetry axis, and writes it into the merge statistics text - which is stdout only. Unless the angle crossed the 15 deg warning threshold it therefore did not survive the run, so the ordinary "the mounting was fine" case could not be greppped, and an incomplete cusp could not be attributed to the mounting or cleared of it after the fact. Section 4 now carries the angle and the axis order as keys on every run that determined a space group. REPORT_VERSION moves to 7. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/RUGNUX_REPORT.md | 8 ++++++++ rugnux/ResultReport.cpp | 11 ++++++++++- rugnux/Rugnux.cpp | 2 ++ rugnux/Rugnux.h | 7 +++++++ 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 2dcb84bfe..2d16af421 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -241,6 +241,14 @@ files are **byte for byte** what a run with no model would have written — same `_unmerged.mtz`. A rejected model is therefore safe to try: it costs the null's compute and changes nothing else. +`SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` and `SPINDLE_SYMMETRY_AXIS_ORDER=` in section 4 say how the crystal +sat on the goniometer: the angle between the spindle and the nearest symmetry axis, and that axis's +order. Below about 15 deg the axis maps the sweep's blind cone onto itself and the reflections in +that cone stay missing however long the sweep runs, which is why the same number also raises a +warning there. It is written on every rotation run that determined a space group, including the +ordinary well-mounted case, so an incomplete cusp can be attributed to the mounting or cleared of it. +(New in `REPORT_VERSION= 7`.) + `SPACE_GROUP_ENANTIOMORPH=` in section 4 reads **`ASSUMED_FROM_MODEL`** when the hand written in the files is the model's. Assumed, not determined: merged intensities cannot see the hand at all — |F| is invariant under the change of hand — so an accepted model asserts it out of prior chemical knowledge. diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 3d1e32947..30e142da7 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -21,7 +21,7 @@ namespace { // The version of this file format. Bumped when a key is renamed or removed, a table column moves, // or a reason code changes meaning - a consumer can gate on it. - constexpr int REPORT_VERSION = 6; + constexpr int REPORT_VERSION = 7; const char *BANNER = " ******************************************************************************"; @@ -306,6 +306,15 @@ std::string RenderResultReport(const std::string &output_prefix, refused ? result.space_group_search->refused_point_group_hm : std::string("NONE")); if (refused) Key(os, "SPACE_GROUP_REFUSED_REASON", result.space_group_search->refused_reason); + // How the crystal sat on the spindle. Until now this reached only stdout unless it crossed the + // warning threshold, so the ordinary "the mounting was fine" case - the one a user needs when + // deciding whether an incomplete cusp is the mounting's fault or the data's - was not greppable. + if (result.spindle_symmetry_axis_deg.has_value()) { + Key(os, "SPINDLE_SYMMETRY_AXIS_ANGLE_DEG", + fmt::format("{:.1f}", *result.spindle_symmetry_axis_deg)); + Key(os, "SPINDLE_SYMMETRY_AXIS_ORDER", std::to_string(result.spindle_symmetry_axis_order)); + } + os << "\n SPACE_GROUP_ALTERNATIVES names every group these data cannot separate from the one\n" << " adopted (enantiomorphic partners, origin-ambiguous pairs, groups a gap in the data\n" << " leaves untested); NONE means the absences single the answer out. SPACE_GROUP_ENANTIOMORPH\n" diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 298157b90..d2f717c31 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -4713,6 +4713,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // case shows the loss while a 16.2 deg one is 99.7% complete. constexpr double WARN_DEG = 15.0; const auto [angle, order] = *closest; + result.spindle_symmetry_axis_deg = angle; + result.spindle_symmetry_axis_order = order; if (angle < WARN_DEG) { const std::string msg = fmt::format( "The crystal's {}-fold axis is only {:.1f} deg from the spindle. A rotation sweep " diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 721aec11a..bc56961a9 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -317,6 +317,13 @@ struct ProcessResult { // Conditions that need a person's attention, one plain sentence each (an ambiguous space group, a // symmetry axis on the spindle, an indexing ambiguity). The same messages the log warns about. std::vector warnings; + + // How the crystal sat on the spindle: the angle to the nearest symmetry axis and that axis's + // order. A property of the mounting rather than of the data, and the one number that says whether + // a single sweep could have been complete, so it is reported whether or not it triggered a + // warning. Absent on stills, or where no space group was determined. + std::optional spindle_symmetry_axis_deg; + int spindle_symmetry_axis_order = 0; }; // How far a cell stands from the metric its space group requires. A group's own rotations must leave -- 2.54.0 From 0dfb1232197639401c4d5acbcc953e62375ed338 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 06:09:26 +0200 Subject: [PATCH 08/75] merge: a median of two is not a median, and the time axis wants a width Three changes to the rotation merge, measured together over 106 paired datasets: 31 better, 10 worse, 65 unchanged in R_meas, and two space groups rescued to their references. OUTLIER REJECTION. The per-reflection median was taken from two observations up. At multiplicity two the median IS one of the two, so it is immune to its own test and only the other can be rejected - against its OWN sigma. On a decaying crystal that pairs one live observation with one dead one, and wherever a dead-frame noise excursion is the larger of the pair the median becomes the noise and the LIVE observation is deleted. Measured on a room-temperature sweep at multiplicity 1.96: 92% of rejected observations came from the live half against a 60% baseline, and correlation with an external reference merge fell from 0.966 to 0.845 overall and from 0.876 to 0.265 in the outer shell. The threshold is not the problem - 12 sigma rejects 1.35% of observations and gives the same answer as 6 - the rule is, and a median is only defined from three. Take it from three. THE REJECTION COUNT IS NOW REPORTED. Rejected observations are excluded from the merge and from R_meas and the CC(1/2) half-sets, which is right - those describe the data as merged - but it means an over-rejecting run looks BETTER by every number it prints. Without the count the failure above is silent, and it stayed silent for as long as the rule was wrong. Over the corpus the new key shows 325652 observations rejected, one dataset dropping 51971 of them. THE TIME-DEPENDENT ABSORPTION GRID. Its time axis was a fixed bin COUNT, so twelve bins is 7.5-30 deg on an ordinary sweep and over 100 deg on a multi-turn one, where a surface meant to follow a crystal drifting through the beam can no longer see the drift. Make it a fixed angular WIDTH, floored at the old count so it is bit-identical below 120 deg of sweep. On a four-turn sweep: R_meas 0.098 -> 0.086, ISa 10.9 -> 12.8. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/scale_merge/Merge.h | 6 ++++ .../scale_merge/RotationScaleMerge.cpp | 31 ++++++++++++++++--- rugnux/ResultReport.cpp | 6 ++++ 3 files changed, 39 insertions(+), 4 deletions(-) diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index 8b45d5640..c2da284bf 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -106,6 +106,12 @@ struct MergeStatistics { // when not determined. batch_deg is the rotation width per batch of the curve. A batch the data cannot // measure is NaN in the curve, and delta_b is NaN when the curve is not a trend a single number // summarises - damage is progressive, so a curve that is not is telling the user about something else. + // Observations the outlier rejection dropped before merging. Rejected observations are excluded + // from the merge AND from R_meas and the CC(1/2) half-sets, which is the XDS convention and is + // right - those statistics describe the data as merged. But it means an over-rejecting run looks + // BETTER by every number it reports, so the count has to be visible or the failure is silent. + // Zero when rejection is off. + size_t n_observations_rejected = 0; double radiation_damage_delta_b = NAN; std::vector radiation_damage_b_batch; double radiation_damage_batch_deg = 0.0; diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 293d1ead6..0d879ddd9 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -1957,7 +1957,18 @@ void RotationScaleMerge::RefineAbsorptionTime(int n_iter, int n_groups) { // 2400 cells, so the grid is second order; 12 x 10 is where the net across the set is largest and where // fewest crystals are hurt, 3 of the 22 it engages on. The time axis wants to be coarse either way - // the crystal drifts through the beam slowly. - constexpr int NT = 12; // time bins across the sweep + // The time axis wants a fixed angular WIDTH, not a fixed bin count. Twelve bins is 7.5-30 deg + // on an ordinary sweep, which is what it was tuned on; on a multi-turn sweep the same twelve are + // 100+ deg each, and a surface meant to follow a crystal drifting through the beam can no longer + // see the drift. Floored at twelve so this is bit-identical below 120 deg of sweep. Measured on a + // four-turn sweep at 10 deg/bin: R_meas 0.098 -> 0.084, ISa 10.9 -> 13.5, and the existing + // cross-validation still refuses the grid once the bins hold too little (2 deg/bin scores -3.6%). + constexpr double TIME_BIN_DEG = 10.0; + const auto gon_t = x.GetGoniometer(); + const double osc_t = gon_t ? std::fabs(gon_t->GetIncrement_deg()) : 0.0; + const int NT = (osc_t > 1e-6) + ? std::max(12, static_cast(std::lround(n_frames * osc_t / TIME_BIN_DEG))) + : 12; constexpr int ND = 10; // detector bins per axis auto usable = [&](const Obs &o) { return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && o.partiality >= min_partiality @@ -2946,7 +2957,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool } // Per-group outlier-rejection median of I*corr (host both paths - a per-group median is awkward on - // the GPU; cheap here, cnt >= 2 filter from the em pass). Fed to the merge accumulate. + // the GPU; cheap here, cnt >= 3 filter from the em pass). Fed to the merge accumulate. if (reject_outliers) { // One flat array with a per-group span, not a vector per group: n_groups is over a million // on a P1 pass, so a vector each is a million allocations, their growth, and a million frees @@ -2955,15 +2966,26 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // cnt[g] IS how many usable fulls group g has - the pass above counted exactly the ones // this walk would - so the spans come straight off it and the first of the two walks over // every full disappears. + // THREE, not two. At multiplicity two the median IS one of the two observations, so it + // is immune to its own test by construction and only the other one can be rejected - + // against its OWN sigma. On a decaying crystal that pairs one live observation with one + // dead one, and wherever a dead-frame noise excursion is the larger of the pair the + // median becomes the noise and the LIVE observation is the one deleted. Measured on a + // room-temperature sweep at multiplicity 1.96: 92% of rejected observations came from + // the live half against a 60% baseline, and correlation with an external reference merge + // fell from 0.966 to 0.845 overall and from 0.876 to 0.265 in the outer shell. The + // threshold is not the problem - 12 sigma rejects 1.35% of observations and gives the + // same answer as 6 sigma - the rule is, and it only has a defined median from three. + // Groups below the cut keep a NaN median and the test below skips them. std::vector start(n_groups + 1, 0); for (int g = 0; g < n_groups; ++g) - start[g + 1] = start[g] + (cnt[g] >= 2 ? cnt[g] : 0); + start[g + 1] = start[g] + (cnt[g] >= 3 ? cnt[g] : 0); std::vector iv(start[n_groups]); { std::vector fill(start.begin(), start.end() - 1); for (int i = 0; i < n_full; ++i) { const int g = mf.group[i]; - if (g >= 0 && cnt[g] >= 2) iv[fill[g]++] = mf.I[i] * mf.corr[i]; + if (g >= 0 && cnt[g] >= 3) iv[fill[g]++] = mf.I[i] * mf.corr[i]; } } ParallelChunks(n_groups, ThreadsForWork(iv.size(), nthreads), [&](int glo, int ghi) { @@ -3550,6 +3572,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // Radiation-damage monitor (measured before any correction by MeasureRadiationDamageB): carry the // first->last relative-B change and the per-batch curve into the reported statistics / mmCIF. + out.n_observations_rejected = reject_count; out.radiation_damage_delta_b = rad_damage_delta_b; out.radiation_damage_b_batch = rad_damage_b_batch; out.radiation_damage_batch_deg = rad_damage_batch_deg; diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 30e142da7..624783fac 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -367,6 +367,12 @@ std::string RenderResultReport(const std::string &output_prefix, Key(os, "FRIEDELS_LAW", experiment.GetScalingSettings().GetMergeFriedel() ? "TRUE" : "FALSE"); Key(os, "UNIQUE_REFLECTIONS", o.unique_reflections); Key(os, "TOTAL_OBSERVATIONS", o.total_observations); + // Outlier rejection drops observations from the merge AND from R_meas and the CC(1/2) + // half-sets, so a run that rejects too much scores BETTER on every other number in this + // block. Without this key that failure is silent, and it is not hypothetical: a merge whose + // rejection rule was wrong at low multiplicity dropped observations that halved the + // correlation with an external reference while its own R_meas improved. + Key(os, "OBSERVATIONS_REJECTED", result.merge_statistics.n_observations_rejected); // One rule for every quantity here: a run that did not measure it writes NO key, rather than // the word "nan" or a zero that reads as a measured absence. SIGANO is the common case - a // Friedel-merged run splits no Bijvoet pair - and it sat one line from CC_ANOM, which already -- 2.54.0 From bb2a42774a18b351f330da821e76de7cb9cadaa4 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 01:54:13 +0200 Subject: [PATCH 09/75] rugnux: the rotation post-refinement fits the crystal and the detector at once Post-refinement scaled the pass-1 lattice by one scalar against the observed rocking angles (step A), then read the detector distance off that scaled cell (step B). The split was there because the positional residual is degenerate with the cell scale - which is true of the positions ALONE, and is exactly what the excitation residual already computed in the same struct breaks. So the degeneracy that motivated the split was resolvable inside one problem all along, and splitting it manufactured error twice over: * Pass 1 frees the whole lattice against a frozen distance, so what it absorbs of a header distance error is ANISOTROPIC (0.16 % on the crystal measured), and no single scale can undo it. * Whatever bias is left in that scale goes straight into the distance, since the distance is only ever determined relative to the cell. Over six runs spanning three wavelengths of one crystal, distance error (%) = 1.33 x step-A bias (%/axis), r = 0.9947 - and the bias changed SIGN with the wavelength (-0.067 % at 1.9 A, +0.049 % at 2.7 A, +0.19 % at 3.3 A), reproducing the whole wavelength ordering of the distance error. The two steps' own printed numbers say it, on 112 of the corpus's datasets. Comparing each run's held-out POSITIONAL residual at the unrefined cell with the same residual at step A's scaled cell and at step B's committed geometry: step A leaves it WORSE than the unrefined cell on 110 of 112 (median 3.41x), and step B - which does fit that residual - cannot get back: its committed value is still worse than the unrefined cell on 110 of 112 (median 2.97x). Step B was improving on a baseline step A had corrupted. This part of the measurement is of the old code alone and owes nothing to the replacement. Now one Ceres problem: free crystal orientation, the cell (every parameter the crystal system leaves free), goniometer axis, detector distance and beam centre; residuals are the positional detector<->reciprocal one at each partial's observed spot and the distance-independent Ewald excitation one at each rocking centroid. Same deterministic hkl-hash split-half gate as before, with the move bounded: distance and every cell length within 1 %, beam within the existing bound. Detector tilt stays fixed (gauge-coupled to the orientation on one crystal). Committed, it leaves the held-out positional residual BELOW the unrefined cell's on the same 110 of 112 (median 0.90x) - though that residual is half its own objective, so read that line as a sanity check, not as an independent arbiter. The rotation-scale fit that followed step A now runs on the jointly committed lattice and axis; it is unchanged otherwise and reproduces its previous k to five digits. MEASURED. Three wavelengths of one rotation crystal, scored against an independent determination of the cell (unconstrained joint fit to XDS spot centroids) and against XDS's own refined cell: set cell error vs XDS committed distance R_meas ISa 1.9 A -0.048 % -> -0.041 % 60.274 -> 60.279 mm .0725/.0725 31.2/31.1 2.7 A +0.057 % -> -0.058 % 60.360 -> 60.266 mm .0681/.0676 25.8/26.3 3.3 A +0.230 % -> -0.033 % 60.491 -> 60.267 mm .1193/.0835 9.2/14.9 The three committed distances now agree with each other to 0.013 mm and with the external value (60.26 +- 0.04) instead of drifting 0.22 mm with the wavelength. Anomalous peak heights at the known sulphur sites rise from 7.82/8.18 to 10.09/9.04 - the best previously reachable state was 10.08/9.10 and required knowing the answer. The bistability documented on the long-wavelength set is gone. Sweeping the header distance over 1.0 mm, the two-step commits 60.49-60.54 however it starts (or refuses outright, losing the dataset: R_meas 0.276, ISa 2.9); the joint fit commits 60.256-60.313, tracking the truth, and lands in the correct cell basin every time. Same from two starting beam centres. CORPUS. 115 datasets, paired against the tip, 113 processed in both arms: * cell deviation from the reference: 58 better, 38 worse, 16 unchanged (sign test p = 0.052); median 0.171 % -> 0.128 %. The asymmetry is in the magnitudes: the largest improvement is 2.4 percentage points, the largest regression 0.125. The 24.4-point figure at the head of that list is not a cell change at all - it is a hexagonal lattice the base arm described in its C-centred orthorhombic setting, which a sorted-axis comparison reports as a large difference; the scorer's own cause for that dataset is an under-called point group, and the primitive volume ratio passed in both arms. * POINT GROUP == reference 105 -> 106. Exact space group 82 -> 81: one screw-axis call (P2 -> P2_1) on a crystal whose R_meas is above 139 %, where the cell itself improved. * V ratio in [0.97, 1.03]: 106 -> 106. Datasets processed: 113 -> 113. * R_meas and ISa are confounded here: of the nine material regressions, seven reached a FINER resolution limit, which raises R_meas and lowers ISa mechanically. The two that did not are small (+0.55 pp R_meas, -0.9 ISa). One unconfounded rescue is large - R_meas 105 % -> 27 % at 0.72 -> 0.51 A on a small-molecule set whose uniform scale step A had refused. Cost: 90-150 ms on runs of minutes. Bit-identical across repeat runs. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 1 + docs/CPU_DATA_ANALYSIS_INDEXING.md | 16 +- image_analysis/geom_refinement/PostRefine.cpp | 535 ++++++++++-------- image_analysis/geom_refinement/PostRefine.h | 24 +- image_analysis/geom_refinement/XtalResidual.h | 33 ++ rugnux/ResultReport.cpp | 6 +- 6 files changed, 354 insertions(+), 261 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 2b331a909..03ff33cde 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -8,6 +8,7 @@ * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, and open masters written at other facilities, including Eiger 1.x and third-party NXmx. * `rugnux` determines the lattice and the space group more reliably - the true cell where the first pass offers a whole-number multiple of it, so a pseudo-translated crystal keeps its full-length axis and a small molecule is indexed on its own cell rather than a protein-sized one, and the point group, the setting and the systematic absences - and `-S` refuses or re-seats a fixed space group whose symmetry axes the indexed cell does not carry. * `rugnux` asks the space-group search again on the cell the lattice metric supports, whenever that metric carries more rotational symmetry than the Bravais class the indexer named, so a lattice whose reduction landed in a lower-symmetry sub-cell can still reach its true point group; the higher symmetry is adopted only where the intensities confirm it. +* `rugnux` post-refines the rotation geometry with one joint fit of the crystal and the detector against both the observed spot positions and the observed rocking angles, in place of scaling the cell against the angles and then reading the detector distance off the scaled cell; the refined distance no longer depends on how wrong the file's distance was. * `rugnux` measures the beam centre on every run and indexes with it when the file's value indexes nothing, refines only the detector-tilt component the data determine - a beam-centre error is no longer reported as a tilt - and places a detector swung out on a 2theta arm where the file says it stands. * `rugnux` writes the unmerged MTZ by default, with a P1 merge beside it, a batch header for every image the observations span, and events kept to the same `--min-captured-fraction` as the merge, so a wrong space group can be re-merged in a scaling program without reprocessing. * `rugnux` writes reflection files in the conventions downstream programs read: `FreeR_flag` is 0 for the test set and 1 for the working set - it was the other way round - the merged and P1 MTZ carry the reserved `HKL_base` dataset so a CCP4 program reads the wavelength instead of falling back to 1.54187 A, and the merged mmCIF marks the free set as `_refln.status` = `f`. diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index b6da4f498..9037bc34e 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -165,7 +165,7 @@ The refinement jointly optimizes, depending on mode and constraints: - crystal orientation (a global rotation), - unit-cell parameters, with constraints determined by inferred crystal system. -The detector distance is not refined against one crystal's spots at all: the positional residual leaves it degenerate with the cell scale, so it is fitted separately - by the rotation post-refinement, which holds the cell at the value its own cell/axis step settled on, and by the stills `--refine-geometry` bundle. Per image, the beam centre and the crystal orientation are refined, and the unit cell as well for stills. The first-pass rotation indexing refines the detector tilt and the rotation-axis direction too, against the spots accumulated across the sweep; everywhere else both are held fixed, because on a single crystal a tilt is absorbed almost exactly by the crystal orientation. A lighter **orientation-only** mode refines just the crystal orientation, for stills whose geometry is already trusted. It carries a weak small-rotation prior penalising the whole angle-axis vector (all three components, at a low weight); what it is there for is the poorly-determined out-of-plane component, which is the one the data barely constrain. +The detector distance is not refined against one crystal's spots at all: the positional residual leaves it degenerate with the cell scale, so it is fitted elsewhere - by the rotation post-refinement, which frees it alongside the whole crystal and adds the distance-independent rocking-angle residual that breaks the degeneracy, and by the stills `--refine-geometry` bundle. Per image, the beam centre and the crystal orientation are refined, and the unit cell as well for stills. The first-pass rotation indexing refines the detector tilt and the rotation-axis direction too, against the spots accumulated across the sweep; everywhere else both are held fixed, because on a single crystal a tilt is absorbed almost exactly by the crystal orientation. A lighter **orientation-only** mode refines just the crystal orientation, for stills whose geometry is already trusted. It carries a weak small-rotation prior penalising the whole angle-axis vector (all three components, at a low weight); what it is there for is the poorly-determined out-of-plane component, which is the one the data barely constrain. For higher symmetries, constraints are enforced, e.g. - cubic: $a=b=c,\ \alpha=\beta=\gamma=90^\circ$, @@ -206,18 +206,20 @@ The loose first stage necessarily admits some spots that are not reflections of The refinement above (§7.2) runs per image against that image's spots. For rotation data an additional **post-refinement** (on by default; `--rotation-no-postrefine` disables it) improves the detector distance, beam centre and crystal cell/axis using **all** frames at once, then re-integrates: 1. **Pass 1** integrates, scales and merges at the header geometry. -2. From pass-1's integrated reflections, the geometry is refined over all frames (Ceres, robust loss) in **two separate steps** rather than one joint fit: - - **Step A**: crystal cell scale + goniometer-axis direction, from the observed rotation angles (a distance-independent excitation residual). - - **Step B**: shared detector distance + beam centre, from the observed spot positions, with the cell held at step A — so the positional residual is no longer degenerate with the cell scale. +2. From pass-1's integrated reflections, the crystal and the detector are refined together over all frames (Ceres, robust loss) in **one joint fit**, against both residuals at once: + - the **positional** detector↔reciprocal residual at each partial's observed spot, and + - a distance-independent **Ewald excitation** residual at each reflection's observed rocking centroid $\phi_\mathrm{obs}$. - Each step is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual, otherwise left at nominal. The solver bounds the move — distance within ±5 %, beam centre within ±15 px — and detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal. -3. **Pass 2** re-indexes de novo and re-integrates at the committed geometry. Only the **detector distance and beam centre** carry over: the refined cell and axis are used to make step B well-posed, but pass 2 re-indexes from scratch, so they are not propagated. + Free: the crystal orientation, the unit cell (every parameter the crystal system leaves free, not one overall scale), the goniometer-axis direction, the detector distance and the beam centre. The positional residual on its own *is* degenerate with the cell scale — that is why this used to be split into a cell-scale step and a distance step — but the excitation residual does not involve the detector at all, so it fixes the absolute size of the reciprocal lattice and breaks the degeneracy inside the same problem. Splitting it instead cost accuracy twice over: pass 1 frees the whole lattice against a frozen distance, so the distortion it absorbs is *anisotropic* and no single scale can undo it; and whatever bias is left in that scale goes straight into the distance, which is only ever determined relative to the cell. + + The fit is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual and the move stays small — distance within 1 %, every cell length within 1 %, beam centre within 15 px of the nearest centre anything already believes — otherwise left at nominal. Detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal. +3. **Pass 2** re-indexes de novo and re-integrates at the committed geometry. Only the **detector distance and beam centre** carry over: the refined cell, orientation and axis are what make the distance identifiable, but pass 2 re-indexes from scratch, so they are not propagated. The space group is determined **after** pass 2, on the geometry the run refined, and pass 1 does not search at all: a decision taken on the worse of the two passes and then carried forward is a constraint on the better one, and had to be reconciled with what pass 2 later found. The guard that chooses which pass is written compares each pass's **first** merge — $P1$ on both sides, full resolution range, before the correction surfaces — which both passes produce anyway, so it never compares statistics computed in two different space groups. One index-time veto remains and is keyed to pass 1's **lattice** rather than its group: a centred pass-1 lattice against a primitive pass-2 one. Only pass 2 is written, as the canonical `_*` output. Pass 1's merge exists to give the guard something to judge pass 2 against, so it stops short of the parts of the merge that only fill in a file — the correction surfaces, the twinning and radiation-damage analyses, the R-free flags and the amplitudes — and writes no merged files of its own. -**Goniometer rotation scale (report only).** A stage that turns further than it was commanded to leaves no trace in the file, because the stored $\omega$ values *are* the commanded ones; the excess then presents as the crystal drifting, in this program and in others. Step A already measures it without a new degree of freedom: its residual rotates by $-\phi\,\mathbf{u}$ with $\mathbf{u}$ an **unnormalised** 3-vector, so $|\mathbf{u}|$ is the factor by which the stage actually turned, and normalising the axis throws it away. It is reported, and warned about beyond 0.5 %, under the same cross-validation that gates the cell move — a fold that merely soaked up noise cannot raise the flag. It is a detector, not a calibration: nothing corrects the data, and it **under-reads** the true magnitude, because the fit only sees reflections that indexed at the nominal angle and per-frame orientation refinement has already absorbed part of the error. +**Goniometer rotation scale (report only).** A stage that turns further than it was commanded to leaves no trace in the file, because the stored $\omega$ values *are* the commanded ones; the excess then presents as the crystal drifting, in this program and in others. The excitation residual already measures it without a new degree of freedom: it rotates by $-\phi\,\mathbf{u}$ with $\mathbf{u}$ an **unnormalised** 3-vector, so $|\mathbf{u}|$ is the factor by which the stage actually turned, and normalising the axis throws it away. It is reported, and warned about beyond 0.5 %, under its own leave-a-fifth-of-the-sweep-out check — a fold that merely soaked up noise cannot raise the flag. It is a detector, not a calibration: nothing corrects the data, and it **under-reads** the true magnitude, because the fit only sees reflections that indexed at the nominal angle and per-frame orientation refinement has already absorbed part of the error. ### 7.6 Detector geometry from powder rings diff --git a/image_analysis/geom_refinement/PostRefine.cpp b/image_analysis/geom_refinement/PostRefine.cpp index a900a9499..f9b0a9a05 100644 --- a/image_analysis/geom_refinement/PostRefine.cpp +++ b/image_analysis/geom_refinement/PostRefine.cpp @@ -13,7 +13,7 @@ #include #include "../../common/JFJochMath.h" // PI -#include "XtalResidual.h" // XtalResidual (the positional detector<->reciprocal residual, step B) +#include "XtalResidual.h" // the positional detector<->reciprocal residual, and the cell it is parameterised by #include "LatticeReduction.h" #include "ceres/ceres.h" #include "ceres/rotation.h" @@ -45,26 +45,36 @@ struct Partial { float obs_x, obs_y; // observed spot centroid (pixels); NAN if the box sum found no centroid }; -// A rocking event and its precomputed reference reciprocal vector (phi=0 frame, from the indexed lattice). +// A rocking event: one reflection's intensity-weighted centroid over the frames it spans. struct Event { double phi_obs; // rad, intensity-weighted rocking centroid double weight; // sqrt(sum I / sum sigma) - double e_ref[3]; // h*a* + k*b* + l*c* at the reference (unrefined) cell/orientation int h, k, l; }; -// Distance-INDEPENDENT Ewald excitation residual for a uniform cell-scale parameter s and a refined -// goniometer axis (3-vector). The header-distance miscalibration leaves a uniform cell scale; the axis is -// the other phi_obs lever. Both are phi_obs-constrained (distance-independent). e_ref is the reference -// reciprocal (h*a* + k*b* + l*c* at the indexed cell). On the Ewald sphere <=> |p|^2 + 2 p_z/lambda == 0. -struct ScaleAxisExcitationResidual { - ScaleAxisExcitationResidual(double lambda, double angle_rad, double weight, const double e_ref[3]) +// Distance-INDEPENDENT Ewald excitation residual with the whole crystal free. The observed rocking +// centroid phi_obs says where the reflection actually crossed the Ewald sphere, which fixes the +// ABSOLUTE size of the reciprocal lattice with the detector never entering - that is what breaks the +// distance <-> cell-scale degeneracy the spot positions alone leave open, and it is why the two can be +// fitted together. The parameter blocks are XtalResidual's crystal half (rotation axis, orientation, +// cell lengths, cell angles), so one problem can share them between the two residuals. +// On the Ewald sphere <=> |p|^2 + 2 p_z/lambda == 0. +struct JointExcitationResidual { + JointExcitationResidual(double lambda, double angle_rad, double weight, + int h, int k, int l, gemmi::CrystalSystem symmetry) : inv_lambda(1.0 / lambda), angle_rad(angle_rad), weight(weight), - ex(e_ref[0]), ey(e_ref[1]), ez(e_ref[2]) {} + h(h), k(k), l(l), symmetry(symmetry) {} template - bool operator()(const T *const s, const T *const axis, T *residual) const { - const T inv_s = T(1) / s[0]; - const T p_ref[3] = {T(ex) * inv_s, T(ey) * inv_s, T(ez) * inv_s}; + bool operator()(const T *const axis, const T *const p0, const T *const p1, const T *const p2, + T *residual) const { + Eigen::Matrix bxc, cxa, axb; + T invV; + XtalResidual::ReciprocalBasis(p1, p2, symmetry, bxc, cxa, axb, invV); + const Eigen::Matrix unrot = (bxc * T(h) + cxa * T(k) + axb * T(l)) * invV; + const T recip_unrot[3] = {unrot[0], unrot[1], unrot[2]}; + T p_ref[3]; + const AngleAxisRotator rot_p0(p0); + rot_p0.Rotate(recip_unrot, p_ref); const T aa[3] = {T(-angle_rad) * axis[0], T(-angle_rad) * axis[1], T(-angle_rad) * axis[2]}; T p_lab[3]; ceres::AngleAxisRotatePoint(aa, p_ref, p_lab); @@ -73,21 +83,23 @@ struct ScaleAxisExcitationResidual { residual[0] = T(weight) * zeta * T(0.5) / T(inv_lambda); return true; } - const double inv_lambda, angle_rad, weight, ex, ey, ez; + const double inv_lambda, angle_rad, weight; + const double h, k, l; + const gemmi::CrystalSystem symmetry; }; -// GONIOMETER ROTATION SCALE k: the same Ewald excitation residual, but with the cell scale and the axis -// DIRECTION already committed by step A, so the single free quantity is how far the stage actually turned -// per unit of commanded angle. Two differences from step A matter: +// GONIOMETER ROTATION SCALE k: the same Ewald excitation residual, but with the crystal and the axis +// DIRECTION already committed by the joint fit, so the single free quantity is how far the stage actually +// turned per unit of commanded angle. Two differences from that fit matter: // * the angle is measured from the CENTRE of the sweep, not from the goniometer's zero. The reference // orientation is the one rotation indexing fitted against the commanded angles, so it has already // absorbed the MEAN angle error; only the part that varies across the sweep is left to fit. Scaling the -// absolute angle instead - which is what reading k off the length of step A's axis vector does - asks +// absolute angle instead - which is what reading k off the length of the fitted axis vector does - asks // the fit to also produce a constant offset it has no parameter for, and the least-squares compromise // shrinks k towards 1 by var(phi) / (var(phi) + phi_centre^2): exactly a factor of four for the common // case of a sweep starting at zero. -// * e_mid is the reference reciprocal vector already turned to the sweep centre and divided by the -// committed cell scale, so nothing but k is free. +// * e_mid is the committed reciprocal vector already turned to the sweep centre, so nothing but k is +// free. struct RotationScaleResidual { RotationScaleResidual(double lambda, double dangle_rad, const double u[3], const double e_mid[3]) : inv_lambda(1.0 / lambda), dangle_rad(dangle_rad), @@ -124,7 +136,6 @@ PostRefineResult PostRefineRotationGeometry(const std::vector=2-frame events carry an - // unbiased phi_obs (a single-frame centroid is just the frame centre); precompute e_ref per event. + // unbiased phi_obs (a single-frame centroid is just the frame centre). constexpr float MAX_FRAME_GAP = 2.0f; const auto run_end = [&](size_t i, size_t end) { size_t j = i + 1; @@ -258,10 +269,7 @@ PostRefineResult PostRefineRotationGeometry(const std::vector 0.0 && sumSig > 0.0)) return false; - const Coord e = Astar * static_cast(pts[i].h) + Bstar * static_cast(pts[i].k) - + Cstar * static_cast(pts[i].l); - out = Event{sumIphi / sumI, std::sqrt(sumI / sumSig), {e.x, e.y, e.z}, - pts[i].h, pts[i].k, pts[i].l}; + out = Event{sumIphi / sumI, std::sqrt(sumI / sumSig), pts[i].h, pts[i].k, pts[i].l}; return true; }; // An event never crosses an h boundary - h is the leading sort key - so the buckets can be @@ -325,15 +333,24 @@ PostRefineResult PostRefineRotationGeometry(const std::vectorreciprocal residual at each partial's observed spot, and + // * the distance-INDEPENDENT Ewald excitation residual at each rocking centroid phi_obs. + // It replaces a two-step fit that scaled the whole cell by ONE scalar against phi_obs and then + // read the distance off that scaled cell. The two were separated because the positional residual + // is degenerate with the cell scale - which is true of the positions ALONE, and is exactly what + // the excitation residual breaks, so the degeneracy that motivated the split is already resolved + // inside the same problem. Splitting it cost accuracy twice over: the first pass frees the whole + // lattice against a frozen distance, so the distortion it absorbs is ANISOTROPIC and no single + // scale can undo it; and whatever bias is left in that scale goes straight into the distance, + // which is only ever determined relative to the cell. Measured over three wavelengths of one + // crystal, that scale's bias changed SIGN with the wavelength and the distance error followed it + // with a slope of 1.3 and a correlation of 0.99, while the rocking centroids on their own fixed + // the cell volume to 0.06 %. + // Detector tilt is held fixed (gauge-coupled to the orientation on a single crystal). + // Committed only if it lowers a HELD-OUT (deterministic split-half) residual and the move stays + // within the bounds below - otherwise the geometry is left at nominal ("quit when things go wrong"). if (settings.refine_geometry) { const gemmi::CrystalSystem sys = (settings.crystal_system == gemmi::CrystalSystem::Trigonal) ? gemmi::CrystalSystem::Hexagonal @@ -345,13 +362,16 @@ PostRefineResult PostRefineRotationGeometry(const std::vector(h) * 2654435761u + static_cast(k) * 2246822519u + static_cast(l) * 3266489917u; @@ -362,53 +382,246 @@ PostRefineResult PostRefineRotationGeometry(const std::vector(std::clamp(nthreads, 1, + std::max(1, n_pts))); + const size_t obs_chunk = (n_pts + n_obs_chunks - 1) / n_obs_chunks; + std::vector obs_offset(n_obs_chunks + 1, 0); + const auto keep_obs = [&](size_t i) { + return std::isfinite(pts[i].obs_x) && std::isfinite(pts[i].obs_y); }; - auto solve_scale_axis = [&](Subset s, double &s_out, double ax_out[3]) { - double sc = 1.0, axv[3] = {ax0[0], ax0[1], ax0[2]}; - ceres::Problem p; + ParallelChunks(static_cast(n_pts), nthreads, [&](int lo, int hi) { + size_t keep = 0; + for (int i = lo; i < hi; ++i) if (keep_obs(i)) keep++; + obs_offset[static_cast(lo) / obs_chunk + 1] = keep; + }); + for (int c = 0; c < n_obs_chunks; ++c) obs_offset[c + 1] += obs_offset[c]; + size_t n_obs = obs_offset[n_obs_chunks]; + std::unique_ptr obs(new const Partial *[n_obs]); + ParallelChunks(static_cast(n_pts), nthreads, [&](int lo, int hi) { + size_t at = obs_offset[static_cast(lo) / obs_chunk]; + for (int i = lo; i < hi; ++i) if (keep_obs(i)) obs[at++] = &pts[i]; + }); + constexpr size_t MAX_OBS = 20000; + if (n_obs > MAX_OBS) { + std::nth_element(obs.get(), obs.get() + MAX_OBS, obs.get() + n_obs, + [](const Partial *a, const Partial *b) { + return a->I / std::max(1e-9, static_cast(a->sigma)) + > b->I / std::max(1e-9, static_cast(b->sigma)); }); + n_obs = MAX_OBS; + } + result.obs_used = static_cast(n_obs); + + // ===== The joint fit ===== + // Cost of a whole geometry over one subset, the two residual families kept apart so the log + // can say which of them moved. Each is a mean per residual VALUE, and the number they are + // summed over below is the same weighting the solver itself applies. + auto joint_cost = [&](Subset s, const double bm[2], const double ds[1], const double rv[3], + const double q0[3], const double q1[3], const double q2[3], + double &pos_out, double &exc_out) { + double cp = 0.0, ce = 0.0; + size_t np = 0, ne = 0; + for (size_t oi = 0; oi < n_obs; ++oi) { + const Partial *pp = obs[oi]; + if (!in(pp->h, pp->k, pp->l, s)) continue; + XtalResidual r(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, cos_rot3, sin_rot3, + angle_rad(pp->img), pp->h, pp->k, pp->l, sys, orientation); + double res[3] = {0, 0, 0}; + r(bm, ds, det_rot, rv, q0, q1, q2, res); + cp += res[0] * res[0] + res[1] * res[1] + res[2] * res[2]; + np += 3; + } for (const int32_t i : selected) { const Event &ev = events[i]; if (!in(ev.h, ev.k, ev.l, s)) continue; - p.AddResidualBlock(new ceres::AutoDiffCostFunction( - new ScaleAxisExcitationResidual(lambda_l, ev.phi_obs, settings.excitation_weight, ev.e_ref)), - new ceres::CauchyLoss(0.02), &sc, axv); + JointExcitationResidual r(lambda_l, ev.phi_obs, settings.excitation_weight, + ev.h, ev.k, ev.l, sys); + double res = 0.0; + r(rv, q0, q1, q2, &res); + ce += res * res; + ++ne; } - p.SetParameterLowerBound(&sc, 0, 0.9); p.SetParameterUpperBound(&sc, 0, 1.1); - for (int j = 0; j < 3; ++j) { p.SetParameterLowerBound(axv, j, ax0[j] - 0.05); - p.SetParameterUpperBound(axv, j, ax0[j] + 0.05); } - ceres::Solver::Options o; o.linear_solver_type = ceres::DENSE_QR; o.max_num_iterations = 50; + pos_out = np ? cp / np : 0.0; + exc_out = ne ? ce / ne : 0.0; + return (np + ne) ? (cp + ce) / static_cast(np + ne) : 0.0; + }; + + // One problem, both residual families, every block seeded at nominal. The detector tilt is + // declared and held constant so that a non-zero rot1/rot2 still acts on the observed side. + auto solve_joint = [&](Subset s, double bm[2], double ds[1], double rv[3], + double q0[3], double q1[3], double q2[3]) { + bm[0] = beam_x0; bm[1] = beam_y0; ds[0] = dist0; + for (int j = 0; j < 3; ++j) { + rv[j] = ax0[j]; q0[j] = p0_0[j]; q1[j] = p1_0[j]; q2[j] = p2_0[j]; + } + ceres::Problem p; + for (size_t oi = 0; oi < n_obs; ++oi) { + const Partial *pp = obs[oi]; + if (!in(pp->h, pp->k, pp->l, s)) continue; + p.AddResidualBlock(new ceres::AutoDiffCostFunction( + new XtalResidual(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, cos_rot3, sin_rot3, + angle_rad(pp->img), pp->h, pp->k, pp->l, sys, orientation)), + new ceres::CauchyLoss(0.02), bm, ds, det_rot, rv, q0, q1, q2); + } + for (const int32_t i : selected) { + const Event &ev = events[i]; + if (!in(ev.h, ev.k, ev.l, s)) continue; + p.AddResidualBlock(new ceres::AutoDiffCostFunction( + new JointExcitationResidual(lambda_l, ev.phi_obs, settings.excitation_weight, + ev.h, ev.k, ev.l, sys)), + new ceres::CauchyLoss(0.02), rv, q0, q1, q2); + } + if (p.NumResidualBlocks() == 0) return false; + p.SetParameterBlockConstant(det_rot); + if (!p2_free) p.SetParameterBlockConstant(q2); + p.SetParameterLowerBound(ds, 0, dist0 * 0.95); p.SetParameterUpperBound(ds, 0, dist0 * 1.05); + // The box has to reach everywhere the gate below would accept, or the gate is never the + // thing that decides: a fit pinned at a box face lands exactly ON the bound and is then + // refused for being there. + for (int j = 0; j < 2; ++j) { + const double m = settings.measured_beam_px ? (*settings.measured_beam_px)[j] : bm[j]; + p.SetParameterLowerBound(bm, j, std::min(bm[j], m) - BEAM_BOUND_PXL); + p.SetParameterUpperBound(bm, j, std::max(bm[j], m) + BEAM_BOUND_PXL); + } + for (int j = 0; j < 3; ++j) { + p.SetParameterLowerBound(rv, j, ax0[j] - 0.05); + p.SetParameterUpperBound(rv, j, ax0[j] + 0.05); + p.SetParameterLowerBound(q1, j, 0.97 * p1_0[j]); + p.SetParameterUpperBound(q1, j, 1.03 * p1_0[j]); + } + if (p2_free) + for (int j = 0; j < 3; ++j) { + p.SetParameterLowerBound(q2, j, p2_0[j] - 0.05); + p.SetParameterUpperBound(q2, j, p2_0[j] + 0.05); + } + ceres::Solver::Options o; o.linear_solver_type = ceres::DENSE_QR; o.max_num_iterations = 60; o.num_threads = std::max(1, settings.num_threads); o.logging_type = ceres::LoggingType::SILENT; ceres::Solver::Summary sum; ceres::Solve(o, &p, &sum); - s_out = sc; ax_out[0] = axv[0]; ax_out[1] = axv[1]; ax_out[2] = axv[2]; return sum.IsSolutionUsable(); }; - double s_fit = 1.0, ax_fit[3]; - const bool convA = solve_scale_axis(FIT, s_fit, ax_fit); - const double cvA_nom = excit_cost(VAL, 1.0, ax0); - const double cvA_ref = excit_cost(VAL, s_fit, ax_fit); - // Commit the cell scale only for a small, credible move: a well-calibrated header needs < ~0.6 %, - // so a > 1 % scale is a red flag (on multi-lattice / noisy data the excitation fit is biased the - // same way in every cross-validation fold, so the relative-improvement gate cannot catch it). - result.cell_refined = convA && cvA_ref < 0.98 * cvA_nom && std::fabs(s_fit - 1.0) < 0.01; - double s = 1.0, axv[3] = {ax0[0], ax0[1], ax0[2]}; - if (result.cell_refined) solve_scale_axis(ALL, s, axv); // commit: re-fit on all data + + double beam[2] = {beam_x0, beam_y0}, dist[1] = {dist0}; + double axv[3] = {ax0[0], ax0[1], ax0[2]}; + bool commit = false; + if (n_obs >= static_cast(settings.min_events)) { + double pos_nom = 0.0, exc_nom = 0.0, pos_ref = 0.0, exc_ref = 0.0; + const double cv_nom = joint_cost(VAL, beam, dist, axv, p0, p1, p2, pos_nom, exc_nom); + double bm_f[2], ds_f[1], rv_f[3], q0_f[3], q1_f[3], q2_f[3]; + const bool convJ = solve_joint(FIT, bm_f, ds_f, rv_f, q0_f, q1_f, q2_f); + const double cv_ref = joint_cost(VAL, bm_f, ds_f, rv_f, q0_f, q1_f, q2_f, pos_ref, exc_ref); + // Commit only for a small, credible move: distance < 1 % (a calibrated header needs + // < ~0.6 %), each cell length < 1 %, and the beam inside the bound measured from + // whichever centre anything already believes is nearer. A larger move is the red flag for + // an unreliable fit - typically a second lattice whose spots bias every cross-validation + // fold identically, so the relative "it improved" gate is blind to it. The absolute size + // of the move discriminates a genuine header correction from that failure far better than + // the absolute residual, which real marginal (noisy / iced) data shares with it. + const double from_nominal = std::hypot(bm_f[0] - beam_x0, bm_f[1] - beam_y0); + const double from_measured = settings.measured_beam_px + ? std::hypot(bm_f[0] - (*settings.measured_beam_px)[0], + bm_f[1] - (*settings.measured_beam_px)[1]) + : std::numeric_limits::infinity(); + double len_shift = 0.0; + for (int j = 0; j < 3; ++j) + len_shift = std::max(len_shift, std::fabs(q1_f[j] - p1_0[j]) / std::max(1e-9, p1_0[j])); + const bool in_bounds = std::fabs(ds_f[0] - dist0) < 0.01 * dist0 + && std::min(from_nominal, from_measured) < BEAM_BOUND_PXL + && len_shift < 0.01; + commit = convJ && cv_ref < 0.98 * cv_nom && in_bounds; + if (commit) solve_joint(ALL, beam, dist, axv, p0, p1, p2); // commit: re-fit on all data + // Name which test refused it. Four different things reject here and the geometry that + // comes out is the same in all four, so a run that silently keeps its header geometry + // says nothing about whether the fit was bad, the improvement too small, or the move too + // large for the bound - which is the one case where the number worth reading is the one + // that was thrown away. + const char *verdict = + commit ? "COMMIT" + : !convJ ? "reject (the fit did not converge)" + : !(cv_ref < 0.98 * cv_nom) ? "reject (the held-out residual did not improve enough)" + : std::fabs(ds_f[0] - dist0) >= 0.01 * dist0 + ? "reject (the distance moved more than 1 %)" + : len_shift >= 0.01 ? "reject (a cell length moved more than 1 %)" + : "reject (the beam moved further than the bound from " + "every centre anything believes)"; + // The cell the log names is the EFFECTIVE one - what the residual's B matrix builds + // from the blocks - not the blocks themselves, whose unused components a high-symmetry + // system leaves at whatever the seed happened to put there. + double len_nom[3], ang_nom[3], len_fit[3], ang_fit[3]; + EffectiveCellFromParams(sys, p1_0, p2_0, len_nom, ang_nom); + EffectiveCellFromParams(sys, q1_f, q2_f, len_fit, ang_fit); + logger.Info("Post-refine GEOM (joint crystal + detector): dist {:.3f} -> {:.3f} mm, beam " + "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), cell {:.3f} {:.3f} {:.3f} -> " + "{:.3f} {:.3f} {:.3f}, held-out positional {:.3e} -> {:.3e}, excitation " + "{:.3e} -> {:.3e} => {}", + dist0, ds_f[0], beam_x0, beam_y0, bm_f[0], bm_f[1], + len_nom[0], len_nom[1], len_nom[2], len_fit[0], len_fit[1], len_fit[2], + pos_nom, pos_ref, exc_nom, exc_ref, verdict); + } else { + logger.Info("Post-refine GEOM: only {} positional observations - the joint fit needs the " + "spot positions as well as the rocking angles, so nothing is refined", n_obs); + } + result.cell_refined = commit; + result.detector_refined = commit; + const double axlen = std::sqrt(axv[0]*axv[0] + axv[1]*axv[1] + axv[2]*axv[2]); const double axdev = std::acos(std::clamp((axv[0]*ax0[0]+axv[1]*ax0[1]+axv[2]*ax0[2]) / std::max(1e-9, axlen), -1.0, 1.0)) * 180.0 / PI; - logger.Info("Post-refine GEOM step A (cell/axis): s = {:.5f}, rot-axis {:.3f} deg, held-out excit " - "{:.3e} -> {:.3e} => {}", s, axdev, cvA_nom, cvA_ref, - result.cell_refined ? "COMMIT" : "reject (kept nominal cell)"); + // The committed crystal, as a lattice again. Where nothing was committed this is the lattice + // rotation indexing handed in, unchanged - not its round trip through the symmetry-constrained + // parameterisation, which would move the cell for a fit that was refused. + double eff_len[3], eff_ang[3]; + EffectiveCellFromParams(sys, p1, p2, eff_len, eff_ang); + const CrystalLattice committed_latt = commit + ? AngleAxisAndCellToLattice(p0, eff_len, eff_ang[0], eff_ang[1], eff_ang[2]) + : reference_latt; + const Coord As = committed_latt.Astar(), Bs = committed_latt.Bstar(), Cs = committed_latt.Cstar(); + if (commit) { + result.cell = UnitCell{static_cast(eff_len[0]), static_cast(eff_len[1]), + static_cast(eff_len[2]), + static_cast(eff_ang[0] * 180.0 / PI), + static_cast(eff_ang[1] * 180.0 / PI), + static_cast(eff_ang[2] * 180.0 / PI)}; + logger.Info("Post-refine GEOM: committed cell {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} {:.2f}, " + "rotation axis moved {:.3f} deg (the cell is the fit's own; the second pass " + "re-indexes at the refined detector geometry)", result.cell.a, result.cell.b, + result.cell.c, result.cell.alpha, result.cell.beta, result.cell.gamma, axdev); + } // ===== Goniometer rotation SCALE k, its own one-parameter fit on the same rocking events ===== // The angles stored in the file are the COMMANDED ones, so a stage that turned k times as far @@ -425,7 +638,7 @@ PostRefineResult PostRefineRotationGeometry(const std::vector(n_events); const double sweep_deg = (phi_hi - phi_lo) * 180.0 / PI; - // The reference reciprocal vector turned to the sweep centre, at the committed cell scale. The + // The committed reciprocal vector turned to the sweep centre. The // angle then enters the fit measured FROM that centre. A constant crystal missetting about the // spindle is k with a slope in phi, so measuring the angle from the goniometer's zero instead // lets a missetting leak into k with gain / - which depends only on where the sweep @@ -450,8 +663,10 @@ PostRefineResult PostRefineRotationGeometry(const std::vector(n_events), nthreads, [&](int lo, int hi) { for (int e = lo; e < hi; ++e) { - const double p[3] = {events[e].e_ref[0] / s, events[e].e_ref[1] / s, - events[e].e_ref[2] / s}; + const Coord ec = As * static_cast(events[e].h) + + Bs * static_cast(events[e].k) + + Cs * static_cast(events[e].l); + const double p[3] = {ec.x, ec.y, ec.z}; double em[3]; ceres::AngleAxisRotatePoint(aa_c, p, em); const double ue = u[0] * em[0] + u[1] * em[1] + u[2] * em[2]; @@ -609,177 +824,15 @@ PostRefineResult PostRefineRotationGeometry(const std::vector 1.0 ? "further" : "less far"); - // Cell (scale s, shape fixed) as the XtalResidual parameter blocks p0/p1/p2, held CONSTANT in step B. - double p0[3] = {0, 0, 0}, p1[3] = {0, 0, 0}, p2[3] = {0, 0, 0}; - double beta = r0.beta; - switch (sys) { - case gemmi::CrystalSystem::Tetragonal: - LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1); - p1[0] = (p1[0] + p1[1]) / 2.0; break; - case gemmi::CrystalSystem::Cubic: - LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1); - p1[0] = (p1[0] + p1[1] + p1[2]) / 3.0; break; - case gemmi::CrystalSystem::Hexagonal: - LatticeToRodriguesAndLengths_Hex(reference_latt, p0, p1); break; - case gemmi::CrystalSystem::Monoclinic: - LatticeToRodriguesLengthsBeta_Mono(reference_latt, p0, p1, beta); - p2[0] = beta; break; - case gemmi::CrystalSystem::Orthorhombic: - LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1); break; - default: - LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1); - p2[0] = r0.alpha * PI / 180.0; p2[1] = r0.beta * PI / 180.0; p2[2] = r0.gamma * PI / 180.0; break; - } - for (int j = 0; j < 3; ++j) p1[j] *= s; // apply the committed cell scale - double rot_vec[3] = {axv[0], axv[1], axv[2]}; // committed (or nominal) axis - - // ===== Step B: detector distance + beam from the observed positions, cell fixed ===== - // Count first, then fill, exactly as the partial gather above does and for the same reason: - // this walks the same tens of millions of partials, and a pointer vector grown by push_back - // copies itself every time it doubles. new[] rather than a sized vector so the array is not - // zeroed on one thread before the parallel fill overwrites it. The fill lands in the order - // the serial loop produced, so the selection below sees the same sequence it always did. - const int n_obs_chunks = static_cast(std::clamp(nthreads, 1, - std::max(1, n_pts))); - const size_t obs_chunk = (n_pts + n_obs_chunks - 1) / n_obs_chunks; - std::vector obs_offset(n_obs_chunks + 1, 0); - const auto keep_obs = [&](size_t i) { - return std::isfinite(pts[i].obs_x) && std::isfinite(pts[i].obs_y); - }; - ParallelChunks(static_cast(n_pts), nthreads, [&](int lo, int hi) { - size_t keep = 0; - for (int i = lo; i < hi; ++i) if (keep_obs(i)) keep++; - obs_offset[static_cast(lo) / obs_chunk + 1] = keep; - }); - for (int c = 0; c < n_obs_chunks; ++c) obs_offset[c + 1] += obs_offset[c]; - size_t n_obs = obs_offset[n_obs_chunks]; - std::unique_ptr obs(new const Partial *[n_obs]); - ParallelChunks(static_cast(n_pts), nthreads, [&](int lo, int hi) { - size_t at = obs_offset[static_cast(lo) / obs_chunk]; - for (int i = lo; i < hi; ++i) if (keep_obs(i)) obs[at++] = &pts[i]; - }); - constexpr size_t MAX_OBS = 20000; - if (n_obs > MAX_OBS) { - std::nth_element(obs.get(), obs.get() + MAX_OBS, obs.get() + n_obs, - [](const Partial *a, const Partial *b) { - return a->I / std::max(1e-9, static_cast(a->sigma)) - > b->I / std::max(1e-9, static_cast(b->sigma)); }); - n_obs = MAX_OBS; - } - result.obs_used = static_cast(n_obs); - const double beam_x0 = nominal_geom.GetBeamX_pxl(), beam_y0 = nominal_geom.GetBeamY_pxl(); - const double dist0 = nominal_geom.GetDetectorDistance_mm(); - auto pos_cost = [&](Subset s, const double beam[2], const double dist[1]) { - double c = 0.0; int n = 0; - for (size_t oi = 0; oi < n_obs; ++oi) { - const Partial *pp = obs[oi]; - if (!in(pp->h, pp->k, pp->l, s)) continue; - XtalResidual r(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, cos_rot3, sin_rot3, - angle_rad(pp->img), - pp->h, pp->k, pp->l, sys, orientation); - double resid[3] = {0, 0, 0}; - r(beam, dist, det_rot, rot_vec, p0, p1, p2, resid); - c += resid[0]*resid[0] + resid[1]*resid[1] + resid[2]*resid[2]; ++n; - } - return n ? c / n : 0.0; - }; - auto solve_detector = [&](Subset s, double beam_out[2], double &dist_out) { - double beam[2] = {beam_x0, beam_y0}, dist[1] = {dist0}; - // What the reduced residual holds fixed varies with the frame alone, so build one set of - // constants per image rather than one per observation. A map node keeps its address as - // the map grows, and the map outlives the problem that points into it. - std::map frame_const; - ceres::Problem p; - for (size_t oi = 0; oi < n_obs; ++oi) { - const Partial *pp = obs[oi]; - if (!in(pp->h, pp->k, pp->l, s)) continue; - const XtalFrameConstants &fc = frame_const.try_emplace( - pp->img, det_rot, rot_vec, angle_rad(pp->img), p1, p2, sys).first->second; - p.AddResidualBlock(new ceres::AutoDiffCostFunction( - new XtalResidualBeamDistance( - XtalResidual(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, cos_rot3, sin_rot3, - angle_rad(pp->img), pp->h, pp->k, pp->l, sys, orientation), - fc, p0)), - new ceres::CauchyLoss(0.02), beam, dist); - } - if (p.NumResidualBlocks() == 0) { beam_out[0] = beam_x0; beam_out[1] = beam_y0; dist_out = dist0; return false; } - // Everything but the beam and the distance is held at its step-A value, and the residual - // above bakes those five blocks in rather than declaring them and freezing them, so there - // is nothing left to hold constant here. - p.SetParameterLowerBound(dist, 0, dist0 * 0.95); p.SetParameterUpperBound(dist, 0, dist0 * 1.05); - // The box has to reach everywhere the gate below would accept, or the gate is never the - // thing that decides: a fit pinned at a box face lands exactly ON the bound and is then - // refused for being there. - for (int j = 0; j < 2; ++j) { - const double m = settings.measured_beam_px ? (*settings.measured_beam_px)[j] : beam[j]; - p.SetParameterLowerBound(beam, j, std::min(beam[j], m) - BEAM_BOUND_PXL); - p.SetParameterUpperBound(beam, j, std::max(beam[j], m) + BEAM_BOUND_PXL); - } - ceres::Solver::Options o; o.linear_solver_type = ceres::DENSE_QR; o.max_num_iterations = 60; - o.num_threads = std::max(1, settings.num_threads); o.logging_type = ceres::LoggingType::SILENT; - ceres::Solver::Summary sum; ceres::Solve(o, &p, &sum); - beam_out[0] = beam[0]; beam_out[1] = beam[1]; dist_out = dist[0]; - return sum.IsSolutionUsable(); - }; - double beam[2] = {beam_x0, beam_y0}, dist = dist0; - if (n_obs >= static_cast(settings.min_events)) { - double beam_fit[2], dist_fit; - const bool convB = solve_detector(FIT, beam_fit, dist_fit); - const double b_nom[2] = {beam_x0, beam_y0}, d_nom[1] = {dist0}; - const double b_ref[2] = {beam_fit[0], beam_fit[1]}, d_ref[1] = {dist_fit}; - const double cvB_nom = pos_cost(VAL, b_nom, d_nom); - const double cvB_ref = pos_cost(VAL, b_ref, d_ref); - // Commit the detector geometry only for a small, credible move: distance < 1 % (a calibrated - // header needs < ~0.6 %). A larger move is the red flag for an unreliable fit - typically a - // second lattice whose spots bias every cross-validation fold identically, so the relative - // "it improved" gate is blind to it and pulls a spurious distance<->cell pair (the radial - // degeneracy) far off. The absolute size of the move discriminates a genuine header correction - // from that failure far better than the absolute residual, which real marginal (noisy / iced) - // data shares with the multi-lattice case. - const double from_nominal = std::hypot(beam_fit[0] - beam_x0, beam_fit[1] - beam_y0); - const double from_measured = settings.measured_beam_px - ? std::hypot(beam_fit[0] - (*settings.measured_beam_px)[0], - beam_fit[1] - (*settings.measured_beam_px)[1]) - : std::numeric_limits::infinity(); - const bool in_bounds = std::fabs(dist_fit - dist0) < 0.01 * dist0 - && std::min(from_nominal, from_measured) < BEAM_BOUND_PXL; - result.detector_refined = convB && cvB_ref < 0.98 * cvB_nom && in_bounds; - if (result.detector_refined) { double bo[2]; solve_detector(ALL, bo, dist); beam[0] = bo[0]; beam[1] = bo[1]; } - // Name which test refused it. Three different things reject here and the geometry that - // comes out is the same in all three, so a run that silently keeps its header geometry - // says nothing about whether the fit was bad, the improvement too small, or the move - // too large for the bound - which is the one case where the number worth reading is the - // one that was thrown away. - const char *verdict = - result.detector_refined ? "COMMIT" - : !convB ? "reject (the fit did not converge; kept nominal detector)" - : !(cvB_ref < 0.98 * cvB_nom) ? "reject (the held-out residual did not improve enough; " - "kept nominal detector)" - : std::fabs(dist_fit - dist0) >= 0.01 * dist0 - ? "reject (the distance moved more than 1 %; kept nominal detector)" - : "reject (the beam moved further than the bound from every " - "centre anything believes; kept nominal detector)"; - logger.Info("Post-refine GEOM step B (distance/beam): dist {:.3f} -> {:.3f} mm, beam " - "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), held-out pos {:.3e} -> {:.3e} => {}", - dist0, result.detector_refined ? dist : dist0, beam_x0, beam_y0, - result.detector_refined ? beam[0] : beam_x0, result.detector_refined ? beam[1] : beam_y0, - cvB_nom, cvB_ref, verdict); - } else { - logger.Info("Post-refine GEOM step B: only {} positional observations - skipped", n_obs); - } - // Assemble the committed geometry. - UnitCell cellA = r0; - if (result.cell_refined) { cellA.a = static_cast(r0.a * s); cellA.b = static_cast(r0.b * s); - cellA.c = static_cast(r0.c * s); } - result.cell = cellA; - result.distance_after_mm = dist; + result.distance_after_mm = dist[0]; result.beam_x_before_px = beam_x0; result.beam_x_after_px = beam[0]; result.beam_y_before_px = beam_y0; result.beam_y_after_px = beam[1]; result.events_used = static_cast(selected.size()); - result.ok = result.cell_refined || result.detector_refined; + result.ok = commit; if (!result.ok) - logger.Info("Post-refine GEOM: neither step passed cross-validation - geometry left at nominal"); + logger.Info("Post-refine GEOM: the joint fit did not pass cross-validation - geometry " + "left at nominal"); return result; } diff --git a/image_analysis/geom_refinement/PostRefine.h b/image_analysis/geom_refinement/PostRefine.h index ab29c6c6c..b3cf0a707 100644 --- a/image_analysis/geom_refinement/PostRefine.h +++ b/image_analysis/geom_refinement/PostRefine.h @@ -17,13 +17,15 @@ // Post-integration geometry refinement for rotation data. Unlike the at-indexing XtalOptimizer, this runs // AFTER integration/merge, where each reflection has an OBSERVED rocking centroid phi_obs (the intensity- -// weighted mean goniometer angle over the frames it spans) and an observed spot position. It refines one -// shared crystal orientation + cell (+ optionally the detector distance) against two residuals: +// weighted mean goniometer angle over the frames it spans) and an observed spot position. One JOINT fit +// refines the crystal (orientation, cell, rotation axis) and the detector (distance, beam centre) together +// against two residuals: // * an Ewald excitation residual evaluated at phi_obs (distance-independent) -> pins the absolute cell // scale that the positional residual leaves degenerate with the distance. Because phi_obs is the real // rocking angle (not a frame centre) it is unbiased. // * the positional detector<->reciprocal residual at each partial's observed spot -> pins the distance. -// Reflections are weighted by their merged I/sigma (strong, well-measured reflections dominate). +// Where there are more reflections than the fit's caps, the strongest by I/sigma are the ones kept; +// inside the fit every reflection carries the same weight. struct PostRefineResult { bool ok = false; @@ -34,11 +36,13 @@ struct PostRefineResult { double distance_before_mm = 0.0, distance_after_mm = 0.0; double beam_x_before_px = 0.0, beam_x_after_px = 0.0; // refined beam centre (GEOM mode) double beam_y_before_px = 0.0, beam_y_after_px = 0.0; - bool cell_refined = false; // GEOM step A (cell scale + axis) passed cross-validation - bool detector_refined = false; // GEOM step B (distance + beam) passed cross-validation + // The joint fit is one decision - the crystal and the detector are refined together and committed + // together - so these two are always equal. Both are kept because the report names them separately. + bool cell_refined = false; // the joint fit passed cross-validation (crystal half) + bool detector_refined = false; // the joint fit passed cross-validation (detector half) // GONIOMETER ROTATION SCALE: the factor by which the stage actually turned relative to the angle // stored in the file (which is the COMMANDED value, hence a stage calibration error is invisible in - // the header). Fitted after step A as a single free parameter, with the cell scale and the axis + // the header). Fitted after the joint fit as a single free parameter, with the crystal and the axis // direction held at their committed values. Always the fitted value; 1.0 = header and stage agree. double rotation_scale = 1.0; // Whether the fit passed every test needed to ACT on it: enough sweep and events, a significant and @@ -48,14 +52,14 @@ struct PostRefineResult { struct PostRefineSettings { gemmi::CrystalSystem crystal_system = gemmi::CrystalSystem::Triclinic; - bool refine_geometry = false; // XtalOptimizer-equivalent: cell scale + axis (from phi_obs) and detector - // distance + beam centre (from the observed spot positions X,Y), as two - // separate cross-validated steps. The only supported refinement mode. + bool refine_geometry = false; // XtalOptimizer-equivalent: the crystal (from phi_obs and the observed + // spot positions) and the detector distance + beam centre, in one + // cross-validated fit. The only supported refinement mode. double excitation_weight = 1.0; // weight of the phi/excitation residual vs the positional one int min_events = 50; int num_threads = 1; // An independent measurement of the beam centre in pixels, where the run has one (rugnux's pre-scan - // fit to the isotropy of the scattered background, which runs on every rotation run). Step B bounds + // fit to the isotropy of the scattered background, which runs on every rotation run). The fit bounds // how far it may move the beam from whichever of this and the nominal centre is NEARER; see // BEAM_BOUND_PXL in PostRefine.cpp for why the nominal centre alone is not enough to bound it. std::optional> measured_beam_px; diff --git a/image_analysis/geom_refinement/XtalResidual.h b/image_analysis/geom_refinement/XtalResidual.h index 3d206210f..9abcda661 100644 --- a/image_analysis/geom_refinement/XtalResidual.h +++ b/image_analysis/geom_refinement/XtalResidual.h @@ -76,6 +76,39 @@ struct AngleAxisRotator { bool at_zero; }; +// The cell that XtalResidual::ReciprocalBasis builds out of the parameter blocks p1 (lengths) and p2 +// (angles, radians): the components a system does not read are filled in from its symmetry, so a caller +// holding a refined parameter set can turn it back into a cell or a lattice. The inverse of what that +// function reads, and kept beside it so the two conventions cannot drift apart. +inline void EffectiveCellFromParams(gemmi::CrystalSystem symmetry, const double p1[3], const double p2[3], + double lengths[3], double angles_rad[3]) { + const double right = M_PI / 2.0; + lengths[0] = p1[0]; + lengths[1] = p1[1]; + lengths[2] = p1[2]; + angles_rad[0] = right; angles_rad[1] = right; angles_rad[2] = right; + switch (symmetry) { + case gemmi::CrystalSystem::Hexagonal: + lengths[1] = p1[0]; + angles_rad[2] = 2.0 * M_PI / 3.0; + break; + case gemmi::CrystalSystem::Tetragonal: + lengths[1] = p1[0]; + break; + case gemmi::CrystalSystem::Cubic: + lengths[1] = p1[0]; lengths[2] = p1[0]; + break; + case gemmi::CrystalSystem::Monoclinic: + angles_rad[1] = p2[0]; + break; + case gemmi::CrystalSystem::Triclinic: + angles_rad[0] = p2[0]; angles_rad[1] = p2[1]; angles_rad[2] = p2[2]; + break; + default: // orthorhombic + break; + } +} + // Detector -> reciprocal geometry residual, shared by the per-image XtalOptimizer (one lattice, one // frame) and the offline GeometryRefiner (shared beam/distance/cell blocks, one orientation block per // frame). Parameter blocks: beam(2), distance_mm(1), detector_rot(2 = rot1,rot2), rotation_axis(3), diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 624783fac..c77dffd06 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -231,9 +231,9 @@ std::string RenderResultReport(const std::string &output_prefix, if (result.post_refine.has_value()) { const auto &pr = *result.post_refine; Section(os, "3. GEOMETRY POST-REFINEMENT"); - os << " The rotation two-pass fits the detector distance and beam centre from the observed spot\n" - << " positions, and the cell scale and rotation axis from the observed rocking angles. Each\n" - << " step is committed only if it improves a held-out residual.\n\n"; + os << " The rotation two-pass fits the crystal (orientation, cell, rotation axis) and the detector\n" + << " (distance, beam centre) together, against the observed spot positions and the observed\n" + << " rocking angles at once. It is committed only if it improves a held-out residual.\n\n"; Key(os, "POSTREFINE_EVENTS_USED", pr.events_used); Key(os, "POSTREFINE_OBS_USED", pr.obs_used); Key(os, "POSTREFINE_CELL_COMMITTED", pr.cell_refined ? "TRUE" : "FALSE"); -- 2.54.0 From 8b880740012ae5412bb483ffb685bdcc499845dc Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 09:34:16 +0200 Subject: [PATCH 10/75] rugnux: the joint post-refinement's gate is asked of the geometry it commits Five corrections to the joint fit, none of them meant to move a well-conditioned result. EffectiveCellFromParams used M_PI, which is not standard C++ and which MSVC does not define without _USE_MATH_DEFINES. image_analysis/ is on the viewer's dependency path, so it takes PI from JFJochMath.h like the rest of the tree. The bounds test ran on the fit-half solution, and what the run actually commits is the re-fit on all the reflections that follows it. That second solve moves the answer, and nothing re-asked the bounds of it, so the committed distance was held only by the solver's +-5 % box rather than by the 1 % the gate advertises. The test is now a function of a candidate geometry and is asked twice - once of the half that earns the right to fit, once of what will be written - and a re-fit that lands outside leaves the run at its header geometry, named in the log like the other refusals. It is inert on the corpus: over the 98 committed fits the largest committed move is 0.709 % of the distance, 0.885 % of a cell length and 1.07 px of the beam, so nothing that commits today stops committing. The bounds covered the distance, the beam and the cell lengths and left the cell ANGLES - beta for monoclinic, all three for triclinic - free to the solver's +-0.05 rad, which is +-2.86 deg. The uniform scale this replaced could not move an angle at all, so that was a new unguarded degree of freedom on the two systems whose conditioning is weakest. The bound is one degree, chosen from what the fit actually does: the largest angle it moves anywhere on the corpus is 0.35 deg, on a monoclinic beta, and the two triclinic cases with the largest moves are 0.09 and 0.07 deg. A degree is about three times the worst of those and still well inside the solver's box, so the box goes on reaching everywhere the gate accepts and no fit that commits today is refused. The gate pooled the two residual families into one held-out mean. With both caps saturated that is some 30000 positional values against 10000 excitation ones, so the excitation residual - the only thing in the problem that identifies the cell SCALE - is outvoted three to one in the decision that commits the cell. Neither family may now degrade. This one is not free. It refuses 1 of the 98 fits the corpus commits, a cubic case whose held-out excitation rose 15 % while its cell scale moved 0.75 %; refused, that dataset keeps its header geometry and lands where the two-step arm did, R_meas 0.2699 -> 0.2724 and ISa 3.15 -> 3.10 with the space group unchanged, and its cell deviation from the reference goes back from 0.05 % to 0.40 %. That is the shape of failure the split is meant to catch, and on the one dataset where the two disagree it costs a real if small improvement. The changelog claimed the refined distance "no longer depends on how wrong the file's distance was". It does. Swept over a millimetre of header distance the committed distance still moves monotonically and the merge degrades with it; the dependence is reduced about thirteenfold, not removed, and the line now says reduced. Finally the log printed the half-data fit's distance, beam and cell beside the word COMMIT while different numbers were committed - 107.908 mm printed against 107.913 mm applied two lines further down, on one sweep. It now prints whichever geometry the verdict is about, and prints the cell angles beside the lengths, since the angles are refined now. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 2 +- docs/CPU_DATA_ANALYSIS_INDEXING.md | 2 +- image_analysis/geom_refinement/PostRefine.cpp | 129 ++++++++++++------ image_analysis/geom_refinement/XtalResidual.h | 5 +- 4 files changed, 96 insertions(+), 42 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 03ff33cde..1c5fba125 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -8,7 +8,7 @@ * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, and open masters written at other facilities, including Eiger 1.x and third-party NXmx. * `rugnux` determines the lattice and the space group more reliably - the true cell where the first pass offers a whole-number multiple of it, so a pseudo-translated crystal keeps its full-length axis and a small molecule is indexed on its own cell rather than a protein-sized one, and the point group, the setting and the systematic absences - and `-S` refuses or re-seats a fixed space group whose symmetry axes the indexed cell does not carry. * `rugnux` asks the space-group search again on the cell the lattice metric supports, whenever that metric carries more rotational symmetry than the Bravais class the indexer named, so a lattice whose reduction landed in a lower-symmetry sub-cell can still reach its true point group; the higher symmetry is adopted only where the intensities confirm it. -* `rugnux` post-refines the rotation geometry with one joint fit of the crystal and the detector against both the observed spot positions and the observed rocking angles, in place of scaling the cell against the angles and then reading the detector distance off the scaled cell; the refined distance no longer depends on how wrong the file's distance was. +* `rugnux` post-refines the rotation geometry with one joint fit of the crystal and the detector against both the observed spot positions and the observed rocking angles, in place of scaling the cell against the angles and then reading the detector distance off the scaled cell, so the refined distance depends far less on how wrong the file's distance was. * `rugnux` measures the beam centre on every run and indexes with it when the file's value indexes nothing, refines only the detector-tilt component the data determine - a beam-centre error is no longer reported as a tilt - and places a detector swung out on a 2theta arm where the file says it stands. * `rugnux` writes the unmerged MTZ by default, with a P1 merge beside it, a batch header for every image the observations span, and events kept to the same `--min-captured-fraction` as the merge, so a wrong space group can be re-merged in a scaling program without reprocessing. * `rugnux` writes reflection files in the conventions downstream programs read: `FreeR_flag` is 0 for the test set and 1 for the working set - it was the other way round - the merged and P1 MTZ carry the reserved `HKL_base` dataset so a CCP4 program reads the wavelength instead of falling back to 1.54187 A, and the merged mmCIF marks the free set as `_refln.status` = `f`. diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index 9037bc34e..412043095 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -212,7 +212,7 @@ The refinement above (§7.2) runs per image against that image's spots. For rota Free: the crystal orientation, the unit cell (every parameter the crystal system leaves free, not one overall scale), the goniometer-axis direction, the detector distance and the beam centre. The positional residual on its own *is* degenerate with the cell scale — that is why this used to be split into a cell-scale step and a distance step — but the excitation residual does not involve the detector at all, so it fixes the absolute size of the reciprocal lattice and breaks the degeneracy inside the same problem. Splitting it instead cost accuracy twice over: pass 1 frees the whole lattice against a frozen distance, so the distortion it absorbs is *anisotropic* and no single scale can undo it; and whatever bias is left in that scale goes straight into the distance, which is only ever determined relative to the cell. - The fit is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual and the move stays small — distance within 1 %, every cell length within 1 %, beam centre within 15 px of the nearest centre anything already believes — otherwise left at nominal. Detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal. + The fit is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual — both families of it, since the excitation residual is the only evidence of the cell scale and the positional values outnumber it about three to one — and the move stays small: distance within 1 %, every cell length within 1 %, every free cell angle within 1°, beam centre within 15 px of the nearest centre anything already believes. The geometry the run commits is re-fitted on all the reflections once the held-out half has approved it, and the bounds are asked again of that fit rather than only of the half that earned it; a move outside them leaves the geometry at nominal, as every other refusal does. Detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal. 3. **Pass 2** re-indexes de novo and re-integrates at the committed geometry. Only the **detector distance and beam centre** carry over: the refined cell, orientation and axis are what make the distance identifiable, but pass 2 re-indexes from scratch, so they are not propagated. The space group is determined **after** pass 2, on the geometry the run refined, and pass 1 does not search at all: a decision taken on the worse of the two passes and then carried forward is a constraint on the better one, and had to be reconciled with what pass 2 later found. The guard that chooses which pass is written compares each pass's **first** merge — $P1$ on both sides, full resolution range, before the correction surfaces — which both passes produce anyway, so it never compares statistics computed in two different space groups. One index-time veto remains and is keyed to pass 1's **lattice** rather than its group: a centred pass-1 lattice against a primitive pass-2 one. diff --git a/image_analysis/geom_refinement/PostRefine.cpp b/image_analysis/geom_refinement/PostRefine.cpp index f9b0a9a05..a4ba669c7 100644 --- a/image_analysis/geom_refinement/PostRefine.cpp +++ b/image_analysis/geom_refinement/PostRefine.cpp @@ -11,6 +11,7 @@ #include #include #include +#include #include "../../common/JFJochMath.h" // PI #include "XtalResidual.h" // the positional detector<->reciprocal residual, and the cell it is parameterised by @@ -20,7 +21,7 @@ namespace { -// How far post-refinement's step B may move the beam centre away from a value something else already +// How far the joint fit may move the beam centre away from a value something else already // believes. The bound is not about how large a real correction can be - it is the backstop for a fit // corrupted by something every cross-validation fold shares (a second lattice, most often), which the // relative "the held-out residual improved" gate cannot see. The absolute size of the move is what @@ -34,6 +35,14 @@ namespace { // the old bound rejected a fit that improved the held-out positional residual nine-fold. constexpr double BEAM_BOUND_PXL = 15.0; +// How far the joint fit may move a cell ANGLE. Only monoclinic (beta) and triclinic leave one free, +// and those are the two systems whose conditioning is weakest, so the angles need the same absolute +// backstop the lengths and the distance already have rather than being left to the solver's box. The +// largest genuine angle move anywhere on the corpus is 0.35 deg, on a monoclinic beta, measured +// against the same run's own second pass; one degree is about twice that, and still well inside the +// +-2.86 deg box the solver works in, so the box reaches everywhere the gate accepts. +constexpr double ANGLE_BOUND_DEG = 1.0; + // One integrated partial, flattened across all images. Kept as narrow as the sort and the event // split allow: on a large cell this array is gigabytes, and the scatter and every level of the // per-bucket sort move all of it. The goniometer angle is not stored - it is a function of the @@ -545,52 +554,96 @@ PostRefineResult PostRefineRotationGeometry(const std::vector::infinity(); - double len_shift = 0.0; - for (int j = 0; j < 3; ++j) - len_shift = std::max(len_shift, std::fabs(q1_f[j] - p1_0[j]) / std::max(1e-9, p1_0[j])); - const bool in_bounds = std::fabs(ds_f[0] - dist0) < 0.01 * dist0 - && std::min(from_nominal, from_measured) < BEAM_BOUND_PXL - && len_shift < 0.01; - commit = convJ && cv_ref < 0.98 * cv_nom && in_bounds; - if (commit) solve_joint(ALL, beam, dist, axv, p0, p1, p2); // commit: re-fit on all data - // Name which test refused it. Four different things reject here and the geometry that - // comes out is the same in all four, so a run that silently keeps its header geometry + // < ~0.6 %), each cell length < 1 %, each free cell angle < 1 deg (ANGLE_BOUND_DEG), and + // the beam inside the bound measured from whichever centre anything already believes is + // nearer. A larger move is the red flag for an unreliable fit - typically a second + // lattice whose spots bias every cross-validation fold identically, so the relative "it + // improved" gate is blind to it. The absolute size of the move discriminates a genuine + // header correction from that failure far better than the absolute residual, which real + // marginal (noisy / iced) data shares with it. Names the test it failed, or nullptr. + const auto out_of_bounds = [&](const double bm[2], const double ds[1], + const double q1c[3], const double q2c[3]) -> const char * { + if (std::fabs(ds[0] - dist0) >= 0.01 * dist0) + return "the distance moved more than 1 %"; + double len_shift = 0.0, ang_shift = 0.0; + for (int j = 0; j < 3; ++j) { + len_shift = std::max(len_shift, + std::fabs(q1c[j] - p1_0[j]) / std::max(1e-9, p1_0[j])); + ang_shift = std::max(ang_shift, std::fabs(q2c[j] - p2_0[j])); + } + if (len_shift >= 0.01) + return "a cell length moved more than 1 %"; + if (ang_shift >= ANGLE_BOUND_DEG * PI / 180.0) + return "a cell angle moved more than 1 deg"; + const double from_nominal = std::hypot(bm[0] - beam_x0, bm[1] - beam_y0); + const double from_measured = settings.measured_beam_px + ? std::hypot(bm[0] - (*settings.measured_beam_px)[0], + bm[1] - (*settings.measured_beam_px)[1]) + : std::numeric_limits::infinity(); + if (std::min(from_nominal, from_measured) >= BEAM_BOUND_PXL) + return "the beam moved further than the bound from every centre anything believes"; + return nullptr; + }; + + // The two residual families are asked separately as well as together. Pooled, the + // positional values outnumber the excitation ones about three to one where both caps + // saturate, and the excitation residual is the only one that identifies the cell SCALE - + // so a pooled mean can improve on the strength of the positions alone while the one + // quantity the cell is committed for has got worse. Neither may degrade. + const char *refused = + !convJ ? "the fit did not converge" + : !(cv_ref < 0.98 * cv_nom) ? "the held-out residual did not improve enough" + : !(pos_ref < pos_nom) ? "the held-out positional residual did not improve" + : !(exc_ref < exc_nom) ? "the held-out excitation residual did not improve" + : out_of_bounds(bm_f, ds_f, q1_f, q2_f); + + // What the half that was held out earned the right to fit is re-fitted on all of it, and + // that second solve moves the geometry - so the bounds are asked again of what will + // actually be committed, not only of the half that passed the gate. Refused here, the + // run keeps its header geometry exactly as the other refusals leave it. + double ds_log[1] = {ds_f[0]}, bm_log[2] = {bm_f[0], bm_f[1]}; + double q1_log[3], q2_log[3]; + for (int j = 0; j < 3; ++j) { q1_log[j] = q1_f[j]; q2_log[j] = q2_f[j]; } + if (refused == nullptr) { + solve_joint(ALL, beam, dist, axv, p0, p1, p2); // commit: re-fit on all data + ds_log[0] = dist[0]; bm_log[0] = beam[0]; bm_log[1] = beam[1]; + for (int j = 0; j < 3; ++j) { q1_log[j] = p1[j]; q2_log[j] = p2[j]; } + refused = out_of_bounds(beam, dist, p1, p2); + if (refused != nullptr) { + beam[0] = beam_x0; beam[1] = beam_y0; dist[0] = dist0; + for (int j = 0; j < 3; ++j) { + axv[j] = ax0[j]; p0[j] = p0_0[j]; p1[j] = p1_0[j]; p2[j] = p2_0[j]; + } + } + } + commit = (refused == nullptr); + // Name which test refused it. Several different things reject here and the geometry that + // comes out is the same in all of them, so a run that silently keeps its header geometry // says nothing about whether the fit was bad, the improvement too small, or the move too // large for the bound - which is the one case where the number worth reading is the one // that was thrown away. - const char *verdict = - commit ? "COMMIT" - : !convJ ? "reject (the fit did not converge)" - : !(cv_ref < 0.98 * cv_nom) ? "reject (the held-out residual did not improve enough)" - : std::fabs(ds_f[0] - dist0) >= 0.01 * dist0 - ? "reject (the distance moved more than 1 %)" - : len_shift >= 0.01 ? "reject (a cell length moved more than 1 %)" - : "reject (the beam moved further than the bound from " - "every centre anything believes)"; + const std::string verdict = commit ? std::string("COMMIT") + : "reject (" + std::string(refused) + ")"; // The cell the log names is the EFFECTIVE one - what the residual's B matrix builds // from the blocks - not the blocks themselves, whose unused components a high-symmetry - // system leaves at whatever the seed happened to put there. + // system leaves at whatever the seed happened to put there. The geometry beside the + // verdict is the one the verdict is about: the all-data re-fit where it committed, the + // half-data fit where it did not. The residual pair stays the split-half gate's, since + // the fit that is committed has no held-out half of its own. double len_nom[3], ang_nom[3], len_fit[3], ang_fit[3]; EffectiveCellFromParams(sys, p1_0, p2_0, len_nom, ang_nom); - EffectiveCellFromParams(sys, q1_f, q2_f, len_fit, ang_fit); + EffectiveCellFromParams(sys, q1_log, q2_log, len_fit, ang_fit); logger.Info("Post-refine GEOM (joint crystal + detector): dist {:.3f} -> {:.3f} mm, beam " - "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), cell {:.3f} {:.3f} {:.3f} -> " - "{:.3f} {:.3f} {:.3f}, held-out positional {:.3e} -> {:.3e}, excitation " - "{:.3e} -> {:.3e} => {}", - dist0, ds_f[0], beam_x0, beam_y0, bm_f[0], bm_f[1], - len_nom[0], len_nom[1], len_nom[2], len_fit[0], len_fit[1], len_fit[2], + "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), cell {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} " + "{:.2f} -> {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} {:.2f}, held-out positional " + "{:.3e} -> {:.3e}, excitation {:.3e} -> {:.3e} => {}", + dist0, ds_log[0], beam_x0, beam_y0, bm_log[0], bm_log[1], + len_nom[0], len_nom[1], len_nom[2], + ang_nom[0] * 180.0 / PI, ang_nom[1] * 180.0 / PI, ang_nom[2] * 180.0 / PI, + len_fit[0], len_fit[1], len_fit[2], + ang_fit[0] * 180.0 / PI, ang_fit[1] * 180.0 / PI, ang_fit[2] * 180.0 / PI, pos_nom, pos_ref, exc_nom, exc_ref, verdict); } else { logger.Info("Post-refine GEOM: only {} positional observations - the joint fit needs the " diff --git a/image_analysis/geom_refinement/XtalResidual.h b/image_analysis/geom_refinement/XtalResidual.h index 9abcda661..b67e055c8 100644 --- a/image_analysis/geom_refinement/XtalResidual.h +++ b/image_analysis/geom_refinement/XtalResidual.h @@ -12,6 +12,7 @@ #include "gemmi/symmetry.hpp" #include "../../common/JFJochException.h" +#include "../../common/JFJochMath.h" // PI (M_PI is not standard, and MSVC does not define it) #include "../../common/CrystalLattice.h" #include "../../common/DetectorOrientation.h" @@ -82,7 +83,7 @@ struct AngleAxisRotator { // function reads, and kept beside it so the two conventions cannot drift apart. inline void EffectiveCellFromParams(gemmi::CrystalSystem symmetry, const double p1[3], const double p2[3], double lengths[3], double angles_rad[3]) { - const double right = M_PI / 2.0; + const double right = PI / 2.0; lengths[0] = p1[0]; lengths[1] = p1[1]; lengths[2] = p1[2]; @@ -90,7 +91,7 @@ inline void EffectiveCellFromParams(gemmi::CrystalSystem symmetry, const double switch (symmetry) { case gemmi::CrystalSystem::Hexagonal: lengths[1] = p1[0]; - angles_rad[2] = 2.0 * M_PI / 3.0; + angles_rad[2] = 2.0 * PI / 3.0; break; case gemmi::CrystalSystem::Tetragonal: lengths[1] = p1[0]; -- 2.54.0 From 91775cc4fb0c46d9a48309d3860e5a88d92af2fb Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 09:39:38 +0200 Subject: [PATCH 11/75] geometry refinement: at coarse slicing, forgive the miss the exposure could have supplied The acceptance gate that decides which spots enter the geometry refinement compares a fractional-index miss against a fixed tolerance, and that comparison assumes the spot diffracted at the frame's midpoint. It did not. The rotation coordinate is bracketed, not observed - the spot diffracted somewhere inside the exposure - and profiling that unknown angle out of the least squares has a closed form. Writing u = m x q for the direction a rotation delta moves the observation along, the component of the miss along u is free up to the exposure's rms half-width and only the excess is charged; no other direction is touched, so |q|, and with it every d-spacing, is unaffected. Without it the gate is a resolution cut that tightens with the frame width, the miss growing as a/d. Where the benefit lives is the per-frame refinement, which holds one frame and therefore deliberately does not back-rotate - the frame's angle is a gauge its orientation block absorbs - and passes no goniometer at all. The spots on that frame still span the exposure, and that spread does not go away with the gauge, so the spindle and the wedge are handed over separately, for the gate alone. The spindle is also written into the rotation vector the frame constants carry, which was left at the lab x axis whenever no goniometer was passed; with a zero angle that vector is multiplied out and the block is held constant, so nothing changes by it today, but it was a trap for the next reader. The width is k = 1/sqrt(12), the rms of a rotation coordinate uniform over the frame, which is what a least-squares is calibrated on. Half the exposure - the worst case a spot could sit at - degrades accuracy against a ground truth, and so does forgiving the direction outright with no bound: the bound is load-bearing, and what it protects is fine slicing. Measured on 112 paired datasets, scored against deposited structure factors because neither ISa nor R_meas can arbitrate this - both move opposite to accuracy along the neighbouring partiality knob. At 0.5 degrees per image or coarser, n = 16: CC to the deposited data better on 9 of 12, median dISa +0.87%, and at the top of the a x oscillation lever better on 9 of 10 with the per-shell CC improving in all ten shells, most in the outer half. The win rises with that lever and the population below it is inert. It is applied only at 0.5 degrees per image and above. The band is principled - the dead zone matters once the exposure's rms rotation ambiguity, 0.29 of the wedge, is comparable to the crystal's own along-u rocking spread, which measures around 0.26 degrees, putting the boundary between 0.25 and 1.0 degrees - while the point inside that band is empirical and taken at the conservative end. It has to be there: the two datasets the corpus lost outright both sit at 0.2 degrees or finer, one of them finely sliced, and the response is not monotone in the width, so no bound on the width alone makes them safe. Below the trigger the block is skipped and the gate computes what it always computed; a finely sliced sweep reprocessed across this change is byte for byte identical. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 1 + image_analysis/IndexAndRefine.cpp | 8 +++ .../geom_refinement/XtalOptimizer.cpp | 63 +++++++++++++++++-- .../geom_refinement/XtalOptimizer.h | 9 +++ 4 files changed, 77 insertions(+), 4 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 1c5fba125..869b72efa 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -10,6 +10,7 @@ * `rugnux` asks the space-group search again on the cell the lattice metric supports, whenever that metric carries more rotational symmetry than the Bravais class the indexer named, so a lattice whose reduction landed in a lower-symmetry sub-cell can still reach its true point group; the higher symmetry is adopted only where the intensities confirm it. * `rugnux` post-refines the rotation geometry with one joint fit of the crystal and the detector against both the observed spot positions and the observed rocking angles, in place of scaling the cell against the angles and then reading the detector distance off the scaled cell, so the refined distance depends far less on how wrong the file's distance was. * `rugnux` measures the beam centre on every run and indexes with it when the file's value indexes nothing, refines only the detector-tilt component the data determine - a beam-centre error is no longer reported as a tilt - and places a detector swung out on a 2theta arm where the file says it stands. +* `rugnux` accepts a spot into the per-frame geometry refinement where the rotation the exposure could have supplied accounts for the miss, on data collected at 0.5 degrees per image or coarser. * `rugnux` writes the unmerged MTZ by default, with a P1 merge beside it, a batch header for every image the observations span, and events kept to the same `--min-captured-fraction` as the merge, so a wrong space group can be re-merged in a scaling program without reprocessing. * `rugnux` writes reflection files in the conventions downstream programs read: `FreeR_flag` is 0 for the test set and 1 for the working set - it was the other way round - the merged and P1 MTZ carry the reserved `HKL_base` dataset so a CCP4 program reads the wavelength instead of falling back to 1.54187 A, and the merged mmCIF marks the free set as `_refln.status` = `f`. * The rugnux results report carries the space groups the data cannot separate and the enantiomorph state, the model's verdict and what it was allowed to decide, the detector geometry measured and what a single sweep cannot determine, the resolution the CC1/2 fit reached, which reciprocal axis each anisotropic diffraction limit belongs to, and twinning measured before and after the space group was decided; `REPORT_VERSION` is 6, and `SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL` is now `ASSUMED_FROM_MODEL`. diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index 4d4d072f3..98b6f33b5 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -325,6 +325,14 @@ void IndexAndRefine::RefineGeometryIfNeeded(DataMessage &msg, IndexAndRefine::In .max_iterations = real_time ? 0 : OFFLINE_REFINE_ITERATIONS }; + // This refinement holds one frame, so it deliberately does not back-rotate - the frame's angle is + // a gauge the orientation block absorbs, and no `axis` is passed above. The spots on that frame + // still diffracted at different angles across the exposure, though, and that spread does not go + // away with the gauge: hand the rocking geometry over so the acceptance gate can profile it out. + if (const auto &gon = experiment.GetGoniometer(); gon.has_value() && gon->IsScanning()) { + data.rocking_spindle = gon->GetAxis(); + data.rocking_wedge_deg = gon->GetWedge_deg(); + } if (outcome.symmetry.crystal_system == gemmi::CrystalSystem::Trigonal) data.crystal_system = gemmi::CrystalSystem::Hexagonal; diff --git a/image_analysis/geom_refinement/XtalOptimizer.cpp b/image_analysis/geom_refinement/XtalOptimizer.cpp index 55d1756bb..dca95df18 100644 --- a/image_analysis/geom_refinement/XtalOptimizer.cpp +++ b/image_analysis/geom_refinement/XtalOptimizer.cpp @@ -133,6 +133,19 @@ static std::vector SpotConfidenceWeights(const std::vector & return weight; } +// The oscillation width at which the acceptance gate starts profiling out the rotation coordinate. +// The BAND is principled: the dead zone below matters once the exposure's own rms rotation ambiguity, +// wedge/sqrt(12) = 0.29*wedge, is comparable to the crystal's intrinsic along-u rocking spread, which +// measures ~0.26 deg, and that puts the boundary somewhere between 0.25 and 1.0 deg. The POINT is +// empirical and is taken at the conservative end of that band, because fine slicing is the core case +// and coarse slicing is compatibility: below this the gate is left exactly as it was. +constexpr float COARSE_SLICING_WEDGE_DEG = 0.5f; + +// The dead zone's half-width as a fraction of the exposure: the rms of a rotation coordinate uniform +// over the frame, which is the width a least-squares is calibrated on. Half the exposure - the worst +// case a spot could sit at - and forgiving the direction outright were both measured worse. +const double DEAD_ZONE_K = 1.0 / std::sqrt(12.0); + bool XtalOptimizerInternal(XtalOptimizerData &data, std::span> spots, const std::vector> &weights, @@ -205,12 +218,33 @@ bool XtalOptimizerInternal(XtalOptimizerData &data, break; } - if (data.axis) { - rot_vec[0] = data.axis->GetAxis().x; - rot_vec[1] = data.axis->GetAxis().y; - rot_vec[2] = data.axis->GetAxis().z; + // The spindle. `rocking_spindle` is the fallback for a caller that holds one frame and so + // passes no `axis` to back-rotate by: the back-rotation is the identity there either way + // (angle_rad is zero and an AngleAxisRotator of a zero angle-axis ignores the vector, so the + // block is also held constant), but leaving the {1,0,0} initialiser standing would hand any + // later reader of this vector the LAB X AXIS in place of the spindle. + if (const auto spindle = data.axis ? std::optional(data.axis->GetAxis()) : data.rocking_spindle) { + rot_vec[0] = spindle->x; + rot_vec[1] = spindle->y; + rot_vec[2] = spindle->z; } + // The exposure this refinement's spots are spread over, and the spindle they are spread + // along. Taken from the explicit rocking fields where the caller set them - the per-frame + // refinement, which does not back-rotate but whose spots still span an exposure - and + // otherwise from the axis this call does back-rotate by. + const float rocking_wedge_deg = data.rocking_wedge_deg > 0.0f + ? data.rocking_wedge_deg + : ((data.axis && data.axis->IsScanning()) + ? data.axis->GetWedge_deg() : 0.0f); + const Coord rocking_spindle = data.rocking_spindle.value_or( + data.axis ? data.axis->GetAxis() : Coord()); + // Zero everywhere below the trigger, which switches the dead zone off and leaves the gate + // computing the plain fractional-index miss. + const double dead_zone_rad = rocking_wedge_deg >= COARSE_SLICING_WEDGE_DEG + ? rocking_wedge_deg * PI / 180.0 * DEAD_ZONE_K + : 0.0; + const float tolerance_sq = tolerance * tolerance; // The same for every spot of every frame, so taken once here rather than per residual. @@ -272,6 +306,27 @@ bool XtalOptimizerInternal(XtalOptimizerData &data, double norm_sq = (h - h_fp) * (h - h_fp) + (k - k_fp) * (k - k_fp) + (l - l_fp) * (l - l_fp); + // At coarse slicing the spot diffracted somewhere inside the exposure, not at its + // midpoint, and that unknown angle is a real part of the miss. Charge only the part + // of it the exposure cannot supply: a rotation delta about the spindle moves the + // fractional index along u = m x q, so the component of the miss along u is free up + // to the exposure's rms half-width and only the excess counts. Every other direction + // is untouched - |q| among them, so every d-spacing is unaffected. Without this the + // gate is a resolution cut that tightens with the frame width, since the miss grows + // as a/d. + if (dead_zone_rad > 0.0) { + const Coord u = rocking_spindle % recip; + const double u0 = u * vec0, u1 = u * vec1, u2 = u * vec2; + const double u_sq = u0 * u0 + u1 * u1 + u2 * u2; + if (u_sq > 1e-24) { + const double inv_u = 1.0 / std::sqrt(u_sq); + const double d_par = ((h - h_fp) * u0 + (k - k_fp) * u1 + (l - l_fp) * u2) * inv_u; + const double dead = dead_zone_rad * std::sqrt(u_sq); + const double excess = std::max(0.0, std::fabs(d_par) - dead); + norm_sq = std::max(0.0, norm_sq - d_par * d_par) + excess * excess; + } + } + if (norm_sq > tolerance_sq) continue; diff --git a/image_analysis/geom_refinement/XtalOptimizer.h b/image_analysis/geom_refinement/XtalOptimizer.h index bc9ddec0b..cbcc78636 100644 --- a/image_analysis/geom_refinement/XtalOptimizer.h +++ b/image_analysis/geom_refinement/XtalOptimizer.h @@ -44,6 +44,15 @@ struct XtalOptimizerData { std::optional axis; + // The rocking geometry, for the acceptance gate's dead zone alone - NOT for back-rotation. A + // refinement that holds a single frame does not back-rotate (the frame's angle is a gauge its + // orientation block absorbs) and so passes no `axis`, but the spots on that frame still + // diffracted at different angles spanning the exposure, and the gate still has to be told it + // does not know which. Left unset - or a wedge below the coarse-slicing trigger - the gate is + // the plain fractional-index test. + std::optional rocking_spindle; + float rocking_wedge_deg = 0.0f; + // output std::optional beam_corr_x; std::optional beam_corr_y; -- 2.54.0 From 555c686205615bc031ef5688ccc32ed8f955009c Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 09:40:00 +0200 Subject: [PATCH 12/75] merge: bridge a rocking event by an angle rather than by a frame count A rotation reflection arrives as a run of partials and is cut into rocking events wherever two successive frames are more than MAX_FRAME_GAP apart. That constant was 2.0 frames, in five places, and the quantity it stands for is not a frame count at all: it is an angle, a reflecting range of a tenth to half a degree. At a tenth of a degree per image two frames is a fifth of a degree of dead rotation and the bridge does what it was meant to; at a degree per image it is two degrees, as wide as a whole event, and two genuine crossings of the Ewald sphere are joined into one "full". Reconstructing the events on a paired control - the same photons sliced two ways - the fraction of events whose partialities sum past 1.5 is 14.3% at 1.0 degree against 6.0% at 0.1, the fused ones spanning a median of five frames. So the gap becomes half a degree of rotation, floored at one frame so an event is never cut at its own neighbours and capped at the two frames that were always allowed. For any oscillation of 0.25 degrees or less the quotient is at least two and the cap returns the literal 2.0f, so every finely sliced sweep is bridged exactly as before, on both the CPU and the GPU path. The value is taken once per run and carried to the device in CombineParams rather than recomputed in the kernel, so the two paths compare the same float by construction and the bit-parity contract the combine documents is preserved. The defect above is measured; the repair is not. No arm has been run with this change, which is why it is a commit of its own. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 1 + image_analysis/IntegrationOutcome.h | 16 +++++++++++++++- image_analysis/WriteReflections.cpp | 12 +++++++----- image_analysis/geom_refinement/PostRefine.cpp | 4 ++-- .../scale_merge/AnisotropyAnalysis.cpp | 6 +++--- image_analysis/scale_merge/AnisotropyAnalysis.h | 1 + .../scale_merge/RotationScaleMerge.cpp | 9 +++++---- image_analysis/scale_merge/RotationScaleMerge.h | 4 ++++ .../scale_merge/RotationScaleMergeGPU.cu | 11 +++++++---- .../scale_merge/RotationScaleMergeGPU.h | 4 ++-- rugnux/Rugnux.cpp | 4 +++- rugnux/rugnux_cli.cpp | 5 ++++- 12 files changed, 54 insertions(+), 23 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 869b72efa..f2915b50b 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -12,6 +12,7 @@ * `rugnux` measures the beam centre on every run and indexes with it when the file's value indexes nothing, refines only the detector-tilt component the data determine - a beam-centre error is no longer reported as a tilt - and places a detector swung out on a 2theta arm where the file says it stands. * `rugnux` accepts a spot into the per-frame geometry refinement where the rotation the exposure could have supplied accounts for the miss, on data collected at 0.5 degrees per image or coarser. * `rugnux` writes the unmerged MTZ by default, with a P1 merge beside it, a batch header for every image the observations span, and events kept to the same `--min-captured-fraction` as the merge, so a wrong space group can be re-merged in a scaling program without reprocessing. +* `rugnux` decides which partials belong to one rocking event from an angle rather than a frame count, so a coarsely sliced sweep no longer joins two crossings of the Ewald sphere into a single full reflection. * `rugnux` writes reflection files in the conventions downstream programs read: `FreeR_flag` is 0 for the test set and 1 for the working set - it was the other way round - the merged and P1 MTZ carry the reserved `HKL_base` dataset so a CCP4 program reads the wavelength instead of falling back to 1.54187 A, and the merged mmCIF marks the free set as `_refln.status` = `f`. * The rugnux results report carries the space groups the data cannot separate and the enantiomorph state, the model's verdict and what it was allowed to decide, the detector geometry measured and what a single sweep cannot determine, the resolution the CC1/2 fit reached, which reciprocal axis each anisotropic diffraction limit belongs to, and twinning measured before and after the space group was decided; `REPORT_VERSION` is 6, and `SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL` is now `ASSUMED_FROM_MODEL`. * `rugnux --mode calibration` writes `.json` beside the `.poni`, holding the geometry as a `jfjoch_broker` `dataset_settings` body, and refuses a fit that is not a measurement - no `.poni`, a non-zero exit, `converged` recorded in the `.json`; `--no-refine-tilt` holds the detector tilt at the file's value instead of zeroing it. diff --git a/image_analysis/IntegrationOutcome.h b/image_analysis/IntegrationOutcome.h index 75b2a09cd..c0d8855fa 100644 --- a/image_analysis/IntegrationOutcome.h +++ b/image_analysis/IntegrationOutcome.h @@ -3,6 +3,8 @@ #pragma once +#include + #include "../common/Reflection.h" #include "../common/CrystalLattice.h" #include "../common/DiffractionGeometry.h" @@ -16,4 +18,16 @@ struct IntegrationOutcome { std::optional image_scale_cc_n; std::optional image_scale_g; std::optional image_scale_wedge_deg; -}; \ No newline at end of file +}; + +// How far apart, in frames, two partials of one raw hkl may sit and still belong to the same rocking +// event. The quantity being bridged is an ANGLE - a reflecting range, a tenth to half a degree - so +// spelling it as a frame count makes the bridge grow with the slicing: at 1 deg per frame the two +// frames that were always allowed are 2 deg of dead rotation, as wide as a whole event, and two +// genuine Ewald crossings fuse into one "full". Half a degree of bridge instead, floored at one frame +// (never cut an event at its own neighbours) and capped at the two frames that were always used. For +// any wedge of 0.25 deg or less the quotient is at least two and the cap returns the literal 2.0f, so +// finely sliced data is bridged exactly as before. +inline float RockingEventFrameGap(float wedge_deg) { + return std::min(2.0f, std::max(1.0f, 0.5f / wedge_deg)); +} \ No newline at end of file diff --git a/image_analysis/WriteReflections.cpp b/image_analysis/WriteReflections.cpp index 8332288e4..e14cc2a4b 100644 --- a/image_analysis/WriteReflections.cpp +++ b/image_analysis/WriteReflections.cpp @@ -598,7 +598,7 @@ float DetectorY(const Reflection &r) { return std::isfinite(r.observed_y) ? r.ob // Sum each rocking event into one full observation. A rotation reflection is integrated image by // image, so it arrives here as a run of partials over consecutive frames; the run is cut where the -// 3D combine cuts it - same raw hkl, frames no further apart than MAX_FRAME_GAP - so the exported +// 3D combine cuts it - same raw hkl, frames no further apart than RockingEventFrameGap - so the exported // file and rugnux's own merge see exactly the same events. // The parts are added, plainly, with their variances in quadrature, which is what every other // rotation program writes as a full. Nothing is divided by the partiality: FRACTIONCALC carries the @@ -630,8 +630,9 @@ float DetectorY(const Reflection &r) { return std::isfinite(r.observed_y) ? r.ob // mosaicity, RotationScaleMerge::SmoothMosaicity), so clamping it to 1 would hide the spread and buy // a reading program nothing. std::vector SumRockingEvents(const std::vector &outcomes, - double min_partiality, double min_captured_fraction) { - constexpr float MAX_FRAME_GAP = 2.0f; // == RotationScaleMerge's: what makes one rocking event + double min_partiality, double min_captured_fraction, + float wedge_deg) { + const float max_frame_gap = RockingEventFrameGap(wedge_deg); // == RotationScaleMerge's // The sort key travels with the part instead of being read back through the pointer, the way the // merge's own ingest sort carries it (RotationScaleMerge's SortKey): there are millions of parts @@ -661,7 +662,7 @@ std::vector SumRockingEvents(const std::vector & size_t j = i + 1; while (j < parts.size() && parts[j].h == parts[i].h && parts[j].k == parts[i].k && parts[j].l == parts[i].l - && parts[j].image_number - parts[j - 1].image_number <= MAX_FRAME_GAP) + && parts[j].image_number - parts[j - 1].image_number <= max_frame_gap) ++j; double sum_p = 0.0, sum_I = 0.0, sum_var = 0.0, sum_var_bkg = 0.0; @@ -815,7 +816,8 @@ void WriteUnmergedMtzReflections(const std::vector &outcomes if (scanning && sum_partials) { for (const auto &r : SumRockingEvents(outcomes, experiment.GetScalingSettings().GetMinPartiality(), - experiment.GetScalingSettings().GetMinCapturedFraction())) + experiment.GetScalingSettings().GetMinCapturedFraction(), + wedge_deg)) add_row(r); } else { for (const auto &outcome : outcomes) diff --git a/image_analysis/geom_refinement/PostRefine.cpp b/image_analysis/geom_refinement/PostRefine.cpp index a4ba669c7..6ef913795 100644 --- a/image_analysis/geom_refinement/PostRefine.cpp +++ b/image_analysis/geom_refinement/PostRefine.cpp @@ -260,11 +260,11 @@ PostRefineResult PostRefineRotationGeometry(const std::vector=2-frame events carry an // unbiased phi_obs (a single-frame centroid is just the frame centre). - constexpr float MAX_FRAME_GAP = 2.0f; + const float max_frame_gap = RockingEventFrameGap(axis.GetWedge_deg()); const auto run_end = [&](size_t i, size_t end) { size_t j = i + 1; while (j < end && pts[j].h == pts[i].h && pts[j].k == pts[i].k && pts[j].l == pts[i].l - && pts[j].img - pts[j - 1].img <= MAX_FRAME_GAP) + && pts[j].img - pts[j - 1].img <= max_frame_gap) ++j; return j; }; diff --git a/image_analysis/scale_merge/AnisotropyAnalysis.cpp b/image_analysis/scale_merge/AnisotropyAnalysis.cpp index 893389bf1..c490f11dd 100644 --- a/image_analysis/scale_merge/AnisotropyAnalysis.cpp +++ b/image_analysis/scale_merge/AnisotropyAnalysis.cpp @@ -1029,7 +1029,7 @@ namespace { std::vector ScaledObservations(const std::vector &outcomes, bool rotation, const gemmi::SpaceGroup *space_group, - double min_partiality) { + float wedge_deg, double min_partiality) { // Per-image scale, indexed the way the outcomes are. std::vector g(outcomes.size(), 0.0); for (size_t i = 0; i < outcomes.size(); ++i) @@ -1070,7 +1070,7 @@ std::vector ScaledObservations(const std::vector ScaledObservations(const std::vector ScaledObservations(const std::vector &outcomes, bool rotation, const gemmi::SpaceGroup *space_group = nullptr, + float wedge_deg = 0.0f, double min_partiality = 0.5); // What the caller knows about the run and the merge that the reflections alone do not say. diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 0d879ddd9..f61da9be2 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -91,7 +91,6 @@ namespace { // guarantees the modulation costs data is the CC dip) constexpr double SWEEP_HARMONIC_CC_DIP = 0.8; // ... and cost real signal at its trough - constexpr float MAX_FRAME_GAP = 2.0f; // a rocking event is a run of frames no more apart than this constexpr double CHI2_1_MEDIAN = 0.454936; // A post-scale-fulls correction surface (decay / absorption) is applied only if its held-out // cross-validation improvement exceeds this fraction of the held-out scatter. A margin (not just >0) @@ -228,6 +227,8 @@ RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment, merge_friedel = s.GetMergeFriedel(); capture_uncertainty_coeff = s.GetCaptureUncertaintyCoeff(); min_captured_fraction = s.GetMinCapturedFraction(); + if (const auto gon = x.GetGoniometer(); gon.has_value()) + max_frame_gap = RockingEventFrameGap(gon->GetWedge_deg()); min_cc_for_image = s.GetMinCCForImage(); search_min_zeta = s.GetSearchMinZeta(); reject_nsigma = s.GetOutlierRejectNsigma(); @@ -874,7 +875,7 @@ void RotationScaleMerge::SmoothMosaicityAndPartiality() { while (i < hi) { int j = i + 1; while (j < hi && partials[perm[j]].image_number - - partials[perm[j - 1]].image_number <= MAX_FRAME_GAP) + - partials[perm[j - 1]].image_number <= max_frame_gap) ++j; if (j - i >= 2) { const float f0 = partials[perm[i]].image_number; @@ -2458,7 +2459,7 @@ void RotationScaleMerge::Combine() { float last_frame = partials[ev[i]].image_number; while (kk < ev.size()) { const float frame = partials[ev[kk]].image_number; - if (frame - last_frame > MAX_FRAME_GAP) break; + if (frame - last_frame > max_frame_gap) break; last_frame = frame; ++kk; } @@ -3850,7 +3851,7 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st if (gpu_active_ && observation_dump_path.empty()) { // The smoothed corr is already resident (scaling + smooth-G ran on the device, no round-trip). const int nf = gpu_->Combine(rawrun_group.data(), min_partiality, capture_uncertainty_coeff, - min_captured_fraction); + min_captured_fraction, max_frame_gap); g_full.assign(n_frames, 1.0); if (scale_fulls && nf > 0) { diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index 3ee86b5ee..13e78a5c9 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -154,6 +154,10 @@ private: bool merge_friedel = true; double capture_uncertainty_coeff = 0.0; double min_captured_fraction = 0.0; + // RockingEventFrameGap of this run's oscillation width, taken once so that the two places that + // cut events on the CPU and the GPU kernel that cuts them on the device all compare the same + // float (RotationScaleMergeGPU.cu documents that bit-parity contract). + float max_frame_gap = 2.0f; // Drop a frame's observations entirely when the frame disagrees with the merged reference below this // correlation (--min-image-cc). A mis-centred or off-crystal frame still produces spots, still diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.cu b/image_analysis/scale_merge/RotationScaleMergeGPU.cu index 8b778ec7d..e6b2821f8 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.cu +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.cu @@ -275,8 +275,6 @@ namespace { __device__ __forceinline__ double Dmin(double a, double b) { return (b < a) ? b : a; } __device__ __forceinline__ float Fmax(float a, float b) { return (a < b) ? b : a; } - constexpr float COMBINE_MAX_FRAME_GAP = 2.0f; // == RotationScaleMerge::MAX_FRAME_GAP - // A partial is usable for the combine iff its corr and (I, sigma) are finite with corr>0, sigma>0. __device__ __forceinline__ bool CombineUsable(int i, const float *I, const float *sigma, const float *corr) { @@ -289,6 +287,9 @@ namespace { struct CombineParams { int n_runs; double min_partiality, capture_uncertainty_coeff, min_captured_fraction; + // RotationScaleMerge::max_frame_gap, passed in rather than recomputed here so that the host + // and the device compare the same float and the combine stays bit-identical on both paths. + float max_frame_gap; const float *__restrict__ I, *__restrict__ sigma, *__restrict__ corr, *__restrict__ partiality, *__restrict__ bkg, *__restrict__ var_bkg, *__restrict__ image_number, *__restrict__ d, *__restrict__ px, *__restrict__ py; @@ -328,7 +329,7 @@ namespace { while (probe < hi && !CombineUsable(p.perm[probe], p.I, p.sigma, p.corr)) ++probe; if (probe >= hi) break; const float img = p.image_number[p.perm[probe]]; - if (img - last_img > COMBINE_MAX_FRAME_GAP) break; + if (img - last_img > p.max_frame_gap) break; last_img = img; ev_end = probe; ++probe; @@ -1024,7 +1025,8 @@ void RotationScaleMergeGPU::SetRawRuns(int n_runs, int n_perm, const int32_t *pe } int RotationScaleMergeGPU::Combine(const int32_t *rawrun_group, double min_partiality, - double capture_uncertainty_coeff, double min_captured_fraction) { + double capture_uncertainty_coeff, double min_captured_fraction, + float max_frame_gap) { DeviceGuard guard(impl_->device, impl_->available); auto &d = *impl_; CudaCheck(cudaMemcpy(d.rr_group.get(), rawrun_group, size_t(d.n_runs) * sizeof(int32_t), @@ -1035,6 +1037,7 @@ int RotationScaleMergeGPU::Combine(const int32_t *rawrun_group, double min_parti p.min_partiality = min_partiality; p.capture_uncertainty_coeff = capture_uncertainty_coeff; p.min_captured_fraction = min_captured_fraction; + p.max_frame_gap = max_frame_gap; p.I = d.I.get(); p.sigma = d.sigma.get(); p.corr = d.corr.get(); p.partiality = d.partiality.get(); p.bkg = d.bkg.get(); p.var_bkg = d.var_bkg.get(); p.image_number = d.image_number.get(); p.d = d.d_obs.get(); p.px = d.px_obs.get(); p.py = d.py_obs.get(); diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.h b/image_analysis/scale_merge/RotationScaleMergeGPU.h index e821ae916..50bf0c23f 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.h +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.h @@ -118,13 +118,13 @@ public: // Combine the resident partials (reading the current resident corr) into fulls on the device, // mirroring RotationScaleMerge::Combine: one thread per raw-hkl run splits its usable partials into - // rocking events (frame gap <= 2), pools background, seeds F, does 3 de-biased Poisson reweights and + // rocking events (frame gap <= max_frame_gap, the host's RockingEventFrameGap), pools background, seeds F, does 3 de-biased Poisson reweights and // adds the capture-uncertainty term. rawrun_group (length n_runs) is the current space group's ASU // id per raw hkl (it becomes the full's group). Deterministic: fulls are emitted in raw-run-major, // event order (a count pass -> host prefix sum -> emit-at-offset), matching the CPU path. Returns the // number of fulls (call GetFulls with buffers of that length). int Combine(const int32_t *rawrun_group, double min_partiality, double capture_uncertainty_coeff, - double min_captured_fraction); + double min_captured_fraction, float max_frame_gap); // Download the combined fulls SoA (length = Combine()'s return). The working corr is downloaded // separately by GetFullsCorr (it is only meaningful after ScaleFulls; otherwise the caller sets it). diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index d2f717c31..72a07c51e 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -4810,7 +4810,9 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b sm.statistics.anisotropy = AnalyzeAnisotropy( sm.merged, ScaledObservations(indexer->GetIntegrationOutcome(), - experiment_.IsRotationIndexing(), twin_sg), + experiment_.IsRotationIndexing(), twin_sg, + experiment_.GetGoniometer() + ? experiment_.GetGoniometer()->GetWedge_deg() : 0.0f), *result.consensus_cell, twin_sg, aniso_run); stats_text << AnisotropyToText(sm.statistics.anisotropy) << "\n"; } diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index d672d9a80..a403ca728 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -1770,7 +1770,10 @@ static int RunRugnux(int argc, char **argv) { aniso_run.dose_term_in_scale_model = experiment.GetScalingSettings().GetCorrectionSurfaces(); aniso_run.radiation_damage_relative_b = merged_statistics.radiation_damage_delta_b; merged_statistics.anisotropy = AnalyzeAnisotropy(merged_reflections, - ScaledObservations(reflections, is_rotation, twin_sg), + ScaledObservations(reflections, is_rotation, twin_sg, + experiment.GetGoniometer() + ? experiment.GetGoniometer()->GetWedge_deg() + : 0.0f), *experiment.GetUnitCell(), twin_sg, aniso_run); std::cout << AnisotropyToText(merged_statistics.anisotropy) << std::endl; } -- 2.54.0 From 441b802e0c8a7a2210fea4e3613a655e1f6c6424 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 07:22:22 +0200 Subject: [PATCH 13/75] docs: a large spindle-to-symmetry-axis angle does not clear the mounting The paragraph introduced with the two new report keys said an incomplete cusp could be "attributed to the mounting or cleared of it". The first half holds; the second does not. A symmetry axis within theta_max of PERPENDICULAR to the spindle carries the blind cone onto its opposite lobe, which a sweep leaves equally unmeasured, so a large angle is not on its own evidence that the mounting was harmless - and the key reports only the nearest axis. Also record the two keys in the changelog, and correct REPORT_VERSION there: the entry still said 6 after the bump to 7. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 3 ++- docs/RUGNUX_REPORT.md | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index f2915b50b..385c3ef00 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -14,7 +14,8 @@ * `rugnux` writes the unmerged MTZ by default, with a P1 merge beside it, a batch header for every image the observations span, and events kept to the same `--min-captured-fraction` as the merge, so a wrong space group can be re-merged in a scaling program without reprocessing. * `rugnux` decides which partials belong to one rocking event from an angle rather than a frame count, so a coarsely sliced sweep no longer joins two crossings of the Ewald sphere into a single full reflection. * `rugnux` writes reflection files in the conventions downstream programs read: `FreeR_flag` is 0 for the test set and 1 for the working set - it was the other way round - the merged and P1 MTZ carry the reserved `HKL_base` dataset so a CCP4 program reads the wavelength instead of falling back to 1.54187 A, and the merged mmCIF marks the free set as `_refln.status` = `f`. -* The rugnux results report carries the space groups the data cannot separate and the enantiomorph state, the model's verdict and what it was allowed to decide, the detector geometry measured and what a single sweep cannot determine, the resolution the CC1/2 fit reached, which reciprocal axis each anisotropic diffraction limit belongs to, and twinning measured before and after the space group was decided; `REPORT_VERSION` is 6, and `SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL` is now `ASSUMED_FROM_MODEL`. +* The rugnux results report carries the space groups the data cannot separate and the enantiomorph state, the model's verdict and what it was allowed to decide, the detector geometry measured and what a single sweep cannot determine, the resolution the CC1/2 fit reached, which reciprocal axis each anisotropic diffraction limit belongs to, and twinning measured before and after the space group was decided; `REPORT_VERSION` is 7, and `SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL` is now `ASSUMED_FROM_MODEL`. +* The rugnux results report records how the crystal sat on the goniometer as `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` and `SPINDLE_SYMMETRY_AXIS_ORDER=` - the angle from the spindle to the nearest symmetry axis and that axis's order - on every rotation run that determined a space group, not only when the angle was small enough to warn about. * `rugnux --mode calibration` writes `.json` beside the `.poni`, holding the geometry as a `jfjoch_broker` `dataset_settings` body, and refuses a fit that is not a measurement - no `.poni`, a non-zero exit, `converged` recorded in the `.json`; `--no-refine-tilt` holds the detector tilt at the file's value instead of zeroing it. * A snake grid scan with a negative slow step and an even number of rows no longer has its positions mirrored along the fast axis in the HDF5 master and the grid map, so the positions recorded for that configuration change; `jfjoch_viewer` draws grid scan cells in the proportion of the scan steps, labels the merge-statistics plot over the range the axis is drawn on, and builds its powder-calibration ring list from the loaded dataset's space group as well as its cell, so a centred sample cell no longer scales the whole fit. * The HDF5 master records `direct_beam_x`/`direct_beam_y` - where the undeflected beam lands, sent on the CBOR start message too - the beam size at the sample as `incident_beam_size` from the new `dataset_settings` `beam_size_x_um`/`beam_size_y_um`, and `/entry/MX/peakCountUnfiltered`; `dataset_settings` accepts any `smargon.chi_deg`, which was restricted to 0-90 degrees. diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 2d16af421..9f047eccc 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -246,8 +246,10 @@ sat on the goniometer: the angle between the spindle and the nearest symmetry ax order. Below about 15 deg the axis maps the sweep's blind cone onto itself and the reflections in that cone stay missing however long the sweep runs, which is why the same number also raises a warning there. It is written on every rotation run that determined a space group, including the -ordinary well-mounted case, so an incomplete cusp can be attributed to the mounting or cleared of it. -(New in `REPORT_VERSION= 7`.) +ordinary well-mounted case, so an incomplete cusp can be attributed to the mounting. A large angle +does not on its own clear it: an axis within the same margin of perpendicular to the spindle carries +the blind cone onto its opposite half, which is equally unmeasured, and only the nearest axis is +reported. (New in `REPORT_VERSION= 7`.) `SPACE_GROUP_ENANTIOMORPH=` in section 4 reads **`ASSUMED_FROM_MODEL`** when the hand written in the files is the model's. Assumed, not determined: merged intensities cannot see the hand at all — |F| is -- 2.54.0 From c477cf0be39e815030a05b833778f29d425399d2 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 07:30:56 +0200 Subject: [PATCH 14/75] spindle: a row perpendicular to the axis is as blind as a row on it The blind region a sweep leaves is a double cone, so a 2-fold sends it to two places: 2*beta away, and 180-2*beta away. The severity took only the first, and scored a row perpendicular to the spindle as 0 - "symmetry repairs everything" - when such a 2-fold in fact carries the cone onto its opposite lobe, which the sweep leaves equally unmeasured. Folding the miss-angle to min(beta, 90-beta) covers both images and reproduces a Monte-Carlo of the true overlap to 0.002. The failure was silent and in the dangerous direction, and it fired on the more common geometry: for a random axis the perpendicular band is several times wider than the aligned one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/IndexAndRefine.cpp | 13 ++++++-- image_analysis/indexing/FFTIndexer.cpp | 12 +++++-- .../indexing/SpindleBlindFraction.cpp | 10 +++++- .../indexing/SpindleBlindFraction.h | 27 ++++++++++------ tests/SpindleBlindFractionTest.cpp | 31 ++++++++++++++----- 5 files changed, 69 insertions(+), 24 deletions(-) diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index 98b6f33b5..1c95af4d8 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -155,6 +155,7 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data const float idx_tol_sq = idx_tol * idx_tol; constexpr float SEED_STOP_FRACTION = 0.9f; // seed explained this well -> stop escalating IndexerResult indexer_result; + std::optional spindle_blind_fraction; bool any_executed = false; float best_frac = -1.0f; // Read per call rather than cached at construction: the run measures for itself whether the @@ -172,6 +173,14 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data } auto res = indexer_->Run(experiment, recip); any_executed |= res.executed; + // Kept from whichever run could measure it - the largest seed that answered - rather than from + // the run whose lattice won: the severity describes the frame's lattice rows, not the cell that + // closed on them, so a frame that indexed nothing still has one. It stays absent while the + // escalation never leaves the lean seed, which is below the spot floor the score needs + // (SPINDLE_MIN_SPOTS in FFTIndexer.cpp) - so on frames that index cleanly at 30 spots there is + // no value at all. That is a gap, not a "no problem": see the note at SPINDLE_MIN_SPOTS. + if (res.spindle_blind_fraction) + spindle_blind_fraction = res.spindle_blind_fraction; if (!res.lattice.empty()) { // Keep the seed the lattice explains the largest FRACTION of: a lean clean seed a good // lattice indexes almost fully beats a flooded seed it fits only in small part. This @@ -202,9 +211,7 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data if (any_executed) msg.indexing_result = false; - // The severity is a property of the frame's lattice rows, not of a cell having closed, so it is - // reported whether or not the frame indexed. - msg.spindle_blind_fraction = indexer_result.spindle_blind_fraction; + msg.spindle_blind_fraction = spindle_blind_fraction; if (!indexer_result.lattice.empty()) { auto latt = indexer_result.lattice[0]; diff --git a/image_analysis/indexing/FFTIndexer.cpp b/image_analysis/indexing/FFTIndexer.cpp index 4cbbfd26b..64bbac844 100644 --- a/image_analysis/indexing/FFTIndexer.cpp +++ b/image_analysis/indexing/FFTIndexer.cpp @@ -485,9 +485,15 @@ std::vector FFTIndexer::RunInternal(const std::vector &co // Free ride on that shortlist: how much of a single sweep's blind cone this orientation makes // unrecoverable. Needs the spindle, the wavelength and the frame's own resolution, all of which // are already here; costs one pass over 30-odd rows. - // Below this the row list is no longer trustworthy: measured over 22 solved crystals, the - // fraction of severe orientations reported as harmless falls from 0.22 at 30 spots to 0.05 at 60, - // and flat above. Fewer spots means no value, not a middling one. + // Below this the row list is no longer trustworthy: measured over solved crystals, the fraction of + // severe orientations reported as harmless falls from 0.22 at 30 spots to 0.05 at 60, and is flat + // above. Fewer spots means no value, not a middling one. + // The floor was calibrated on a pass over a frame's WHOLE spot list, but nspots here is whatever + // the caller fed this call. IndexAndRefine escalates 30 -> 80 -> all and usually stops at 30, so + // on a frame that indexes cleanly the score is simply never computed. Closing that gap means + // measuring the severity on the full list rather than riding on the indexing seed, which costs a + // pass this deliberately does not spend; until then the value is absent far more often than the + // floor alone implies. constexpr size_t SPINDLE_MIN_SPOTS = 60; spindle_severity = {}; if (spindle_axis && wavelength_A > 0 && nspots >= SPINDLE_MIN_SPOTS) { diff --git a/image_analysis/indexing/SpindleBlindFraction.cpp b/image_analysis/indexing/SpindleBlindFraction.cpp index f969674d1..abbfce535 100644 --- a/image_analysis/indexing/SpindleBlindFraction.cpp +++ b/image_analysis/indexing/SpindleBlindFraction.cpp @@ -76,7 +76,15 @@ std::optional SpindleBlindFraction(const std::vector &ro continue; const float cos_beta = std::min(1.0f, std::fabs(rows[i] * axis) / length); const float beta_deg = static_cast(std::acos(cos_beta) * 180.0 / M_PI); - const float score = BlindConeSelfOverlap(beta_deg / theta_max_deg); + // A row PERPENDICULAR to the spindle is as damaging as one along it, and far more common: a + // 2-fold about it carries the blind cone onto the cone's opposite lobe, which the same sweep + // leaves equally unmeasured. That is Friedel's rescue, which is no rescue - the cone is + // double-sided. Both ends of the range are the bad case and the safe zone lies between them, + // so the miss-angle is folded about 45 deg. Checked against a Monte-Carlo of the true + // spherical overlap the folded form is within 0.006 to theta_max = 20 deg and 0.024 to 45 deg; + // unfolded it is wrong by a full 1.0 at beta = 90 deg, reporting the worst case as the best. + const float fold_deg = std::min(beta_deg, 90.0f - beta_deg); + const float score = BlindConeSelfOverlap(fold_deg / theta_max_deg); if (score > ret.score || ret.row_length_A == 0) { ret.score = score; ret.row_length_A = length; diff --git a/image_analysis/indexing/SpindleBlindFraction.h b/image_analysis/indexing/SpindleBlindFraction.h index 2d98d2690..55f321d46 100644 --- a/image_analysis/indexing/SpindleBlindFraction.h +++ b/image_analysis/indexing/SpindleBlindFraction.h @@ -18,16 +18,25 @@ // still cannot know the point group, but it can measure where the crystal's short lattice rows // are, and a symmetry axis is always one of them. // -// Score = the fraction of the blind cone that a 2-fold about the nearest short row carries back -// into the blind cone: two equal caps of angular radius theta_max whose centres are 2*beta apart. -// 0 means any operator about that row moves the cone entirely off itself and one sweep loses -// nothing symmetry could have given; 1 means the row is on the spindle and the whole cone is -// lost. Nothing about the goniometer enters, so the number describes the problem and leaves the -// remedy - a chi offset, a second sweep, accepting the loss - to the beamline. +// Score = the fraction of the blind cone that a 2-fold about the worst plausible short row carries +// back into the blind cone: two equal caps of angular radius theta_max whose centres are 2*beta +// apart. 0 means any operator about that row moves the cone entirely off itself and one sweep loses +// nothing symmetry could have given; 1 means the whole cone is lost. Nothing about the goniometer +// enters, so the number describes the problem and leaves the remedy - a chi offset, a second sweep, +// accepting the loss - to the beamline. +// +// BOTH ends of the miss-angle range are the bad case. A row ON the spindle maps the cone onto itself; +// a row PERPENDICULAR to the spindle maps it onto the cone's opposite lobe, which is equally +// unmeasured, so that is no rescue either - and being a band rather than a cap it is the commoner of +// the two. The safe zone is the middle, and the miss-angle is folded about 45 deg accordingly. -// Fraction of a cap of angular radius theta that its own image under a 2-fold at beta covers. -// x = beta / theta_max. Exact in the flat limit and within 0.035 of the spherical value even at -// theta_max = 55 deg (the widest cone a 3.3 A beam produces), so the closed form is used as is. +// Fraction of a cap of angular radius theta that its own image under a 2-fold covers, as a function +// of x = (folded miss-angle) / theta_max; the caller does the folding. Exact in the flat limit. +// Checked against a Monte-Carlo of the true spherical overlap the folded score is within 0.006 up to +// theta_max = 20 deg and within 0.024 up to 45 deg, so the closed form is used as is. Only past +// 45 deg - which needs lambda > 1.41 * d_min, so a long-wavelength beamline - does the sphere's +// curvature start to tell, and there it UNDER-reports, by 0.05 at theta_max = 50 deg and 0.13 at +// 55 deg. There the score is a lower bound on the loss rather than an estimate of it. float BlindConeSelfOverlap(float x); struct SpindleSeverity { diff --git a/tests/SpindleBlindFractionTest.cpp b/tests/SpindleBlindFractionTest.cpp index d53d71aed..02dce82ec 100644 --- a/tests/SpindleBlindFractionTest.cpp +++ b/tests/SpindleBlindFractionTest.cpp @@ -9,7 +9,8 @@ using Catch::Matchers::WithinAbs; TEST_CASE("SpindleBlindFraction_Overlap", "[Indexing][Spindle]") { - // A row on the spindle leaves the whole cone unrecoverable; a row on the cone edge leaves none. + // x is the FOLDED miss-angle over theta_max: 0 both on the spindle and perpendicular to it. + // A row at either end leaves the whole cone unrecoverable; one on the cone edge leaves none. CHECK_THAT(BlindConeSelfOverlap(0.0f), WithinAbs(1.0f, 1e-6)); CHECK_THAT(BlindConeSelfOverlap(1.0f), WithinAbs(0.0f, 1e-6)); CHECK_THAT(BlindConeSelfOverlap(2.0f), WithinAbs(0.0f, 1e-6)); @@ -34,8 +35,21 @@ TEST_CASE("SpindleBlindFraction_Rows", "[Indexing][Spindle]") { CHECK_THAT(s->row_length_A, WithinAbs(50.0f, 1e-3)); } - SECTION("no short row inside the cone scores zero") { - const std::vector rows = {Coord(50, 0, 0), Coord(0, 60, 0), Coord(0, 60, 25)}; + SECTION("a row perpendicular to the spindle is the worst case too") { + // The 2-fold about it carries the blind cone onto the cone's opposite lobe, which the sweep + // leaves equally unmeasured. Reporting this as harmless was the bug the fold fixes. + const std::vector rows = {Coord(50, 0, 0), Coord(0, 60, 0), Coord(0, 0, 70)}; + const std::vector mag = {100, 90, 80}; + const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + CHECK_THAT(s->miss_angle_deg, WithinAbs(90.0f, 1e-3)); + } + + SECTION("no short row near either end of the range scores zero") { + // Rows well away from both the spindle and its perpendicular plane: any 2-fold about them + // swings the cone clear of itself, and one sweep loses nothing symmetry could have returned. + const std::vector rows = {Coord(50, 0, 50), Coord(0, 60, 60), Coord(40, 40, 56)}; const std::vector mag = {100, 90, 80}; const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); REQUIRE(s.has_value()); @@ -43,9 +57,9 @@ TEST_CASE("SpindleBlindFraction_Rows", "[Indexing][Spindle]") { } SECTION("a row too long to be a symmetry axis is ignored") { - // 300 A along the spindle, in a crystal whose own rows are 50-70 A: 6x the shortest row is + // 300 A along the spindle, in a crystal whose own rows are 70-85 A: 4x the shortest row is // not a plausible symmetry axis, and the cone it sits in is not the crystal's problem. - const std::vector rows = {Coord(50, 0, 0), Coord(0, 60, 0), Coord(0, 0, 300)}; + const std::vector rows = {Coord(50, 0, 50), Coord(0, 60, 60), Coord(0, 0, 300)}; const std::vector mag = {100, 90, 80}; const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); REQUIRE(s.has_value()); @@ -53,17 +67,18 @@ TEST_CASE("SpindleBlindFraction_Rows", "[Indexing][Spindle]") { } SECTION("the same row in a crystal that IS that big is not ignored") { - const std::vector rows = {Coord(250, 0, 0), Coord(0, 280, 0), Coord(0, 0, 300)}; + const std::vector rows = {Coord(180, 0, 180), Coord(0, 200, 200), Coord(0, 0, 300)}; const std::vector mag = {100, 90, 80}; const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); REQUIRE(s.has_value()); CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + CHECK_THAT(s->miss_angle_deg, WithinAbs(0.0f, 1e-3)); } SECTION("a weak spurious short row does not shrink the length window") { // What a long-cell still produces: the real rows near 300 A plus a weaker short peak. Taking // the window off that peak would hide the aligned row and report the orientation harmless. - const std::vector rows = {Coord(100, 5, 0), Coord(250, 0, 0), Coord(0, 0, 300)}; + const std::vector rows = {Coord(70, 70, 30), Coord(180, 0, 180), Coord(0, 0, 300)}; const std::vector mag = {40, 100, 95}; const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); REQUIRE(s.has_value()); @@ -84,7 +99,7 @@ TEST_CASE("SpindleBlindFraction_Rows", "[Indexing][Spindle]") { } SECTION("a wider cone at long wavelength makes the same miss-angle worse") { - const std::vector rows = {Coord(0, 20, 50), Coord(60, 0, 0)}; + const std::vector rows = {Coord(0, 20, 50), Coord(50, 0, 50)}; const std::vector mag = {100, 90}; const auto narrow = SpindleBlindFraction(rows, mag, spindle, 10.0f); const auto wide = SpindleBlindFraction(rows, mag, spindle, 35.0f); -- 2.54.0 From f5c2720b022c055e33b98dcc52f4123405511ce8 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 09:53:38 +0200 Subject: [PATCH 15/75] spindle: the cone is the sweep's, so its width comes from the detector, not the still The severity took theta_max from the still's own spot resolution - the 95th percentile of |q| - so a weak, attenuated grid still reaching 3 A scored a row at an 11 deg miss-angle as exactly 0, while the real sweep collected afterwards at the detector's 1.8 A loses a fifth of a cone that wide. The number exists to decide whether that sweep needs a second orientation, so the cone it must describe is the sweep's, and the still's spot list systematically understates it in the one direction that produces silent misses. theta_max is now asin(lambda/2d) at the geometric resolution of the setup - the detector corner at the recorded distance and wavelength. That is an upper bound on any sweep collected without moving the detector, it is available on every frame however weak, and where it overstates what the sweep will reach it errs towards reporting loss, which is the cheap error. Computed once per Setup in the Indexer base rather than per frame in the FFT implementation, because nothing about it is specific to either the frame or the indexer. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/indexing/FFTIndexer.cpp | 34 ++++--------------- image_analysis/indexing/Indexer.cpp | 1 + image_analysis/indexing/Indexer.h | 9 +++++ .../indexing/SpindleBlindFraction.cpp | 7 ++++ .../indexing/SpindleBlindFraction.h | 13 +++++++ 5 files changed, 37 insertions(+), 27 deletions(-) diff --git a/image_analysis/indexing/FFTIndexer.cpp b/image_analysis/indexing/FFTIndexer.cpp index 64bbac844..773183545 100644 --- a/image_analysis/indexing/FFTIndexer.cpp +++ b/image_analysis/indexing/FFTIndexer.cpp @@ -483,40 +483,20 @@ std::vector FFTIndexer::RunInternal(const std::vector &co const auto shortlist = FilterFFTResults(30, &magnitudes); // Free ride on that shortlist: how much of a single sweep's blind cone this orientation makes - // unrecoverable. Needs the spindle, the wavelength and the frame's own resolution, all of which - // are already here; costs one pass over 30-odd rows. - // Below this the row list is no longer trustworthy: measured over solved crystals, the fraction of - // severe orientations reported as harmless falls from 0.22 at 30 spots to 0.05 at 60, and is flat - // above. Fewer spots means no value, not a middling one. + // unrecoverable. Needs the spindle and the cone width, both already computed in Setup; costs one + // pass over 30-odd rows. The cone is the GEOMETRIC one (see spindle_theta_max_deg in Indexer.h), + // not the still's own spot resolution, which on a weak attenuated frame understates the sweep's + // real loss. // The floor was calibrated on a pass over a frame's WHOLE spot list, but nspots here is whatever // the caller fed this call. IndexAndRefine escalates 30 -> 80 -> all and usually stops at 30, so // on a frame that indexes cleanly the score is simply never computed. Closing that gap means // measuring the severity on the full list rather than riding on the indexing seed, which costs a // pass this deliberately does not spend; until then the value is absent far more often than the // floor alone implies. - constexpr size_t SPINDLE_MIN_SPOTS = 60; spindle_severity = {}; - if (spindle_axis && wavelength_A > 0 && nspots >= SPINDLE_MIN_SPOTS) { - float d_min = 0; - { // the spots' own resolution: the 95th percentile of |q|, so a few stray outliers do not - // set the cone width for the whole frame - std::vector q; - q.reserve(nspots); - for (size_t i = 0; i < nspots; i++) - q.push_back(coord[i].Length()); - if (!q.empty()) { - const size_t k = std::min(q.size() - 1, static_cast(0.95 * q.size())); - std::nth_element(q.begin(), q.begin() + k, q.end()); - if (q[k] > 0) - d_min = 1.0f / q[k]; - } - } - if (d_min > 0) { - const float sin_theta = std::min(1.0f, wavelength_A / (2.0f * d_min)); - const float theta_max_deg = static_cast(std::asin(sin_theta) * 180.0 / M_PI); - spindle_severity = SpindleBlindFraction(shortlist, magnitudes, *spindle_axis, theta_max_deg); - } - } + if (spindle_axis && spindle_theta_max_deg > 0 && nspots >= SPINDLE_MIN_SPOTS) + spindle_severity = SpindleBlindFraction(shortlist, magnitudes, *spindle_axis, + spindle_theta_max_deg); auto lattices = ReduceAndRefine(coord, nspots, shortlist, false); diff --git a/image_analysis/indexing/Indexer.cpp b/image_analysis/indexing/Indexer.cpp index 91d052676..82dc6dd90 100644 --- a/image_analysis/indexing/Indexer.cpp +++ b/image_analysis/indexing/Indexer.cpp @@ -13,6 +13,7 @@ void Indexer::Setup(const DiffractionExperiment& experiment) { spindle_axis = {}; if (const auto goniometer = experiment.GetGoniometer()) spindle_axis = goniometer->GetAxis(); + spindle_theta_max_deg = SpindleThetaMax_deg(wavelength_A, experiment.GetDetectorMaxResolution_A()); SetupUnitCell(experiment.GetUnitCell()); } diff --git a/image_analysis/indexing/Indexer.h b/image_analysis/indexing/Indexer.h index bdbe915ed..4f5377c08 100644 --- a/image_analysis/indexing/Indexer.h +++ b/image_analysis/indexing/Indexer.h @@ -41,6 +41,15 @@ protected: // defined but stationary - that is exactly the case the spindle severity is for. std::optional spindle_axis; float wavelength_A = 0; + // Half-angle of the blind double cone a sweep would leave: asin(lambda/2d) at the GEOMETRIC + // resolution of the setup - the detector corner at the recorded distance and wavelength - not at + // the still's own spot resolution. A weak, attenuated grid still reaching 3 A would score a row + // at an 11 deg miss-angle as exactly 0 while the real sweep, at the detector's 1.8 A, loses a + // fifth of its wider cone. The corner is an upper bound on any sweep collected without moving + // the detector, it is there on every frame, and it is unbiased where the still's spot list is + // systematically shallow; a wider cone errs towards reporting loss, which is the cheap error. + // 0 when the geometry cannot give it. + float spindle_theta_max_deg = 0; virtual void SetupUnitCell(const std::optional& cell) = 0; virtual std::vector RunInternal(const std::vector &coord, size_t nspots) = 0; diff --git a/image_analysis/indexing/SpindleBlindFraction.cpp b/image_analysis/indexing/SpindleBlindFraction.cpp index abbfce535..879dda38f 100644 --- a/image_analysis/indexing/SpindleBlindFraction.cpp +++ b/image_analysis/indexing/SpindleBlindFraction.cpp @@ -26,6 +26,13 @@ namespace { constexpr float MAX_REFERENCE_LENGTH_RATIO = 3.0f; } +float SpindleThetaMax_deg(float wavelength_A, float d_min_A) { + if (wavelength_A <= 0 || d_min_A <= 0) + return 0; + const float sin_theta = std::min(1.0f, wavelength_A / (2.0f * d_min_A)); + return static_cast(std::asin(sin_theta) * 180.0 / M_PI); +} + float BlindConeSelfOverlap(float x) { if (x >= 1.0f) return 0.0f; diff --git a/image_analysis/indexing/SpindleBlindFraction.h b/image_analysis/indexing/SpindleBlindFraction.h index 55f321d46..e91f43587 100644 --- a/image_analysis/indexing/SpindleBlindFraction.h +++ b/image_analysis/indexing/SpindleBlindFraction.h @@ -3,6 +3,7 @@ #pragma once +#include #include #include @@ -39,6 +40,18 @@ // 55 deg. There the score is a lower bound on the loss rather than an estimate of it. float BlindConeSelfOverlap(float x); +// Below this many spots the row shortlist is no longer trustworthy: measured on the stills of +// 22 solved crystals - 22 independent mounts, however many frames each contributed - the fraction +// of severe orientations reported harmless falls from 0.22 at 30 spots to 0.05 at 60 and is flat +// above. Fewer spots means NO value - which automation must treat as "engage", the recoverable +// error - never a middling score. +constexpr size_t SPINDLE_MIN_SPOTS = 60; + +// asin(lambda/2d) in degrees; 0 when either input is unusable. d_min_A should be the setup's +// GEOMETRIC resolution (DiffractionExperiment::GetDetectorMaxResolution_A), the upper bound on any +// sweep collected without moving the detector - see spindle_theta_max_deg in Indexer.h for why. +float SpindleThetaMax_deg(float wavelength_A, float d_min_A); + struct SpindleSeverity { float score = 0.0f; // [0,1] float row_length_A = 0; // the row the score was taken on -- 2.54.0 From 0ee1e5e07057273f12c366554080911eb30c44a4 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 09:56:07 +0200 Subject: [PATCH 16/75] spindle: an axis too long to see still shows its direction in the normals of the rows that are not The length window is deliberate - a row 2.5x the crystal's shortest is not a plausible symmetry axis in a crystal that small - but it made the score blind to a lone 2-fold on an axis LONGER than the window: a monoclinic crystal with a long unique axis, mounted near-perpendicular at an unlucky azimuth, returned a confident 0 rather than a refusal. The axis is recoverable without ever seeing its row: the normal to two direct-lattice rows is itself a reciprocal-lattice row, and a symmetry axis is parallel in the direct and reciprocal bases, so cross(a, c) is the unique-axis direction whatever the length of b. The normals of the strong in-window row pairs are now scored alongside the rows themselves, with row_length_A = 0 marking a direction the frame inferred rather than measured. Measured on a synthetic lone-diad crystal with a 300 A unique axis over random mounts, the fraction of severe mounts reported severe at the 0.5 trigger rises from 0.60 to 1.00, the engagement rate on harmless mounts of that class does not move, and the recovered direction reproduces the true axis exactly (every visible row is perpendicular to it). Also state the shortlist-consistency calibration on its per-crystal basis - 22 independent mounts, not the several hundred frames they contributed - and carry the conditioning the perpendicular case needs: an axis of order >= 3 there fully repairs the cone (measured 0.000 unrepaired for orders 3, 4, 6 against 1.000 for a diad), which a still cannot know, so the lone diad stays the operative worst case and the bound stays deliberately pessimistic on higher-symmetry crystals. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- .../indexing/SpindleBlindFraction.cpp | 62 ++++++++++++++----- .../indexing/SpindleBlindFraction.h | 35 +++++++---- tests/SpindleBlindFractionTest.cpp | 16 +++++ 3 files changed, 87 insertions(+), 26 deletions(-) diff --git a/image_analysis/indexing/SpindleBlindFraction.cpp b/image_analysis/indexing/SpindleBlindFraction.cpp index 879dda38f..9d61cbd78 100644 --- a/image_analysis/indexing/SpindleBlindFraction.cpp +++ b/image_analysis/indexing/SpindleBlindFraction.cpp @@ -19,8 +19,10 @@ namespace { constexpr float MIN_MAGNITUDE_RATIO = 0.5f; // Where the search grid can no longer resolve the crystal's rows, the pass stops returning them // and starts returning short spurious ones instead, and the shortlist becomes internally - // inconsistent: its strong rows are many times longer than its shortest entry. Measured over 384 - // frames of 22 solved crystals the ratio never exceeded 2.34 and was 1.00 at the median; on a + // inconsistent: its strong rows are many times longer than its shortest entry. Measured on the + // stills of 22 solved crystals - 22 independent mounts, so 22 is the sample size, not the + // several hundred frames they contributed - the ratio never exceeded 2.34 and was 1.00 at the + // median; on a // synthetic still whose cell is past the grid's reach it runs 4-13. Past this the score would be // reporting a cone it cannot see into, so it reports nothing instead. constexpr float MAX_REFERENCE_LENGTH_RATIO = 3.0f; @@ -74,29 +76,61 @@ std::optional SpindleBlindFraction(const std::vector &ro if (reference_length > MAX_REFERENCE_LENGTH_RATIO * shortest_length) return {}; - SpindleSeverity ret; + std::vector eligible; for (size_t i = 0; i < rows.size(); i++) { const float length = rows[i].Length(); if (length <= 0 || length > MAX_ROW_LENGTH_RATIO * reference_length) continue; if (magnitudes[i] < MIN_MAGNITUDE_RATIO * max_magnitude) continue; - const float cos_beta = std::min(1.0f, std::fabs(rows[i] * axis) / length); + eligible.push_back(i); + } + + SpindleSeverity ret; + bool scored = false; + const auto consider = [&](const Coord &direction, float row_length_A) { + const float len = direction.Length(); + const float cos_beta = std::min(1.0f, std::fabs(direction * axis) / len); const float beta_deg = static_cast(std::acos(cos_beta) * 180.0 / M_PI); - // A row PERPENDICULAR to the spindle is as damaging as one along it, and far more common: a - // 2-fold about it carries the blind cone onto the cone's opposite lobe, which the same sweep - // leaves equally unmeasured. That is Friedel's rescue, which is no rescue - the cone is - // double-sided. Both ends of the range are the bad case and the safe zone lies between them, - // so the miss-angle is folded about 45 deg. Checked against a Monte-Carlo of the true - // spherical overlap the folded form is within 0.006 to theta_max = 20 deg and 0.024 to 45 deg; - // unfolded it is wrong by a full 1.0 at beta = 90 deg, reporting the worst case as the best. + // A direction PERPENDICULAR to the spindle is as damaging as one along it, and far more + // common: a lone 2-fold about it carries the blind cone onto the cone's opposite lobe, which + // the same sweep leaves equally unmeasured. That is Friedel's rescue, which is no rescue - + // the cone is double-sided. (An axis of order >= 3 there DOES repair the cone - measured + // unrepaired fraction 0.000 for orders 3, 4 and 6 against 1.000 for order 2 - but a still + // cannot know the order, and the lone diad is the worst case this bound assumes.) Both ends + // of the range are the bad case and the safe zone lies between them, so the miss-angle is + // folded about 45 deg. Checked against a Monte-Carlo of the true spherical overlap the + // folded form is within 0.006 to theta_max = 20 deg and 0.024 to 45 deg; unfolded it is + // wrong by a full 1.0 at beta = 90 deg, reporting the worst case as the best. const float fold_deg = std::min(beta_deg, 90.0f - beta_deg); const float score = BlindConeSelfOverlap(fold_deg / theta_max_deg); - if (score > ret.score || ret.row_length_A == 0) { + if (!scored || score > ret.score) { ret.score = score; - ret.row_length_A = length; + ret.row_length_A = row_length_A; ret.miss_angle_deg = beta_deg; + scored = true; } - } + }; + + for (const auto i : eligible) + consider(rows[i], rows[i].Length()); + + // A lone 2-fold on an axis LONGER than the length window is invisible above - not among the + // shortlist's rows, and excluded by the window even when it is - but its direction is still + // recoverable: the normal to two direct-lattice rows is itself a reciprocal-lattice row, and a + // symmetry axis is parallel in the direct and reciprocal bases, so for a monoclinic cell + // cross(a, c) IS the unique-axis direction whatever the length of b. Score the normals of the + // strong in-window row pairs alongside the rows themselves; measured on a synthetic lone-diad + // crystal with a 300 A unique axis, the fraction of severe mounts reported severe at the 0.5 + // trigger rises from 0.60 to 1.00 and the engagement rate on harmless mounts of that class + // does not move. The guard only rejects a numerically degenerate normal; the shortlist already + // keeps its rows 5 deg apart. + for (size_t a = 0; a < eligible.size(); a++) + for (size_t b = a + 1; b < eligible.size(); b++) { + const Coord n = rows[eligible[a]] % rows[eligible[b]]; + if (n.Length() > 1e-4f * rows[eligible[a]].Length() * rows[eligible[b]].Length()) + consider(n, 0.0f); // 0 = a direction inferred from a pair, not a measured row + } + return ret; } diff --git a/image_analysis/indexing/SpindleBlindFraction.h b/image_analysis/indexing/SpindleBlindFraction.h index e91f43587..2f80366c5 100644 --- a/image_analysis/indexing/SpindleBlindFraction.h +++ b/image_analysis/indexing/SpindleBlindFraction.h @@ -17,19 +17,29 @@ // repaired by the point group, which maps the cone onto measured territory. It is not repaired // when the operator's axis lies inside the cone, because then the cone maps onto itself - and a // still cannot know the point group, but it can measure where the crystal's short lattice rows -// are, and a symmetry axis is always one of them. +// are, and a symmetry axis is always a lattice row, and usually among the short ones (over 107 +// solved cells, 88% of symmetry axes fall inside the 2.5x length window below). // -// Score = the fraction of the blind cone that a 2-fold about the worst plausible short row carries -// back into the blind cone: two equal caps of angular radius theta_max whose centres are 2*beta -// apart. 0 means any operator about that row moves the cone entirely off itself and one sweep loses -// nothing symmetry could have given; 1 means the whole cone is lost. Nothing about the goniometer -// enters, so the number describes the problem and leaves the remedy - a chi offset, a second sweep, -// accepting the loss - to the beamline. +// Score = the fraction of the blind cone that a 2-fold about the worst plausible row direction +// carries back into the blind cone: two equal caps of angular radius theta_max whose centres are +// 2*beta apart. 0 means any operator about that direction moves the cone entirely off itself and +// one sweep loses nothing symmetry could have given; 1 means the whole cone is lost. Nothing about +// the goniometer enters, so the number describes the problem and leaves the remedy - a chi offset, +// a second sweep, accepting the loss - to the beamline. // -// BOTH ends of the miss-angle range are the bad case. A row ON the spindle maps the cone onto itself; -// a row PERPENDICULAR to the spindle maps it onto the cone's opposite lobe, which is equally -// unmeasured, so that is no rescue either - and being a band rather than a cap it is the commoner of -// the two. The safe zone is the middle, and the miss-angle is folded about 45 deg accordingly. +// The score is a WORST-CASE BOUND under an assumption of no symmetry, not an estimate: a still +// cannot rule out that the nearest plausible row is a lone 2-fold, so it is scored as one. BOTH +// ends of the miss-angle range are that diad's bad case. A row ON the spindle maps the cone onto +// itself whatever its order; a row PERPENDICULAR to the spindle maps it onto the cone's opposite +// lobe - equally unmeasured, and a band rather than a cap, so the commoner of the two - but only +// for a LONE DIAD: an axis of order >= 3 there fully repairs the cone (measured unrepaired +// fraction 0.000 for orders 3, 4 and 6 against 1.000 for order 2), which a still cannot know. +// The safe zone is the middle, and the miss-angle is folded about 45 deg accordingly. On +// higher-symmetry crystals the bound is therefore deliberately pessimistic, and the cost model +// accepts that: at theta_max = 15 deg it engages, at the 0.5 trigger threshold, on roughly a +// quarter of harmless mountings (a single strong row's perpendicular band alone covers ~11% of +// orientation space), against a false negative that costs the dataset a hole no amount of sweep +// can fill. // Fraction of a cap of angular radius theta that its own image under a 2-fold covers, as a function // of x = (folded miss-angle) / theta_max; the caller does the folding. Exact in the flat limit. @@ -54,7 +64,8 @@ float SpindleThetaMax_deg(float wavelength_A, float d_min_A); struct SpindleSeverity { float score = 0.0f; // [0,1] - float row_length_A = 0; // the row the score was taken on + float row_length_A = 0; // the row the score was taken on; 0 = a direction inferred from the + // normal of a row pair, whose own row length the frame cannot see float miss_angle_deg = 0; }; diff --git a/tests/SpindleBlindFractionTest.cpp b/tests/SpindleBlindFractionTest.cpp index 02dce82ec..67ee26063 100644 --- a/tests/SpindleBlindFractionTest.cpp +++ b/tests/SpindleBlindFractionTest.cpp @@ -85,6 +85,22 @@ TEST_CASE("SpindleBlindFraction_Rows", "[Indexing][Spindle]") { CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); } + SECTION("a lone 2-fold on an axis too long to see is recovered from the visible rows' normal") { + // A monoclinic-like crystal: the unique axis is far beyond the length window, so no + // shortlist row points along it, but every visible row is perpendicular to it, and the + // normal of any two of them is its direction - a symmetry axis is parallel in the direct + // and reciprocal bases. Here that direction is perpendicular to the spindle, the case the + // fold exists for; before the pair-normal search this scored 0, a silent "safe". + const std::vector rows = {Coord(0, 45, 45), Coord(0, 60, 25)}; + const std::vector mag = {100, 90}; + const auto s = SpindleBlindFraction(rows, mag, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + CHECK_THAT(s->miss_angle_deg, WithinAbs(90.0f, 1e-3)); + // 0 marks a direction inferred from a pair of rows rather than a measured row. + CHECK_THAT(s->row_length_A, WithinAbs(0.0f, 1e-6)); + } + SECTION("a shortlist the pass could not resolve gives no answer at all") { // Strong rows ten times longer than the shortest entry: the grid has lost the crystal's real // rows and is returning spurious short ones. Reporting zero here would be a silent "safe". -- 2.54.0 From 89962574ef5dddff7452ab8aa18bba2773b9952e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 09:58:43 +0200 Subject: [PATCH 17/75] spindle: the severity no longer rides on the indexing seed or on which indexer is configured The score gated on 60 spots, but the seed escalation stops at the leanest seed that indexes - 30 spots on precisely the clean frames a grid scan produces - so the value was absent exactly where beamline automation most needs it, and absence maps to "engage": the protocol would have fired on every good frame, which degenerates the trigger into "always". The floor itself stays where it was calibrated; what changes is what it gates. When no escalation pass could answer, one severity-only pass runs over the full spot list - the row search alone, no reduction, no refinement - purely to produce the number. The same was true of the indexer choice: only the FFT family computes a row shortlist, so a deployment configured with the known-cell indexer - the ordinary online stills path - never produced the score at all. Where the severity-only pass has no row search to run, the severity is read off the rows of the winning lattice instead, which any indexer produces: the lattice's shortest few distinct directions, as many as the FFT shortlist resolves in practice, fed through the same window and scoring with equal magnitudes. The count parity is load-bearing - a worst case over every enumerable lattice direction fires on 100% of harmless mounts of a generic triclinic cell against 74% for this selection at theta_max = 15 deg, and an always-firing trigger decides nothing - while the diad-detection rate stays 1.00 on the monoclinic classes either way, a dropped axis row being recovered by the pair normals exactly as an invisible one is. A frame that neither indexed nor reached the spot floor still reports nothing, which is the honest answer and maps to the recoverable error. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/IndexAndRefine.cpp | 42 ++++++++++++++++-- image_analysis/indexing/FFTIndexer.cpp | 18 +++++--- image_analysis/indexing/FFTIndexer.h | 1 + image_analysis/indexing/Indexer.cpp | 14 ++++-- image_analysis/indexing/Indexer.h | 10 ++++- image_analysis/indexing/IndexerThreadPool.cpp | 11 ++--- image_analysis/indexing/IndexerThreadPool.h | 8 +++- .../indexing/SpindleBlindFraction.cpp | 43 +++++++++++++++++++ .../indexing/SpindleBlindFraction.h | 10 +++++ tests/IndexingUnitTest.cpp | 43 +++++++++++++++++++ tests/SpindleBlindFractionTest.cpp | 32 ++++++++++++++ 11 files changed, 211 insertions(+), 21 deletions(-) diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index 1c95af4d8..67c969705 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -175,10 +175,9 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data any_executed |= res.executed; // Kept from whichever run could measure it - the largest seed that answered - rather than from // the run whose lattice won: the severity describes the frame's lattice rows, not the cell that - // closed on them, so a frame that indexed nothing still has one. It stays absent while the - // escalation never leaves the lean seed, which is below the spot floor the score needs - // (SPINDLE_MIN_SPOTS in FFTIndexer.cpp) - so on frames that index cleanly at 30 spots there is - // no value at all. That is a gap, not a "no problem": see the note at SPINDLE_MIN_SPOTS. + // closed on them, so a frame that indexed nothing still has one. When the escalation never + // leaves a seed below the score's spot floor it stays absent here, and the severity-only pass + // after the loop supplies it instead. if (res.spindle_blind_fraction) spindle_blind_fraction = res.spindle_blind_fraction; if (!res.lattice.empty()) { @@ -211,6 +210,41 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data if (any_executed) msg.indexing_result = false; + // The severity's spot floor (SPINDLE_MIN_SPOTS) was calibrated on a frame's whole spot list, + // but the escalation stops at the leanest seed that indexes - 30 spots on precisely the clean + // frames a grid scan produces - so riding on the indexing seed left the score absent exactly + // where automation most needs it, and absence maps to "engage" (see SpindleBlindFraction.h). + // Decouple the two: when no escalation pass could answer, spend one severity-only row pass over + // the full spot list (no reduction, no refinement); where that path does not exist - an indexer + // without a row search - or refuses, read the severity off the rows of the winning lattice, + // which any indexer produces. + if (!spindle_blind_fraction && experiment.GetGoniometer().has_value()) { + std::vector recip; + recip.reserve(std::min(msg.spots.size(), FFT_MAX_SPOTS)); + for (const auto &i : msg.spots) { + if (index_ice_rings || !i.ice_ring) { + recip.push_back(i.ReciprocalCoord(geom_)); + if (recip.size() >= FFT_MAX_SPOTS) + break; + } + } + if (recip.size() >= SPINDLE_MIN_SPOTS) { + const auto algorithm = experiment.GetIndexingAlgorithm(); + if (algorithm == IndexingAlgorithmEnum::FFT || algorithm == IndexingAlgorithmEnum::FFTW) { + const auto res = indexer_->Run(experiment, recip, /*severity_only=*/true); + spindle_blind_fraction = res.spindle_blind_fraction; + } + if (!spindle_blind_fraction && !indexer_result.lattice.empty()) { + const float theta_max_deg = SpindleThetaMax_deg(experiment.GetWavelength_A(), + experiment.GetDetectorMaxResolution_A()); + if (const auto severity = SpindleBlindFractionFromLattice( + indexer_result.lattice.front(), experiment.GetGoniometer()->GetAxis(), + theta_max_deg)) + spindle_blind_fraction = severity->score; + } + } + } + msg.spindle_blind_fraction = spindle_blind_fraction; if (!indexer_result.lattice.empty()) { diff --git a/image_analysis/indexing/FFTIndexer.cpp b/image_analysis/indexing/FFTIndexer.cpp index 773183545..7bec2f92d 100644 --- a/image_analysis/indexing/FFTIndexer.cpp +++ b/image_analysis/indexing/FFTIndexer.cpp @@ -466,6 +466,16 @@ std::optional FFTIndexer::SearchCap(const std::vector &coord, size return found; } +std::optional FFTIndexer::RunSeverityOnly(const std::vector &coord) { + if (!spindle_axis || spindle_theta_max_deg <= 0 || coord.size() < SPINDLE_MIN_SPOTS) + return {}; + const size_t nspots = std::min(coord.size(), static_cast(FFT_MAX_SPOTS)); + ExecuteFFT(coord, nspots); + std::vector magnitudes; + const auto shortlist = FilterFFTResults(30, &magnitudes); + return SpindleBlindFraction(shortlist, magnitudes, *spindle_axis, spindle_theta_max_deg); +} + std::vector FFTIndexer::RunInternal(const std::vector &coord, size_t nspots) { if (nspots > coord.size()) nspots = coord.size(); @@ -488,11 +498,9 @@ std::vector FFTIndexer::RunInternal(const std::vector &co // not the still's own spot resolution, which on a weak attenuated frame understates the sweep's // real loss. // The floor was calibrated on a pass over a frame's WHOLE spot list, but nspots here is whatever - // the caller fed this call. IndexAndRefine escalates 30 -> 80 -> all and usually stops at 30, so - // on a frame that indexes cleanly the score is simply never computed. Closing that gap means - // measuring the severity on the full list rather than riding on the indexing seed, which costs a - // pass this deliberately does not spend; until then the value is absent far more often than the - // floor alone implies. + // the caller fed this call - IndexAndRefine escalates 30 -> 80 -> all and usually stops at 30, + // so on a frame that indexes cleanly nothing is computed here; the caller then asks for the + // severity with a RunSeverityOnly pass over the full list instead of leaving it absent. spindle_severity = {}; if (spindle_axis && spindle_theta_max_deg > 0 && nspots >= SPINDLE_MIN_SPOTS) spindle_severity = SpindleBlindFraction(shortlist, magnitudes, *spindle_axis, diff --git a/image_analysis/indexing/FFTIndexer.h b/image_analysis/indexing/FFTIndexer.h index a483eb9ba..d29cce9bc 100644 --- a/image_analysis/indexing/FFTIndexer.h +++ b/image_analysis/indexing/FFTIndexer.h @@ -57,6 +57,7 @@ protected: // Filled by RunInternal from the shortlist it already computes; read out by Indexer::Run. std::optional spindle_severity; std::optional GetSpindleSeverity() const override { return spindle_severity; } + std::optional RunSeverityOnly(const std::vector &coord) override; virtual void ExecuteFFT(const std::vector &coord, size_t nspots) = 0; // Called after direction_vectors is rewritten, for implementations that keep a copy of it. diff --git a/image_analysis/indexing/Indexer.cpp b/image_analysis/indexing/Indexer.cpp index 82dc6dd90..e7725a8ce 100644 --- a/image_analysis/indexing/Indexer.cpp +++ b/image_analysis/indexing/Indexer.cpp @@ -17,16 +17,22 @@ void Indexer::Setup(const DiffractionExperiment& experiment) { SetupUnitCell(experiment.GetUnitCell()); } -IndexerResult Indexer::Run(const std::vector &coord) { +IndexerResult Indexer::Run(const std::vector &coord, bool severity_only) { IndexerResult ret; auto start = std::chrono::steady_clock::now(); - ret.lattice = RunInternal(coord, coord.size()); + std::optional severity; + if (severity_only) { + severity = RunSeverityOnly(coord); + } else { + ret.lattice = RunInternal(coord, coord.size()); + ret.executed = true; + severity = GetSpindleSeverity(); + } auto end = std::chrono::steady_clock::now(); std::chrono::duration duration = end - start; ret.indexing_time_s = duration.count(); - ret.executed = true; - if (const auto severity = GetSpindleSeverity()) { + if (severity) { ret.spindle_blind_fraction = severity->score; ret.spindle_row_length_A = severity->row_length_A; ret.spindle_miss_angle_deg = severity->miss_angle_deg; diff --git a/image_analysis/indexing/Indexer.h b/image_analysis/indexing/Indexer.h index 4f5377c08..cd2190145 100644 --- a/image_analysis/indexing/Indexer.h +++ b/image_analysis/indexing/Indexer.h @@ -55,10 +55,18 @@ protected: virtual std::vector RunInternal(const std::vector &coord, size_t nspots) = 0; // Set by RunInternal when the implementation computes it; read out by Run. virtual std::optional GetSpindleSeverity() const { return {}; } + // The row pass alone, over the whole of `coord`, to produce the spindle severity without + // reducing or refining anything. Implemented by the FFT-family indexers, which own a row + // search; the others have nothing to answer with and return nothing. + virtual std::optional RunSeverityOnly(const std::vector &coord) { return {}; } public: virtual ~Indexer() = default; void Setup(const DiffractionExperiment& experiment); - IndexerResult Run(const std::vector &coord); + // severity_only = true runs RunSeverityOnly instead of indexing: no lattice comes back and + // `executed` stays false, since nothing was an indexing attempt. Used when the seed escalation + // indexed a frame from fewer spots than the severity's floor - the score must not be absent on + // precisely the frames that index cleanly. + IndexerResult Run(const std::vector &coord, bool severity_only = false); }; diff --git a/image_analysis/indexing/IndexerThreadPool.cpp b/image_analysis/indexing/IndexerThreadPool.cpp index 8684c3307..56b925c18 100644 --- a/image_analysis/indexing/IndexerThreadPool.cpp +++ b/image_analysis/indexing/IndexerThreadPool.cpp @@ -148,7 +148,7 @@ void IndexerThread::Worker(int threadid) { Indexer &indexer = **slot; indexer.Setup(input->experiment); - tmp_result = std::make_unique(indexer.Run(input->recip)); + tmp_result = std::make_unique(indexer.Run(input->recip, input->severity_only)); } catch (std::exception &e) { // Hand the failure back as a result carrying the reason. A nullptr here was // indistinguishable from a worker that was never dispatched, and both then read @@ -178,7 +178,7 @@ void IndexerThread::Finalize() { } std::unique_ptr IndexerThread::Run(const DiffractionExperiment &experiment, - const std::vector &recip) { + const std::vector &recip, bool severity_only) { std::unique_ptr tmp_result; { std::unique_lock lock(m); @@ -186,7 +186,7 @@ std::unique_ptr IndexerThread::Run(const DiffractionExperiment &e return nullptr; if (state != TaskState::IDLE) return nullptr; - task_input = std::make_unique(std::cref(experiment), std::cref(recip)); + task_input = std::make_unique(std::cref(experiment), std::cref(recip), severity_only); state = TaskState::READY; } c_start.notify_one(); @@ -231,7 +231,8 @@ int IndexerThreadPool::GetFreeWorker() { return -1; } -IndexerResult IndexerThreadPool::Run(const DiffractionExperiment &experiment, const std::vector &recip) { +IndexerResult IndexerThreadPool::Run(const DiffractionExperiment &experiment, const std::vector &recip, + bool severity_only) { const auto algorithm = experiment.GetIndexingAlgorithm(); if (algorithm == IndexingAlgorithmEnum::None) return IndexerResult{.lattice = {}, .indexing_time_s = 0, .executed = false}; @@ -285,7 +286,7 @@ IndexerResult IndexerThreadPool::Run(const DiffractionExperiment &experiment, co std::unique_ptr result; if (task >= 0) { try { - result = tasks[task]->Run(experiment, recip); + result = tasks[task]->Run(experiment, recip, severity_only); } catch (const std::exception &e) { spdlog::error("Indexer thread failed: {}", e.what()); result = std::make_unique(IndexerResult{ diff --git a/image_analysis/indexing/IndexerThreadPool.h b/image_analysis/indexing/IndexerThreadPool.h index 2fb319e9e..116444b44 100644 --- a/image_analysis/indexing/IndexerThreadPool.h +++ b/image_analysis/indexing/IndexerThreadPool.h @@ -44,6 +44,7 @@ class IndexerThread { struct TaskInput { const DiffractionExperiment &experiment; const std::vector &recip; + const bool severity_only; }; // Held by value: with IndexerConstruction::OnFirstUse the worker builds its indexer long after @@ -66,7 +67,8 @@ class IndexerThread { public: IndexerThread(const IndexingSettings& settings, int threadid, IndexerConstruction construction); ~IndexerThread(); - std::unique_ptr Run(const DiffractionExperiment &experiment, const std::vector &recip); + std::unique_ptr Run(const DiffractionExperiment &experiment, const std::vector &recip, + bool severity_only = false); void Finalize(); }; @@ -82,7 +84,9 @@ class IndexerThreadPool { public: IndexerThreadPool(const IndexingSettings& settings, IndexerConstruction construction = IndexerConstruction::Preconstruct); - IndexerResult Run(const DiffractionExperiment& experiment, const std::vector& recip); + // severity_only skips indexing and produces just the spindle severity - see Indexer::Run. + IndexerResult Run(const DiffractionExperiment& experiment, const std::vector& recip, + bool severity_only = false); }; diff --git a/image_analysis/indexing/SpindleBlindFraction.cpp b/image_analysis/indexing/SpindleBlindFraction.cpp index 9d61cbd78..d7e9bd623 100644 --- a/image_analysis/indexing/SpindleBlindFraction.cpp +++ b/image_analysis/indexing/SpindleBlindFraction.cpp @@ -134,3 +134,46 @@ std::optional SpindleBlindFraction(const std::vector &ro return ret; } + +std::optional SpindleBlindFractionFromLattice(const CrystalLattice &lattice, + const Coord &spindle, + float theta_max_deg) { + // Candidate rows: the direct lattice's shortest few distinct directions, drawn from the index + // box up to +/-2 - the range in which the symmetry axes of a reduced or conventional basis + // lie. The count matches what the FFT shortlist resolves in practice (four or five distinct + // rows - see FilterFFTResults), so the bound is taken over comparable evidence on either path. + // That parity is load-bearing: a worst case over every enumerable direction saturates towards + // "always engage" - measured on a generic triclinic cell it fires on 100% of harmless mounts, + // against 74% for this selection at theta_max = 15 deg - and an always-firing trigger decides + // nothing. The diad-detection rate stays 1.00 on the monoclinic classes either way, because a + // dropped axis row is recovered by the pair normals exactly as an invisible one is. + std::vector all; + all.reserve(62); + for (int u = 0; u <= 2; u++) + for (int v = (u == 0) ? 0 : -2; v <= 2; v++) + for (int w = (u == 0 && v == 0) ? 1 : -2; w <= 2; w++) + all.push_back(lattice.Vec0() * static_cast(u) + + lattice.Vec1() * static_cast(v) + + lattice.Vec2() * static_cast(w)); + std::sort(all.begin(), all.end(), + [](const Coord &a, const Coord &b) { return a.Length() < b.Length(); }); + + constexpr size_t MAX_LATTICE_ROWS = 6; + const float cos_5_deg = std::cos(5.0f * static_cast(M_PI) / 180.0f); + std::vector rows; + for (const auto &r : all) { + if (rows.size() >= MAX_LATTICE_ROWS + || r.Length() > MAX_ROW_LENGTH_RATIO * all.front().Length()) + break; + bool distinct = true; + for (const auto &k : rows) + if (std::fabs(r * k) / (r.Length() * k.Length()) > cos_5_deg) { + distinct = false; + break; + } + if (distinct) + rows.push_back(r); + } + const std::vector magnitudes(rows.size(), 1.0f); + return SpindleBlindFraction(rows, magnitudes, spindle, theta_max_deg); +} diff --git a/image_analysis/indexing/SpindleBlindFraction.h b/image_analysis/indexing/SpindleBlindFraction.h index 2f80366c5..9ac03a55d 100644 --- a/image_analysis/indexing/SpindleBlindFraction.h +++ b/image_analysis/indexing/SpindleBlindFraction.h @@ -8,6 +8,7 @@ #include #include "../../common/Coord.h" +#include "../../common/CrystalLattice.h" // How much of a single sweep's blind cone this orientation makes unrecoverable. // @@ -75,3 +76,12 @@ std::optional SpindleBlindFraction(const std::vector &ro const std::vector &magnitudes, const Coord &spindle, float theta_max_deg); + +// The same severity read off an indexed lattice instead of an FFT shortlist, for the paths that +// index without a row search (a known-cell indexer on the online stills path). The candidate rows +// are the lattice's shortest few distinct directions - as many as the FFT shortlist resolves in +// practice, so the bound is over comparable evidence on either path - with equal magnitudes: a +// lattice does not rank its rows, and all of them are equally real. +std::optional SpindleBlindFractionFromLattice(const CrystalLattice &lattice, + const Coord &spindle, + float theta_max_deg); diff --git a/tests/IndexingUnitTest.cpp b/tests/IndexingUnitTest.cpp index 545c68c19..49c0365d4 100644 --- a/tests/IndexingUnitTest.cpp +++ b/tests/IndexingUnitTest.cpp @@ -246,6 +246,49 @@ TEST_CASE("FFTIndexer","[Indexing]") { logger.Info("Time: {} ms", std::chrono::duration_cast(end - start).count()); } +TEST_CASE("FFTIndexer_SpindleSeverity", "[Indexing][Spindle]") { + // End to end through the real indexer: a crystal whose shortest row lies on the spindle must + // come back with a severity of 1 from a normal run, from a severity-only run - which must not + // index anything - and not at all when the frame is below the spot floor. + UnitCell uc{39, 45, 78, 90, 90, 90}; + CrystalLattice cl(uc); + + DiffractionExperiment experiment(DetJF4M()); + experiment.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4); + // The 39 A axis of the cell lies along x, and so does the spindle. + experiment.Goniometer(GoniometerAxis("omega", 0, 0.1f, Coord(1, 0, 0), {})); + + IndexingSettings settings; + settings.Algorithm(IndexingAlgorithmEnum::FFT) + .FFT_MaxUnitCell_A(250.0).FFT_HighResolution_A(2 * M_PI / 3.0); + experiment.ImportIndexingSettings(settings).SetUnitCell(uc); + + std::unique_ptr indexer = CreateIndexer(experiment); + REQUIRE(indexer); + indexer->Setup(experiment); + + std::vector vec; + for (int h = -2; h < 10; h++) + for (int k = -5; k < 10; k++) + for (int l = -3; l < 10; l++) + vec.push_back(h * cl.Astar() + k * cl.Bstar() + l * cl.Cstar()); + + auto full = indexer->Run(vec); + REQUIRE(full.spindle_blind_fraction.has_value()); + CHECK_THAT(*full.spindle_blind_fraction, Catch::Matchers::WithinAbs(1.0f, 1e-4)); + + auto severity_only = indexer->Run(vec, /*severity_only=*/true); + CHECK(severity_only.lattice.empty()); + CHECK_FALSE(severity_only.executed); + REQUIRE(severity_only.spindle_blind_fraction.has_value()); + CHECK_THAT(*severity_only.spindle_blind_fraction, Catch::Matchers::WithinAbs(1.0f, 1e-4)); + + // Below the spot floor there is no value - the CANNOT-SAY state - not a middling one. + const std::vector few(vec.begin(), vec.begin() + 40); + auto starved = indexer->Run(few, /*severity_only=*/true); + CHECK_FALSE(starved.spindle_blind_fraction.has_value()); +} + TEST_CASE("PostIndexingRefinement_MultiLattice_TwoCrystals_BraggPrediction","[Indexing]") { Logger logger("PostIndexingRefinement_MultiLattice_TwoCrystals_BraggPrediction"); diff --git a/tests/SpindleBlindFractionTest.cpp b/tests/SpindleBlindFractionTest.cpp index 67ee26063..4823ba27b 100644 --- a/tests/SpindleBlindFractionTest.cpp +++ b/tests/SpindleBlindFractionTest.cpp @@ -124,3 +124,35 @@ TEST_CASE("SpindleBlindFraction_Rows", "[Indexing][Spindle]") { CHECK(wide->score > narrow->score); } } + +TEST_CASE("SpindleBlindFraction_FromLattice", "[Indexing][Spindle]") { + const Coord spindle(0, 0, 1); + const float theta_max = 20.0f; + + SECTION("a known-cell frame answers from its lattice rows") { + // Monoclinic-like basis with the unique axis along the spindle. The severity needs no FFT + // shortlist: the lattice's own short rows carry the answer, here a worst case twice over + // (a row on the spindle and rows perpendicular to it). + const CrystalLattice latt(Coord(50, 0, 0), Coord(0, 0, 60), Coord(20, 70, 0)); + const auto s = SpindleBlindFractionFromLattice(latt, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + } + + SECTION("a long unique axis outside the window is recovered from the pair normals") { + // The 300 A axis is excluded by the length window as a row, but every kept row is + // perpendicular to it, so the normal of any pair recovers its direction - perpendicular + // to the spindle, the lone-diad worst case. + const CrystalLattice latt(Coord(0, 45, 45), Coord(300, 0, 0), Coord(0, -60, 25)); + const auto s = SpindleBlindFractionFromLattice(latt, spindle, theta_max); + REQUIRE(s.has_value()); + CHECK_THAT(s->score, WithinAbs(1.0f, 1e-5)); + CHECK_THAT(s->miss_angle_deg, WithinAbs(90.0f, 1e-3)); + CHECK_THAT(s->row_length_A, WithinAbs(0.0f, 1e-6)); + } + + SECTION("no cone, no answer") { + const CrystalLattice latt(Coord(50, 0, 0), Coord(0, 0, 60), Coord(20, 70, 0)); + CHECK_FALSE(SpindleBlindFractionFromLattice(latt, spindle, 0.0f).has_value()); + } +} -- 2.54.0 From bc3d69372167e07f352b3cdc27afbc186aa84593 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 10:08:15 +0200 Subject: [PATCH 18/75] spindle: three trigger states, and the threshold is geometry rather than tuning The score exists so beamline automation can engage a recovery protocol - a two-sweep collection, a goniometer reorientation - with no human in the loop, so its canonical reading is fixed in one place with nothing for a beamline to tune: engage at 0.5, do not below, and NO VALUE is a third state that automation must treat as engage. The error costs are asymmetric - a false negative leaves the data permanently short, a false positive costs minutes of beamtime - and a frame nobody could measure must not be read as a frame measured safe. The 0.5 is derived, not tuned: the score is monotone in the folded miss-angle, so any threshold is a fold-angle gate, and 0.5 gates at fold <= 0.4040 * theta_max (verified root). Engaging on any overlap at all would gate at fold < theta_max, whose perpendicular band alone spans sin(theta_max) per row - 26% of orientation space at 15 deg - and unions over a frame's rows to well over half of all mountings, degenerating the trigger into "always"; at 0.5 the residual missed loss stays below half the cone. Also correct the message-field comment that said the whole cone means "1 - cos theta_max of every shell": an inner shell loses 1 - cos theta(d) at its own, smaller theta(d). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- common/JFJochMessages.h | 8 +++-- .../indexing/SpindleBlindFraction.h | 29 +++++++++++++++++++ tests/SpindleBlindFractionTest.cpp | 15 ++++++++++ 3 files changed, 49 insertions(+), 3 deletions(-) diff --git a/common/JFJochMessages.h b/common/JFJochMessages.h index a811bca17..317d98eb2 100644 --- a/common/JFJochMessages.h +++ b/common/JFJochMessages.h @@ -130,9 +130,11 @@ struct DataMessage { std::optional ice_ring_score; // strongest ice ring over the smooth radial background (1 = none) // How much of a single sweep's blind cone this orientation makes unrecoverable: 0 = one sweep // about the spindle reaches everything the point group can give, 1 = a short lattice row lies on - // the spindle and the whole cone (1 - cos theta_max of every shell) is lost coherently. Absent - // when the frame gives nothing to decide on - too few spots, no lattice rows, no spindle - which - // is a third state, not a value of one half. + // the spindle and the whole cone is lost coherently - each shell losing its own 1 - cos theta(d), + // theta(d) = asin(lambda/2d), up to theta_max at the resolution edge. Absent when the frame gives + // nothing to decide on - too few spots, no lattice rows, no spindle - which is a third state, not + // a value of one half: automation must treat absence as "engage" (see SpindleTrigger in + // image_analysis/indexing/SpindleBlindFraction.h). std::optional spindle_blind_fraction; std::optional indexing_result; diff --git a/image_analysis/indexing/SpindleBlindFraction.h b/image_analysis/indexing/SpindleBlindFraction.h index 9ac03a55d..197a5e23f 100644 --- a/image_analysis/indexing/SpindleBlindFraction.h +++ b/image_analysis/indexing/SpindleBlindFraction.h @@ -63,6 +63,35 @@ constexpr size_t SPINDLE_MIN_SPOTS = 60; // sweep collected without moving the detector - see spindle_theta_max_deg in Indexer.h for why. float SpindleThetaMax_deg(float wavelength_A, float d_min_A); +// The trigger the score exists for: beamline automation engaging a recovery protocol - a two-sweep +// collection, a goniometer reorientation - with no human in the loop. The stored and transported +// quantity stays the continuous score; these three states are the canonical reading of it, fixed +// by design with nothing for a beamline to tune: +// +// Engage score >= 0.5 +// DontEngage score < 0.5 +// CannotSay no score at all - too few spots, no shortlist, the consistency guard refused, +// the path never computed one. Automation MUST treat CannotSay as Engage: the error +// costs are asymmetric. A false negative is unrecoverable - the protocol does not +// fire, one sweep is collected, and the data are permanently short - while a false +// positive costs an extra wedge, minutes of beamtime. +// +// The 0.5 threshold is geometry, not tuning. The score is monotone in the folded miss-angle, so +// any threshold is a fold-angle gate; 0.5 gates at fold <= 0.4040 * theta_max. Engaging on any +// overlap at all (score > 0) would gate at fold < theta_max, whose perpendicular band alone spans +// sin(theta_max) of orientation space per row - 26% at theta_max = 15 deg - and unions over a +// frame's rows to well over half of all mountings, degenerating the trigger into "always". At 0.5 +// the residual missed loss is bounded below half the cone. +enum class SpindleTrigger { Engage, DontEngage, CannotSay }; + +constexpr float SPINDLE_ENGAGE_THRESHOLD = 0.5f; + +inline SpindleTrigger SpindleTriggerState(const std::optional &score) { + if (!score.has_value()) + return SpindleTrigger::CannotSay; + return *score >= SPINDLE_ENGAGE_THRESHOLD ? SpindleTrigger::Engage : SpindleTrigger::DontEngage; +} + struct SpindleSeverity { float score = 0.0f; // [0,1] float row_length_A = 0; // the row the score was taken on; 0 = a direction inferred from the diff --git a/tests/SpindleBlindFractionTest.cpp b/tests/SpindleBlindFractionTest.cpp index 4823ba27b..3f03d2e8c 100644 --- a/tests/SpindleBlindFractionTest.cpp +++ b/tests/SpindleBlindFractionTest.cpp @@ -156,3 +156,18 @@ TEST_CASE("SpindleBlindFraction_FromLattice", "[Indexing][Spindle]") { CHECK_FALSE(SpindleBlindFractionFromLattice(latt, spindle, 0.0f).has_value()); } } + +TEST_CASE("SpindleTrigger_States", "[Indexing][Spindle]") { + // The 0.5 threshold is a fold-angle gate: the score is monotone in the folded miss-angle and + // crosses 0.5 at fold = 0.4040 * theta_max (verified root of the overlap closed form). + CHECK_THAT(BlindConeSelfOverlap(0.4040f), WithinAbs(0.5f, 1e-3)); + + CHECK(SpindleTriggerState(0.5f) == SpindleTrigger::Engage); + CHECK(SpindleTriggerState(1.0f) == SpindleTrigger::Engage); + CHECK(SpindleTriggerState(0.49f) == SpindleTrigger::DontEngage); + CHECK(SpindleTriggerState(0.0f) == SpindleTrigger::DontEngage); + // No value is a third state, and the one automation must treat as Engage: a measured 0 says + // "one sweep loses nothing", absence says "nobody could look" - taking the second for the + // first is the unrecoverable error. + CHECK(SpindleTriggerState(std::nullopt) == SpindleTrigger::CannotSay); +} -- 2.54.0 From 26a82ddc729bd5585a1c96c3c1f12601c1430970 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 10:23:07 +0200 Subject: [PATCH 19/75] spindle: the per-image severity travels the whole data path, and absence travels with it The score was computed and then thrown away - carried on the per-image message but transported nowhere - so the automation it exists for could not read it. It now flows like bkg_estimate at every layer: CBOR (per-image key, END-block run mean and per-image array), HDF5 (per-image /entry/MX/spindleBlindFraction in the data files, the array and spindleBlindFractionMean in the master), read-back into a re-opened dataset, the scan result, the receiver plots, the REST plot and scan_result schemas, and the viewer and frontend plot menus. Absence is load-bearing and every transport keeps it distinguishable from a measured zero: the CBOR key is simply missing, the HDF5 array holds NaN, and read-back turns NaN back into an absent optional rather than a value. A pipeline that read 0 where the truth is "no value" would take exactly the wrong action - a measured 0 says one sweep loses nothing, absence says nobody could look, and the second must engage the recovery protocol while the first must not. The round-trip tests pin all three states through CBOR and through a written-and-reopened file. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- broker/OpenAPIConvert.cpp | 3 ++ broker/gen/model/Scan_result_images_inner.cpp | 31 ++++++++++++++++++- broker/gen/model/Scan_result_images_inner.h | 9 ++++++ broker/jfjoch_api.yaml | 11 +++++++ common/JFJochMessages.h | 7 +++++ common/JFJochReceiverPlots.cpp | 16 ++++++++++ common/JFJochReceiverPlots.h | 3 ++ common/Plot.h | 2 +- common/ScanResult.h | 1 + common/ScanResultGenerator.cpp | 3 ++ docs/CBOR.md | 3 ++ docs/HDF5.md | 2 ++ frame_serialize/CBORStream2Deserializer.cpp | 6 ++++ frame_serialize/CBORStream2Serializer.cpp | 3 ++ frontend/src/client/types.gen.ts | 10 ++++-- frontend/src/client/zod.gen.ts | 4 +++ .../src/components/DataProcessingPlot.tsx | 2 ++ .../src/components/DataProcessingPlots.tsx | 1 + reader/HDF5MetadataSource.cpp | 10 ++++++ reader/JFJochReaderDataset.h | 2 ++ reader/JFJochReaderImage.cpp | 1 + receiver/JFJochReceiver.cpp | 1 + rugnux/Rugnux.cpp | 1 + tests/CBORTest.cpp | 15 ++++++++- tests/JFJochReaderTest.cpp | 11 +++++++ viewer/JFJochHttpReader.cpp | 1 + viewer/JFJochProcessController.cpp | 1 + viewer/JFJochViewerDatasetInfo.cpp | 2 ++ writer/HDF5DataFilePluginMX.cpp | 5 +++ writer/HDF5DataFilePluginMX.h | 2 ++ writer/HDF5NXmx.cpp | 4 +++ 31 files changed, 168 insertions(+), 5 deletions(-) diff --git a/broker/OpenAPIConvert.cpp b/broker/OpenAPIConvert.cpp index e44b9fca2..8cf6f8e6d 100644 --- a/broker/OpenAPIConvert.cpp +++ b/broker/OpenAPIConvert.cpp @@ -898,6 +898,7 @@ PlotType ConvertPlotType(const std::optional& input) { throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Plot type is compulsory paramater"); if (input == "bkg_estimate") return PlotType::BkgEstimate; + if (input == "spindle_blind_fraction") return PlotType::SpindleBlindFraction; if (input == "ice_ring_score") return PlotType::IceRingScore; if (input == "azint") return PlotType::AzInt; if (input == "azint_1d") return PlotType::AzInt1D; @@ -1134,6 +1135,8 @@ org::openapitools::server::model::Scan_result Convert(const ScanResult& input) { if (i.bkg.has_value()) tmp.setBkg(i.bkg.value()); + if (i.spindle_blind.has_value()) + tmp.setSpindleBlind(i.spindle_blind.value()); if (i.angle_deg.has_value()) tmp.setAngle(i.angle_deg.value()); diff --git a/broker/gen/model/Scan_result_images_inner.cpp b/broker/gen/model/Scan_result_images_inner.cpp index 088d233d5..76197259b 100644 --- a/broker/gen/model/Scan_result_images_inner.cpp +++ b/broker/gen/model/Scan_result_images_inner.cpp @@ -31,6 +31,8 @@ Scan_result_images_inner::Scan_result_images_inner() m_AngleIsSet = false; m_Bkg = 0.0f; m_BkgIsSet = false; + m_Spindle_blind = 0.0f; + m_Spindle_blindIsSet = false; m_Spots = 0L; m_SpotsIsSet = false; m_Spots_low_res = 0L; @@ -84,7 +86,7 @@ bool Scan_result_images_inner::validate(std::stringstream& msg, const std::strin bool success = true; const std::string _pathPrefix = pathPrefix.empty() ? "Scan_result_images_inner" : pathPrefix; - + return success; } @@ -112,6 +114,9 @@ bool Scan_result_images_inner::operator==(const Scan_result_images_inner& rhs) c ((!bkgIsSet() && !rhs.bkgIsSet()) || (bkgIsSet() && rhs.bkgIsSet() && getBkg() == rhs.getBkg())) && + ((!spindleBlindIsSet() && !rhs.spindleBlindIsSet()) || (spindleBlindIsSet() && rhs.spindleBlindIsSet() && getSpindleBlind() == rhs.getSpindleBlind())) && + + ((!spotsIsSet() && !rhs.spotsIsSet()) || (spotsIsSet() && rhs.spotsIsSet() && getSpots() == rhs.getSpots())) && @@ -180,6 +185,8 @@ void to_json(nlohmann::json& j, const Scan_result_images_inner& o) j["angle"] = o.m_Angle; if(o.bkgIsSet()) j["bkg"] = o.m_Bkg; + if(o.spindleBlindIsSet()) + j["spindle_blind"] = o.m_Spindle_blind; if(o.spotsIsSet()) j["spots"] = o.m_Spots; if(o.spotsLowResIsSet()) @@ -239,6 +246,11 @@ void from_json(const nlohmann::json& j, Scan_result_images_inner& o) j.at("bkg").get_to(o.m_Bkg); o.m_BkgIsSet = true; } + if(j.find("spindle_blind") != j.end()) + { + j.at("spindle_blind").get_to(o.m_Spindle_blind); + o.m_Spindle_blindIsSet = true; + } if(j.find("spots") != j.end()) { j.at("spots").get_to(o.m_Spots); @@ -406,6 +418,23 @@ void Scan_result_images_inner::unsetBkg() { m_BkgIsSet = false; } +float Scan_result_images_inner::getSpindleBlind() const +{ + return m_Spindle_blind; +} +void Scan_result_images_inner::setSpindleBlind(float const value) +{ + m_Spindle_blind = value; + m_Spindle_blindIsSet = true; +} +bool Scan_result_images_inner::spindleBlindIsSet() const +{ + return m_Spindle_blindIsSet; +} +void Scan_result_images_inner::unsetSpindle_blind() +{ + m_Spindle_blindIsSet = false; +} int64_t Scan_result_images_inner::getSpots() const { return m_Spots; diff --git a/broker/gen/model/Scan_result_images_inner.h b/broker/gen/model/Scan_result_images_inner.h index 5eab27c97..e99639d74 100644 --- a/broker/gen/model/Scan_result_images_inner.h +++ b/broker/gen/model/Scan_result_images_inner.h @@ -97,6 +97,13 @@ public: bool bkgIsSet() const; void unsetBkg(); /// + /// Fraction (0-1) of a rotation sweep's blind double cone that this orientation makes unrecoverable, assuming the nearest plausible lattice row is a lone 2-fold (a worst-case bound, not an estimate). At 0.5 and above a recovery protocol (second sweep, reorientation) should engage. Absent means the frame could not be assessed, which automation must treat the same as above threshold - it is not a zero. + /// + float getSpindleBlind() const; + void setSpindleBlind(float const value); + bool spindleBlindIsSet() const; + void unsetSpindle_blind(); + /// /// Spot count /// int64_t getSpots() const; @@ -224,6 +231,8 @@ protected: bool m_AngleIsSet; float m_Bkg; bool m_BkgIsSet; + float m_Spindle_blind; + bool m_Spindle_blindIsSet; int64_t m_Spots; bool m_SpotsIsSet; int64_t m_Spots_low_res; diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index 7b3fe7587..cbb5a71bb 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -84,6 +84,7 @@ components: type: string enum: - bkg_estimate + - spindle_blind_fraction - azint - azint_1d - spot_count @@ -1641,6 +1642,16 @@ components: type: number format: float description: Background estimate + spindle_blind: + type: number + format: float + description: > + Fraction (0-1) of a rotation sweep's blind double cone that this orientation + makes unrecoverable, assuming the nearest plausible lattice row is a lone + 2-fold (a worst-case bound, not an estimate). At 0.5 and above a recovery + protocol (second sweep, reorientation) should engage. Absent means the frame + could not be assessed, which automation must treat the same as above + threshold - it is not a zero. spots: type: integer format: int64 diff --git a/common/JFJochMessages.h b/common/JFJochMessages.h index 317d98eb2..c75e9dc45 100644 --- a/common/JFJochMessages.h +++ b/common/JFJochMessages.h @@ -377,6 +377,9 @@ struct EndMessage { std::optional efficiency; std::optional indexing_rate; std::optional bkg_estimate; + // Run mean of the per-image spindle_blind_fraction, over the frames that had a value; absent + // when none did. Written to /entry/MX/spindleBlindFractionMean. + std::optional spindle_blind_fraction; std::optional end_date; @@ -427,6 +430,10 @@ struct EndMessage { std::vector image_indexed; std::vector indexed_lattice_count; std::vector v_bkg_estimate; + // Per-image spindle_blind_fraction; NaN where the frame had no value. NaN is the CANNOT-SAY + // state, which automation must treat as "engage" - it is not a zero (see SpindleTrigger in + // image_analysis/indexing/SpindleBlindFraction.h). + std::vector v_spindle_blind_fraction; std::vector profile_radius; std::vector mosaicity; std::vector bFactor; diff --git a/common/JFJochReceiverPlots.cpp b/common/JFJochReceiverPlots.cpp index 239606614..ebc64a11d 100644 --- a/common/JFJochReceiverPlots.cpp +++ b/common/JFJochReceiverPlots.cpp @@ -61,6 +61,7 @@ void JFJochReceiverPlots::Setup(const DiffractionExperiment &experiment, const A xfel_event_code.reserve(r); } bkg_estimate.Clear(r); + spindle_blind_fraction.Clear(r); ice_ring_score.Clear(r); spot_count.Clear(r); spot_count_low_res.Clear(r); @@ -128,6 +129,7 @@ void JFJochReceiverPlots::Setup(const DiffractionExperiment &experiment, const A void JFJochReceiverPlots::Add(const DataMessage &msg, const AzimuthalIntegrationProfile &profile) { bkg_estimate.AddElement(msg.number, msg.bkg_estimate); + spindle_blind_fraction.AddElement(msg.number, msg.spindle_blind_fraction); ice_ring_score.AddElement(msg.number, msg.ice_ring_score); resolution_estimate.AddElement(msg.number, msg.resolution_estimate); spot_count.AddElement(msg.number, msg.spot_count); @@ -277,6 +279,9 @@ MultiLinePlot JFJochReceiverPlots::GetPlots(const PlotRequest &request) { case PlotType::BkgEstimate: ret = bkg_estimate.GetMeanPlot(nbins, start, incr, request.fill_value); break; + case PlotType::SpindleBlindFraction: + ret = spindle_blind_fraction.GetMeanPlot(nbins, start, incr, request.fill_value); + break; case PlotType::IceRingScore: ret = ice_ring_score.GetMeanPlot(nbins, start, incr, request.fill_value); break; @@ -472,6 +477,14 @@ std::optional JFJochReceiverPlots::GetBkgEstimate() const { return {}; } +std::optional JFJochReceiverPlots::GetSpindleBlindFraction() const { + auto tmp = spindle_blind_fraction.Mean(); + if (std::isfinite(tmp)) + return tmp; + else + return {}; +} + std::optional JFJochReceiverPlots::GetResolutionEstimate() const { std::vector v = resolution_estimate.ExportArray(); std::erase_if(v, [](float x) { return !std::isfinite(x); }); @@ -573,6 +586,9 @@ void JFJochReceiverPlots::GetPlotRaw(std::vector &v, PlotType type, const case PlotType::BkgEstimate: v = bkg_estimate.ExportArray(); break; + case PlotType::SpindleBlindFraction: + v = spindle_blind_fraction.ExportArray(); + break; case PlotType::IceRingScore: v = ice_ring_score.ExportArray(); break; diff --git a/common/JFJochReceiverPlots.h b/common/JFJochReceiverPlots.h index a7ebdddf3..8a1656962 100644 --- a/common/JFJochReceiverPlots.h +++ b/common/JFJochReceiverPlots.h @@ -42,6 +42,7 @@ class JFJochReceiverPlots { AutoIncrVector xfel_event_code; StatusVector bkg_estimate; + StatusVector spindle_blind_fraction; StatusVector ice_ring_score; StatusVector spot_count; StatusVector spot_count_low_res; @@ -122,6 +123,8 @@ public: std::optional GetIndexingRate() const; std::optional GetBkgEstimate() const; + // Run mean of the per-image spindle severity, over the frames that had one; nothing when none did. + std::optional GetSpindleBlindFraction() const; std::optional GetIceRingScore() const; // Pooled over the run: spots on the hexagonal rings over the same q width of ice-free control // flanks. 1 = spots spread evenly, > 1 = they pile up on the rings (textured ice). diff --git a/common/Plot.h b/common/Plot.h index b629c1b44..3ae807f5e 100644 --- a/common/Plot.h +++ b/common/Plot.h @@ -9,7 +9,7 @@ #include enum class PlotType { - BkgEstimate, AzInt, AzInt1D, SpotCount, SpotCountLowRes, SpotCountIndexed, SpotCountIceRing, + BkgEstimate, SpindleBlindFraction, AzInt, AzInt1D, SpotCount, SpotCountLowRes, SpotCountIndexed, SpotCountIceRing, IndexingRate, IndexingUnitCellLength, IndexingUnitCellAngle, ErrorPixels, SaturatedPixels, ImageCollectionEfficiency, ReceiverDelay, ReceiverFreeSendBuf, ROISum, ROIMean, ROIMaxCount, ROIPixels, ROIWeightedX, ROIWeightedY, PacketsReceived, MaxValue, diff --git a/common/ScanResult.h b/common/ScanResult.h index 92a409bdc..a7a96b49a 100644 --- a/common/ScanResult.h +++ b/common/ScanResult.h @@ -27,6 +27,7 @@ struct ScanResultElem { std::optional err_pixels; std::optional sat_pixels; std::optional bkg; + std::optional spindle_blind; std::optional spot_count; std::optional spot_count_low_res; std::optional spot_count_indexed; diff --git a/common/ScanResultGenerator.cpp b/common/ScanResultGenerator.cpp index 1510da683..841312d1c 100644 --- a/common/ScanResultGenerator.cpp +++ b/common/ScanResultGenerator.cpp @@ -43,6 +43,7 @@ void ScanResultGenerator::Add(const DataMessage &message) { v[image_number].pixel_sum = message.pixel_sum; v[image_number].collection_efficiency = message.image_collection_efficiency.value_or(1.0); v[image_number].bkg = message.bkg_estimate; + v[image_number].spindle_blind = message.spindle_blind_fraction; v[image_number].spot_count = message.spot_count; v[image_number].indexing_solution = message.indexing_result; v[image_number].indexed_lattice_count = message.indexing_lattice_count; @@ -99,6 +100,7 @@ void ScanResultGenerator::FillEndMessage(EndMessage &message) const { message.spot_count_indexed.resize(n); message.image_indexed.resize(n); message.v_bkg_estimate.resize(n); + message.v_spindle_blind_fraction.resize(n); message.profile_radius.resize(n); message.mosaicity.resize(n); message.bFactor.resize(n); @@ -131,6 +133,7 @@ void ScanResultGenerator::FillEndMessage(EndMessage &message) const { message.spot_count_indexed[number] = static_cast(value_or_zero(e.spot_count_indexed)); message.image_indexed[number] = static_cast(e.indexing_solution.value_or(0)); message.v_bkg_estimate[number] = e.bkg.value_or(NAN); + message.v_spindle_blind_fraction[number] = e.spindle_blind.value_or(NAN); message.profile_radius[number] = e.profile_radius.value_or(NAN); message.mosaicity[number] = e.mosaicity.value_or(NAN); message.bFactor[number] = e.b_factor.value_or(NAN); diff --git a/docs/CBOR.md b/docs/CBOR.md index 095c256a8..6c4d3e36a 100644 --- a/docs/CBOR.md +++ b/docs/CBOR.md @@ -225,6 +225,7 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s | packets_received | uint64 | Number of packets received per image (in units of 2 kB) | | | | bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A \[photons\] | | | | ice_ring_score | float | Strongest hexagonal-ice ring intensity over the smooth radial background (1 = no ice) | | | +| spindle_blind_fraction | float | Fraction (0-1) of a rotation sweep's blind cone this orientation makes unrecoverable, as a lone-2-fold worst-case bound; >= 0.5 should engage a recovery protocol, and ABSENT means the frame could not be assessed, which automation must treat the same way | | | | spot_count_ice_control | float | Spots in the ice-free flanks beside the hexagonal rings, rescaled to the ring bands' own q width (control for spot_count_ice_rings) | | | | beam_corr_x | float | Beam center correction X applied during processing \[pixel\] | | X | | beam_corr_y | float | Beam center correction Y applied during processing \[pixel\] | | X | @@ -301,6 +302,7 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s | adu_histogram_bin_width | uint64 | Width of bins in the above histogram \[ADU\] | | | max_receiver_delay | uint64 | Internal performance of Jungfraujoch | | | bkg_estimate | float | Mean background estimate for the whole run | | +| spindle_blind_fraction | float | Run mean of the per-image spindle_blind_fraction, over the frames that had one | | | indexing_rate | float | Mean indexing rate for the whole run | | | unit_cell | object (optional) | Unit cell of the system, based on the actual experiment: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\] | | | rotation_lattice_type | object | Bravais lattice classification of the total rotation solution over the run, if available; same schema as `lattice_type` | | @@ -316,6 +318,7 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s | spot_count_indexed | Array(int32) | Per-image number of spots fitting indexing solution | | | image_indexed | Array(uint8) | Per-image indexing result; 0 = not indexed, nonzero = indexed | | | v_bkg_estimate | Array(float) | Per-image background estimate | | +| v_spindle_blind_fraction | Array(float) | Per-image spindle_blind_fraction; NaN where the frame had no value (which is "cannot say", not zero) | | | ice_ring_score | Array(float) | Per-image strongest ice-ring intensity over the smooth radial background (1 = no ice) | | | spot_count_ice_control | Array(float) | Per-image spot count in the ice-free flanks beside the hexagonal rings, rescaled to the ring bands' q width | | | ice_ring_score_mean | float | Mean ice-ring score for the whole run (1 = no ice) | | diff --git a/docs/HDF5.md b/docs/HDF5.md index 1bd2c6ebb..3442b2b22 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -346,6 +346,7 @@ In legacy/VDS mode these live in the data files and are linked/virtual-stacked i | `integratedReflections` | | number of integrated reflections | | `bkgEstimate` | photons | mean background in the 3–5 Å resolution band | | `iceRingScore` | ratio | strongest hexagonal-ice ring intensity over the smooth radial background (1 = no ice) | +| `spindleBlindFraction` | fraction (0-1) | how much of a rotation sweep's blind cone this orientation makes unrecoverable, as a lone-2-fold worst-case bound; NaN = the frame could not be assessed, which automation must treat like a value at or above the 0.5 trigger, never as 0 | | `beam_corr_x`, `beam_corr_y` | pixel | beam-center correction applied during processing | | `imageScaleFactor` | | on-the-fly per-image scale factor *g* | | `imageScaleCC` | | on-the-fly scaling correlation coefficient | @@ -367,6 +368,7 @@ variants. | `rotationLatticeNiggliClass` | | Niggli class of the run lattice | | `imageIndexedMean` | | mean indexing rate over the run | | `bkgEstimateMean` | photons | mean background over the run | +| `spindleBlindFractionMean` | fraction (0-1) | mean `spindleBlindFraction` over the frames that had one | | `iceRingScoreMean` | ratio | mean `iceRingScore` over the run — the single "how icy was this dataset" number (1 = no ice) | | `indexedLatticeCount` | | per-image lattice count summary (master). *Note: data files use `indexingLatticeCount`; readers accept either.* | | `reindexMatrix` | | change of basis from the setting the per-image data are in to the setting of `/entry/sample/unit_cell` (`[9]`, `int32`, flattened 3×3, row major) — see below | diff --git a/frame_serialize/CBORStream2Deserializer.cpp b/frame_serialize/CBORStream2Deserializer.cpp index 658e5a1cb..bb2daba1e 100644 --- a/frame_serialize/CBORStream2Deserializer.cpp +++ b/frame_serialize/CBORStream2Deserializer.cpp @@ -820,6 +820,8 @@ namespace { message.bkg_estimate = GetCBORFloat(value); else if (key == "ice_ring_score") message.ice_ring_score = GetCBORFloat(value); + else if (key == "spindle_blind_fraction") + message.spindle_blind_fraction = GetCBORFloat(value); else if (key == "adu_histogram") GetCBORUInt64Array(value, message.adu_histogram); else if (key == "beam_corr_x") @@ -1475,6 +1477,8 @@ namespace { message.adu_histogram_bin_width = GetCBORUInt(value); else if (key == "bkg_estimate") message.bkg_estimate = GetCBORFloat(value); + else if (key == "spindle_blind_fraction") + message.spindle_blind_fraction = GetCBORFloat(value); else if (key == "indexing_rate") message.indexing_rate = GetCBORFloat(value); else if (key == "indexed_lattice_count") @@ -1493,6 +1497,8 @@ namespace { GetCBORUInt8Array(value, message.image_indexed); else if (key == "v_bkg_estimate") GetCBORFloatArray(value, message.v_bkg_estimate); + else if (key == "v_spindle_blind_fraction") + GetCBORFloatArray(value, message.v_spindle_blind_fraction); else if (key == "ice_ring_score") GetCBORFloatArray(value, message.ice_ring_score); else if (key == "spot_count_ice_control") diff --git a/frame_serialize/CBORStream2Serializer.cpp b/frame_serialize/CBORStream2Serializer.cpp index 7a17da3f9..1ee9bed6b 100644 --- a/frame_serialize/CBORStream2Serializer.cpp +++ b/frame_serialize/CBORStream2Serializer.cpp @@ -801,6 +801,7 @@ void CBORStream2Serializer::SerializeSequenceEnd(const EndMessage& message) { CBOR_ENC(mapEncoder, "max_receiver_delay", message.max_receiver_delay); CBOR_ENC(mapEncoder, "indexing_rate", message.indexing_rate); CBOR_ENC(mapEncoder, "bkg_estimate", message.bkg_estimate); + CBOR_ENC(mapEncoder, "spindle_blind_fraction", message.spindle_blind_fraction); CBOR_ENC(mapEncoder, "rotation_lattice_type", message.rotation_lattice_type); if (message.rotation_lattice.has_value()) @@ -820,6 +821,7 @@ void CBORStream2Serializer::SerializeSequenceEnd(const EndMessage& message) { CBOR_ENC(mapEncoder, "spot_count_indexed", message.spot_count_indexed); CBOR_ENC(mapEncoder, "image_indexed", message.image_indexed); CBOR_ENC(mapEncoder, "v_bkg_estimate", message.v_bkg_estimate); + CBOR_ENC(mapEncoder, "v_spindle_blind_fraction", message.v_spindle_blind_fraction); CBOR_ENC(mapEncoder, "ice_ring_score", message.ice_ring_score); CBOR_ENC(mapEncoder, "ice_ring_score_mean", message.ice_ring_score_mean); CBOR_ENC(mapEncoder, "spot_count_ice_control", message.spot_count_ice_control); @@ -918,6 +920,7 @@ void CBORStream2Serializer::SerializeImageInternal(CborEncoder &mapEncoder, cons CBOR_ENC(mapEncoder, "packets_received", message.packets_received); CBOR_ENC(mapEncoder, "bkg_estimate", message.bkg_estimate); CBOR_ENC(mapEncoder, "ice_ring_score", message.ice_ring_score); + CBOR_ENC(mapEncoder, "spindle_blind_fraction", message.spindle_blind_fraction); CBOR_ENC(mapEncoder, "adu_histogram", message.adu_histogram); CBOR_ENC(mapEncoder, "roi_integrals", message.roi); CBOR_ENC(mapEncoder, "beam_corr_x", message.beam_corr_x); diff --git a/frontend/src/client/types.gen.ts b/frontend/src/client/types.gen.ts index 812ee65da..b6e3e74b3 100644 --- a/frontend/src/client/types.gen.ts +++ b/frontend/src/client/types.gen.ts @@ -1080,6 +1080,11 @@ export type scan_result = { * Background estimate */ bkg?: number; + /** + * Fraction (0-1) of a rotation sweep's blind double cone that this orientation makes unrecoverable, assuming the nearest plausible lattice row is a lone 2-fold (a worst-case bound, not an estimate). At 0.5 and above a recovery protocol (second sweep, reorientation) should engage. Absent means the frame could not be assessed, which automation must treat the same as above threshold - it is not a zero. + * + */ + spindle_blind?: number; /** * Spot count */ @@ -1952,6 +1957,7 @@ export type fill_value = number; */ export const plot_type = { BKG_ESTIMATE: 'bkg_estimate', + SPINDLE_BLIND_FRACTION: 'spindle_blind_fraction', AZINT: 'azint', AZINT_1D: 'azint_1d', SPOT_COUNT: 'spot_count', @@ -3293,7 +3299,7 @@ export type getPreviewPlotData = { /** * Type of requested plot */ - type: 'bkg_estimate' | 'azint' | 'azint_1d' | 'spot_count' | 'spot_count_low_res' | 'spot_count_indexed' | 'spot_count_ice' | 'indexing_rate' | 'indexing_lattice_count' | 'indexing_unit_cell_length' | 'indexing_unit_cell_angle' | 'profile_radius' | 'mosaicity' | 'b_factor' | 'error_pixels' | 'saturated_pixels' | 'image_collection_efficiency' | 'receiver_delay' | 'receiver_free_send_buf' | 'strong_pixels' | 'roi_sum' | 'roi_mean' | 'roi_max_count' | 'roi_pixels' | 'roi_weighted_x' | 'roi_weighted_y' | 'packets_received' | 'max_pixel_value' | 'resolution_estimate' | 'pixel_sum' | 'processing_time' | 'beam_center_x' | 'beam_center_y' | 'integrated_reflections' | 'image_scale_factor' | 'image_scale_cc' | 'compression_ratio' | 'ice_ring_score'; + type: 'bkg_estimate' | 'spindle_blind_fraction' | 'azint' | 'azint_1d' | 'spot_count' | 'spot_count_low_res' | 'spot_count_indexed' | 'spot_count_ice' | 'indexing_rate' | 'indexing_lattice_count' | 'indexing_unit_cell_length' | 'indexing_unit_cell_angle' | 'profile_radius' | 'mosaicity' | 'b_factor' | 'error_pixels' | 'saturated_pixels' | 'image_collection_efficiency' | 'receiver_delay' | 'receiver_free_send_buf' | 'strong_pixels' | 'roi_sum' | 'roi_mean' | 'roi_max_count' | 'roi_pixels' | 'roi_weighted_x' | 'roi_weighted_y' | 'packets_received' | 'max_pixel_value' | 'resolution_estimate' | 'pixel_sum' | 'processing_time' | 'beam_center_x' | 'beam_center_y' | 'integrated_reflections' | 'image_scale_factor' | 'image_scale_cc' | 'compression_ratio' | 'ice_ring_score'; /** * Fill value for elements that were missed during data collection * @@ -3340,7 +3346,7 @@ export type getPreviewPlotBinData = { /** * Type of requested plot */ - type: 'bkg_estimate' | 'azint' | 'azint_1d' | 'spot_count' | 'spot_count_low_res' | 'spot_count_indexed' | 'spot_count_ice' | 'indexing_rate' | 'indexing_lattice_count' | 'indexing_unit_cell_length' | 'indexing_unit_cell_angle' | 'profile_radius' | 'mosaicity' | 'b_factor' | 'error_pixels' | 'saturated_pixels' | 'image_collection_efficiency' | 'receiver_delay' | 'receiver_free_send_buf' | 'strong_pixels' | 'roi_sum' | 'roi_mean' | 'roi_max_count' | 'roi_pixels' | 'roi_weighted_x' | 'roi_weighted_y' | 'packets_received' | 'max_pixel_value' | 'resolution_estimate' | 'pixel_sum' | 'processing_time' | 'beam_center_x' | 'beam_center_y' | 'integrated_reflections' | 'image_scale_factor' | 'image_scale_cc' | 'compression_ratio' | 'ice_ring_score'; + type: 'bkg_estimate' | 'spindle_blind_fraction' | 'azint' | 'azint_1d' | 'spot_count' | 'spot_count_low_res' | 'spot_count_indexed' | 'spot_count_ice' | 'indexing_rate' | 'indexing_lattice_count' | 'indexing_unit_cell_length' | 'indexing_unit_cell_angle' | 'profile_radius' | 'mosaicity' | 'b_factor' | 'error_pixels' | 'saturated_pixels' | 'image_collection_efficiency' | 'receiver_delay' | 'receiver_free_send_buf' | 'strong_pixels' | 'roi_sum' | 'roi_mean' | 'roi_max_count' | 'roi_pixels' | 'roi_weighted_x' | 'roi_weighted_y' | 'packets_received' | 'max_pixel_value' | 'resolution_estimate' | 'pixel_sum' | 'processing_time' | 'beam_center_x' | 'beam_center_y' | 'integrated_reflections' | 'image_scale_factor' | 'image_scale_cc' | 'compression_ratio' | 'ice_ring_score'; /** * Name of ROI for which plot is requested */ diff --git a/frontend/src/client/zod.gen.ts b/frontend/src/client/zod.gen.ts index da9254f7f..a34f337b4 100644 --- a/frontend/src/client/zod.gen.ts +++ b/frontend/src/client/zod.gen.ts @@ -437,6 +437,7 @@ export const zScanResult = z.object({ ny: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }).optional(), angle: z.number().optional(), bkg: z.number().optional(), + spindle_blind: z.number().optional(), spots: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }).optional(), spots_low_res: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }).optional(), spots_indexed: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }).optional(), @@ -893,6 +894,7 @@ export const zFillValue = z.number(); */ export const zPlotType = z.enum([ 'bkg_estimate', + 'spindle_blind_fraction', 'azint', 'azint_1d', 'spot_count', @@ -1202,6 +1204,7 @@ export const zGetPreviewPlotQuery = z.object({ binning: z.int().optional().default(1), type: z.enum([ 'bkg_estimate', + 'spindle_blind_fraction', 'azint', 'azint_1d', 'spot_count', @@ -1257,6 +1260,7 @@ export const zGetPreviewPlotResponse = zPlots; export const zGetPreviewPlotBinQuery = z.object({ type: z.enum([ 'bkg_estimate', + 'spindle_blind_fraction', 'azint', 'azint_1d', 'spot_count', diff --git a/frontend/src/components/DataProcessingPlot.tsx b/frontend/src/components/DataProcessingPlot.tsx index 8843dc1b2..9790624c7 100644 --- a/frontend/src/components/DataProcessingPlot.tsx +++ b/frontend/src/components/DataProcessingPlot.tsx @@ -54,6 +54,8 @@ function AxisTypeY(plot: plot_type) : string | ReactNode { return "Count"; case plot_type.ICE_RING_SCORE: return "Ratio"; + case plot_type.SPINDLE_BLIND_FRACTION: + return "Fraction"; case plot_type.AZINT: case plot_type.AZINT_1D: case plot_type.BKG_ESTIMATE: diff --git a/frontend/src/components/DataProcessingPlots.tsx b/frontend/src/components/DataProcessingPlots.tsx index 7e950811c..bff4a20bf 100644 --- a/frontend/src/components/DataProcessingPlots.tsx +++ b/frontend/src/components/DataProcessingPlots.tsx @@ -54,6 +54,7 @@ function DataProcessingPlots({type: initialType, height}: MyProps) { Azimuthal integration profile Azimuthal integration profile (1D) Background estimate + Spindle blind fraction Diffraction resolution estimate ROI area sum ROI area mean diff --git a/reader/HDF5MetadataSource.cpp b/reader/HDF5MetadataSource.cpp index 8b12c1fab..fa86cf8d6 100644 --- a/reader/HDF5MetadataSource.cpp +++ b/reader/HDF5MetadataSource.cpp @@ -557,6 +557,7 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen dataset->indexing_result = master_file->ReadOptVector("/entry/MX/imageIndexed"); dataset->bkg_estimate = master_file->ReadOptVector("/entry/MX/bkgEstimate"); + dataset->spindle_blind_fraction = master_file->ReadOptVector("/entry/MX/spindleBlindFraction"); dataset->ice_ring_score = master_file->ReadOptVector("/entry/MX/iceRingScore"); dataset->resolution_estimate = master_file->ReadOptVector("/entry/MX/resolutionEstimate"); dataset->profile_radius = master_file->ReadOptVector("/entry/MX/profileRadius"); @@ -700,6 +701,10 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen data_file, "/entry/MX/bkgEstimate", number_of_images, fimages); + ReadVector(dataset->spindle_blind_fraction, + data_file, "/entry/MX/spindleBlindFraction", + number_of_images, fimages); + ReadVector(dataset->ice_ring_score, data_file, "/entry/MX/iceRingScore", number_of_images, fimages); @@ -1327,6 +1332,11 @@ void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_im message.indexing_lattice_count = dataset->indexing_lattice_count[image_number]; if (dataset->bkg_estimate.size() > image_number) message.bkg_estimate = dataset->bkg_estimate[image_number]; + // NaN is how the file stores a frame with no value; the optional must come back absent, not + // carrying a NaN, because absence is the CANNOT-SAY trigger state and a value is not. + if (dataset->spindle_blind_fraction.size() > image_number + && std::isfinite(dataset->spindle_blind_fraction[image_number])) + message.spindle_blind_fraction = dataset->spindle_blind_fraction[image_number]; if (dataset->ice_ring_score.size() > image_number) message.ice_ring_score = dataset->ice_ring_score[image_number]; if (dataset->efficiency.size() > image_number) diff --git a/reader/JFJochReaderDataset.h b/reader/JFJochReaderDataset.h index ab6d36e8a..cc50376eb 100644 --- a/reader/JFJochReaderDataset.h +++ b/reader/JFJochReaderDataset.h @@ -56,6 +56,8 @@ struct JFJochReaderDataset { std::vector indexing_result; std::vector indexing_lattice_count; std::vector bkg_estimate; + // NaN entries are frames with no value (CANNOT SAY), not zeros. + std::vector spindle_blind_fraction; std::vector ice_ring_score; std::vector resolution_estimate; std::vector efficiency; diff --git a/reader/JFJochReaderImage.cpp b/reader/JFJochReaderImage.cpp index 2d22b00a3..bb5688c1e 100644 --- a/reader/JFJochReaderImage.cpp +++ b/reader/JFJochReaderImage.cpp @@ -198,6 +198,7 @@ void JFJochReaderImage::AddImage(const JFJochReaderImage &other) { message.indexing_result = false; message.resolution_estimate = {}; message.bkg_estimate = {}; + message.spindle_blind_fraction = {}; message.spots = {}; error_pixel.clear(); diff --git a/receiver/JFJochReceiver.cpp b/receiver/JFJochReceiver.cpp index 7c1a21799..8c4aca8b3 100644 --- a/receiver/JFJochReceiver.cpp +++ b/receiver/JFJochReceiver.cpp @@ -162,6 +162,7 @@ void JFJochReceiver::SendEndMessage() { message.run_name = experiment.GetRunName(); message.bkg_estimate = plots.GetBkgEstimate(); + message.spindle_blind_fraction = plots.GetSpindleBlindFraction(); message.ice_ring_score_mean = plots.GetIceRingScore(); message.indexing_rate = plots.GetIndexingRate(); diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 72a07c51e..469233cb6 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -3532,6 +3532,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b end_msg.run_number = experiment_.GetRunNumber(); end_msg.run_name = experiment_.GetRunName(); end_msg.bkg_estimate = plots.GetBkgEstimate(); + end_msg.spindle_blind_fraction = plots.GetSpindleBlindFraction(); result.spot_resolution_estimate_A = plots.GetResolutionEstimate(); end_msg.ice_ring_score = plots.GetIceRingScoreArray(); end_msg.ice_ring_score_mean = plots.GetIceRingScore(); diff --git a/tests/CBORTest.cpp b/tests/CBORTest.cpp index 4c36ea360..638506425 100644 --- a/tests/CBORTest.cpp +++ b/tests/CBORTest.cpp @@ -497,6 +497,7 @@ TEST_CASE("CBORSerialize_End", "[CBOR]") { .images_sent_to_write_count = 40000, .max_receiver_delay = 3456, .efficiency = 0.99, + .spindle_blind_fraction = 0.31f, .end_date = "ccc", .run_name = "bla5", .run_number = 45676782, @@ -505,7 +506,8 @@ TEST_CASE("CBORSerialize_End", "[CBOR]") { .niggli_class = 22, .crystal_system = gemmi::CrystalSystem::Tetragonal, }, - .rotation_lattice = CrystalLattice(40, 50, 60, 90, 90, 90) + .rotation_lattice = CrystalLattice(40, 50, 60, 90, 90, 90), + .v_spindle_blind_fraction = {0.25f, NAN, 1.0f} }; REQUIRE_NOTHROW(serializer.SerializeSequenceEnd(message)); @@ -533,6 +535,14 @@ TEST_CASE("CBORSerialize_End", "[CBOR]") { CHECK(output_message.rotation_lattice_type->niggli_class == 22); REQUIRE(output_message.rotation_lattice.has_value()); CHECK(output_message.rotation_lattice->GetUnitCell().c == Catch::Approx(60.0)); + + REQUIRE(output_message.spindle_blind_fraction == message.spindle_blind_fraction); + // The per-image vector holds NaN where a frame had no value (CANNOT SAY); the hole must + // survive the round trip as a hole, not as a number. + REQUIRE(output_message.v_spindle_blind_fraction.size() == 3); + CHECK(output_message.v_spindle_blind_fraction[0] == Catch::Approx(0.25f)); + CHECK(std::isnan(output_message.v_spindle_blind_fraction[1])); + CHECK(output_message.v_spindle_blind_fraction[2] == Catch::Approx(1.0f)); } TEST_CASE("CBORSerialize_End_SpaceGroup", "[CBOR]") { @@ -663,6 +673,7 @@ TEST_CASE("CBORSerialize_Image", "[CBOR]") { .spots = spots, .spot_count_ice_rings = 157, .bkg_estimate = 12.345f, + .spindle_blind_fraction = 0.62f, .indexing_result = true, .indexing_unit_cell = UnitCell{.a = 123, .b = 145, .c = 67.5, .alpha = 90, .beta = 120, .gamma = 134}, .adu_histogram = {3, 4, 5, 8}, @@ -716,6 +727,7 @@ TEST_CASE("CBORSerialize_Image", "[CBOR]") { REQUIRE(image_array.error_pixel_count == message.error_pixel_count); REQUIRE(image_array.strong_pixel_count == message.strong_pixel_count); REQUIRE(image_array.bkg_estimate == message.bkg_estimate); + REQUIRE(image_array.spindle_blind_fraction == message.spindle_blind_fraction); REQUIRE(image_array.image_collection_efficiency == message.image_collection_efficiency); REQUIRE(image_array.user_data == message.user_data); REQUIRE(image_array.original_number == message.original_number); @@ -1310,6 +1322,7 @@ TEST_CASE("CBORSerialize_Metadata", "[CBOR]") { REQUIRE(deserialized->metadata->images.size() == 2); CHECK(deserialized->metadata->images.at(0).number == 172); CHECK(deserialized->metadata->images.at(0).bkg_estimate == 45); + CHECK(!deserialized->metadata->images.at(0).spindle_blind_fraction.has_value()); CHECK(deserialized->metadata->images.at(0).image.GetWidth() == 0); CHECK(deserialized->metadata->images.at(1).number == 173); diff --git a/tests/JFJochReaderTest.cpp b/tests/JFJochReaderTest.cpp index 1b3e422b5..3dfa26fe0 100644 --- a/tests/JFJochReaderTest.cpp +++ b/tests/JFJochReaderTest.cpp @@ -636,6 +636,10 @@ TEST_CASE("JFJochReader_DataI16", "[HDF5][Full]") { message.spots = spots; message.indexing_result = (i % 2 == 0); message.bkg_estimate = i * 345.6; + // Only one frame has a spindle severity: the others must come back ABSENT (stored as + // NaN), because no value is the CANNOT-SAY trigger state and a zero is not. + if (i == 2) + message.spindle_blind_fraction = 0.75f; message.number = i; message.profile_radius = 123.09; generator.Add(message); @@ -656,6 +660,13 @@ TEST_CASE("JFJochReader_DataI16", "[HDF5][Full]") { REQUIRE(dataset->spot_count.size() == 4); REQUIRE(dataset->bkg_estimate.size() == 4); REQUIRE(dataset->profile_radius.size() == 4); + REQUIRE(dataset->spindle_blind_fraction.size() == 4); + for (int i = 0; i < 4; i++) { + if (i == 2) + CHECK(dataset->spindle_blind_fraction[i] == Catch::Approx(0.75f)); + else + CHECK(std::isnan(dataset->spindle_blind_fraction[i])); + } REQUIRE_THROWS(reader.LoadImage(4)); std::shared_ptr reader_image; diff --git a/viewer/JFJochHttpReader.cpp b/viewer/JFJochHttpReader.cpp index 1c0bf8c81..b4c6e0f98 100644 --- a/viewer/JFJochHttpReader.cpp +++ b/viewer/JFJochHttpReader.cpp @@ -267,6 +267,7 @@ std::shared_ptr JFJochHttpReader::UpdateDataset_i() { dataset->experiment.FluorescenceSpectrum(msg->start_message->fluorescence_spectrum); dataset->bkg_estimate = GetPlot_i("bkg_estimate"); + dataset->spindle_blind_fraction = GetPlot_i("spindle_blind_fraction"); dataset->ice_ring_score = GetPlot_i("ice_ring_score"); dataset->spot_count = GetPlot_i("spot_count"); dataset->spot_count_ice_rings = GetPlot_i("spot_count_ice"); diff --git a/viewer/JFJochProcessController.cpp b/viewer/JFJochProcessController.cpp index dd859518e..752ea0a4b 100644 --- a/viewer/JFJochProcessController.cpp +++ b/viewer/JFJochProcessController.cpp @@ -122,6 +122,7 @@ void JFJochProcessController::OnImageProcessed(const DataMessage &msg) { if (msg.indexing_result) put(d.indexing_result, *msg.indexing_result ? 1.0f : 0.0f); if (msg.indexing_lattice_count) put(d.indexing_lattice_count, *msg.indexing_lattice_count); if (msg.bkg_estimate) put(d.bkg_estimate, *msg.bkg_estimate); + if (msg.spindle_blind_fraction) put(d.spindle_blind_fraction, *msg.spindle_blind_fraction); if (msg.resolution_estimate) put(d.resolution_estimate, *msg.resolution_estimate); if (msg.profile_radius) put(d.profile_radius, *msg.profile_radius); if (msg.mosaicity_deg) put(d.mosaicity_deg, *msg.mosaicity_deg); diff --git a/viewer/JFJochViewerDatasetInfo.cpp b/viewer/JFJochViewerDatasetInfo.cpp index cd2bd1b1f..237836734 100644 --- a/viewer/JFJochViewerDatasetInfo.cpp +++ b/viewer/JFJochViewerDatasetInfo.cpp @@ -138,6 +138,7 @@ void JFJochViewerDatasetInfo::UpdateLabels() { if (this->dataset) { combo_box->addItem("Background estimate", 0); combo_box->addItem("Ice ring score", 14); + combo_box->addItem("Spindle blind fraction", 15); combo_box->addItem("Resolution estimate", 7); combo_box->addItem("Spot count", 1); combo_box->addItem("Spot count (indexed)", 2); @@ -237,6 +238,7 @@ std::vector JFJochViewerDatasetInfo::ExtractMetric(const JFJochReaderData else if (val == 12) data = ds.integrated_reflections; else if (val == 13) data = ds.indexing_lattice_count; else if (val == 14) data = ds.ice_ring_score; + else if (val == 15) data = ds.spindle_blind_fraction; else if (val >= 100) { const int roi_index = (val - 100) / 4; if (val % 4 == 0) { diff --git a/writer/HDF5DataFilePluginMX.cpp b/writer/HDF5DataFilePluginMX.cpp index 808a2006a..ea4c7d81a 100644 --- a/writer/HDF5DataFilePluginMX.cpp +++ b/writer/HDF5DataFilePluginMX.cpp @@ -54,6 +54,7 @@ HDF5DataFilePluginMX::HDF5DataFilePluginMX(const StartMessage &msg) void HDF5DataFilePluginMX::OpenFile(HDF5File &data_file, const DataMessage &msg, size_t images_per_file) { bkg_estimate.reserve(images_per_file); ice_ring_score.reserve(images_per_file); + spindle_blind_fraction.reserve(images_per_file); if (max_spots == 0) return; @@ -101,6 +102,8 @@ void HDF5DataFilePluginMX::Write(const DataMessage &msg, uint64_t image_number) bkg_estimate[image_number] = msg.bkg_estimate.value(); if (msg.ice_ring_score.has_value()) ice_ring_score[image_number] = msg.ice_ring_score.value(); + if (msg.spindle_blind_fraction.has_value()) + spindle_blind_fraction[image_number] = msg.spindle_blind_fraction.value(); if (max_spots == 0) return; @@ -252,6 +255,8 @@ void HDF5DataFilePluginMX::WriteFinal(HDF5File &data_file) { data_file.SaveVector("/entry/MX/bkgEstimate", bkg_estimate.vec()); if (!ice_ring_score.empty()) data_file.SaveVector("/entry/MX/iceRingScore", ice_ring_score.vec()); + if (!spindle_blind_fraction.empty()) + data_file.SaveVector("/entry/MX/spindleBlindFraction", spindle_blind_fraction.vec()); if (!profile_radius.empty()) data_file.SaveVector("/entry/MX/profileRadius", profile_radius.vec())->Units("Angstrom^-1"); if (!mosaicity_deg.empty()) diff --git a/writer/HDF5DataFilePluginMX.h b/writer/HDF5DataFilePluginMX.h index 68a568edb..d369f8b0d 100644 --- a/writer/HDF5DataFilePluginMX.h +++ b/writer/HDF5DataFilePluginMX.h @@ -48,6 +48,8 @@ class HDF5DataFilePluginMX : public HDF5DataFilePlugin { // bkg_estimate AutoIncrVector bkg_estimate{NAN}; AutoIncrVector ice_ring_score{NAN}; + // NaN = the frame had no value (CANNOT SAY), which is not a zero - see SpindleTrigger. + AutoIncrVector spindle_blind_fraction{NAN}; // resolution_estimation AutoIncrVector resolution_estimate{NAN}; diff --git a/writer/HDF5NXmx.cpp b/writer/HDF5NXmx.cpp index d1eecac63..a0b1be62b 100644 --- a/writer/HDF5NXmx.cpp +++ b/writer/HDF5NXmx.cpp @@ -1179,6 +1179,9 @@ void NXmx::Finalize(const EndMessage &end) { if (end.bkg_estimate) { SaveScalar(*hdf5_file, "/entry/MX/bkgEstimateMean", end.bkg_estimate.value()); } + if (end.spindle_blind_fraction) { + SaveScalar(*hdf5_file, "/entry/MX/spindleBlindFractionMean", end.spindle_blind_fraction.value()); + } if (end.ice_ring_score_mean) { SaveScalar(*hdf5_file, "/entry/MX/iceRingScoreMean", end.ice_ring_score_mean.value()); } @@ -1258,6 +1261,7 @@ void NXmx::EndResultVectors(const EndMessage &end) { SaveVectorIfMissing(*hdf5_file, "/entry/MX/imageIndexed", end.image_indexed); SaveVectorIfMissing(*hdf5_file, "/entry/MX/indexedLatticeCount", end.indexed_lattice_count); SaveVectorIfMissing(*hdf5_file, "/entry/MX/bkgEstimate", end.v_bkg_estimate); + SaveVectorIfMissing(*hdf5_file, "/entry/MX/spindleBlindFraction", end.v_spindle_blind_fraction); SaveVectorIfMissing(*hdf5_file, "/entry/MX/iceRingScore", end.ice_ring_score); SaveVectorIfMissing(*hdf5_file, "/entry/MX/profileRadius", end.profile_radius, "Angstrom^-1"); SaveVectorIfMissing(*hdf5_file, "/entry/MX/mosaicity", end.mosaicity, "deg"); -- 2.54.0 From 307987c86510a9eb97b794a450b1bc2fe5249e1b Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 10:41:50 +0200 Subject: [PATCH 20/75] rugnux: the mounting's cost is computed exactly from the measured group, not guessed from an angle Offline, a merged rotation run has what a still lacks - a determined point group and an exact indexed orientation - so the run-level number no longer needs the pessimistic presumed-diad bound, and it no longer uses the nearest-axis angle either. That 15-deg warning heuristic was wrong in both directions: an aligned in-plane 2-fold of a dihedral group is repaired by the principal axis, a cubic group is never severe in any orientation, and a lone diad perpendicular to the spindle is severe with no axis anywhere near the spindle at all. The group's proper rotations are applied to the sweep's blind double cone in the crystal's actual orientation, and what no operator maps onto measured territory is counted, weighted by each shell's own cone width so the result is a fraction of unique reflections to this run's resolution limit. Friedel and the improper operators need no separate handling - the cone and the measured region are both inversion-symmetric. The number is machine-readable on purpose: SPINDLE_LOST_UNIQUE_FRACTION in the report (0-1, a bare number a pipeline can act on) and /entry/MX/spindleLostUniqueFraction in the master, with the warning prose only on top of it, fired when the group recovers less than half the cone's content. REPORT_VERSION stays 7: the format's own rule is that adding a key does not move it. This also settles what the nearest-axis keys hedged: with the measured group the mounting is cleared or convicted exactly, so their documentation now calls them descriptive and points at the new key for the verdict. Verified against Monte Carlo: P1 loses 2.0% of unique reflections at theta_max = 15 deg with nothing repaired; a lone diad on or perpendicular to the spindle repairs nothing; an axis of order >= 3 perpendicular to the spindle repairs everything; 622 with an in-plane diad on the spindle loses nothing; cubic loses nothing in any orientation. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- common/JFJochMessages.h | 5 + docs/CBOR.md | 1 + docs/HDF5.md | 1 + docs/RUGNUX_REPORT.md | 20 ++-- frame_serialize/CBORStream2Deserializer.cpp | 2 + frame_serialize/CBORStream2Serializer.cpp | 1 + rugnux/CMakeLists.txt | 2 + rugnux/ResultReport.cpp | 8 ++ rugnux/Rugnux.cpp | 64 +++++++---- rugnux/Rugnux.h | 6 ++ rugnux/SpindleCuspLoss.cpp | 111 ++++++++++++++++++++ rugnux/SpindleCuspLoss.h | 36 +++++++ tests/CBORTest.cpp | 2 + tests/CMakeLists.txt | 1 + tests/SpindleCuspLossTest.cpp | 93 ++++++++++++++++ writer/HDF5NXmx.cpp | 4 + 16 files changed, 331 insertions(+), 26 deletions(-) create mode 100644 rugnux/SpindleCuspLoss.cpp create mode 100644 rugnux/SpindleCuspLoss.h create mode 100644 tests/SpindleCuspLossTest.cpp diff --git a/common/JFJochMessages.h b/common/JFJochMessages.h index c75e9dc45..c4dba77a5 100644 --- a/common/JFJochMessages.h +++ b/common/JFJochMessages.h @@ -380,6 +380,11 @@ struct EndMessage { // Run mean of the per-image spindle_blind_fraction, over the frames that had a value; absent // when none did. Written to /entry/MX/spindleBlindFractionMean. std::optional spindle_blind_fraction; + // The exact run-level counterpart, offline only (rugnux): the fraction (0-1) of unique + // reflections to the run's resolution limit that the MEASURED point group could not recover + // from the sweep's blind cone, in the crystal's indexed orientation. Not the per-image + // worst-case bound. Written to /entry/MX/spindleLostUniqueFraction. + std::optional spindle_lost_unique_fraction; std::optional end_date; diff --git a/docs/CBOR.md b/docs/CBOR.md index 6c4d3e36a..536a11803 100644 --- a/docs/CBOR.md +++ b/docs/CBOR.md @@ -303,6 +303,7 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s | max_receiver_delay | uint64 | Internal performance of Jungfraujoch | | | bkg_estimate | float | Mean background estimate for the whole run | | | spindle_blind_fraction | float | Run mean of the per-image spindle_blind_fraction, over the frames that had one | | +| spindle_lost_unique_fraction | float | Fraction (0-1) of unique reflections the mounting made unmeasurable, exact under the measured point group; offline (rugnux) only | | | indexing_rate | float | Mean indexing rate for the whole run | | | unit_cell | object (optional) | Unit cell of the system, based on the actual experiment: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\] | | | rotation_lattice_type | object | Bravais lattice classification of the total rotation solution over the run, if available; same schema as `lattice_type` | | diff --git a/docs/HDF5.md b/docs/HDF5.md index 3442b2b22..52a46f29c 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -369,6 +369,7 @@ variants. | `imageIndexedMean` | | mean indexing rate over the run | | `bkgEstimateMean` | photons | mean background over the run | | `spindleBlindFractionMean` | fraction (0-1) | mean `spindleBlindFraction` over the frames that had one | +| `spindleLostUniqueFraction` | fraction (0-1) | unique reflections (to the run's resolution limit) the mounting made unmeasurable, exact under the measured point group and indexed orientation; offline (rugnux) only | | `iceRingScoreMean` | ratio | mean `iceRingScore` over the run — the single "how icy was this dataset" number (1 = no ice) | | `indexedLatticeCount` | | per-image lattice count summary (master). *Note: data files use `indexingLatticeCount`; readers accept either.* | | `reindexMatrix` | | change of basis from the setting the per-image data are in to the setting of `/entry/sample/unit_cell` (`[9]`, `int32`, flattened 3×3, row major) — see below | diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 9f047eccc..52e5ac65d 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -243,13 +243,19 @@ nothing else. `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` and `SPINDLE_SYMMETRY_AXIS_ORDER=` in section 4 say how the crystal sat on the goniometer: the angle between the spindle and the nearest symmetry axis, and that axis's -order. Below about 15 deg the axis maps the sweep's blind cone onto itself and the reflections in -that cone stay missing however long the sweep runs, which is why the same number also raises a -warning there. It is written on every rotation run that determined a space group, including the -ordinary well-mounted case, so an incomplete cusp can be attributed to the mounting. A large angle -does not on its own clear it: an axis within the same margin of perpendicular to the spindle carries -the blind cone onto its opposite half, which is equally unmeasured, and only the nearest axis is -reported. (New in `REPORT_VERSION= 7`.) +order. They are descriptive: neither convicts nor clears the mounting on its own, because an aligned +axis of any order maps the sweep's blind cone onto itself while an axis near perpendicular does the +same only when it is a lone 2-fold, and only the nearest axis is reported. (New in +`REPORT_VERSION= 7`.) + +`SPINDLE_LOST_UNIQUE_FRACTION=` in section 4 is the exact verdict the angle cannot give: the fraction +(0-1, so 0.0300 means 3%) of unique reflections, to this run's resolution limit, that the mounting +made unmeasurable - the part of the sweep's blind double cone that no operator of the measured point +group maps onto measured territory, computed in the crystal's actual indexed orientation. 0.0000 +means the mounting cost nothing; the run warns when the group recovers less than half of the cone's +content. The same number is written to the master file as `/entry/MX/spindleLostUniqueFraction`, so a +pipeline can read it from either output without parsing prose. Written on every rotation run that +determined a space group and merged reflections. `SPACE_GROUP_ENANTIOMORPH=` in section 4 reads **`ASSUMED_FROM_MODEL`** when the hand written in the files is the model's. Assumed, not determined: merged intensities cannot see the hand at all — |F| is diff --git a/frame_serialize/CBORStream2Deserializer.cpp b/frame_serialize/CBORStream2Deserializer.cpp index bb2daba1e..367e85615 100644 --- a/frame_serialize/CBORStream2Deserializer.cpp +++ b/frame_serialize/CBORStream2Deserializer.cpp @@ -1479,6 +1479,8 @@ namespace { message.bkg_estimate = GetCBORFloat(value); else if (key == "spindle_blind_fraction") message.spindle_blind_fraction = GetCBORFloat(value); + else if (key == "spindle_lost_unique_fraction") + message.spindle_lost_unique_fraction = GetCBORFloat(value); else if (key == "indexing_rate") message.indexing_rate = GetCBORFloat(value); else if (key == "indexed_lattice_count") diff --git a/frame_serialize/CBORStream2Serializer.cpp b/frame_serialize/CBORStream2Serializer.cpp index 1ee9bed6b..9a86d4091 100644 --- a/frame_serialize/CBORStream2Serializer.cpp +++ b/frame_serialize/CBORStream2Serializer.cpp @@ -802,6 +802,7 @@ void CBORStream2Serializer::SerializeSequenceEnd(const EndMessage& message) { CBOR_ENC(mapEncoder, "indexing_rate", message.indexing_rate); CBOR_ENC(mapEncoder, "bkg_estimate", message.bkg_estimate); CBOR_ENC(mapEncoder, "spindle_blind_fraction", message.spindle_blind_fraction); + CBOR_ENC(mapEncoder, "spindle_lost_unique_fraction", message.spindle_lost_unique_fraction); CBOR_ENC(mapEncoder, "rotation_lattice_type", message.rotation_lattice_type); if (message.rotation_lattice.has_value()) diff --git a/rugnux/CMakeLists.txt b/rugnux/CMakeLists.txt index a8d0e7c24..8883c0df2 100644 --- a/rugnux/CMakeLists.txt +++ b/rugnux/CMakeLists.txt @@ -18,6 +18,8 @@ ADD_LIBRARY(Rugnux STATIC ResultReport.h SpotWidth.cpp SpotWidth.h + SpindleCuspLoss.cpp + SpindleCuspLoss.h WriteModel.cpp WriteModel.h ) diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index c77dffd06..15b9c4875 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -314,6 +314,14 @@ std::string RenderResultReport(const std::string &output_prefix, fmt::format("{:.1f}", *result.spindle_symmetry_axis_deg)); Key(os, "SPINDLE_SYMMETRY_AXIS_ORDER", std::to_string(result.spindle_symmetry_axis_order)); } + // The exact verdict on the mounting, which the angle above cannot give on its own: the + // fraction (0-1) of unique reflections, to this run's resolution limit, that the measured + // point group could not recover from the sweep's blind cone. 0.0000 means the mounting + // cost nothing. Machine-readable on purpose - a downstream pipeline decides on this + // number, not on the warning prose. + if (result.spindle_lost_unique_fraction.has_value()) + Key(os, "SPINDLE_LOST_UNIQUE_FRACTION", + fmt::format("{:.4f}", *result.spindle_lost_unique_fraction)); os << "\n SPACE_GROUP_ALTERNATIVES names every group these data cannot separate from the one\n" << " adopted (enantiomorphic partners, origin-ambiguous pairs, groups a gap in the data\n" diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 469233cb6..4b1a6fd9c 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -5,6 +5,7 @@ #include "Rugnux.h" #include "ModelValidation.h" #include "WriteModel.h" +#include "SpindleCuspLoss.h" #include "SpotWidth.h" #include @@ -4702,38 +4703,63 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // search itself made, so the text cannot claim "no twin law exists" on its own say-so. result.twinning.laue_class_was_chosen_by_promotion = promoted_point_group; - // Symmetry axis vs spindle. Reported always on rotation data (it is a property of how the - // crystal was mounted, which the user can change), warned about when the two nearly coincide. + // How the crystal sat on the spindle. Reported always on rotation data (it is a property of + // the mounting, which the user can change). The angle to the nearest symmetry axis is kept + // as a descriptive key, but the WARNING is decided by the exact orbit computation: the old + // any-proper-axis-within-15-deg rule was wrong in both directions - an aligned in-plane + // 2-fold of a dihedral group is repaired by the principal axis, a cubic group is never + // severe in any orientation, and a lone diad PERPENDICULAR to the spindle is severe while + // no axis is anywhere near it. if (experiment_.IsRotationIndexing() && twin_sg && end_msg.rotation_lattice.has_value()) { if (const auto gonio = experiment_.GetGoniometer()) { const auto closest = ClosestSymmetryAxisToSpindle(*twin_sg, *end_msg.rotation_lattice, gonio->GetAxis()); if (closest.has_value()) { - // Practical bound, not a derived one: the blind cone's half-angle is the maximum - // Bragg angle (~15 deg for 2 A data at 1 A), and measured on this battery a 13.6 deg - // case shows the loss while a 16.2 deg one is 99.7% complete. - constexpr double WARN_DEG = 15.0; const auto [angle, order] = *closest; result.spindle_symmetry_axis_deg = angle; result.spindle_symmetry_axis_order = order; - if (angle < WARN_DEG) { + stats_text << "Closest symmetry axis to the spindle: " << order << "-fold at " + << std::fixed << std::setprecision(1) << angle << " deg\n"; + } + + // The measured resolution of THIS run sets the cone: past merging the question is + // what these data lost, not what the detector could have reached. + double d_min = 0; + for (const auto &r : sm.merged) + if (std::isfinite(r.d) && r.d > 0 && (d_min == 0 || r.d < d_min)) + d_min = r.d; + if (const auto loss = SpindleUnrepairedFraction( + *twin_sg, *end_msg.rotation_lattice, gonio->GetAxis(), + experiment_.GetWavelength_A(), d_min)) { + result.spindle_lost_unique_fraction = loss->lost_unique_fraction; + end_msg.spindle_lost_unique_fraction = + static_cast(loss->lost_unique_fraction); + stats_text << "Unique reflections the mounting made unmeasurable: " + << std::fixed << std::setprecision(2) + << 100.0 * loss->lost_unique_fraction << "% (of the " + << std::setprecision(2) << 100.0 * (1.0 - std::cos(loss->theta_max_deg * PI / 180.0)) + << "% a sweep's blind cone holds at " << std::setprecision(2) + << d_min << " A)\n\n"; + // Warn when the point group recovers less than half the cone - the same half + // that fixes the per-image trigger threshold. The loss is a coherent cap about + // the spindle direction, which costs a map more than the same percentage lost + // at random. + if (loss->cone_fraction >= 0.5) { const std::string msg = fmt::format( - "The crystal's {}-fold axis is only {:.1f} deg from the spindle. A rotation sweep " - "never records the reflections whose reciprocal vector lies within the Bragg angle " - "of the spindle, and symmetry normally supplies them from an equivalent elsewhere; " - "it cannot here, because that axis maps the blind region onto itself. Those " - "reflections stay missing however long the sweep runs. The loss is confined to " - "that cone rather than spread over the data, so overall completeness may still " - "look reasonable - check it near the spindle direction. A second sweep on a " - "different axis, or re-mounting, recovers them.", - order, angle); + "The mounting makes {:.1f}% of unique reflections unmeasurable: the " + "measured point group cannot map that part of the sweep's blind cone " + "onto measured territory, however long the sweep runs. The loss is a " + "coherent cap about the spindle direction rather than a scatter, so " + "overall completeness may still look reasonable - check it near the " + "spindle. A second sweep about a different axis, or re-mounting, " + "recovers it.", + 100.0 * loss->lost_unique_fraction); logger.Warning("{}", msg); stats_text << " !! " << msg << "\n\n"; result.warnings.push_back(msg); - } else { - stats_text << "Closest symmetry axis to the spindle: " << order << "-fold at " - << std::fixed << std::setprecision(1) << angle << " deg\n\n"; } + } else { + stats_text << "\n"; } } } diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index bc56961a9..f1c43e3c5 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -324,6 +324,12 @@ struct ProcessResult { // warning. Absent on stills, or where no space group was determined. std::optional spindle_symmetry_axis_deg; int spindle_symmetry_axis_order = 0; + // Fraction of unique reflections (to the processing resolution limit) the mounting made + // unmeasurable: the part of the sweep's blind double cone that no operator of the measured + // point group maps onto measured territory, in the crystal's actual indexed orientation. + // Exact, not the per-image worst-case bound. 0 = symmetry (or the mounting) recovered + // everything a sweep can lose. + std::optional spindle_lost_unique_fraction; }; // How far a cell stands from the metric its space group requires. A group's own rotations must leave diff --git a/rugnux/SpindleCuspLoss.cpp b/rugnux/SpindleCuspLoss.cpp new file mode 100644 index 000000000..f3adcc027 --- /dev/null +++ b/rugnux/SpindleCuspLoss.cpp @@ -0,0 +1,111 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include +#include + +#include "SpindleCuspLoss.h" +#include "../common/JFJochMath.h" + +std::optional SpindleUnrepairedFraction(const gemmi::SpaceGroup &sg, + const CrystalLattice &lattice, + const Coord &spindle, + double wavelength_A, + double d_min_A) { + const double spindle_len = spindle.Length(); + if (spindle_len < 1e-9 || wavelength_A <= 0 || d_min_A <= 0) + return {}; + const Coord axis = spindle * static_cast(1.0 / spindle_len); + + const double sin_tm = std::min(1.0, wavelength_A / (2.0 * d_min_A)); + const double cos_tm = std::sqrt(std::max(0.0, 1.0 - sin_tm * sin_tm)); + if (sin_tm <= 0) + return {}; + + // Proper rotations only, identity included. Friedel and the improper operators need no separate + // treatment: the blind double cone and the measured region are both inversion-symmetric, so -R + // lands a point inside or outside exactly as R does. + std::vector, 3>> rot; + for (const auto &op : sg.operations().derive_symmorphic().sym_ops) { + const auto &w = op.rot; + const double det = + static_cast(w[0][0]) * (w[1][1] * w[2][2] - w[1][2] * w[2][1]) + - static_cast(w[0][1]) * (w[1][0] * w[2][2] - w[1][2] * w[2][0]) + + static_cast(w[0][2]) * (w[1][0] * w[2][1] - w[1][1] * w[2][0]); + if (det <= 0) + continue; + if (op.rot == gemmi::Op::identity().rot) + continue; // handled exactly below, without the float round trip through the basis + std::array, 3> m{}; + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) + m[i][j] = static_cast(w[i][j]) / gemmi::Op::DEN; + rot.push_back(m); + } + + const Coord a = lattice.Vec0(), b = lattice.Vec1(), c = lattice.Vec2(); + const Coord as = lattice.Astar(), bs = lattice.Bstar(), cs = lattice.Cstar(); + + // Deterministic spiral over ONE lobe of the blind cone; the other lobe contributes identically + // (the orbit of -p is the negated orbit of p, and only |cos| to the spindle enters). Only cone + // directions can be lost - the identity is in every orbit - so nothing outside is sampled. + constexpr int N_DIRECTIONS = 8192; + const double golden_angle = PI * (3.0 - std::sqrt(5.0)); + + // Any unit vector perpendicular to the spindle, to open the cap around it. + const Coord seed = std::fabs(axis.x) < 0.9f ? Coord(1, 0, 0) : Coord(0, 1, 0); + const Coord e1 = (axis % seed).Normalize(); + const Coord e2 = axis % e1; + + double lost_sum = 0, p1_sum = 0; + for (int i = 0; i < N_DIRECTIONS; i++) { + const double z = cos_tm + (1.0 - cos_tm) * (static_cast(i) + 0.5) / N_DIRECTIONS; + const double r = std::sqrt(std::max(0.0, 1.0 - z * z)); + const double phi = golden_angle * i; + const Coord p = axis * static_cast(z) + + e1 * static_cast(r * std::cos(phi)) + + e2 * static_cast(r * std::sin(phi)); + + // h = (a.p, b.p, c.p) are continuous Miller coordinates; an operator sends the reflection + // h to h' = h W (gemmi's row convention) and h' returns to Cartesian through the reciprocal + // basis. No matrix inversion: direct and reciprocal bases are mutually inverse. + const double h0 = a * p, h1 = b * p, h2 = c * p; + + // The orbit's largest folded angle to the spindle, as its sine (the fold to [0, 90] makes + // the sine monotone). A point at direction p and radius q is unmeasured iff every image of + // its orbit is inside the cone of ITS OWN shell, i.e. iff q > 2 sin(a_max) / lambda - so + // the direction loses the radial fraction 1 - (sin a_max / sin theta_max)^3 of its unique + // reflections, reciprocal volume being what unique reflections are proportional to. + // The identity's own angle comes straight from z, exactly; only the non-trivial + // operators go through the basis. + const double sin_id = std::sqrt(std::max(0.0, 1.0 - z * z)); + double max_sin = sin_id; + for (const auto &m : rot) { + const double g0 = h0 * m[0][0] + h1 * m[1][0] + h2 * m[2][0]; + const double g1 = h0 * m[0][1] + h1 * m[1][1] + h2 * m[2][1]; + const double g2 = h0 * m[0][2] + h1 * m[1][2] + h2 * m[2][2]; + const Coord s = as * static_cast(g0) + bs * static_cast(g1) + + cs * static_cast(g2); + const double len = s.Length(); + if (len < 1e-12) + continue; + const double cos_a = std::min(1.0, std::fabs(s * axis) / len); + max_sin = std::max(max_sin, std::sqrt(std::max(0.0, 1.0 - cos_a * cos_a))); + } + if (max_sin < sin_tm) { + const double x = max_sin / sin_tm; + lost_sum += 1.0 - x * x * x; + } + const double y = sin_id / sin_tm; + p1_sum += 1.0 - y * y * y; + } + + SpindleCuspLoss ret; + ret.theta_max_deg = std::asin(sin_tm) * 180.0 / PI; + // The double cone is (1 - cos theta_max) of the sphere, and the sample covers exactly the cone, + // so the cone average rescales by that solid angle to a fraction of ALL unique reflections. + ret.lost_unique_fraction = (1.0 - cos_tm) * lost_sum / N_DIRECTIONS; + ret.cone_fraction = p1_sum > 0 ? lost_sum / p1_sum : 0.0; + return ret; +} diff --git a/rugnux/SpindleCuspLoss.h b/rugnux/SpindleCuspLoss.h new file mode 100644 index 000000000..d5fd0452f --- /dev/null +++ b/rugnux/SpindleCuspLoss.h @@ -0,0 +1,36 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include + +#include + +#include "../common/Coord.h" +#include "../common/CrystalLattice.h" + +// The exact, orbit-based counterpart of the per-image spindle severity. A still has to assume the +// worst - that the nearest plausible lattice row is a lone 2-fold - but by the time a rotation run +// is merged the point group and the indexed orientation are KNOWN, so nothing needs guessing: apply +// the group's proper rotations to the sweep's blind double cone in the crystal's actual orientation +// and count what no operator maps onto measured territory. A unique reflection is lost exactly when +// its whole orbit stays inside the cone (Friedel never helps: the cone is double-sided, so the +// inversion maps it onto itself). +struct SpindleCuspLoss { + // Fraction of all unique reflections to d_min that the mounting made unmeasurable. Weighted by + // reciprocal volume: each shell has its own, narrower cone theta(d) = asin(lambda/2d), so a + // direction at angle a from the spindle is only blind past q = 2 sin(a) / lambda. + double lost_unique_fraction = 0; + // The same loss as a fraction of the blind cone's own unique content: 1 = the group repaired + // nothing (P1, or a lone diad on or perpendicular to the spindle), 0 = symmetry recovers the + // whole cone. This is what severity means independent of how wide the cone happens to be. + double cone_fraction = 0; + double theta_max_deg = 0; +}; + +std::optional SpindleUnrepairedFraction(const gemmi::SpaceGroup &sg, + const CrystalLattice &lattice, + const Coord &spindle, + double wavelength_A, + double d_min_A); diff --git a/tests/CBORTest.cpp b/tests/CBORTest.cpp index 638506425..0b76e1726 100644 --- a/tests/CBORTest.cpp +++ b/tests/CBORTest.cpp @@ -498,6 +498,7 @@ TEST_CASE("CBORSerialize_End", "[CBOR]") { .max_receiver_delay = 3456, .efficiency = 0.99, .spindle_blind_fraction = 0.31f, + .spindle_lost_unique_fraction = 0.021f, .end_date = "ccc", .run_name = "bla5", .run_number = 45676782, @@ -537,6 +538,7 @@ TEST_CASE("CBORSerialize_End", "[CBOR]") { CHECK(output_message.rotation_lattice->GetUnitCell().c == Catch::Approx(60.0)); REQUIRE(output_message.spindle_blind_fraction == message.spindle_blind_fraction); + REQUIRE(output_message.spindle_lost_unique_fraction == message.spindle_lost_unique_fraction); // The per-image vector holds NaN where a frame had no value (CANNOT SAY); the hole must // survive the round trip as a hole, not as a number. REQUIRE(output_message.v_spindle_blind_fraction.size() == 3); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 299a6a503..bb6e33adf 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -83,6 +83,7 @@ ADD_EXECUTABLE(jfjoch_test TimeTest.cpp RotationIndexerTest.cpp SpindleBlindFractionTest.cpp + SpindleCuspLossTest.cpp TopPixelsTest.cpp HKLKeyTest.cpp TCPImagePusherTest.cpp diff --git a/tests/SpindleCuspLossTest.cpp b/tests/SpindleCuspLossTest.cpp new file mode 100644 index 000000000..162476764 --- /dev/null +++ b/tests/SpindleCuspLossTest.cpp @@ -0,0 +1,93 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include + +#include "../rugnux/SpindleCuspLoss.h" + +using Catch::Matchers::WithinAbs; + +namespace { + // lambda = 1 A against d_min = 1 / (2 sin 15 deg) puts theta_max at exactly 15 deg. + constexpr double WVL = 1.0; + const double D_MIN = 1.0 / (2.0 * std::sin(15.0 * M_PI / 180.0)); + + Coord Perpendicular(const Coord &v) { + const Coord seed = std::fabs(v.Normalize().x) < 0.9f ? Coord(1, 0, 0) : Coord(0, 1, 0); + return (v % seed).Normalize(); + } +} + +TEST_CASE("SpindleCuspLoss_Orbit", "[Rugnux][Spindle]") { + SECTION("P1 repairs nothing, anywhere") { + const CrystalLattice latt(UnitCell{40, 50, 60, 83, 95, 102}); + const auto r = SpindleUnrepairedFraction(*gemmi::find_spacegroup_by_name("P 1"), latt, + Coord(0.3f, -0.5f, 0.8f), WVL, D_MIN); + REQUIRE(r.has_value()); + CHECK_THAT(r->cone_fraction, WithinAbs(1.0, 1e-9)); + // Radially weighted double-cone content at theta_max = 15 deg (verified by Monte Carlo). + CHECK_THAT(r->lost_unique_fraction, WithinAbs(0.0203, 0.0005)); + CHECK_THAT(r->theta_max_deg, WithinAbs(15.0, 1e-6)); + } + + SECTION("a lone diad on the spindle, and one perpendicular to it, both lose the whole cone") { + const CrystalLattice latt(UnitCell{40, 50, 60, 90, 100, 90}); // P2, unique axis b + const auto &sg = *gemmi::find_spacegroup_by_name("P 2"); + const Coord diad = latt.Vec1(); // b is perpendicular to a and c, so it IS the axis + const auto on = SpindleUnrepairedFraction(sg, latt, diad, WVL, D_MIN); + REQUIRE(on.has_value()); + CHECK_THAT(on->cone_fraction, WithinAbs(1.0, 1e-4)); + const auto perp = SpindleUnrepairedFraction(sg, latt, Perpendicular(diad), WVL, D_MIN); + REQUIRE(perp.has_value()); + CHECK_THAT(perp->cone_fraction, WithinAbs(1.0, 1e-4)); + } + + SECTION("the same diad at 60 deg from the spindle loses nothing") { + const CrystalLattice latt(UnitCell{40, 50, 60, 90, 100, 90}); + const Coord diad = latt.Vec1().Normalize(); + const Coord spindle = diad * 0.5f + Perpendicular(diad) * static_cast(std::sqrt(0.75)); + const auto r = SpindleUnrepairedFraction(*gemmi::find_spacegroup_by_name("P 2"), latt, + spindle, WVL, D_MIN); + REQUIRE(r.has_value()); + CHECK_THAT(r->lost_unique_fraction, WithinAbs(0.0, 1e-9)); + } + + SECTION("an axis of order >= 3 perpendicular to the spindle repairs the cone completely") { + // The situation the per-image worst-case bound must flag but the known group clears. + const CrystalLattice latt(UnitCell{50, 50, 60, 90, 90, 120}); + const Coord three_fold = latt.Vec2(); // c, the 3-fold of P3 + const auto r = SpindleUnrepairedFraction(*gemmi::find_spacegroup_by_name("P 3"), latt, + Perpendicular(three_fold), WVL, D_MIN); + REQUIRE(r.has_value()); + CHECK_THAT(r->lost_unique_fraction, WithinAbs(0.0, 1e-9)); + } + + SECTION("a dihedral group repairs an in-plane diad the warning heuristic used to flag") { + // 622 with an in-plane 2-fold exactly on the spindle: the old any-axis-within-15-deg rule + // warned here, but the principal 6-fold maps the cone off itself and nothing is lost. + const CrystalLattice latt(UnitCell{50, 50, 60, 90, 90, 120}); + const auto r = SpindleUnrepairedFraction(*gemmi::find_spacegroup_by_name("P 6 2 2"), latt, + latt.Vec0(), WVL, D_MIN); + REQUIRE(r.has_value()); + CHECK_THAT(r->lost_unique_fraction, WithinAbs(0.0, 1e-9)); + } + + SECTION("a cubic group is never severe, whatever the mounting") { + const CrystalLattice latt(UnitCell{60, 60, 60, 90, 90, 90}); + const auto &sg = *gemmi::find_spacegroup_by_name("P 4 3 2"); + for (const auto &spindle : {Coord(1, 0, 0), Coord(0.3f, -0.5f, 0.8f), Coord(1, 1, 1)}) { + const auto r = SpindleUnrepairedFraction(sg, latt, spindle, WVL, D_MIN); + REQUIRE(r.has_value()); + CHECK_THAT(r->lost_unique_fraction, WithinAbs(0.0, 1e-9)); + } + } + + SECTION("no spindle, no wavelength or no resolution gives no answer") { + const CrystalLattice latt(UnitCell{40, 50, 60, 90, 90, 90}); + const auto &sg = *gemmi::find_spacegroup_by_name("P 1"); + CHECK_FALSE(SpindleUnrepairedFraction(sg, latt, Coord(0, 0, 0), WVL, D_MIN).has_value()); + CHECK_FALSE(SpindleUnrepairedFraction(sg, latt, Coord(0, 0, 1), 0.0, D_MIN).has_value()); + CHECK_FALSE(SpindleUnrepairedFraction(sg, latt, Coord(0, 0, 1), WVL, 0.0).has_value()); + } +} diff --git a/writer/HDF5NXmx.cpp b/writer/HDF5NXmx.cpp index a0b1be62b..c37e745b0 100644 --- a/writer/HDF5NXmx.cpp +++ b/writer/HDF5NXmx.cpp @@ -1182,6 +1182,10 @@ void NXmx::Finalize(const EndMessage &end) { if (end.spindle_blind_fraction) { SaveScalar(*hdf5_file, "/entry/MX/spindleBlindFractionMean", end.spindle_blind_fraction.value()); } + if (end.spindle_lost_unique_fraction) { + SaveScalar(*hdf5_file, "/entry/MX/spindleLostUniqueFraction", + end.spindle_lost_unique_fraction.value()); + } if (end.ice_ring_score_mean) { SaveScalar(*hdf5_file, "/entry/MX/iceRingScoreMean", end.ice_ring_score_mean.value()); } -- 2.54.0 From 2adf5d2e28476022aeb093e1fccc43bf20161f3f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 10:43:57 +0200 Subject: [PATCH 21/75] docs: the spindle severity documented as the worst-case trigger it is Section 5.5 of the indexing documentation describes the per-image score as shipped: the folded formula, the geometric cone width, the pair-normal recovery of an axis too long to see, and the three trigger states with the rule that CANNOT SAY must engage. It states plainly that the score is a worst-case bound under an assumption of no symmetry rather than an estimate - an axis of order three or higher perpendicular to the spindle in fact repairs the cone, which a still cannot know - and gives the engagement rates actually measured instead of the decoy-null figure, which showed only that the estimator does not hallucinate rows and was never a false-alarm rate over harmless mountings. The run-level orbit-based fraction is pointed to as the exact measure available once the group is known. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 2 + docs/CPU_DATA_ANALYSIS_INDEXING.md | 80 ++++++++++++++++++++++++++++++ 2 files changed, 82 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 385c3ef00..d4665f8ed 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -16,6 +16,8 @@ * `rugnux` writes reflection files in the conventions downstream programs read: `FreeR_flag` is 0 for the test set and 1 for the working set - it was the other way round - the merged and P1 MTZ carry the reserved `HKL_base` dataset so a CCP4 program reads the wavelength instead of falling back to 1.54187 A, and the merged mmCIF marks the free set as `_refln.status` = `f`. * The rugnux results report carries the space groups the data cannot separate and the enantiomorph state, the model's verdict and what it was allowed to decide, the detector geometry measured and what a single sweep cannot determine, the resolution the CC1/2 fit reached, which reciprocal axis each anisotropic diffraction limit belongs to, and twinning measured before and after the space group was decided; `REPORT_VERSION` is 7, and `SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL` is now `ASSUMED_FROM_MODEL`. * The rugnux results report records how the crystal sat on the goniometer as `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` and `SPINDLE_SYMMETRY_AXIS_ORDER=` - the angle from the spindle to the nearest symmetry axis and that axis's order - on every rotation run that determined a space group, not only when the angle was small enough to warn about. +* The rugnux results report writes `SPINDLE_LOST_UNIQUE_FRACTION=` - the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group - with the same number in the HDF5 master as `/entry/MX/spindleLostUniqueFraction`, and the spindle-mounting warning fires on that exact number instead of on the angle to the nearest symmetry axis. +* Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone the crystal's orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, the HDF5 files as `/entry/MX/spindleBlindFraction`, the REST plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * `rugnux --mode calibration` writes `.json` beside the `.poni`, holding the geometry as a `jfjoch_broker` `dataset_settings` body, and refuses a fit that is not a measurement - no `.poni`, a non-zero exit, `converged` recorded in the `.json`; `--no-refine-tilt` holds the detector tilt at the file's value instead of zeroing it. * A snake grid scan with a negative slow step and an even number of rows no longer has its positions mirrored along the fast axis in the HDF5 master and the grid map, so the positions recorded for that configuration change; `jfjoch_viewer` draws grid scan cells in the proportion of the scan steps, labels the merge-statistics plot over the range the axis is drawn on, and builds its powder-calibration ring list from the loaded dataset's space group as well as its cell, so a centred sample cell no longer scales the whole fit. * The HDF5 master records `direct_beam_x`/`direct_beam_y` - where the undeflected beam lands, sent on the CBOR start message too - the beam size at the sample as `incident_beam_size` from the new `dataset_settings` `beam_size_x_um`/`beam_size_y_um`, and `/entry/MX/peakCountUnfiltered`; `dataset_settings` accepts any `smargon.chi_deg`, which was restricted to 0-90 degrees. diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index 412043095..5e9ca6e11 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -105,6 +105,86 @@ Selection is **not limited to a single lattice**: after the best cell is accepte An optional reference unit cell (if supplied) restricts acceptance to cells within a relative distance tolerance in edge lengths (permutation-invariant). +### 5.5 Spindle alignment: the part of the blind cone symmetry cannot repair + +A sweep about the spindle $\hat{\mathbf{e}}$ never brings a reciprocal point closer than +$\theta_\mathrm{max}=\arcsin(\lambda/2d)$ to the axis onto the Ewald sphere, so a double cone of +half-angle $\theta_\mathrm{max}$ is missing from every resolution shell — each shell losing its own +$1-\cos\theta(d)$ — however long the sweep runs. Crystal symmetry normally repairs that loss by +mapping the cone onto measured territory. It fails to when a symmetry axis lies inside the cone (the +cone maps onto itself) — and, for a **2-fold**, equally when the axis is perpendicular to the +spindle, because the diad carries the cone onto its opposite lobe, which the sweep leaves just as +unmeasured. Friedel never helps: the cone is double-sided. The loss is a coherent cap rather than a +scatter of absences, so it costs a map more than the same percentage lost at random. + +The per-image score asks how much of that cone the frame's own orientation makes unrecoverable. +The crystal's short lattice rows are read off the FFT row shortlist of §5.2 (a symmetry axis is +always a lattice row, and usually among the short ones), and each plausible direction is scored as +if it carried a lone 2-fold: + +$$ \text{spindle blind fraction} = \frac{2}{\pi}\left(\arccos x - x\sqrt{1-x^{2}}\right), +\qquad x = \min(\beta,\,90^\circ-\beta)\,/\,\theta_\mathrm{max}, $$ + +where $\beta$ is the direction's miss-angle from the spindle. The **fold** of $\beta$ about +$45^\circ$ is the diad geometry above: both ends of the range are the bad case, and the closed form +reproduces a Monte Carlo of the true double-cone self-overlap to 0.004 at +$\theta_\mathrm{max}=15^\circ$ and 0.008 at $25^\circ$ (past $45^\circ$ it under-reports, by 0.05 +at $50^\circ$). $\theta_\mathrm{max}$ is taken from the **geometric** resolution of the setup — the +detector corner at the recorded distance and wavelength — an upper bound on any sweep collected +without moving the detector; the still's own spot resolution would understate the cone on exactly the +weak frames that mislead. The worst direction wins, and the directions scored are the strong +in-window rows **and the normals of their pairs** — the normal to two lattice rows is itself a +reciprocal-lattice row and a symmetry axis is parallel in both bases, so a lone 2-fold on an axis far +beyond the length window (a long monoclinic unique axis) is still seen by direction: measured on a +synthetic lone-diad crystal with a 300 Å unique axis, the fraction of severe mounts reported severe +rises from 0.60 to 1.00 with the pair normals, at no extra engagement on that class's harmless +mounts. Nothing about the goniometer enters: the number describes the problem and leaves the remedy — +a second sweep, a reorientation — to the beamline. + +**This is a worst-case bound under an assumption of no symmetry, not an estimate.** A still cannot +know the point group, so the nearest plausible row is scored as a lone 2-fold. An axis of order +$\geq 3$ perpendicular to the spindle in fact **fully repairs** the cone (measured unrepaired +fraction 0.000 for orders 3, 4 and 6, against 1.000 for a diad), which a still cannot see, so the +bound is deliberately pessimistic on higher-symmetry crystals — that is the intended trade, because +the number exists as a **trigger** for beamline automation, not as a physical quantity a user +interprets. + +**Trigger states.** The stored quantity is the continuous score; automation reads it through three +fixed states with nothing to tune (`SpindleTrigger` in `SpindleBlindFraction.h`): **engage** at +score ≥ 0.5, **don't engage** below, and **cannot say** when there is no value at all — too few +spots, no shortlist, the consistency guard refused, the path never computed one. **Automation must +treat CANNOT SAY as ENGAGE**: the error costs are asymmetric — a false negative is unrecoverable +(one sweep is collected and the data stay short forever) while a false positive costs minutes of +beamtime. Every transport keeps absence distinguishable from a measured zero (an absent CBOR key, a +NaN in the HDF5 arrays, an absent optional after read-back). The 0.5 threshold is geometry, not +tuning: the score is monotone in the folded miss-angle, so a threshold is a fold-angle gate, and 0.5 +gates at $\min(\beta,90^\circ-\beta) \le 0.404\,\theta_\mathrm{max}$; engaging on any overlap at +all would gate at the cone edge, whose perpendicular band alone spans $\sin\theta_\mathrm{max}$ of +orientation space per row (26 % at $15^\circ$) and unions over a frame's rows to well over half of +all mountings — a trigger that always fires decides nothing. + +**Reach and honest rates.** The score needs 60 spots (calibrated per crystal — 22 independent +mounts — misses triple below it); below that, a frame that still indexed answers from the winning +lattice's shortest rows, and otherwise the state is *cannot say*. Because the bound is pessimistic +by design, it engages on a substantial share of harmless mountings: a single strong row's +perpendicular band alone covers ~11 % of orientation space at the severe level +($\theta_\mathrm{max}=15^\circ$), and the union over a frame's rows and pair normals reaches +roughly a quarter to three quarters of random mountings depending on cone width and row count +(measured 0.74 on a generic triclinic cell at $15^\circ$ via the lattice path). That is accepted: +the cheap error is the extra wedge. An earlier figure of ~1 % false alarms (AUC 0.948) came from a +null of five *decoy directions per frame* — it shows the estimator does not hallucinate rows near +arbitrary directions, which is worth knowing, but it is **not** a false-alarm rate over harmless +mountings, which geometry forbids to be that low. + +**Offline, the guessing stops.** Once `rugnux` has merged a rotation run it holds the measured +point group and the exact indexed orientation, and the run-level number is computed exactly instead: +the group's proper rotations are applied to the blind double cone in the crystal's actual +orientation, and the fraction of unique reflections no operator can recover is reported as +`SPINDLE_LOST_UNIQUE_FRACTION` in the processing report and `/entry/MX/spindleLostUniqueFraction` in +the master file (§ docs/RUGNUX_REPORT.md). That number clears or convicts a mounting the per-image +bound can only be pessimistic about: a dihedral crystal with an in-plane diad on the spindle, or any +cubic crystal in any orientation, loses nothing at all. + --- ## 6. Bravais lattice / centering inference (“lattice search”) -- 2.54.0 From 77a8a07c51893d13df7bec6646240959a72acdc1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 19:38:14 +0200 Subject: [PATCH 22/75] docs: 20 scouted external datasets recorded, credited, and reconciled Add the datasets pulled from the archive-wide PDB raw-data scout to the external-test-data page: new table rows (each with a DataCite-verified DOI, RCSB-deposited SG/cell/resolution, and the detector read from the image file), their multi-collection layouts, two more image-vs-PDB detector conflicts, and the two small-molecule reference cells. Multi-crystal wedge and non-native sets are excluded. Credit the two new repositories the data came from - MXRDR (ICM Warsaw) and the ESRF data portal - and bring the SBGrid and Zenodo counts in line with the table. All per-repository and per-section totals reconcile against the row count (95 PDB-coded + 7 without = 102). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DSMszqvyZUb6FHnSSS5nXY --- docs/ACKNOWLEDGEMENT.md | 11 +++++-- docs/EXTERNAL_TEST_DATA.md | 64 ++++++++++++++++++++++++++++++++------ 2 files changed, 63 insertions(+), 12 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index c79ec7d9d..f50c63d6c 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -48,17 +48,24 @@ Integrated Resource for Reproducibility in Macromolecular Crystallography: Exper four years" (2019), Struct. Dyn. 6, 064301 [doi:10.1063/1.5128672](https://doi.org/10.1063/1.5128672). -**[SBGrid Data Bank](https://data.sbgrid.org/)** supplied nine of the datasets. P. A. Meyer, +**[SBGrid Data Bank](https://data.sbgrid.org/)** supplied eighteen of the datasets. P. A. Meyer, S. Socias, J. Key, E. Ransey, E. C. Tjon, A. Buschiazzo et al., "Data publication with the structural biology data grid supports live analysis" (2016), Nat. Commun. 7, 10882 [doi:10.1038/ncomms10882](https://doi.org/10.1038/ncomms10882). -**[Zenodo](https://zenodo.org/)** hosts eleven, deposited there directly by the groups that +**[Zenodo](https://zenodo.org/)** hosts sixteen, deposited there directly by the groups that collected them. European Organization for Nuclear Research and OpenAIRE, "Zenodo" (2013), CERN [doi:10.25495/7GXK-RD71](https://doi.org/10.25495/7GXK-RD71). Three of those datasets were published as IUCrData Raw Data Letters; the letters are cited on the [EXTERNAL_TEST_DATA](EXTERNAL_TEST_DATA.md) page, beside the datasets they describe. +**[MXRDR](https://mxrdr.icm.edu.pl/)**, the Macromolecular Xtallography Raw Data Repository +(ICM, University of Warsaw), supplied four, released under CC0 with the dataset DOI to be cited; +those DOIs (`10.18150/…`, `10.60884/…`) are in the table. + +**The [ESRF data portal](https://data.esrf.fr/)** supplied one, released under CC BY 4.0 under the +ESRF data policy with the dataset DOI (`10.15151/…`) to be cited. + The beamline, resolution, space group and unit cell quoted for each dataset are the values deposited with the corresponding PDB entry, read from the RCSB PDB data API. H. M. Berman, J. Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, I. N. Shindyalov and P. E. Bourne, diff --git a/docs/EXTERNAL_TEST_DATA.md b/docs/EXTERNAL_TEST_DATA.md index d01695f57..489331157 100644 --- a/docs/EXTERNAL_TEST_DATA.md +++ b/docs/EXTERNAL_TEST_DATA.md @@ -38,7 +38,6 @@ the table below; the repositories themselves are cited in | [5SRC](https://www.rcsb.org/structure/5SRC) | IRRMC [10.18430/M35SRC](https://doi.org/10.18430/M35SRC) | ALS 8.3.1 | 1.05 | P 43 | 88.7 88.7 39.2 90.0 90.0 90.0 | PILATUS3 6M | PanDDA analysis group deposition -- Crystal structure of SARS-CoV-2 NSP3 macrodomain in complex with Z5198562500 - (R,R) and (R,S) isomers | | [6HV2](https://www.rcsb.org/structure/6HV2) | IRRMC [10.18430/m36hv2](https://doi.org/10.18430/m36hv2) | SLS X06SA | 1.71 | P 61 2 2 | 68.9 68.9 133.6 90.0 90.0 120.0 | Dectris Eiger 16M | MMP-13 in complex with the peptide IMISF | | [6JGJ](https://www.rcsb.org/structure/6JGJ) | IRRMC [10.18430/m36jgj](https://doi.org/10.18430/m36jgj) | SPring-8 BL41XU | 0.77 | P 21 21 21 | 50.9 62.3 68.8 90.0 90.0 90.0 | PILATUS3 300K | Crystal structure of the F99S/M153T/V163A/E222Q variant of GFP at 0.78 A | -| [6LEO](https://www.rcsb.org/structure/6LEO) | Zenodo [10.5281/zenodo.4003042](https://doi.org/10.5281/zenodo.4003042) | SPring-8 BL32XU | 2.52 | C 2 2 21 | 73.5 95.3 101.4 90.0 90.0 90.0 | Dectris Eiger 9M | Crystal structure of thiosulfate transporter YeeE from Spirochaeta thermophila | | [6O2H](https://www.rcsb.org/structure/6O2H) | SBGrid [10.15785/sbgrid/747](https://doi.org/10.15785/sbgrid/747) | CHESS F1 | 1.21 | P 1 | 27.4 32.1 34.5 88.7 108.5 111.9 | PILATUS3 6M | Hen lysozyme in triclinic space group at ambient temperature - diffuse scattering dataset | | [6R72](https://www.rcsb.org/structure/6R72) | Zenodo [10.5281/zenodo.14894181](https://doi.org/10.5281/zenodo.14894181) | SOLEIL PROXIMA 2 | 3.95 | P 1 21 1 | 117.8 110.8 155.6 90.0 93.2 90.0 | Dectris Eiger 9M | Crystal structure of BmrA-E504A in an outward-facing conformation | | [6RLR](https://www.rcsb.org/structure/6RLR) | Zenodo [10.5281/zenodo.5886687](https://doi.org/10.5281/zenodo.5886687) | Diamond I04 | 2.00 | P 1 | 40.0 40.0 63.6 80.4 76.3 68.2 | Eiger 16M | Crystal structure of CD9 large extracellular loop | @@ -108,20 +107,42 @@ the table below; the repositories themselves are cited in | [9ZLO](https://www.rcsb.org/structure/9ZLO) | Zenodo [10.5281/zenodo.18652652](https://doi.org/10.5281/zenodo.18652652) | Australian Synchrotron MX2 | 2.00 | P 21 21 21 | 38.4 90.0 107.0 90.0 90.0 90.0 | Dectris EIGER1 Si 16M | Crystal structure of Proteus mirabilis UreE | | [9ZM0](https://www.rcsb.org/structure/9ZM0) | IRRMC [10.18430/M39ZM0](https://doi.org/10.18430/M39ZM0) | NSLS-II 17-ID-1 | 2.10 | P 1 21 1 | 50.4 30.1 91.2 90.0 97.1 90.0 | Dectris EIGER1 Si 9M | Crystal structure of monomeric Atg23 | | [9ZMU](https://www.rcsb.org/structure/9ZMU) | IRRMC [10.18430/M39ZMU](https://doi.org/10.18430/M39ZMU) | NSLS-II 19-ID | 1.98 | P 65 2 2 | 47.8 47.8 492.6 90.0 90.0 120.0 | Dectris EIGER2 Si 9M | Crystal structure of an Iole protein from Brucella melitensis (hexagonal P form) | +| [5JVN](https://www.rcsb.org/structure/5JVN) | IRRMC [10.18430/m35jvn](https://doi.org/10.18430/m35jvn) | ESRF ID29 | 2.90 | P 6 2 2 | 249.4 249.4 84.1 90.0 90.0 120.0 | PILATUS3 6M | C3-type pyruvate phosphate dikinase: intermediate state of the swiveling-domain mechanism | +| [5M17](https://www.rcsb.org/structure/5M17) | Zenodo [10.5281/zenodo.4300323](https://doi.org/10.5281/zenodo.4300323) | Diamond I02 | 1.03 | I 4 | 108.6 108.6 67.7 90.0 90.0 90.0 | PILATUS 6M-F | Structure of the GH99 endo-alpha-mannanase from Bacteroides xylanisolvens | +| [6FID](https://www.rcsb.org/structure/6FID) | SBGrid [10.15785/sbgrid/541](https://doi.org/10.15785/sbgrid/541) | ESRF ID30B | 2.20 | P 21 21 21 | 59.9 64.1 69.7 90.0 90.0 90.0 | PILATUS3 6M | Bovine trypsin solved by S-SAD on ID30B | +| [6FVZ](https://www.rcsb.org/structure/6FVZ) | IRRMC [10.18430/m36fvz](https://doi.org/10.18430/m36fvz) | ESRF ID23-2 | 1.80 | C 2 2 2 | 131.2 222.8 86.5 90.0 90.0 90.0 | PILATUS3 X 2M | Crystal structure of human monoamine oxidase B (MAO B) in complex with an inhibitor | +| [6HWJ](https://www.rcsb.org/structure/6HWJ) | SBGrid [10.15785/sbgrid/614](https://doi.org/10.15785/sbgrid/614) | ALBA XALOC | 1.98 | P 1 21 1 | 59.8 96.1 80.3 90.0 106.7 90.0 | PILATUS 6M | Glucosamine kinase (crystal form A) | +| [6IU8](https://www.rcsb.org/structure/6IU8) | Zenodo [10.5281/zenodo.2532134](https://doi.org/10.5281/zenodo.2532134) | SPring-8 BL41XU | 2.70 | P 31 | 85.5 85.5 98.4 90.0 90.0 120.0 | PILATUS3 6M | Crystal structure of cytoplasmic metal binding domain with cobalt | +| [6P8P](https://www.rcsb.org/structure/6P8P) | SBGrid [10.15785/sbgrid/673](https://doi.org/10.15785/sbgrid/673) | APS 24-ID-C | 1.64 | P 4 | 97.5 97.5 60.1 90.0 90.0 90.0 | PILATUS 6M-F | Structure of P. aeruginosa ATCC27853 HORMA1 | +| [6PB3](https://www.rcsb.org/structure/6PB3) | SBGrid [10.15785/sbgrid/681](https://doi.org/10.15785/sbgrid/681) | APS 24-ID-E | 2.05 | P 6 | 100.4 100.4 48.9 90.0 90.0 120.0 | Dectris Eiger 16M | Structure of Rhizobiales Trip13 | +| [6WZO](https://www.rcsb.org/structure/6WZO) | SBGrid [10.15785/sbgrid/785](https://doi.org/10.15785/sbgrid/785) | APS 24-ID-E | 1.42 | P 1 | 43.7 50.1 69.3 106.5 90.1 97.1 | Dectris Eiger 16M | Structure of SARS-CoV-2 Nucleocapsid dimerization domain, P1 form | +| [7ARR](https://www.rcsb.org/structure/7ARR) | MXRDR [10.18150/EM87YL](https://doi.org/10.18150/EM87YL) | PETRA III, EMBL c/o DESY P13 (MX1) | 1.10 | P 1 | 30.9 32.1 43.1 114.2 91.9 109.9 | PILATUS 6M-F | The de novo designed hybrid alpha/beta-miniprotein | +| [7L84](https://www.rcsb.org/structure/7L84) | SBGrid [10.15785/sbgrid/816](https://doi.org/10.15785/sbgrid/816) | APS 24-ID-C | 1.60 | P 43 21 2 | 79.3 79.3 37.8 90.0 90.0 90.0 | PILATUS 6M-F | Hen Egg White Lysozyme by Native S-SAD at Room Temperature | +| [7OS3](https://www.rcsb.org/structure/7OS3) | MXRDR [10.18150/74YTYQ](https://doi.org/10.18150/74YTYQ) | PETRA III, EMBL c/o DESY P13 (MX1) | 2.18 | P 21 21 21 | 78.2 91.0 105.8 90.0 90.0 90.0 | PILATUS 6M-F | Crystal structure of Rhizobium etli inducible L-asparaginase | +| [8TYY](https://www.rcsb.org/structure/8TYY) | SBGrid [10.15785/sbgrid/1040](https://doi.org/10.15785/sbgrid/1040) | APS 24-ID-E | 1.68 | F 4 3 2 | 214.9 214.9 214.9 90.0 90.0 90.0 | Dectris Eiger 16M | Structure of a bacterial Ubl-deubiquitinase complex (form 2) | +| [9C18](https://www.rcsb.org/structure/9C18) | Zenodo [10.5281/zenodo.11405662](https://doi.org/10.5281/zenodo.11405662) | NSLS-II 17-ID-1 | 1.90 | P 1 | 41.9 42.0 60.2 84.1 87.2 63.7 | Dectris EIGER1 Si 9M | Human biliverdin IX beta reductase in complex with NADP | +| [9E2T](https://www.rcsb.org/structure/9E2T) | SBGrid [10.15785/sbgrid/1148](https://doi.org/10.15785/sbgrid/1148) | SSRL BL12-1 | 2.28 | P 1 | 75.5 78.1 101.2 94.6 103.4 114.5 | Dectris EIGER2 Si 16M | Structure of a de novo designed interleukin-21 mimetic complex | +| [9HNC](https://www.rcsb.org/structure/9HNC) | MXRDR [10.60884/0K7B68](https://doi.org/10.60884/0K7B68) | PETRA III, EMBL c/o DESY P13 (MX1) | 1.88 | P 1 2 1 | 123.8 123.6 187.7 90.0 90.1 90.0 | PILATUS 6M-F | Crystal structure of potassium-independent L-asparaginase | +| [9QW8](https://www.rcsb.org/structure/9QW8) | ESRF [10.15151/ESRF-DC-2127908021](https://doi.org/10.15151/ESRF-DC-2127908021) | ESRF ID23-1 | 1.80 | P 1 | 35.6 35.6 100.9 86.5 84.2 72.5 | Dectris EIGER2 CdTe 16M | FKBP12 in complex with bifunctional ligand 1ad | +| [9RCI](https://www.rcsb.org/structure/9RCI) | Zenodo [10.5281/zenodo.15615368](https://doi.org/10.5281/zenodo.15615368) | SOLEIL PROXIMA 2 | 1.66 | P 1 | 35.9 39.3 100.9 98.3 90.3 90.1 | Dectris Eiger 9M | Crystal Structure of Flap Endonuclease FEN1 with Compound 28 | +| [8OWM](https://www.rcsb.org/structure/8OWM) | MXRDR [10.18150/II5MT4](https://doi.org/10.18150/II5MT4) | PETRA III, EMBL c/o DESY P13 (MX1) | 1.70 | P 1 | 95.5 95.6 95.8 90.4 93.6 117.8 | Dectris Eiger 16M | Crystal structure of glutamate dehydrogenase 2 from Arabidopsis thaliana binding Ca, NAD and 2,2-dihydroxyglutarate | | — | Zenodo [10.5281/zenodo.1036416](https://doi.org/10.5281/zenodo.1036416) | Diamond Light Source I19-1 | | | | PILATUS 2M | 0.48 Angstrom 3,5-dinitrobenzoic acid (3,5-DNBA) C2/c polymorph single crystal X-ray diffraction data set recorded at Diamond Light Source I19-1 | | — | Zenodo [10.5281/zenodo.14894181](https://doi.org/10.5281/zenodo.14894181) | | | | | Dectris Eiger 9M | Dataset for PDB 6r72 Crystal structure of BmrA-E504A in an outward-facing conformation | | — | Zenodo [10.5281/zenodo.20041091](https://doi.org/10.5281/zenodo.20041091) | Diamond Light Source I19-2 | | | | Eiger 2X 4M (CdTe) | Single-crystal X-ray diffractometry data for a sample of Ni(dppe)Cl₂ collected on beamline I19-2 at Diamond Light Source with an Eiger 2X 4M with CdTe sensor | | — | Zenodo [10.5281/zenodo.20135265](https://doi.org/10.5281/zenodo.20135265) | Diamond Light Source I19-2 | | | | Eiger 2X 4M (CdTe) | Single-crystal X-ray diffractometry data for a sample of metformin collected on beamline I19-2 at Diamond Light Source with an Eiger 2X 4M with CdTe sensor | | — | Zenodo [10.5281/zenodo.6347466](https://doi.org/10.5281/zenodo.6347466) | Diamond Light Source I19-2 | | | | Eiger 2X 4M (CdTe) | Single-crystal X-ray diffractometry data for a sample of [Cu(HF₂)(pyrazine)₂]PF₆ collected on beamline I19-2 at Diamond Light Source | +| — | Zenodo [10.5281/zenodo.33555](https://doi.org/10.5281/zenodo.33555) | Diamond Light Source I19-1 | | | | PILATUS 2M | Example Cytidine data set from I19-1 at Diamond Light Source | +| — | Zenodo [10.5281/zenodo.11946282](https://doi.org/10.5281/zenodo.11946282) | Diamond Light Source I19 | | | | PILATUS 2M | RODIN X-ray Diffraction Data 2360282 (L-alanine) | -Five rows have no PDB code. Four are small-molecule / chemical-crystallography datasets, kept -because they exercise short wavelengths, CdTe sensors and fine slicing; the fifth is the second -collection in the 6R72 Zenodo record, described below. They have no deposited macromolecular -values, so those columns are blank, and their titles are the repository record titles verbatim. +Seven rows have no PDB code. Six are small-molecule / chemical-crystallography datasets, kept +because they exercise short wavelengths, CdTe sensors, fine slicing and non-zero detector +2θ; the seventh is the second collection in the 6R72 Zenodo record, described below. They have +no deposited macromolecular values, so those columns are blank, and their titles are the +repository record titles verbatim. ## Archives that are not a single sweep -Most rows above are a single continuous rotation. Eleven archives are not; their layout is read +Most rows above are a single continuous rotation. Twenty-one archives are not; their layout is read from the image files themselves, from the repository file listings and from the depositors' own description of the record. Where an archive held more than one collection, only one is kept - the repository's project page is not a reliable guide to this, because it describes the project @@ -160,6 +181,23 @@ the rest were deleted, so a run over the data directory sees a single collection | 9CRW | a dose pair on one crystal 37 min apart - 0.025 s at 289 mm, 0.010 s at 276 mm | the 0.025 s sweep, whose 2.5 Å target matches the deposited 2.49 Å | | 7RIS | two crystals at two wavelengths - 1.53494 Å (Ho derivative) and 1.03329 Å (the deposited native) | **both** | +**Ten of the scout archives hold more than one collection.** Their layout was read from the +image files and repository listings; one sweep is kept for a run over the data directory unless +noted. + +| PDB / dataset | What the archive holds | Kept | +|---|---|---| +| 5JVN | two 360° sweeps of one crystal, 3600 × 0.1° each (`w1_3`, `w1_4`) | the `w1_3` sweep | +| 6FID | two 360° sweeps of one crystal, 3600 × 0.1° each | the first | +| 6IU8 | a two-wavelength MAD pair, 720 × 0.5° each at 1.605 Å (low remote) and 1.740 Å (peak) | **both** - the pair is the point | +| 7OS3 | four 360° sweeps at λ 2.066 Å, 3600 × 0.1° each, from two crystal positions (`pos2_1/2`, `pos3_1/2`) | all four are kept as separate sweep directories `pos*/` | +| 7L84 | two ~720° helical sweeps, 1439 × 0.5° each at λ 1.892 Å, room temperature | the `301_helical_1` sweep | +| 5M17 | seven crystals in one tar (5M03/5M17/5MEL/5MC8/5M5D/5M3W/5LYR), one 1800-frame sweep each | only the 5M17 tar was downloaded | +| cytidine | six scans, three ω and three φ, at 2θ = 30° (I19-1 commissioning) | the 1800-frame φ scan | +| lalanine | four runs of the RODIN L-alanine deposition at 2θ = 20° | the 900-frame `pgw240050_01` run | +| 9E2T | one continuous sweep plus screening images | the 2700-frame sweep | +| 8OWM | three MXRDR zips covering one 1800-frame sweep, plus a processed-data zip | the three sweep zips (proc zip skipped) | + ## Datasets published as Raw Data Letters Three of the datasets - 6R72, 8QQ7 and 6RLR - were published as IUCrData Raw Data Letters, a @@ -183,11 +221,11 @@ The authors of the second letter also published their own reciprocal-space recon ## Detector: image file vs PDB entry -For 76 of the 77 PDB-coded rows both the image file and the PDB entry name a detector. (For +For 94 of the 95 PDB-coded rows both the image file and the PDB entry name a detector. (For 8XTG neither can be compared - the header reads `PILATUS XXX, S/N XX-XXX`.) The table above uses the file value in every case, because the entry's label is often approximate. -**Seven of the 76 genuinely conflict** - the two sources name detectors that cannot both be +**Nine of the 94 genuinely conflict** - the two sources name detectors that cannot both be right: | PDB | PDB entry says | Image file says | Conflict | @@ -199,6 +237,8 @@ right: | 7ATG | DECTRIS PILATUS3 S 6M | PILATUS 6M-F, S/N 60-0117-F | generation | | 9O0H | DECTRIS EIGER X 16M | Dectris EIGER2 Si 16M, S/N D021324 | generation | | 9Z44 | DECTRIS EIGER X 9M | Dectris EIGER2 Si 9M, S/N E-18-0131 | generation | +| 9HNC | DECTRIS EIGER X 16M | PILATUS 6M-F, S/N 60-0117-F | model / size | +| 6P8P | DECTRIS PILATUS3 S 6M | PILATUS 6M-F, S/N 60-0112-F | generation / size | For 9SL0 the file is decisive and the entry is wrong: 3108 x 3262 pixels of 75 um on 450 um silicon, written by EIGER2 firmware `release-2022.1.2`, is an EIGER2 9M and not a PILATUS4 4M. @@ -211,7 +251,7 @@ generation (`PILATUS3 6M`) that the entry leaves off (`DECTRIS PILATUS 6M`) - 6Y ## Deposited models and structure factors -77 of the 82 datasets have a released PDB entry, and RCSB reports released structure factors +95 of the 102 datasets have a released PDB entry, and RCSB reports released structure factors (`status_code_sf = REL`) for every one of them. A merged result from this pipeline can therefore be checked against the deposited model or against the deposited intensities. @@ -236,8 +276,10 @@ reader that globs `*.img` will pick them up, so they are named here rather than | `dnba` | Zenodo record 10.5281/zenodo.1036416 | a small-molecule dataset, not a PDB deposition | | `metformin` | Zenodo record 10.5281/zenodo.20135265 | a small-molecule dataset, not a PDB deposition | | `nidppe` | Zenodo record 10.5281/zenodo.20041091 | a small-molecule dataset, not a PDB deposition | +| `cytidine` | Zenodo record 10.5281/zenodo.33555 | a small-molecule dataset, not a PDB deposition | +| `lalanine` | Zenodo record 10.5281/zenodo.11946282 | a small-molecule dataset, not a PDB deposition | -Three of the four small-molecule sets have a published structure to check a run against. These are +Five of the six small-molecule sets have a published structure to check a run against. These are reference values from the literature, not results obtained here. | Dataset | Space group | Cell (A, deg) | T | Reference | @@ -245,6 +287,8 @@ reference values from the literature, not results obtained here. | `dnba` | `C 1 2/c 1` (15) | 20.2635 8.7575 9.6697 / 90 109.941 90 | 30 K | the Zenodo record's own title and the `xia2.html` the depositors ship inside it, corroborated by COD 4510614/4510615 - Cryst. Growth Des. **13** (2013) 1861-1871 [doi:10.1021/cg300906j](https://doi.org/10.1021/cg300906j) | | `metformin` | `P 1 21/c 1` (14) | 7.9104 13.8794 7.9310 / 90 114.606 90 | 100 K | the hydrochloride, form I; COD 2108029 - Acta Cryst. B**73** (2017) 10-22 [doi:10.1107/S2052520616017844](https://doi.org/10.1107/S2052520616017844) | | `nidppe` | `P 1 21/c 1` (14) | 11.2779 13.3386 15.8739 / 90 98.7953 90 | 150 K | COD 2012031 - Acta Cryst. C**57** (2001) 690-693 [doi:10.1107/S0108270101003961](https://doi.org/10.1107/S0108270101003961) | +| `cytidine` | `P 21 21 21` (19) | 13.98 14.788 5.119 / 90 90 90 | 296 K | β-cytidine; COD 2001311 - D. L. Ward, Acta Cryst. C**49** (1993) 1789-1792 [doi:10.1107/S0108270193003464](https://doi.org/10.1107/S0108270193003464) | +| `lalanine` | `P 21 21 21` (19) | 5.7952 5.933 12.362 / 90 90 90 | ambient | COD 2104782 - N. A. Tumanov et al., Acta Cryst. B**66** (2010) 458-471 [doi:10.1107/S010876811001983X](https://doi.org/10.1107/S010876811001983X) | `cuhf2` has no confirmed cell. Its space group is published as `P 4/n m m` (Phys. Rev. B **81**, 064422 (2010) [doi:10.1103/PhysRevB.81.064422](https://doi.org/10.1103/PhysRevB.81.064422)) but no -- 2.54.0 From 7e7103f49f575ca1c99f00b9040052373aed7db6 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 4 Sep 2026 20:00:20 +0200 Subject: [PATCH 23/75] spindle: the blind-cone score compiles on MSVC, and the report version test says 7 Two CI failures from the spindle series, both self-inflicted. The severity used M_PI, which is not standard and which MSVC does not define, in a file under image_analysis - a tree the Windows viewer builds. common/JFJochMath.h exists for exactly this and is now included. The same defect was found and fixed in the post-refinement residual earlier in the day; the grep that confirmed it looked at a worktree that did not yet carry this file, so the second instance survived. The report-version test pins the version deliberately - adding a key to the report is a contract change and that line is where it has to be acknowledged - but the key added with the spindle report keys bumped the constant to 7 without the test following. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/indexing/SpindleBlindFraction.cpp | 10 ++++++---- tests/ResultReportTest.cpp | 5 +++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/image_analysis/indexing/SpindleBlindFraction.cpp b/image_analysis/indexing/SpindleBlindFraction.cpp index d7e9bd623..3cee55c6e 100644 --- a/image_analysis/indexing/SpindleBlindFraction.cpp +++ b/image_analysis/indexing/SpindleBlindFraction.cpp @@ -6,6 +6,8 @@ #include "SpindleBlindFraction.h" +#include "../../common/JFJochMath.h" // PI - M_PI is not standard, and MSVC does not define it + namespace { // Only rows short enough to be a plausible symmetry axis count. The cut is relative to the // crystal's own shortest row, not an absolute length, so it works the same for a 40 A cell and @@ -32,7 +34,7 @@ float SpindleThetaMax_deg(float wavelength_A, float d_min_A) { if (wavelength_A <= 0 || d_min_A <= 0) return 0; const float sin_theta = std::min(1.0f, wavelength_A / (2.0f * d_min_A)); - return static_cast(std::asin(sin_theta) * 180.0 / M_PI); + return static_cast(std::asin(sin_theta) * 180.0 / PI); } float BlindConeSelfOverlap(float x) { @@ -40,7 +42,7 @@ float BlindConeSelfOverlap(float x) { return 0.0f; if (x <= 0.0f) return 1.0f; - return static_cast(2.0 / M_PI) * + return static_cast(2.0 / PI) * (std::acos(x) - x * std::sqrt(1.0f - x * x)); } @@ -91,7 +93,7 @@ std::optional SpindleBlindFraction(const std::vector &ro const auto consider = [&](const Coord &direction, float row_length_A) { const float len = direction.Length(); const float cos_beta = std::min(1.0f, std::fabs(direction * axis) / len); - const float beta_deg = static_cast(std::acos(cos_beta) * 180.0 / M_PI); + const float beta_deg = static_cast(std::acos(cos_beta) * 180.0 / PI); // A direction PERPENDICULAR to the spindle is as damaging as one along it, and far more // common: a lone 2-fold about it carries the blind cone onto the cone's opposite lobe, which // the same sweep leaves equally unmeasured. That is Friedel's rescue, which is no rescue - @@ -159,7 +161,7 @@ std::optional SpindleBlindFractionFromLattice(const CrystalLatt [](const Coord &a, const Coord &b) { return a.Length() < b.Length(); }); constexpr size_t MAX_LATTICE_ROWS = 6; - const float cos_5_deg = std::cos(5.0f * static_cast(M_PI) / 180.0f); + const float cos_5_deg = std::cos(5.0f * static_cast(PI) / 180.0f); std::vector rows; for (const auto &r : all) { if (rows.size() >= MAX_LATTICE_ROWS diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index 4f4462fe0..d63348ec1 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -68,8 +68,9 @@ TEST_CASE("ResultReport_Render", "[Diagnostics]") { const auto text = RenderResultReport("prefix", "in.h5", x, result); - // The stable keys a consumer greps for. - CHECK(text.find("\nREPORT_VERSION= 6\n") != std::string::npos); + // The stable keys a consumer greps for. The version is pinned on purpose: a key added to the + // report is a contract change, and this line is where it has to be acknowledged. + CHECK(text.find("\nREPORT_VERSION= 7\n") != std::string::npos); CHECK(text.find("\nOUTPUT_PREFIX= prefix\n") != std::string::npos); CHECK(text.find("\nINDEXING_RATE= 0.8700\n") != std::string::npos); CHECK(text.find("\nSPACE_GROUP_NUMBER= 96\n") != std::string::npos); -- 2.54.0 From 814f864daf22761999c87fe575488308327976fb Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 07:30:14 +0200 Subject: [PATCH 24/75] rotation: a pass-2 floor miss falls back on pass 1 rather than throwing Two-pass rotation indexing re-indexes de novo at the post-refined geometry, and where the second pass lands on an integer harmonic of a long axis it can fall below the validation-frame floor and abort - discarding a pass-1 result that was already correct. The frame count cannot arbitrate: a spurious axis multiple indexes every frame its true cell does, so the harmonic is not recognised as one. Where the second pass misses the floor and the first pass produced a lattice, its whole result is substituted and integration continues. This is the floor-side companion of the supercell-collapse guard just below, which the throw pre-empted. Measured on the 152-dataset corpus: every run that completed before completes with bit-identical results - 143 of 143 unchanged, no space group and no R_meas moved - and three runs that aborted now finish, one of them recovering the deposited cell and passing. The other two carry a pass-1 lattice that was itself on a harmonic; the substitution is faithful to pass 1 and does not claim to repair it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- rugnux/Rugnux.cpp | 57 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 43 insertions(+), 14 deletions(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 4b1a6fd9c..daf87222a 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -3183,20 +3183,49 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // the long-axis rescue, so a metric that rescue recovers is never rejected on its pre-rescue // score. if (best.score < static_cast(validation.size()) / 6) { - // Name the cell and Bravais class that was rejected. The commonest cause is a metric - // symmetry promoted one class too far - the constrained cell then misses every - // reflection by the small angle the constraint snapped away - and without the cell in - // the message there is nothing to see that from. - const auto &c = best.result->search_result.conventional.GetUnitCell(); - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - fmt::format("Two-pass rotation indexing found only a lattice that " - "indexes {}/{} validation frames - too few for it to be " - "this crystal's lattice. It was {}-centred {}, " - "{:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f}.{}", - best.score, static_cast(validation.size()), - best.result->search_result.centering, - gemmi::crystal_system_str(best.result->search_result.system), - c.a, c.b, c.c, c.alpha, c.beta, c.gamma, next_steps)); + // The second (refined-geometry) pass re-indexes DE NOVO, and the post-refined geometry - + // fitted to the lattice pass 1 already found - can tip the blind FFT onto an axis harmonic + // of a long cell that indexes too few frames to clear this floor (measured: a tripled + // ~100 A c-axis at 7/60). Pass 1 found and integrated the true cell at the header geometry, + // so prefer its whole result rather than abort the run on a re-index only the refined + // geometry made fail. This is the floor-side companion of the supercell-collapse guard + // below: that one only reaches a pass-2 lattice that DID clear the floor, whereas here the + // harmonic fell BELOW it and the throw would run before the guard could ever see it. + if (!geometry_prepass && prepass_result_.has_value()) { + const auto &pc = prepass_result_->search_result.conventional.GetUnitCell(); + const auto &bc = best.result->search_result.conventional.GetUnitCell(); + logger.Warning("Two-pass: the refined-geometry re-index indexes only {}/{} validation " + "frames ({}-centred {}, {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f}) - too " + "few for this crystal's lattice; integrating with pass-1's lattice " + "instead ({}-centred {}, {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f})", + best.score, static_cast(validation.size()), + best.result->search_result.centering, + gemmi::crystal_system_str(best.result->search_result.system), + bc.a, bc.b, bc.c, bc.alpha, bc.beta, bc.gamma, + prepass_result_->search_result.centering, + gemmi::crystal_system_str(prepass_result_->search_result.system), + pc.a, pc.b, pc.c, pc.alpha, pc.beta, pc.gamma); + best.result = *prepass_result_; + best.score = count_indexed(*indexer, *best.result); // re-score at the refined geometry + best.vol = std::abs(best.result->lattice + .ToPrimitive(best.result->search_result.centering).CalcVolume()); + best.name = "pass-1 lattice (refined-geometry re-index too sparse)"; + } else { + // Name the cell and Bravais class that was rejected. The commonest cause is a metric + // symmetry promoted one class too far - the constrained cell then misses every + // reflection by the small angle the constraint snapped away - and without the cell in + // the message there is nothing to see that from. + const auto &c = best.result->search_result.conventional.GetUnitCell(); + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + fmt::format("Two-pass rotation indexing found only a lattice that " + "indexes {}/{} validation frames - too few for it to be " + "this crystal's lattice. It was {}-centred {}, " + "{:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f}.{}", + best.score, static_cast(validation.size()), + best.result->search_result.centering, + gemmi::crystal_system_str(best.result->search_result.system), + c.a, c.b, c.c, c.alpha, c.beta, c.gamma, next_steps)); + } } indexer->ForceRotationIndexerResult(*best.result); logger.Info("First-pass spot finding: {} frames in {} batches, {:.2f} s", -- 2.54.0 From e1aa84cc9ec63ae1fa6f344475dedd1aacb2937e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 07:31:43 +0200 Subject: [PATCH 25/75] symmetry: an operator is judged against a reference a false operator cannot contaminate The promotion decision compared each candidate operator against its parent's mean H, and took the maximal-order confirmed subgroup as that parent. Where a false operator was already confirmed, the mean pooled it with the real one, the true lower group was barred from being the reference at all, and the systematic-absence veto - gated on the H test failing - was disarmed by the same false confirmation. The first step out of P1 had no H parent to compare against and stood on the operator correlation floor alone. Each candidate now also carries an intensity-weighted R over the pairs the H test already collects, summed rather than averaged so strong reflections decide it, and judged against two references a false operator cannot move: the best-agreeing operator where two or more are confirmed, and the half-dataset noise floor on the first step out of P1. The gate only ever refuses, so a wrong refusal leaves P1 and the run recoverable. Measured on the 152-dataset corpus: three point-group over-calls corrected, one of them to an exact match with the deposited group; four datasets improve R_meas by 0.15 to 0.27 and none worsens; the remaining 139 are bit-identical. One crystal whose 2-fold was already being over-called is now under-called instead - the bounds were fitted on few exemplars and the report prints them per run so the population can be measured. Screw axes are untouched: they cannot be called from I/sigma and wait on their own rewrite. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- .../scale_merge/SearchSpaceGroup.cpp | 150 +++++++++++++++++- image_analysis/scale_merge/SearchSpaceGroup.h | 61 +++++++ 2 files changed, 204 insertions(+), 7 deletions(-) diff --git a/image_analysis/scale_merge/SearchSpaceGroup.cpp b/image_analysis/scale_merge/SearchSpaceGroup.cpp index 51846557d..d4d4f4df8 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.cpp +++ b/image_analysis/scale_merge/SearchSpaceGroup.cpp @@ -475,6 +475,26 @@ SearchSpaceGroupResult SearchSpaceGroup( // crystal, so the population this is normalised over means the same thing on all of them. const std::vector Ecc = shell_normalised(pass_cc); + // The merge's own random-noise R floor from the two half-dataset merges (see + // SearchSpaceGroupResult::merge_r_floor). Over the same present (pass_cc) reflections the operator + // R is measured on, so the two are at the same intensity range and multiplicity and their ratio + // means the same on every crystal - intensity-weighted, sigma-free, and built without applying any + // candidate symmetry so a false operator cannot inflate it. + { + double r_num = 0.0, r_den = 0.0; + for (size_t i = 0; i < n; ++i) { + if (!pass_cc[i]) + continue; + const double a = merged[i].I_half[0], b = merged[i].I_half[1]; + if (std::isfinite(a) && std::isfinite(b) && a + b > 0.0) { + r_num += std::fabs(a - b); + r_den += a + b; + } + } + if (r_den > 0.0) + result.merge_r_floor = r_num / r_den; + } + std::unordered_map key_to_index; key_to_index.reserve(n * 2); for (size_t i = 0; i < n; ++i) @@ -521,16 +541,22 @@ SearchSpaceGroupResult SearchSpaceGroup( // a margin of 1.8%. H is calibrated on raw I and stays there. std::vector hv; hv.reserve(x.size()); + double r_num = 0.0, r_den = 0.0; // intensity-weighted R across the operator (see r_stat) for (size_t p = 0; p < x.size(); ++p) { const double denom = x[p] + y[p]; - if (denom > 0.0) + if (denom > 0.0) { hv.push_back(std::fabs(x[p] - y[p]) / denom); + r_num += std::fabs(x[p] - y[p]); + r_den += denom; + } } if (!hv.empty()) { const size_t mid = hv.size() / 2; std::nth_element(hv.begin(), hv.begin() + mid, hv.end()); s.h_stat = hv[mid]; } + if (r_den > 0.0) + s.r_stat = r_num / r_den; s.present = s.n_pairs >= opt.min_pairs_per_operator && std::isfinite(s.cc) && s.cc >= opt.min_operator_cc; return s; @@ -736,7 +762,13 @@ SearchSpaceGroupResult SearchSpaceGroup( struct PGCand { const PointGroupInfo* pg; int order; double min_class_cc; double chi2; double b_extra; // Filled by the selection loop below and carried so the adopted candidate's H // ratio can be reported whether or not the bound had anything to say about it. - double h_ratio = std::numeric_limits::quiet_NaN(); }; + double h_ratio = std::numeric_limits::quiet_NaN(); + // The added operators' mean intensity-weighted R, and that over the merge's random + // R floor - the full-resolution merge-degradation gate's numbers (see r_stat, + // max_operator_r_over_best and max_operator_r_over_floor). Carried for the same + // reason as h_ratio: to report the adopted candidate's whether or not it decided. + double r_added = std::numeric_limits::quiet_NaN(); + double r_over_floor = std::numeric_limits::quiet_NaN(); }; int refused_order = 0; std::string refused_pg_hm, refused_why; std::vector pg_cands; @@ -779,6 +811,22 @@ SearchSpaceGroupResult SearchSpaceGroup( chi2_ref = std::min(chi2_ref, cand_chi2[i]); } + // The best-agreeing operator anywhere in the data: the smallest intensity-weighted R over every + // CC-confirmed operator (see the merge-degradation gate below). A genuine symmetry operator + // relates equal intensities, so its R is small; a low R is thus itself the mark of a genuine + // operator, and the smallest one is the cleanest reference the data offers. It is GLOBAL, not + // per-candidate, because a false point group can be built entirely from operators that agree + // among THEMSELVES - a pseudo-tetragonal 4 whose 4-folds relate the same wrong intensities - and + // only an operator outside that group (the one true 2-fold) exposes them. Confirmed operators + // only: an unconfirmed (low-CC) one has, by construction, a high R and never sets this minimum. + double global_best_r = std::numeric_limits::infinity(); + int n_confirmed_ops = 0; + for (const auto& [rk, s] : op_cache) + if (s.present && s.n_pairs >= opt.min_pairs_per_operator && s.r_stat > 0.0) { + global_best_r = std::min(global_best_r, s.r_stat); + ++n_confirmed_ops; + } + // Choose the largest point group that is both operator-confirmed AND self-consistent (its merge // chi^2 is not inflated past the miscalibration-widened bound below; ties -> higher min class CC). // Identity (no operators) is always consistent, so it stays the P1 fallback. @@ -846,8 +894,9 @@ SearchSpaceGroupResult SearchSpaceGroup( // tie (see above), the promotion is judged on the most damning of them. double h_ratio = std::numeric_limits::quiet_NaN(); double h_added = 0.0; // the added operators' own H, for the twin fraction below + double r_added = std::numeric_limits::quiet_NaN(); // added operators' mean R (full-res gate) for (const auto *parent : parents) { - double h_new = 0.0, h_par = 0.0; + double h_new = 0.0, h_par = 0.0, r_new = 0.0; int n_new = 0, n_par = 0, pairs_new = 0, pairs_par = 0; for (const auto &rot : c.pg->rotations) { if (rot.rot == gemmi::Op::identity().rot) @@ -858,17 +907,42 @@ SearchSpaceGroupResult SearchSpaceGroup( const bool in_parent = std::binary_search(parent->rotation_set.begin(), parent->rotation_set.end(), RotKey(rot)); if (in_parent) { h_par += os.h_stat; ++n_par; pairs_par += os.n_pairs; } - else { h_new += os.h_stat; ++n_new; pairs_new += os.n_pairs; } + else { h_new += os.h_stat; ++n_new; pairs_new += os.n_pairs; r_new += os.r_stat; } } if (n_new > 0 && n_par > 0 && pairs_new >= opt.min_pairs_for_h && pairs_par >= opt.min_pairs_for_h && h_par > 0.0) { const double r = (h_new / n_new) / (h_par / n_par); + // Record the added operators' R at the same (max-H-ratio) parent the H ratio reports. + if (!std::isfinite(h_ratio) || r > h_ratio) + r_added = r_new / n_new; if (!std::isfinite(h_ratio) || r > h_ratio) { h_ratio = r; h_added = h_new / n_new; } } } + // The first step out of P1 has no parent with operators, so the H loop above left r_added + // unset: the added set is then ALL of this candidate's operators, and the only reference is + // the merge's random-noise floor (there is no other operator to compare against). This is the + // one guard that step has - the H ratio and the b veto both need a parent group. + if (!std::isfinite(r_added) && !c.pg->rotations.empty()) { + double r_new = 0.0; int n_new = 0, pairs_new = 0; + for (const auto &rot : c.pg->rotations) { + if (rot.rot == gemmi::Op::identity().rot) + continue; + const auto &os = operator_score(rot); + if (os.n_pairs < opt.min_pairs_per_operator) + continue; + r_new += os.r_stat; ++n_new; pairs_new += os.n_pairs; + } + if (n_new > 0 && pairs_new >= opt.min_pairs_for_h) + r_added = r_new / n_new; + } + c.r_added = r_added; + c.r_over_floor = (std::isfinite(r_added) && std::isfinite(result.merge_r_floor) + && result.merge_r_floor > 0.0) + ? r_added / result.merge_r_floor + : std::numeric_limits::quiet_NaN(); c.h_ratio = h_ratio; // The chi^2 ratio is only trustworthy when the error model is calibrated. When even the best // subgroup's reduced chi^2 (chi2_ref) is far above 1 - weak, low-resolution data whose merged @@ -908,6 +982,33 @@ SearchSpaceGroupResult SearchSpaceGroup( if (h_refused) consistent = false; + // Full-resolution merge-degradation test. A necessary condition like H, but read against a + // CLEAN reference rather than against a parent that can itself be contaminated - so it catches + // a pseudo-symmetric cascade the parent-normalised gates wave through (a 2 -> 222 -> 422 built + // by pooling one real 2-fold with false ones), and it is the only gate that can act on the + // first step out of P1. Not rescuable: the b rescue above only lifts a chi^2-borderline case. + // - When the data confirm more than one operator, judge the added operators' R against the + // GLOBALLY best-agreeing operator (the smallest R anywhere): on a genuine group every + // operator agrees about as well as that best one, so the ratio is ~1; a false operator - + // even one whose group's members all agree among themselves, like a pseudo-tetragonal 4 - + // sits far above it (see max_operator_r_over_best). + // - The first step out of P1, where only one operator is confirmed at all, has no other + // operator to be the best - fall back to the merge's own random-noise R floor + // (max_operator_r_over_floor). + bool r_refused = false; + double r_gate_value = std::numeric_limits::quiet_NaN(); + if (std::isfinite(c.r_added)) { + if (n_confirmed_ops >= 2 && std::isfinite(global_best_r) && global_best_r > 0.0) { + r_gate_value = c.r_added / global_best_r; + r_refused = r_gate_value > opt.max_operator_r_over_best; + } else if (std::isfinite(result.merge_r_floor) && result.merge_r_floor > 0.0) { + r_gate_value = c.r_added / result.merge_r_floor; + r_refused = r_gate_value > opt.max_operator_r_over_floor; + } + } + if (r_refused) + consistent = false; + if (!consistent) { // Record the highest-order refusal so the caller can say WHY it is processing lower. if (c.order > refused_order && c.pg->representative) { @@ -924,6 +1025,16 @@ SearchSpaceGroupResult SearchSpaceGroup( + ") - the added operator relates unequal intensities, as a twin law of " "fraction " + FormatDouble(std::max(0.0, 0.5 - h_added), 2) + " or more would"; + else if (r_refused) + refused_why = "its added operators' intensity-weighted R is " + + FormatDouble(r_gate_value, 1) + "x " + + (n_confirmed_ops >= 2 + ? "the best-agreeing operator in the data (bound " + + FormatDouble(opt.max_operator_r_over_best, 1) + : "the merge's own random-noise floor (bound " + + FormatDouble(opt.max_operator_r_over_floor, 1)) + + "x) - at full resolution the added operator relates reflections that " + "disagree far beyond measurement error, i.e. it is not a real symmetry"; else if (b_refused) // Against the FLOORED parent b, which is what the test compared - quoting the raw // parent b reads as a far bigger balloon than the one that actually tripped the veto. @@ -977,9 +1088,16 @@ SearchSpaceGroupResult SearchSpaceGroup( // The H ratio of the promotion that was ADOPTED, reported whether the bound had anything to say // about it or not. Read after the choice is final, so a fixed_point_group override reports the // ratio of the group it forced rather than of the one Stage A would have taken. + result.global_best_operator_r = std::isfinite(global_best_r) ? global_best_r + : std::numeric_limits::quiet_NaN(); for (const auto& c : pg_cands) - if (c.pg == best_pg) + if (c.pg == best_pg) { result.h_ratio = c.h_ratio; + result.r_added = c.r_added; + result.r_over_floor = c.r_over_floor; + if (std::isfinite(c.r_added) && std::isfinite(global_best_r) && global_best_r > 0.0) + result.r_over_best = c.r_added / global_best_r; + } result.h_ratio_bound = opt.max_operator_h_ratio; // Only report a refusal that is actually ABOVE what was adopted. @@ -1343,13 +1461,14 @@ std::string SearchSpaceGroupResultToText(const SearchSpaceGroupResult& result, os << " " << std::setw(14) << std::left << "operator" << std::right << std::setw(9) << "CC" << std::setw(10) << "pairs" << std::setw(9) << "symm" - << std::setw(9) << "H" << "\n"; + << std::setw(9) << "H" << std::setw(9) << "R" << "\n"; for (const auto& s : result.operator_scores) { os << " " << std::setw(14) << std::left << s.op_triplet_hkl << std::right << std::setw(9) << std::fixed << std::setprecision(3) << s.cc << std::setw(10) << s.n_pairs << std::setw(9) << (s.present ? "yes" : "no") - << std::setw(9) << std::fixed << std::setprecision(3) << s.h_stat << "\n"; + << std::setw(9) << std::fixed << std::setprecision(3) << s.h_stat + << std::setw(9) << std::fixed << std::setprecision(3) << s.r_stat << "\n"; } os << " H = median |I1-I2|/(I1+I2) over the operator's pairs - the disagreement it implies, with\n" " no sigma in it. The promotion gate is the RATIO of the mean H over the operators a\n" @@ -1360,6 +1479,23 @@ std::string SearchSpaceGroupResultToText(const SearchSpaceGroupResult& result, << FormatDouble(result.h_ratio_bound, 2) << ").\n"; else os << " H ratio not available (no parent group to normalise against, or too few pairs).\n"; + // R = intensity-weighted sum|I1-I2|/sum(I1+I2) across the operator's pairs (strong reflections + // dominate, where the median H under-weights them). The merge-degradation gate scores the added + // operators' R against a clean reference - the globally best-agreeing operator, or the merge's + // own random-noise R floor (from the half-dataset merges) on the first step out of P1 - so it + // catches false symmetry the parent-normalised H waves through. + if (std::isfinite(result.r_added)) { + os << " Added-operator R " << FormatDouble(result.r_added, 3); + if (std::isfinite(result.r_over_best)) + os << " = " << FormatDouble(result.r_over_best, 2) << "x the best operator (" + << FormatDouble(result.global_best_operator_r, 3) << ")"; + if (std::isfinite(result.r_over_floor)) + os << ", " << FormatDouble(result.r_over_floor, 2) << "x the random-noise floor (" + << FormatDouble(result.merge_r_floor, 3) << ")"; + os << " for the adopted point group.\n"; + } else if (std::isfinite(result.merge_r_floor)) { + os << " Merge random-noise R floor " << FormatDouble(result.merge_r_floor, 3) << ".\n"; + } os << "\nSpace-group candidates\n"; os << " " << std::setw(10) << std::left << "SG" << std::right diff --git a/image_analysis/scale_merge/SearchSpaceGroup.h b/image_analysis/scale_merge/SearchSpaceGroup.h index 0382fdd9e..5e2f29411 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.h +++ b/image_analysis/scale_merge/SearchSpaceGroup.h @@ -41,6 +41,13 @@ struct SpaceGroupOperatorScore { // twin law (see max_operator_h_ratio). The median rather than the mean because a merohedral twin // perturbs EVERY pair while a badly integrated minority perturbs only the tail. double h_stat = 0.0; + // INTENSITY-WEIGHTED disagreement across this operator's pairs: sum|I1-I2| / sum(I1+I2), an + // R-factor form. Unlike h_stat (a median) it is dominated by the STRONG reflections, which is where + // a false operator relating unequal intensities shows the largest contrast and where a genuine one + // agrees best - the same reflections a median under-weights. Sigma-free, so it does not saturate + // with the merge's ISa the way the chi^2 and systematic-b gates do on a starved search merge. Used + // as the full-resolution merge-degradation gate (see max_operator_r_over_floor). + double r_stat = 0.0; }; struct SpaceGroupCandidateScore { @@ -345,6 +352,42 @@ struct SearchSpaceGroupOptions { // The H test needs at least this many pairs on both sides to mean anything. int min_pairs_for_h = 200; + // Full-resolution merge-degradation gate. The operators a promotion ADDS are scored by their + // intensity-weighted R (r_stat: a SUM, so the strong low-resolution reflections that carry the + // contrast dominate it, where the median H under-weights them), and that R is judged against a + // CLEAN reference. This is the one Stage-A gate whose reference cannot be contaminated by false + // operators: the H ratio and the systematic-b veto both normalise against the PARENT group, and a + // parent can itself be a false promotion (a pseudo-tetragonal 2 -> 222 -> 422 cascade confirms + // 222 by pooling the one real 2-fold with two false ones, so the 422 step is then measured + // against an already-contaminated 222, and every ratio reads ~1.4 and passes). It also acts on + // the first step out of P1, where there is no parent to normalise against at all and the operator + // CC floor stood alone. Sigma-free, so unlike the chi^2 and b gates it does not saturate with the + // merge's ISa on a starved search merge. + // + // Two references, each scale-free so the bound means the same on every crystal (the property an + // absolute R bound lacks): + // + // - max_operator_r_over_best: when the data confirm more than one operator, the added operators' + // mean R over the GLOBALLY best-agreeing operator (the smallest r_stat anywhere). A genuine + // operator relates equal intensities, so a small R is itself the signature of a real operator + // and the smallest one is the cleanest reference the data hold. On a genuine group EVERY + // operator agrees about as well as the best (ratio ~1.0-1.2 measured, cubic 3-folds and a + // genuine 6 included); a false operator sits far above it, INCLUDING one whose own group's + // members agree among themselves (a pseudo-tetragonal 4 reads ~3.8, because the reference is + // the true 2-fold OUTSIDE that group). Measured over the probe crystals: genuine promotions + // reach 1.2, the false 2-fold/4-fold/6-fold-holohedry steps 2.2-3.8. 2.0 sits in that gap. + double max_operator_r_over_best = 2.0; + // + // - max_operator_r_over_floor: the first step out of P1 confirms only the one operator, so there + // is no other to be the best - fall back to the merge's own random-noise R floor (merge_r_floor, + // from the half-dataset merges). The floor is a genuine-symmetry disagreement (two halves of + // the SAME reflections) at the same multiplicity, so it too is a clean scale-free reference. + // Measured: a genuine first 2-fold sits at ~1.3-2.3x the floor, a false one (a pseudo-C-centred + // metric coincidence) at ~4.7. 3.5 sits in that gap. Wider than the best-operator bound because + // the floor is a noisier reference than a confirmed operator (it can also be inflated by an + // indexing ambiguity that mixes hands), and the safe direction here is to stay in P1. + double max_operator_r_over_floor = 3.5; + // Above this reduced chi^2 for the best subgroup (chi2_ref), the merged error model is treated as // badly miscalibrated (weak, low-resolution data whose sigmas are far too small): the fixed-sigma // chi^2 ratio then grows with point-group order for genuine high symmetry too and can no longer @@ -498,6 +541,24 @@ struct SearchSpaceGroupResult { double h_ratio = std::numeric_limits::quiet_NaN(); double h_ratio_bound = 0.0; + // The merge's own RANDOM-noise R floor: sum|Ihalf0-Ihalf1| / sum(Ihalf0+Ihalf1) over the present + // reflections, from the two half-dataset merges. This is the intensity-weighted disagreement two + // GENUINELY equivalent groups of observations of the SAME reflection show - i.e. the value the + // r_stat of a real symmetry operator would tend to (plus a modest systematic floor), measured on + // this very crystal at this very multiplicity. It is the clean, uncontaminated reference the + // operator R is judged against (see max_operator_r_over_floor), the one a false operator cannot + // move because it is built without applying any candidate symmetry. NaN if the merge carried no + // half-dataset intensities. Reported so a run shows the margin the gate had. + double merge_r_floor = std::numeric_limits::quiet_NaN(); + // The intensity-weighted operator R of the promotion that was ADOPTED (mean over the operators it + // adds), that R over merge_r_floor, and that R over the globally best-agreeing operator - the + // numbers the merge-degradation gate acts on (best-operator where two or more operators are + // confirmed, floor on the first step out of P1). Reported so a run shows the margin the gate had. + double r_added = std::numeric_limits::quiet_NaN(); + double r_over_floor = std::numeric_limits::quiet_NaN(); + double r_over_best = std::numeric_limits::quiet_NaN(); + double global_best_operator_r = std::numeric_limits::quiet_NaN(); + // A HIGHER point group whose operators the intensities confirmed (Stage A) but whose promotion the // consistency tests refused, with the reason. Processing continues in the lower group, which is the // safe direction: merging a twinned crystal in the twin's holohedry averages non-equivalent -- 2.54.0 From d4fc09277e3c197685707649cf4fe23b5429784f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 10:11:07 +0200 Subject: [PATCH 26/75] twinning: the promotion-circularity flag is set on the struct the text is rendered from AnalyzeTwinning returns a fresh result with the flag clear, so setting it only afterwards left the statistics text asserting "No twinning: the Laue class is holohedral" as a fact while the report, rendering the same struct once the flag had been set, said the opposite. One run, two twin verdicts, and the flat one landed on exactly the case the flag exists to mark. Assigning before the call would not do either: the call replaces the whole struct. Reproduced on 18 of 95 stored runs from the current corpus. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- rugnux/Rugnux.cpp | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index daf87222a..b669806bc 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -4726,6 +4726,13 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // thing - it is what the SEARCH did, the second pass reads it, and it is set either way. if (!geometry_prepass) { result.twinning = AnalyzeTwinning(sm.merged, twin_sg); + // Set on the struct the text below is rendered from. AnalyzeTwinning returns a fresh result + // with this flag clear, so setting it only afterwards left the statistics text asserting + // "No twinning: the Laue class is holohedral" as a fact while the report, rendering the same + // struct once the flag was set, said the opposite - two twin verdicts from one run, with the + // flat one landing on exactly the case the flag exists for. (Assigning it before the call + // would not do: the call replaces the whole struct.) + result.twinning.laue_class_was_chosen_by_promotion = promoted_point_group; stats_text << TwinningAnalysisToText(result.twinning) << "\n"; } // Mark the conclusion as non-authoritative when the Laue class was reached by a promotion the -- 2.54.0 From 431ac4e3d99e1cdcea0b5ba3c5f89b99ad15648e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 10:11:19 +0200 Subject: [PATCH 27/75] symmetry: the systematic-b fit divides by its degrees of freedom, as its own chi2 does Each deviation is taken from a mean fitted on its own orbit, so one degree of freedom per orbit is spent: dof = sum(n-1) = N - G, the divisor chi2_under a few lines above already uses. Dividing by N understated the reduced chi^2 and so overstated the b that brings it to 1 - unevenly, because a low-multiplicity parent loses a larger fraction of its dof than the higher-multiplicity candidate it is compared against, and the difference landed on the b RATIO the gates read. Measured three times independently on disjoint corpora: median ratio inflation +23.7%, +23.7% and +23.8%, with the corrected ratio lower in 42 of 42 promotions, so the error pushed toward over-calling. Expect no verdict to move: the b veto has not been observed to fire on any corpus it has been measured over, and refused_by=b fired zero times in 25 refusals. The bounds are left at 1.78 / 2.00 / 0.05 rather than rescaled. They were calibrated in the old convention, but re-measured on the corrected statistic against independent space groups they still separate: the genuine maximum is 1.640 and the nearest false step 3.208, so both bounds sit inside the only clean gap the corrected distribution offers. A second derivation from population medians argues for 1.38 / 1.56 / 0.07 instead; that is a calibration question for a battery, not a condition of fixing the arithmetic. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/scale_merge/SearchSpaceGroup.cpp | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/image_analysis/scale_merge/SearchSpaceGroup.cpp b/image_analysis/scale_merge/SearchSpaceGroup.cpp index d4d4f4df8..de1790b91 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.cpp +++ b/image_analysis/scale_merge/SearchSpaceGroup.cpp @@ -731,6 +731,10 @@ SearchSpaceGroupResult SearchSpaceGroup( // floor a-dependent, on a quantity that has no a. Leave it alone. auto merge_systematic_b = [&](const Orbits& orb) -> double { std::vector> obs; // I, sigma, deviation-from-orbit-mean + long n_orbits = 0; // orbits contributing, for the degrees of freedom + for (const Acc& g : orb.acc) + if (g.n >= 2) + ++n_orbits; for (size_t i = 0; i < n; ++i) { if (orb.orbit[i] < 0) continue; @@ -739,13 +743,19 @@ SearchSpaceGroupResult SearchSpaceGroup( continue; obs.push_back({I[i], Sigma[i], I[i] - g.swI / g.sw}); } - if (obs.size() < 20) + // Each deviation is taken from a mean fitted on its own orbit, so one degree of freedom per + // orbit is spent: dof = sum(n-1) = N - G, the same denominator chi2_under uses above. Dividing + // by N instead understates the reduced chi^2 and so overstates the b that brings it to 1, and + // it does so unevenly - a low-multiplicity parent loses a larger fraction of its dof than the + // higher-multiplicity candidate, which inflates the b RATIO the gates read. + const double dof = static_cast(obs.size()) - static_cast(n_orbits); + if (obs.size() < 20 || dof <= 0.0) return 0.0; auto reduced_chi2 = [&](double b) { double s = 0.0; for (const auto& o : obs) s += o[2] * o[2] / (o[1] * o[1] + (b * o[0]) * (b * o[0])); - return s / static_cast(obs.size()); + return s / dof; }; if (reduced_chi2(0.0) <= 1.0) return 0.0; -- 2.54.0 From 5761bcc94dcd37680e13c80be48317c4f9d41068 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 10:11:35 +0200 Subject: [PATCH 28/75] lattice: the metric re-ask reads the centring from the alternatives, and its own arm's ISa Two defects in the Le Page re-ask, both of which refused promotions the intensities had confirmed. The centring cannot be read from the absences in this setting: reindexing into the metric candidate's conventional cell leaves its centring-absent class holding no data at all, so among the point-group-equivalent candidates the intensities cannot separate, the tie-break takes the lowest number, and that is the primitive one. The re-ask then compared that primitive representative against the metric's centred candidate and refused. Pick the candidate matching the metric's own centring out of the alternatives first, exactly as the centred-lattice test above already does, and leave the displaced candidate among the alternatives so the reported "or ..." list stays the set of groups the data cannot separate. The refusal message was wrong as well, and said so in a way that read as an intensity verdict: a promotion refused only for the centring technicality above now says the point group IS higher and names what this setting cannot test. Separately, the filtered arm's search was handed the all-observation arm's ISa, which is re-set for that merge a few hundred lines earlier and differs by up to a factor of six - so the filtered merge's "genuinely present" cut was judged on the other arm's error model. Capture it before it is replaced. A census of every run log in the external corpus bounds the effect: the re-ask fires six times, two already adopt, three are honestly refused at equal order, and exactly one dataset can move - which it does, from a primitive triclinic group to the centred monoclinic one deposited for it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- rugnux/Rugnux.cpp | 49 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 8 deletions(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index b669806bc..b4f70e8d6 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -4139,6 +4139,10 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // two-arm rule: a promotion is refused by the most damning statistic it can be shown, and // only one of the two merges has to be starved for that to happen. std::vector merged_filtered; + // ...and that arm's OWN ISa. sg_opts.merge_isa is re-set below for the all-observation + // merge, and the two differ by up to a factor of six, so the re-ask has to keep this one + // or it judges the filtered merge's "genuinely present" cut on the other arm's error model. + double merged_filtered_isa = 0.0; // Second opinion from a merge that keeps only well-measured observations (see // RotationScaleMerge::search_min_zeta). Where the two disagree the all-observation merge @@ -4160,6 +4164,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b rsm->SetSearchMinZeta(0.0); auto sm_all = scale_and_merge("P1, all observations", true); rsm->SetSearchMinZeta(zeta); + merged_filtered_isa = sg_opts.merge_isa; // the filtered arm's, before it is replaced sg_opts.merge_isa = result.error_model_isa; // this arm's own error model const auto alt = SearchSpaceGroup(sm_all.merged, sg_opts); // The order of the point group each arm confirmed. Read from the search, not from the @@ -4430,16 +4435,31 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // Where it finds the larger point group, the absences are still judged on all // observations, with the point group pinned. if (!merged_filtered.empty()) { - const auto filtered = SearchSpaceGroup(reindexed(merged_filtered), o2); + SearchSpaceGroupOptions of = o2; + of.merge_isa = merged_filtered_isa; // this arm's own error model + const auto filtered = SearchSpaceGroup(reindexed(merged_filtered), of); if (filtered.point_group_order > s2.point_group_order) { o2.fixed_point_group = filtered.point_group_representative; s2 = SearchSpaceGroup(merged_all, o2); o2.fixed_point_group.reset(); } } - const bool higher = s2.best_space_group.has_value() - && s2.point_group_order > sg_search.point_group_order - && s2.best_space_group->centring_type() == cand->centering; + // The centring cannot be read from the absences in this setting: reindexing + // into the metric candidate's conventional cell leaves its centring-absent class + // holding no data at all, so among the point-group-equivalent candidates the + // intensities cannot separate, the tie-break takes the lowest number, and that is + // the primitive one. Pick the candidate matching the metric's own centring out of + // the alternatives first, exactly as the centred-lattice test above does - + // otherwise a promotion the intensities DID confirm is refused for naming the + // same lattice in its equivalent primitive setting. + std::optional chosen = s2.best_space_group; + if (chosen.has_value() && chosen->centring_type() != cand->centering) + for (const auto &alt : s2.alternatives) + if (alt.centring_type() == cand->centering) { chosen = alt; break; } + const bool point_group_higher = s2.best_space_group.has_value() + && s2.point_group_order > sg_search.point_group_order; + const bool higher = point_group_higher + && chosen->centring_type() == cand->centering; const auto &uc = cand->conventional.GetUnitCell(); logger.Info("The cell metric carries {} rotations where the {} lattice the " "indexer named has {}, so the search was asked again on the metric's " @@ -4454,12 +4474,25 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b s2.best_space_group.has_value() ? s2.best_space_group->xhm() : "nothing", s2.point_group_order, sg_search.point_group_order, higher ? " - adopted" - : (s2.refused_reason.empty() - ? " - the intensities do not support it, keeping the group " - "already found" - : " - keeping the group already found; " + s2.refused_reason)); + : (point_group_higher + ? " - keeping the group already found; the point group IS " + "higher, but no candidate of it carries the centring the " + "metric cell names, which this setting cannot test" + : (s2.refused_reason.empty() + ? " - the intensities do not support it, keeping the " + "group already found" + : " - keeping the group already found; " + s2.refused_reason))); if (higher) { sg_search = s2; + // Leave the candidate it displaced among the alternatives, so the reported + // "or ..." list stays the set of groups the data cannot separate. + if (chosen->number != s2.best_space_group->number) { + sg_search.alternatives.push_back(*s2.best_space_group); + std::erase_if(sg_search.alternatives, [&](const gemmi::SpaceGroup &a) { + return a.number == chosen->number; + }); + } + sg_search.best_space_group = chosen; commit_reindex = reindex; commit_cell = cand->conventional.GetUnitCell(); commit_lattice = end_msg.rotation_lattice->Multiply(reindex); -- 2.54.0 From efa8ab1221d67c67b621891336450386a1d5a4d3 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 11:07:01 +0200 Subject: [PATCH 29/75] integration: each reflection is corrected for the sensor's efficiency at the angle it arrives A photon entering a flat sensor at an angle alpha to its normal crosses t/cos(alpha) of material instead of t, so the absorbed fraction rises toward the detector edge. The correction is QE(0)/QE(alpha) taken on the diffracted-beam direction against the detector normal, not on the scattering angle, so it follows a tilted or swung-out detector rather than assuming the two coincide. On an untilted detector this is a function of |s| alone: it is 99.7% a Wilson B offset and cancels exactly within a resolution shell, so merged protein data barely moves and no gain is claimed. It stops cancelling the moment the detector is tilted, because the incidence angle then acquires an azimuthal dependence: on a 30 degree swung-out geometry at 18 keV the within-shell spread reaches 21% median and 31% peak, and the anisotropy tensor moves with it. Attenuation lengths are the tabulated NIST coefficients rather than a wavelength-cubed approximation, which is within 0.2% for silicon above 10 keV but wrong for CdTe by a factor of two, and by six above the cadmium K edge. Photoelectric branching cancels in the ratio; K-fluorescence escape is not modelled, and the header says so. The correction self-disables where the physics makes it meaningless - an opaque sensor - so it needs no flag and is exactly neutral on all long-wavelength data and on thick CdTe. That also makes it a no-op on a file that stores its sensor thickness in the wrong unit, of which the corpus holds one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/ACKNOWLEDGEMENT.md | 9 ++ docs/CHANGELOG.md | 1 + image_analysis/SensorAbsorption.h | 143 ++++++++++++++++++ .../bragg_prediction/BraggPrediction.cpp | 17 ++- .../bragg_prediction/BraggPredictionRot.cpp | 22 ++- .../bragg_prediction/BraggPredictionRotGPU.cu | 24 ++- .../bragg_prediction/BraggPredictionRotGPU.h | 5 + reader/HDF5MetadataSource.cpp | 14 ++ 8 files changed, 229 insertions(+), 6 deletions(-) create mode 100644 image_analysis/SensorAbsorption.h diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index f50c63d6c..ceef72480 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -281,3 +281,12 @@ Report of the IUCr Subcommittee on Statistical Descriptors" (1989), Acta Cryst. S. C. Abrahams, H. D. Flack, E. Prince and A. J. C. Wilson, "Statistical descriptors in crystallography. II. Report of a Working Group on Expression of Uncertainty in Measurement" (1995), Acta Cryst. A51, 565-569 [doi:10.1107/S0108767395002340](https://doi.org/10.1107/S0108767395002340). + +**Sensor absorption at oblique incidence** — the angle-dependent quantum efficiency of a flat +sensor, and the radial parallax variance that comes from the same integral, are the Beer-Lambert law +taken along a ray that crosses t/cos(alpha) of sensor and converts at a random depth. The +attenuation coefficients are the NIST tabulation: J. H. Hubbell and S. M. Seltzer, "Tables of X-Ray +Mass Attenuation Coefficients and Mass Energy-Absorption Coefficients from 1 keV to 20 MeV for +Elements Z = 1 to 92 and 48 Additional Substances of Dosimetric Interest" (1995, data updated 2004), +NIST Standard Reference Database 126 +[doi:10.18434/T4D01F](https://doi.org/10.18434/T4D01F). diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index d4665f8ed..cdfdf9501 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -3,6 +3,7 @@ ### 1.0.0-rc.166 +* `rugnux` corrects each reflection for the sensor's quantum efficiency at the angle the diffracted beam meets the detector. * `rugnux --model` treats the model as a hypothesis: it decides the enantiomorph and the indexing only where its R-work beats that of the same model in random orientations, and a model the data reject is still scored, placed and mapped, but leaves the reflection files byte for byte what a run with no model writes. * `rugnux --model` places the model against the data as a rigid body before scoring it, writes sigma_A-weighted 2mFo-DFc and mFo-DFc maps in place of the unweighted 2Fo-Fc and Fo-Fc, and writes the model as it was placed - `_model.cif`, and `_model.pdb` where the PDB format can express the cell - in the cell and space group of the reflection files beside it. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, and open masters written at other facilities, including Eiger 1.x and third-party NXmx. diff --git a/image_analysis/SensorAbsorption.h b/image_analysis/SensorAbsorption.h new file mode 100644 index 000000000..6b72f6c51 --- /dev/null +++ b/image_analysis/SensorAbsorption.h @@ -0,0 +1,143 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#ifndef JUNGFRAUJOCH_SENSORABSORPTION_H +#define JUNGFRAUJOCH_SENSORABSORPTION_H + +#include +#include + +// How much of the beam a flat sensor stops, as a function of the angle at which the beam enters it. +// +// A photon arriving at incidence angle alpha to the sensor normal crosses t/cos(alpha) of sensor +// instead of t, so more of it is absorbed: the detector is MORE efficient at high angle than at +// normal incidence, and an uncorrected reflection out at the detector edge reads high. Dividing by +// that ratio is the whole correction. +// +// a(alpha) = t / (L cos alpha) crossing length in attenuation lengths +// QE(alpha) = 1 - exp(-a) absorbed fraction +// correction to I = QE(0) / QE(alpha) normalised so normal incidence is untouched +// +// Two limits, both physical and both reached in this corpus: +// - L << t (long wavelength): every photon stops in the entrance skin, QE = 1 at every angle and +// the correction is exactly 1. This is why the low-energy case is inert rather than gated off. +// - L >> t (thin sensor, hard X-rays): absorption is proportional to path length and the +// correction tends to cos(alpha) - a factor of 2 at 2theta = 60 degrees. This is the regime +// small-molecule work at 18-25 keV lives in. +// +// alpha is the angle to the DETECTOR NORMAL, not the scattering angle. The two coincide only on an +// untilted detector. On a tilted one the correction acquires an azimuthal dependence at fixed +// resolution, which is the only part of it that is not degenerate with an overall Wilson B. +// +// Attenuation coefficients: NIST X-Ray Mass Attenuation Coefficients (Hubbell & Seltzer), total +// mu/rho with coherent scattering. Photoelectric absorption dominates over the whole range used +// here, and because the correction is a RATIO of two absorbed fractions the photoelectric branching +// ratio cancels exactly; only the attenuation length enters. Coherent scattering, which attenuates +// without converting, is a few per cent of mu/rho in Si below 20 keV and is kept in the total - it +// biases L slightly short, and the correction depends on L only through t/L. +// +// NOT modelled, and negligible where this is used: K-fluorescence escape from the sensor (in CdTe +// above the Cd K edge at 26.71 keV a fluorescence photon can leave the sensor, so the charge is +// recorded in the wrong pixel or not at all); charge sharing between pixels; and the obliquity of +// the entrance window. Above the Cd/Te K edges the escape term is large and this model should not +// be trusted as it stands. +namespace sensor_absorption { + +// NIST mass attenuation coefficient mu/rho [cm^2/g] against photon energy [keV]. Absorption edges +// are the repeated energies - the table is walked in order and interpolated log-log, which is how +// these tables are meant to be read. Si has only the K edge at 1.839 keV, far below anything used. +struct MuRow { double E_keV, mu_rho; }; + +inline constexpr MuRow SI_TABLE[] = { + {1.0, 1570.0}, {1.5, 535.5}, {1.8389, 309.2}, {1.8389, 3192.0}, {2.0, 2777.0}, + {3.0, 978.4}, {4.0, 452.9}, {5.0, 245.0}, {6.0, 147.0}, {8.0, 64.68}, {10.0, 33.89}, + {15.0, 10.34}, {20.0, 4.464}, {30.0, 1.436}, {40.0, 0.7012}, {50.0, 0.4385}}; + +inline constexpr MuRow CD_TABLE[] = { + {1.0, 7350.0}, {1.5, 2931.0}, {2.0, 1473.0}, {3.0, 541.4}, {3.5375, 357.5}, + {3.5375, 1152.0}, {3.63101, 1083.0}, {3.727, 1013.0}, {3.727, 1389.0}, {4.0, 1170.0}, + {4.018, 1157.0}, {4.018, 1324.0}, {5.0, 768.5}, {6.0, 479.3}, {8.0, 225.4}, {10.0, 124.4}, + {15.0, 41.78}, {20.0, 19.20}, {26.7112, 8.809}, {26.7112, 50.65}, {30.0, 37.65}, + {40.0, 17.78}, {50.0, 9.779}}; + +inline constexpr MuRow TE_TABLE[] = { + {1.0, 8434.0}, {1.5, 3608.0}, {2.0, 1832.0}, {3.0, 679.2}, {4.0, 329.7}, {4.3414, 267.8}, + {4.3414, 788.2}, {4.47465, 750.4}, {4.612, 699.5}, {4.612, 944.5}, {4.7728, 878.2}, + {4.9392, 806.2}, {4.9392, 929.2}, {5.0, 901.4}, {6.0, 572.1}, {8.0, 270.2}, {10.0, 150.1}, + {15.0, 50.78}, {20.0, 23.41}, {30.0, 7.878}, {31.8138, 6.738}, {31.8138, 37.19}, + {40.0, 20.64}, {50.0, 11.45}}; + +// Log-log interpolation, clamped at both ends of the table. +template +double InterpMuRho(const MuRow (&tab)[N], double E_keV) { + if (E_keV <= tab[0].E_keV) + return tab[0].mu_rho; + if (E_keV >= tab[N - 1].E_keV) + return tab[N - 1].mu_rho; + std::size_t i = 1; + while (i < N - 1 && tab[i].E_keV < E_keV) + i++; + const double e0 = tab[i - 1].E_keV, e1 = tab[i].E_keV; + if (!(e1 > e0)) // the two rows of an absorption edge: take the upper side + return tab[i].mu_rho; + const double f = (std::log(E_keV) - std::log(e0)) / (std::log(e1) - std::log(e0)); + return std::exp(std::log(tab[i - 1].mu_rho) + f * (std::log(tab[i].mu_rho) - std::log(tab[i - 1].mu_rho))); +} + +// Attenuation length 1/mu [um] of a sensor material at a given wavelength. Si and CdTe are the +// sensors in use; an unrecognised material is treated as silicon, which is what DetectorSetup +// defaults to anyway. +inline double AttenuationLength_um(const std::string &material, double lambda_A) { + if (!(lambda_A > 0.0)) + return 0.0; + const double E_keV = 12.39842 / lambda_A; + double mu_rho_cm2_g, rho_g_cm3; + if (material == "CdTe") { + // Mass fractions from the atomic weights, Cd 112.414 and Te 127.60. + constexpr double w_cd = 112.414 / 240.014, w_te = 127.60 / 240.014; + mu_rho_cm2_g = w_cd * InterpMuRho(CD_TABLE, E_keV) + w_te * InterpMuRho(TE_TABLE, E_keV); + rho_g_cm3 = 5.85; + } else { + mu_rho_cm2_g = InterpMuRho(SI_TABLE, E_keV); + rho_g_cm3 = 2.3290; + } + const double mu_cm = mu_rho_cm2_g * rho_g_cm3; + return mu_cm > 0.0 ? 1e4 / mu_cm : 0.0; +} + +// Everything the per-reflection correction needs, reduced to the two numbers that are constant for +// a dataset. Built once on the host; the GPU predictor takes the two floats. +struct SensorQE { + float a0 = 0.0f; // t / L, the optical thickness at normal incidence + float qe0 = 1.0f; // absorbed fraction at normal incidence + bool active = false; // false where the sensor is opaque and the correction is exactly 1 + + // Opaque beyond this: exp(-20) = 2e-9, so QE(0)/QE(alpha) rounds to exactly 1.0f for every + // incidence angle and the correction is bit-identical to not applying it. This is what makes + // the long-wavelength case inert without a flag or a threshold anyone has to choose. + static constexpr float OPAQUE_A0 = 20.0f; + + static SensorQE Build(const std::string &material, double thickness_um, double lambda_A) { + SensorQE q; + const double L = AttenuationLength_um(material, lambda_A); + if (!(thickness_um > 0.0) || !(L > 0.0)) + return q; + q.a0 = static_cast(thickness_um / L); + q.qe0 = static_cast(1.0 - std::exp(-q.a0)); + q.active = q.a0 < OPAQUE_A0; + return q; + } + + // The multiplicative correction to an intensity recorded at incidence angle alpha: QE(0)/QE(alpha). + // Always <= 1, because a sensor is more efficient off-normal than head-on. + [[nodiscard]] float Factor(float cos_alpha) const { + if (!active || !(cos_alpha > 1e-3f)) + return 1.0f; + const float qe = 1.0f - std::exp(-a0 / cos_alpha); + return qe > 0.0f ? qe0 / qe : 1.0f; + } +}; + +} // namespace sensor_absorption + +#endif // JUNGFRAUJOCH_SENSORABSORPTION_H diff --git a/image_analysis/bragg_prediction/BraggPrediction.cpp b/image_analysis/bragg_prediction/BraggPrediction.cpp index de48b63a3..a43e1cc05 100644 --- a/image_analysis/bragg_prediction/BraggPrediction.cpp +++ b/image_analysis/bragg_prediction/BraggPrediction.cpp @@ -6,6 +6,7 @@ #include "../../common/JFJochMath.h" #include "../../common/Logger.h" #include "BraggPrediction.h" +#include "../SensorAbsorption.h" #include "../bragg_integration/SystematicAbsence.h" void BraggPrediction::GrowCapacity(int count) { @@ -92,6 +93,12 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal float pixel_size = geom.GetPixelSize_mm(); float F = det_distance / pixel_size; + // Angle-dependent sensor efficiency; per-dataset constants collapsed to two numbers. Inert + // wherever the sensor is opaque, which is every long wavelength. + const auto &det_setup = experiment.GetDetectorSetup(); + const auto sensor_qe = sensor_absorption::SensorQE::Build( + det_setup.GetSensorMaterial(), det_setup.GetSensorThickness_um(), geom.GetWavelength_A()); + const float epsilon = 1e-5f; const float s0_sq = S0 * S0; const float rad_to_deg = 180.0f / static_cast(PI); @@ -185,6 +192,12 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal if ((x < 0) || (x >= det_width_pxl) || (y < 0) || (y >= det_height_pxl)) continue; + // Sensor quantum efficiency at this reflection's angle of incidence on the + // detector - the diffracted direction against the detector normal, so the + // detector tilt is carried for free. See sensor_absorption::SensorQE. + const float qe_corr = sensor_qe.Factor( + S_rot_z / std::sqrt(S_x * S_x + S_y * S_y + S_z * S_z)); + float d = 1.0f / sqrtf(recip_sq); reflections[i] = Reflection{ .h = h, @@ -197,10 +210,10 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal .observed_y = NAN, .d = d, .dist_ewald = dist_ewald_sphere, - .rlp = 1.0, + .rlp = qe_corr, .partiality = 1.0f, .zeta = 1.0, - .image_scale_corr = 1.0 + .image_scale_corr = qe_corr }; ++i; } diff --git a/image_analysis/bragg_prediction/BraggPredictionRot.cpp b/image_analysis/bragg_prediction/BraggPredictionRot.cpp index 59fb41c81..b660ac1fa 100644 --- a/image_analysis/bragg_prediction/BraggPredictionRot.cpp +++ b/image_analysis/bragg_prediction/BraggPredictionRot.cpp @@ -3,6 +3,7 @@ #include "../../common/JFJochMath.h" #include "BraggPredictionRot.h" +#include "../SensorAbsorption.h" #include "../bragg_integration/SystematicAbsence.h" @@ -60,6 +61,14 @@ int BraggPredictionRot::Calc(const DiffractionExperiment &experiment, const Crys const float bandwidth_sigma = settings.bandwidth_sigma; const float half_wavelength_A = geom.GetWavelength_A() / 2.0f; + // Angle-dependent sensor efficiency. Per-dataset constants (thickness, material, wavelength) + // collapse to two numbers here; the per-reflection part is one exponential below. Inert - and + // bit-identical to not applying it - wherever the sensor is opaque, which is every long + // wavelength, so it needs no flag and no threshold anyone has to choose. + const auto &det = experiment.GetDetectorSetup(); + const auto sensor_qe = sensor_absorption::SensorQE::Build( + det.GetSensorMaterial(), det.GetSensorThickness_um(), geom.GetWavelength_A()); + for (int h = -settings.max_h; h <= settings.max_h; h++) { // Precompute A* h contribution @@ -165,6 +174,15 @@ int BraggPredictionRot::Calc(const DiffractionExperiment &experiment, const Crys float dist_ewald_sphere = std::fabs(S.Length() - one_over_wavelength); + // Sensor quantum efficiency at this reflection's own angle of incidence on the + // detector. The angle is taken against the DETECTOR NORMAL - S_rot is the + // diffracted direction in the detector's own frame, so its z component over its + // length is that cosine already, at no cost. Taking it here rather than from the + // resolution is what makes it right on a tilted detector, where the incidence + // angle stops being a function of resolution and the correction stops + // cancelling within a resolution shell. + const float qe_corr = sensor_qe.Factor(S_rot_z / S.Length()); + float d = 1.0f / sqrtf(p0_sq); reflections[i] = Reflection{ .h = h, @@ -177,10 +195,10 @@ int BraggPredictionRot::Calc(const DiffractionExperiment &experiment, const Crys .observed_y = NAN, .d = d, .dist_ewald = dist_ewald_sphere, - .rlp = lorentz_reciprocal, + .rlp = lorentz_reciprocal * qe_corr, .partiality = partiality, .zeta = zeta_abs, - .image_scale_corr = lorentz_reciprocal / partiality, + .image_scale_corr = lorentz_reciprocal * qe_corr / partiality, }; i++; } diff --git a/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu b/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu index 63126406b..b114fee5b 100644 --- a/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu +++ b/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu @@ -4,6 +4,7 @@ #include #include "../../common/JFJochMath.h" #include "BraggPredictionRotGPU.h" +#include "../SensorAbsorption.h" #ifdef JFJOCH_USE_CUDA #include "../indexing/CUDAMemHelpers.h" @@ -166,6 +167,19 @@ namespace { float dist_ewald = fabsf(sqrtf(Sx * Sx + Sy * Sy + Sz * Sz) - C.one_over_wavelength); + // Sensor quantum efficiency at this reflection's angle of incidence on the detector. + // Sr is the diffracted direction in the detector's own frame, so Srz over its length is + // the cosine to the detector NORMAL - which carries detector tilt for free. Mirrors + // sensor_absorption::SensorQE::Factor on the CPU side. + float qe_corr = 1.0f; + if (C.qe_a0 > 0.0f) { + float cos_alpha = Srz / sqrtf(Sx * Sx + Sy * Sy + Sz * Sz); + if (cos_alpha > 1e-3f) { + float qe = 1.0f - expf(-C.qe_a0 / cos_alpha); + if (qe > 0.0f) qe_corr = C.qe_qe0 / qe; + } + } + out[count].h = h; out[count].k = k; out[count].l = l; @@ -176,10 +190,10 @@ namespace { out[count].observed_y = NAN; out[count].d = 1.0f / sqrtf(p0_sq); out[count].dist_ewald = dist_ewald; - out[count].rlp = lorentz; + out[count].rlp = lorentz * qe_corr; out[count].partiality = partiality; out[count].zeta = zeta_abs; - out[count].image_scale_corr = lorentz / partiality; + out[count].image_scale_corr = lorentz * qe_corr / partiality; count++; } return count; @@ -234,6 +248,12 @@ namespace { kc.bandwidth_sigma = settings.bandwidth_sigma; kc.half_wavelength_A = geom.GetWavelength_A() / 2.0f; + const auto &det = experiment.GetDetectorSetup(); + const auto sensor_qe = sensor_absorption::SensorQE::Build( + det.GetSensorMaterial(), det.GetSensorThickness_um(), geom.GetWavelength_A()); + kc.qe_a0 = sensor_qe.active ? sensor_qe.a0 : 0.0f; + kc.qe_qe0 = sensor_qe.qe0; + kc.Astar = lattice.Astar(); kc.Bstar = lattice.Bstar(); kc.Cstar = lattice.Cstar(); diff --git a/image_analysis/bragg_prediction/BraggPredictionRotGPU.h b/image_analysis/bragg_prediction/BraggPredictionRotGPU.h index f58ec0abb..7e06a1e58 100644 --- a/image_analysis/bragg_prediction/BraggPredictionRotGPU.h +++ b/image_analysis/bragg_prediction/BraggPredictionRotGPU.h @@ -22,6 +22,11 @@ struct KernelConstsRot { float mosaicity_multiplier; float bandwidth_sigma; // relative dlambda/lambda as a sigma; 0 = monochromatic float half_wavelength_A; + // Angle-dependent sensor efficiency, reduced to the two per-dataset numbers the kernel needs: + // the optical thickness at normal incidence and the absorbed fraction there. qe_a0 = 0 leaves + // every reflection untouched (see sensor_absorption::SensorQE). + float qe_a0; + float qe_qe0; Coord Astar, Bstar, Cstar, S0; Coord m1, m2, m3; float m2_S0; diff --git a/reader/HDF5MetadataSource.cpp b/reader/HDF5MetadataSource.cpp index fa86cf8d6..ef24274ec 100644 --- a/reader/HDF5MetadataSource.cpp +++ b/reader/HDF5MetadataSource.cpp @@ -1006,11 +1006,25 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen .value_or(0)))); // Sensor thickness/material drive the parallax/absorption model, so take them from the file // rather than the DetectorSetup default. + // Say so when they are absent rather than let the DetectorSetup default stand in silently. + // The default (320 um Si) is right for every JUNGFRAU and wrong for the 450 um Si and 750 um + // CdTe sensors this code also reads, and both the angle-dependent efficiency correction and + // the parallax variance term are computed from these two numbers - so a file that omits them + // gets a correction derived from an assumption, which the user should be told about. The + // correction is still applied: a 320 um Si assumption is closer to every real sensor than + // switching the physics off, and declining would silently disagree with the same data read + // from a file that does state its sensor. if (master_file->Exists("/entry/instrument/detector/sensor_thickness")) detector.SensorThickness_um( ReadLength_m(*master_file, "/entry/instrument/detector/sensor_thickness") * 1e6); + else + Logger("HDF5Reader").Warning("No sensor_thickness in the file; assuming {:.0f} um for the " + "sensor absorption model", detector.GetSensorThickness_um()); if (master_file->Exists("/entry/instrument/detector/sensor_material")) detector.SensorMaterial(master_file->GetString("/entry/instrument/detector/sensor_material")); + else + Logger("HDF5Reader").Warning("No sensor_material in the file; assuming {} for the sensor " + "absorption model", detector.GetSensorMaterial()); // Optional, because a file that states no saturation value anywhere is a real and common // thing: an Eiger master links saturation_value into a companion _meta.h5, and a deposited // dataset frequently does not include that file, leaving neither the NXmx name nor the -- 2.54.0 From 9b96906150192f3f99585db331ef8574097f99fa Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 11:07:17 +0200 Subject: [PATCH 30/75] integration: the parallax variance takes its attenuation length from the tables, not from lambda^3 The radial parallax broadening is the variance of the depth at which a photon converts, so it scales with the attenuation length. That length was approximated as photoelectric-dominated and scaled by lambda^3 from a single 13 keV reference per material. For silicon above 10 keV that is within 0.2%, but for CdTe it overstates the attenuation length by up to a factor of two, and by six above the cadmium K edge - which made this variance term 1.9x too large on 750 um CdTe data. Take the length from the same tabulated coefficients the efficiency correction uses. Silicon data is unaffected to within the approximation's own error; CdTe data gets a spot-width variance that matches the sensor it was recorded on. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 1 + .../bragg_integration/BraggIntegrationEngine.cpp | 13 ++++++++----- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index cdfdf9501..49176581a 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -4,6 +4,7 @@ ### 1.0.0-rc.166 * `rugnux` corrects each reflection for the sensor's quantum efficiency at the angle the diffracted beam meets the detector. +* `rugnux` takes the sensor attenuation length from tabulated coefficients rather than a wavelength-cubed approximation, which corrects the parallax term of the spot-width variance on CdTe sensors. * `rugnux --model` treats the model as a hypothesis: it decides the enantiomorph and the indexing only where its R-work beats that of the same model in random orientations, and a model the data reject is still scored, placed and mapped, but leaves the reflection files byte for byte what a run with no model writes. * `rugnux --model` places the model against the data as a rigid body before scoring it, writes sigma_A-weighted 2mFo-DFc and mFo-DFc maps in place of the unweighted 2Fo-Fc and Fo-Fc, and writes the model as it was placed - `_model.cif`, and `_model.pdb` where the PDB format can express the cell - in the cell and space group of the reflection files beside it. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, and open masters written at other facilities, including Eiger 1.x and third-party NXmx. diff --git a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp index c45df8fcf..9120dacb2 100644 --- a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp +++ b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp @@ -12,20 +12,23 @@ #include #include "../../common/JFJochMath.h" // PI (M_PI is not standard, and MSVC does not define it) +#include "../SensorAbsorption.h" namespace { // Radial parallax broadening as the coefficient of tan^2(2theta), i.e. Var(z)/pixel^2 [px^2]. // Copied verbatim from ProfileIntegrate2D: a photon converts at a random depth z (exponential, // attenuation length L, truncated at the sensor thickness), shifting the recorded spot radially by -// z*tan(2theta). L is photoelectric-dominated (~lambda^3), so a per-material reference (13 keV) is -// scaled by lambda^3; Si and CdTe are the sensors in use. +// z*tan(2theta). L comes from the tabulated NIST attenuation coefficients (SensorAbsorption.h); the +// lambda^3 approximation this used before is within 0.2% for silicon above 10 keV but overstates +// the attenuation length of CdTe by up to a factor of two, and by six above the Cd K edge, which +// made this variance 1.9x too large on 750 um CdTe data. double parallax_var_px2(const std::string &material, double thickness_um, double lambda_A, double pixel_um) { if (!(thickness_um > 0.0) || !(pixel_um > 0.0) || !(lambda_A > 0.0)) return 0.0; - const double L_ref = material == "CdTe" ? 42.6 : 273.0; // attenuation length [um] at 0.953 A - const double s = lambda_A / 0.953; - const double L = L_ref / (s * s * s); + const double L = sensor_absorption::AttenuationLength_um(material, lambda_A); + if (!(L > 0.0)) + return 0.0; const double a = thickness_um / L, e = std::exp(-a); if (1.0 - e <= 0.0) return 0.0; -- 2.54.0 From 0f9d6a1bdeca2e242adf77a779ef16135010cd86 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 11:25:33 +0200 Subject: [PATCH 31/75] symmetry: --finalist-ledger reports the evidence for every group the search considered The search folds orbits and refits the error model for every point group it confirms, ranks them, adopts one and discards the rest. The discarded numbers are the only per-run measurement of how much worse the rivals actually are, and until now the only way to see them was to re-run with -S and compare by hand. Behind --finalist-ledger the run prints them: for the adopted group, the refused higher point group's best, the parent's best and any re-ask candidate that scored and lost, an evidence row folded from the P1 cross-check merge. That merge is the substrate the audit kept asking for and nothing else in the run has - the same observations at full resolution, production-scaled, folded under no symmetry, so every hypothesis is folded from it on equal terms. It also carries half-set intensities, which is what lets the noise floor be formed for the first step out of P1, and the reason this cannot be done from the stored reflection file afterwards. Report-only, and deliberately so: the accept side has 0.10 of headroom between the largest genuine ratio measured on the corpus and the bound the gate already uses, so reading these numbers as a decision before they are calibrated is the one way this can do harm. The merged output is byte for byte what a run without the flag writes. Two of the six channels are printed as evidence but marked as measured-contaminated: a ratio taken against the parent is meaningless when the parent is itself a false promotion, and on this corpus both false cases sit strictly inside the genuine range for them. A screw over-call is invisible here by construction - the folds are identical - and the table says so rather than reading clean. Every path that cannot produce a ledger now declines out loud instead of printing nothing: stills, --no-p1-crosscheck, a fixed centred group whose absences were never predicted, --no-merge, and --mode scale. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- .../scale_merge/SearchSpaceGroup.cpp | 143 +++++++++++++++++- image_analysis/scale_merge/SearchSpaceGroup.h | 46 ++++++ rugnux/CMakeLists.txt | 5 + rugnux/Rugnux.cpp | 40 +++++ rugnux/Rugnux.h | 1 + rugnux/rugnux_cli.cpp | 24 +++ rugnux/rugnux_ledger.cpp | 107 +++++++++++++ 7 files changed, 365 insertions(+), 1 deletion(-) create mode 100644 rugnux/rugnux_ledger.cpp diff --git a/image_analysis/scale_merge/SearchSpaceGroup.cpp b/image_analysis/scale_merge/SearchSpaceGroup.cpp index de1790b91..c546177f1 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.cpp +++ b/image_analysis/scale_merge/SearchSpaceGroup.cpp @@ -778,7 +778,14 @@ SearchSpaceGroupResult SearchSpaceGroup( // max_operator_r_over_best and max_operator_r_over_floor). Carried for the same // reason as h_ratio: to report the adopted candidate's whether or not it decided. double r_added = std::numeric_limits::quiet_NaN(); - double r_over_floor = std::numeric_limits::quiet_NaN(); }; + double r_over_floor = std::numeric_limits::quiet_NaN(); + // Report-only, for the finalist ledger: the subgroup b the tests below compared + // this candidate against, whether it survived every consistency test, and - when + // it was the highest refusal - why it did not. The selection loop computes all + // three already and drops them on the floor; the ledger is that table kept. + double parent_b_used = -1.0; + bool eligible = false; + std::string why; }; int refused_order = 0; std::string refused_pg_hm, refused_why; std::vector pg_cands; @@ -948,6 +955,7 @@ SearchSpaceGroupResult SearchSpaceGroup( if (n_new > 0 && pairs_new >= opt.min_pairs_for_h) r_added = r_new / n_new; } + c.parent_b_used = parent_b; c.r_added = r_added; c.r_over_floor = (std::isfinite(r_added) && std::isfinite(result.merge_r_floor) && result.merge_r_floor > 0.0) @@ -1020,6 +1028,10 @@ SearchSpaceGroupResult SearchSpaceGroup( consistent = false; if (!consistent) { + // Same three sentences the highest refusal gets below, but kept per candidate for the + // ledger: short, because the long form is written once for the group the user is told about. + c.why = h_refused ? "H ratio" : r_refused ? "added-operator R" : b_refused ? "systematic b" + : "merge chi^2"; // Record the highest-order refusal so the caller can say WHY it is processing lower. if (c.order > refused_order && c.pg->representative) { refused_order = c.order; @@ -1061,6 +1073,7 @@ SearchSpaceGroupResult SearchSpaceGroup( } continue; } + c.eligible = true; if (c.order > best_pg_order || (c.order == best_pg_order && c.min_class_cc > best_pg_min_cc)) { best_pg = c.pg; best_pg_order = c.order; @@ -1109,6 +1122,38 @@ SearchSpaceGroupResult SearchSpaceGroup( result.r_over_best = c.r_added / global_best_r; } result.h_ratio_bound = opt.max_operator_h_ratio; + result.r_over_best_bound = opt.max_operator_r_over_best; + result.r_over_floor_bound = opt.max_operator_r_over_floor; + + // The FINALIST LEDGER: every operator-confirmed hypothesis with its evidence vector, adopted and + // refused alike. Report-only - nothing below reads it back, and the choice above is already final. + // The numbers are the ones the selection loop formed anyway; what is new is that the losers keep + // theirs instead of being reduced to a single "refused" line. + for (const auto& c : pg_cands) { + PointGroupLedgerEntry e; + e.point_group_hm = c.pg->representative ? c.pg->representative->point_group_hm() : "1"; + e.order = c.order; + e.min_class_cc = c.min_class_cc; + e.chi2 = c.chi2; + if (std::isfinite(c.chi2) && std::isfinite(chi2_ref) && chi2_ref > 0.0) + e.chi2_over_best = c.chi2 / chi2_ref; + e.b_extra = c.b_extra; + if (c.parent_b_used > 1e-4) + e.b_over_parent = c.b_extra / c.parent_b_used; + e.h_ratio = c.h_ratio; + e.r_added = c.r_added; + if (std::isfinite(c.r_added) && std::isfinite(global_best_r) && global_best_r > 0.0) + e.r_over_best = c.r_added / global_best_r; + e.r_over_floor = c.r_over_floor; + e.adopted = (c.pg == best_pg); + e.eligible = c.eligible; + e.refused_reason = c.why; + result.point_group_ledger.push_back(e); + } + std::sort(result.point_group_ledger.begin(), result.point_group_ledger.end(), + [](const PointGroupLedgerEntry& a, const PointGroupLedgerEntry& b) { + return a.order > b.order; + }); // Only report a refusal that is actually ABOVE what was adopted. if (refused_order > best_pg_order) { @@ -1456,6 +1501,102 @@ SearchSpaceGroupResult SearchSpaceGroup( return result; } +static std::string LedgerCell(double v, int prec) { + return std::isfinite(v) ? FormatDouble(v, prec) : std::string("-"); +} + +std::string FinalistLedgerToText(const SearchSpaceGroupResult& result) { + std::ostringstream os; + os << "Finalist ledger - every point group whose operators the intensities confirmed on this " + "merge,\n adopted and refused alike. REPORT-ONLY: the group this run processed in was " + "chosen\n without any of it, and nothing below reads it back.\n"; + + // A search that confirmed no operator has no finalists, and an empty table under a header reads as + // "nothing was wrong" when it means "nothing was asked". Say which it is, and stop. + int confirmed_above_p1 = 0; + for (const auto& e : result.point_group_ledger) + if (e.order > 1) + ++confirmed_above_p1; + if (result.point_group_ledger.empty()) { + os << " No finalists: the search confirmed no point-group operator on this merge, so there is " + "no\n hypothesis to tabulate. This is the expected table for a crystal that is genuinely " + "P1,\n and for one whose operators the correlation stage never confirmed.\n"; + return os.str(); + } + if (confirmed_above_p1 == 0) { + os << " No finalists above P1: the only row is the trivial group, so the ledger has nothing to " + "weigh\n against anything. The table follows for completeness.\n"; + } + + if (std::isfinite(result.merge_r_floor)) { + os << " Reference R floor (random noise, half-set) " << FormatDouble(result.merge_r_floor, 4) + << "; best-agreeing operator R " << LedgerCell(result.global_best_operator_r, 4); + // On the FIRST step out of P1 there is no second operator to be the reference, so this ratio - + // not any row below - carries the whole of the evidence. Spell it out rather than leaving the + // reader to divide two numbers, because that is the case where the table itself is empty. + if (std::isfinite(result.global_best_operator_r) && result.merge_r_floor > 0.0) + os << " (" << FormatDouble(result.global_best_operator_r / result.merge_r_floor, 2) + << "x the floor, bound " << FormatDouble(result.r_over_floor_bound, 2) << ")"; + os << ".\n"; + } + else + os << " Best-agreeing operator R " << LedgerCell(result.global_best_operator_r, 4) + << " (no half-set intensities in this merge, so no random-noise floor;\n every R/floor " + "reads '-' and the first step out of P1 is not measurable here).\n"; + + os << "\n PG ord minCC R_add R/best R/floor | H chi2/best b b/par verdict\n" + " <-- calibrated evidence --> | <--- diagnostic only, NOT evidence --->\n"; + for (const auto& e : result.point_group_ledger) { + os << " " << std::left << std::setw(8) << e.point_group_hm << std::right << std::setw(3) + << e.order + << std::setw(8) << LedgerCell(e.min_class_cc, 3) + << std::setw(8) << LedgerCell(e.r_added, 4) + << std::setw(7) << LedgerCell(e.r_over_best, 2) + << std::setw(8) << LedgerCell(e.r_over_floor, 2) + << " |" << std::setw(6) << LedgerCell(e.h_ratio, 2) + << std::setw(10) << LedgerCell(e.chi2_over_best, 2) + << std::setw(9) << LedgerCell(e.b_extra, 4) + << std::setw(7) << LedgerCell(e.b_over_parent, 2) + << " " << (e.adopted ? "ADOPTED" : e.eligible ? "eligible" : "refused") + << (e.refused_reason.empty() ? "" : " (" + e.refused_reason + ")"); + // The margin, on the row that decided. The value alone does not say how close the run came to + // being refused, and on this corpus that distance is small enough to matter (below). + if (e.adopted && std::isfinite(e.r_over_best) && result.r_over_best_bound > 0.0) + os << " [" << FormatDouble(result.r_over_best_bound - e.r_over_best, 2) + << " below the " << FormatDouble(result.r_over_best_bound, 2) << " bound]"; + else if (e.adopted && std::isfinite(e.r_over_floor) && result.r_over_floor_bound > 0.0) + os << " [" << FormatDouble(result.r_over_floor_bound - e.r_over_floor, 2) + << " below the " << FormatDouble(result.r_over_floor_bound, 2) << " floor bound]"; + os << "\n"; + } + + // How to read it. Every sentence here is something the numbers above have been measured to NOT + // support, and each was a wrong reading somebody actually made. + os << "\n READ IT AS AN ADMISSION TEST, NOT A RANKING: R/best rises with point-group order by\n" + " construction - a larger group adds more operators - so the smallest non-trivial subgroup\n" + " very often has the lowest R/best in the table. Sorting these rows and taking the minimum\n" + " would demote nearly every genuinely high-symmetry crystal. The question the table answers\n" + " is 'the largest order whose R/best stays near 1', not 'which row is smallest'.\n" + "\n ONLY R/best AND R/floor ARE EVIDENCE. They are ratios to a reference a false hypothesis\n" + " cannot move: the best-agreeing operator anywhere in the data, and the crystal's own\n" + " half-set noise floor. H, chi2/best, b and b/par are printed because they are free, and are\n" + " DIAGNOSTIC ONLY - on the calibration set chi2/best and b/par put their LARGEST value on a\n" + " GENUINE crystal, with both known false cases inside the genuine range. A ratio to the\n" + " PARENT is contaminated whenever the parent is itself false, which is exactly the case\n" + " these columns would have to catch. Do not read them as a ranking or a second opinion.\n" + "\n BLIND SPOTS. This table is folded from merged intensities, so it can only see a\n" + " hypothesis that DEGRADES A MERGE. It is structurally blind to the other two ways a group\n" + " is over-called: a screw-axis over-call is in the same Laue class and folds a byte-identical\n" + " merge, and a wrong LATTICE is not a hypothesis here at all - the cell is an input to this\n" + " search, not something it weighs. A crystal whose only error is one of those has a\n" + " clean-looking ledger, and a clean ledger is therefore not a clean bill of health.\n" + "\n MARGIN. On the corpus this was calibrated on, the largest R/best ever seen on a GENUINE\n" + " adopted group is 1.90, against a refusal bound of 2.00. The accept side has ~0.10 of\n" + " headroom, not the wide gap the refused rows suggest, so a margin printed above as small is\n" + " a real one - and any future tightening of this bound has almost none to spend.\n"; + return os.str(); +} + std::string SearchSpaceGroupResultToText(const SearchSpaceGroupResult& result, size_t max_candidates_to_print) { std::ostringstream os; diff --git a/image_analysis/scale_merge/SearchSpaceGroup.h b/image_analysis/scale_merge/SearchSpaceGroup.h index 5e2f29411..c574ff112 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.h +++ b/image_analysis/scale_merge/SearchSpaceGroup.h @@ -510,6 +510,36 @@ struct SearchSpaceGroupOptions { size_t nthreads = 1; }; +// One row of the FINALIST LEDGER: what the intensities say about ONE point-group hypothesis, on the +// merge this search was handed. Every candidate the operator CCs confirmed gets a row, adopted or +// refused, so a run can show what the rival was and how far behind it came - the current flow keeps +// only the winner and the single highest refusal. +// +// Every number here is PAIRED or a RATIO TO A REFERENCE THE HYPOTHESIS CANNOT MOVE, never an absolute +// per-operator statistic: r_over_best divides by the best-agreeing operator anywhere in the data, +// r_over_floor by the merge's own random-noise floor, chi2_over_best and b_over_parent by the same +// quantity under a subgroup of this very candidate. An absolute score would be the joint-likelihood +// design that was already refuted; a ratio to a clean within-crystal reference is the form thread1_A +// measured to separate genuine operators (1.06-1.15x) from false ones (2.4-4.66x). +// +// REPORT-ONLY. Nothing in the search reads these back; the adoption is made exactly as before. +struct PointGroupLedgerEntry { + std::string point_group_hm; + int order = 0; + double min_class_cc = 0.0; // weakest operator CC among the group's own operators + double chi2 = std::numeric_limits::quiet_NaN(); // merge chi^2 folding under it + double chi2_over_best = std::numeric_limits::quiet_NaN(); // ... over the most consistent subgroup's + double b_extra = 0.0; // systematic-b refit under it + double b_over_parent = std::numeric_limits::quiet_NaN(); // ... over its largest confirmed subgroup's + double h_ratio = std::numeric_limits::quiet_NaN(); + double r_added = std::numeric_limits::quiet_NaN(); // added operators' mean intensity-weighted R + double r_over_best = std::numeric_limits::quiet_NaN(); // ... over the best-agreeing operator anywhere + double r_over_floor = std::numeric_limits::quiet_NaN(); // ... over the merge's random-noise R floor + bool adopted = false; // this is the group the search took + bool eligible = false; // it passed every consistency test (so it COULD have been taken) + std::string refused_reason; // why it did not, when it was the highest refusal +}; + struct SearchSpaceGroupResult { std::optional best_space_group; // Other space groups that fit the data equally well (same systematic absences): enantiomorphic @@ -567,12 +597,28 @@ struct SearchSpaceGroupResult { // Empty when nothing was refused. Surfaced to the user - a silent demotion is how a twin gets missed. std::string refused_point_group_hm; std::string refused_reason; + + // Every operator-confirmed point-group hypothesis with its full evidence vector, ranked by order. + // Report-only (see PointGroupLedgerEntry); the adoption above is unchanged by it. Empty on a search + // that confirmed no operator at all, which the report renders as "no finalists" rather than as an + // empty table - a blank one reads as "nothing was wrong" when it means "nothing was asked". + std::vector point_group_ledger; + // The refusal bounds the ledger's own channels were judged against, copied from the options so the + // report can print the MARGIN the adopted row had rather than only its value. The accept-side + // headroom is thin (measured: the largest genuine adopted r_over_best on a 63-dataset corpus is + // 1.90 against a bound of 2.00), and a margin is the only form in which that is visible per run. + double r_over_best_bound = 0.0; + double r_over_floor_bound = 0.0; }; SearchSpaceGroupResult SearchSpaceGroup( const std::vector& merged, const SearchSpaceGroupOptions& opt = {}); +// The finalist ledger as a ranked table, highest point group first. Report-only; printed behind +// --finalist-ledger so the numbers cannot be read as a decision before they have been calibrated. +std::string FinalistLedgerToText(const SearchSpaceGroupResult& result); + std::string SearchSpaceGroupResultToText( const SearchSpaceGroupResult& result, size_t max_candidates_to_print = 20); diff --git a/rugnux/CMakeLists.txt b/rugnux/CMakeLists.txt index 8883c0df2..b437e2e67 100644 --- a/rugnux/CMakeLists.txt +++ b/rugnux/CMakeLists.txt @@ -32,6 +32,11 @@ ADD_EXECUTABLE(rugnux rugnux_cli.cpp) TARGET_LINK_LIBRARIES(rugnux Rugnux JFJochReader JFJochImageAnalysis JFJochWriter) INSTALL(TARGETS rugnux RUNTIME COMPONENT rugnux) +# Offline finalist-ledger instrument (triage only, not installed): runs the space-group evidence +# table over a stored P1-merged MTZ. +ADD_EXECUTABLE(rugnux_ledger rugnux_ledger.cpp) +TARGET_LINK_LIBRARIES(rugnux_ledger JFJochImageAnalysis JFJochReader gemmi) + # libcufft_static.a carries a relocatable-device-code object (separate_callback.o), so an executable # linking it needs a CUDA device-link step -- without it the host link fails on an undefined # __cudaRegisterLinkedBinary_* symbol. CUDA 13 no longer ships libcufft_static_nocallback.a, which diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index b4f70e8d6..9e37ca379 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -5163,6 +5163,19 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b logger.Warning("No P1 cross-check dataset written: the fixed space group's centring " "absences were never predicted, so a P1 merge of this run would be " "missing whole centring classes. Re-run without -S to get one."); + // The ledger runs off the P1 cross-check merge below, so every reason that merge is not + // built is also a reason there is no ledger. Say which one it was: a user who gave + // --finalist-ledger and got no table would otherwise have to read this function to find out + // whether the instrument declined, or the crystal simply had nothing to report. + if (config_.finalist_ledger + && !(config_.write_p1_crosscheck && p1_integration_complete && is_rotation)) + logger.Warning("No finalist ledger: it is folded from the P1 cross-check merge, and " + "this run builds none ({}).", + !is_rotation ? "stills - the cross-check is rotation-only for now" + : !config_.write_p1_crosscheck ? "--no-p1-crosscheck was given" + : "a fixed centred space group's absences were never predicted, so a P1 " + "merge would be missing whole centring classes"); + if (config_.write_p1_crosscheck && p1_integration_complete && is_rotation) { // The whole GROUP, not its number: the search can adopt a non-reference setting, and // a number only ever names the reference one - so restoring through the number would @@ -5186,6 +5199,33 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // the whole of it and restored after. experiment_.SpaceGroupNumber(1); const auto p1 = scale_and_merge("P1 cross-check", false); + + // FINALIST LEDGER (--finalist-ledger), report-only and deliberately behind a flag: these + // numbers are an instrument, and reading them as a decision before they are calibrated + // is the one way this can do harm. + // + // It runs here because THIS merge is the substrate the whole space-group audit says is + // missing and nothing else in the run has: the same observations at full resolution, + // production-scaled with the correction surfaces fitted, and folded under NO symmetry - + // so every hypothesis can be folded from it on equal terms. It is already being built + // for the P1 cross-check file, so the ledger costs one search over it and no merge at + // all. It also carries half-set intensities, which is what lets the random-noise R + // floor be formed - the only reference available to the first step out of P1, and the + // reason this cannot be done from the stored MTZ afterwards. + if (config_.finalist_ledger) { + SearchSpaceGroupOptions lopt; + lopt.nthreads = static_cast(std::max(1, config_.nthreads)); + lopt.merge_friedel = experiment_.GetScalingSettings().GetMergeFriedel(); + if (result.consensus_cell.has_value()) + lopt.cell = gemmi::UnitCell(*result.consensus_cell); + lopt.enumerate_all_settings = true; + lopt.enumerate_all_rotation_sets = true; + lopt.merge_isa = result.error_model_isa; + const auto led = SearchSpaceGroup(p1.merged, lopt); + logger.Info("Finalist ledger on the full-resolution P1 merge ({} reflections):\n{}", + p1.merged.size(), FinalistLedgerToText(led)); + } + const std::string path = config_.output_prefix + "_P1.mtz"; WriteMtzReflections(p1.merged, *result.consensus_cell, experiment_, path); experiment_.SetSpaceGroup(determined_group); diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index f1c43e3c5..2f460af2c 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -179,6 +179,7 @@ struct ProcessConfig { // declines it). Every de-novo rotation run writes it, including one that determined P1 itself; // a user-fixed -S writes nothing, because its centring absences were never integrated. bool write_p1_crosscheck = true; + bool finalist_ledger = false; // --finalist-ledger; report-only symmetry evidence table }; struct ProcessResult { diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index a403ca728..f37c27b2a 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -148,6 +148,7 @@ void print_usage() { std::cout << " --scale-fulls rot3d: after the 3D combine, refit a per-frame scale on the fulls (XDS order, Unity model). Default ON for rot3d" << std::endl; std::cout << " --no-scale-fulls Disable the rot3d scale-fulls refit (it is on by default for rot3d)" << std::endl; std::cout << " --write-process-h5 Also write the (large) _process.h5 when merging (default: only .mtz/.cif when merging)" << std::endl; + std::cout << " --finalist-ledger Report the full-resolution evidence for each space group the search considered, not only the one it adopted (report-only; the decision is unchanged)" << std::endl; std::cout << " --smooth-g[=deg] rot3d: smooth per-frame scale G over a deg-degree rotation range (XDS DELPHI-like) before the combine (default: 5 for rot3d; 0 = off)" << std::endl; std::cout << " --relative-b[=deg] rot3d: fit a per-batch relative-B (beyond the single decay slope) over deg-degree batches; cross-validated (default: 10 deg when bare; off otherwise)" << std::endl; std::cout << " --no-scaling-corrections rot3d: disable the (default-on) decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls" << std::endl; @@ -279,6 +280,7 @@ enum { OPT_ICE_MIN_SPOT_RATIO, OPT_NO_SCALE_FULLS, OPT_WRITE_PROCESS_H5, + OPT_FINALIST_LEDGER, OPT_FORCE_STILL, OPT_AZIM_MIN_Q, OPT_AZIM_MAX_Q, @@ -332,6 +334,7 @@ static option long_options[] = { {"scale-fulls", no_argument, nullptr, OPT_SCALE_FULLS}, {"no-scale-fulls", no_argument, nullptr, OPT_NO_SCALE_FULLS}, {"write-process-h5", no_argument, nullptr, OPT_WRITE_PROCESS_H5}, + {"finalist-ledger", no_argument, nullptr, OPT_FINALIST_LEDGER}, {"smooth-g", optional_argument, nullptr, OPT_SMOOTH_G}, {"relative-b", optional_argument, nullptr, OPT_RELATIVE_B}, {"no-scaling-corrections", no_argument, nullptr, OPT_NO_SCALING_CORRECTIONS}, @@ -681,6 +684,7 @@ static int RunRugnux(int argc, char **argv) { bool write_p1_crosscheck = true; // _P1.mtz is written by default; --no-p1-crosscheck declines it std::optional scale_fulls_arg; // --scale-fulls / --no-scale-fulls; default on for rot3d bool write_process_h5_flag = false; // --write-process-h5; also write _process.h5 when merging + bool finalist_ledger_flag = false; // --finalist-ledger; report-only symmetry evidence table std::optional detect_ice_rings; // --detect-ice-rings[=on|off]; unset => use the dataset (file) value bool index_ice_rings = false; // --index-ice-rings[=on|off]; index on the ice-band spots too std::optional ice_min_score_arg; // --ice-min-score: ice-presence gate on the measured score @@ -1123,6 +1127,9 @@ static int RunRugnux(int argc, char **argv) { case OPT_WRITE_PROCESS_H5: write_process_h5_flag = true; break; + case OPT_FINALIST_LEDGER: + finalist_ledger_flag = true; + break; case OPT_SMOOTH_G: smooth_g_deg_arg = optarg ? parse_double_arg(optarg, "--smooth-g", logger) : SMOOTH_G_DEFAULT_DEG; break; @@ -1443,6 +1450,14 @@ static int RunRugnux(int argc, char **argv) { // --mode scale: re-scale and merge the already-integrated reflections stored in the input file, // without re-running spot finding or integration (folded in from the former rugnux_scale tool). if (mode == RugnuxMode::Scale) { + // --mode scale re-merges reflections that are already integrated and already indexed; it runs + // no space-group search and writes no P1 cross-check, so there is no ledger to print here. Said + // out loud because this mode is the cheap CPU instrument someone calibrating the ledger will + // reach for first, and silence would read as "the ledger found nothing". + if (finalist_ledger_flag) + logger.Warning("No finalist ledger: --mode scale re-merges an already-determined group and " + "runs no space-group search. Use --mode mx, which builds the P1 cross-check " + "merge the ledger is folded from."); // Re-scaling reads reflections a previous run integrated, which only a _process.h5 holds. if (input_is_cbf) { logger.Error("--mode scale needs the integrated reflections in a _process.h5; " @@ -2300,6 +2315,14 @@ static int RunRugnux(int argc, char **argv) { // Scaling and merging are on by default (run_scaling initialised true); --no-merge turns them off // for both rotation and stills, in which case only the per-image _process.h5 is written. + // The finalist ledger is folded from the P1 cross-check merge, so a run that does no merging has + // none to fold. Said here, where the flag that decided it is in hand: the pipeline's own decline + // message sits inside the scaling block and is unreachable once that block is switched off, and + // silence would read as "the ledger found nothing to report" rather than "it never ran". + if (finalist_ledger_flag && !run_scaling) + logger.Warning("No finalist ledger: --no-merge skips scaling and merging, and the ledger is " + "folded from the P1 cross-check merge. Drop --no-merge to get one."); + // Configure Indexing IndexingSettings indexing_settings; indexing_settings.Algorithm(indexing_algorithm); @@ -2592,6 +2615,7 @@ static int RunRugnux(int argc, char **argv) { // When merging, the merged reflections (.mtz/.cif) are the wanted output; skip the large // _process.h5 unless explicitly requested. Without merging, the _process.h5 is the only output. config.write_process_h5 = run_scaling ? write_process_h5_flag : true; + config.finalist_ledger = finalist_ledger_flag; Rugnux process(reader, experiment, *dataset->pixel_mask, config); diff --git a/rugnux/rugnux_ledger.cpp b/rugnux/rugnux_ledger.cpp new file mode 100644 index 000000000..463049b75 --- /dev/null +++ b/rugnux/rugnux_ledger.cpp @@ -0,0 +1,107 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +// Offline finalist-ledger instrument. Reads a P1-merged MTZ - the P1 cross-check dataset every +// rotation run already writes, which is the run's own observations at FULL resolution, production +// scaled, and folded under no symmetry at all - and prints the ledger for it. +// +// The point of reading THAT file is the one the space-group audit keeps arriving at: the search +// decides on a low-resolution, zeta-filtered SEARCH merge, and the full-resolution merge where a +// false operator shows itself is never consulted. This tool consults it, changes nothing, and is +// only an instrument: it exists to measure whether the separation is there before any gate is +// allowed to read it. +// +// A resolution limit may be passed as the 4th argument. That is the paired contrast the whole +// hypothesis rests on: the production search decides on a low-resolution merge, so running the same +// vector twice - once at full range, once cut back to the search range - asks whether the contrast a +// false operator shows is actually being thrown away by the cut, on one crystal, with everything +// else held identical. +// +// Usage: rugnux_ledger [tag] [nthreads] [d_min_A] + +#include +#include +#include +#include +#include + +#include + +#include "../common/Reflection.h" +#include "../image_analysis/scale_merge/SearchSpaceGroup.h" + +int main(int argc, char** argv) { + if (argc < 2) { + std::cout << "Usage: rugnux_ledger [tag] [nthreads]" << std::endl; + return 1; + } + const std::string path = argv[1]; + const std::string tag = argc > 2 ? argv[2] : "-"; + const int nthreads = argc > 3 ? std::atoi(argv[3]) : 4; + const double d_min = argc > 4 ? std::atof(argv[4]) : 0.0; + + gemmi::Mtz mtz; + try { + mtz.read_file_gz(path, true); + } catch (const std::exception& e) { + std::cerr << "LEDGER\t" << tag << "\tREAD_FAIL\t" << e.what() << std::endl; + return 1; + } + + const gemmi::Mtz::Column* ci = mtz.column_with_label("IMEAN", nullptr, 'J'); + const gemmi::Mtz::Column* cs = mtz.column_with_label("SIGIMEAN", nullptr, 'Q'); + if (ci == nullptr || cs == nullptr) { + std::cerr << "LEDGER\t" << tag << "\tNO_IMEAN" << std::endl; + return 1; + } + + std::vector merged; + merged.reserve(static_cast(mtz.nreflections)); + for (int i = 0; i < mtz.nreflections; ++i) { + const size_t row = static_cast(i) * mtz.columns.size(); + MergedReflection r; + r.h = static_cast(std::lround(mtz.data[row + 0])); + r.k = static_cast(std::lround(mtz.data[row + 1])); + r.l = static_cast(std::lround(mtz.data[row + 2])); + r.I = mtz.data[row + ci->idx]; + r.sigma = mtz.data[row + cs->idx]; + if (!std::isfinite(r.I) || !std::isfinite(r.sigma) || r.sigma <= 0.0f) + continue; + r.d = static_cast(mtz.cell.calculate_d( + {{static_cast(r.h), static_cast(r.k), static_cast(r.l)}})); + // I_half stays NaN: the merged MTZ carries no half-set columns, so the random-noise R floor + // cannot be formed here. Every ratio that needs it reads "-" in the table below, and the + // first step out of P1 - the one case with no other operator to be the reference - is + // therefore NOT measurable from a stored MTZ. It needs the in-process hook. + merged.push_back(r); + } + + SearchSpaceGroupOptions opt; + opt.nthreads = static_cast(std::max(1, nthreads)); + opt.cell = mtz.cell; + opt.enumerate_all_settings = true; + opt.enumerate_all_rotation_sets = true; + opt.d_min_limit_A = d_min; + // No lattice_system cap: the metric class this run indexed on is not in the file. Measured over + // 40 crystals when the same question was asked before, the cap was non-binding on every one of + // them, so releasing it here costs nothing and keeps the tool independent of the run's lattice. + + const auto result = SearchSpaceGroup(merged, opt); + + std::cout << "LEDGER\t" << tag << "\tMETA\tdmin=" << d_min << "\tn=" << merged.size() + << "\tcell=" << mtz.cell.a << "," << mtz.cell.b << "," << mtz.cell.c + << "," << mtz.cell.alpha << "," << mtz.cell.beta << "," << mtz.cell.gamma + << "\tpg=" << (result.point_group_hm.empty() ? "?" : result.point_group_hm) + << "\torder=" << result.point_group_order + << "\tsg=" << (result.best_space_group ? result.best_space_group->xhm() : "-") + << "\tbest_op_r=" << result.global_best_operator_r << std::endl; + for (const auto& e : result.point_group_ledger) + std::cout << "LEDGER\t" << tag << "\tROW\t" << e.point_group_hm << "\t" << e.order + << "\t" << e.min_class_cc << "\t" << e.r_added << "\t" << e.r_over_best + << "\t" << e.h_ratio << "\t" << e.chi2_over_best << "\t" << e.b_extra + << "\t" << e.b_over_parent + << "\t" << (e.adopted ? "ADOPTED" : e.eligible ? "eligible" : "refused") + << "\t" << (e.refused_reason.empty() ? "-" : e.refused_reason) << std::endl; + std::cout << std::endl << FinalistLedgerToText(result) << std::endl; + return 0; +} -- 2.54.0 From eeb66ae5b447d898324792901a5fd88b787abaaf Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 13:30:27 +0200 Subject: [PATCH 32/75] symmetry: the systematic-b bounds follow the divisor that was changed under them Dividing the b fit by its degrees of freedom rather than by the observation count rescaled the statistic those bounds are compared against, and the bounds were left where they were. The header said what would happen: it records the rescue bound's calibration as a genuine step at 1.71 against a twin at 1.85 - a 4% margin - names a trigonal 3 -> 32 twin as the case it was fitted on, and instructs that it be re-measured whenever the scaling changes. It was not, and the margin closed on the wrong side. On a merohedrally twinnable crystal the b ratio moved 1.74 -> 1.40 under the new divisor, because the low-multiplicity parent gained 43% where the candidate gained 16%; with chi^2 failing at 1.97 but inside the rescue band and H confirming, the b rescue alone then promoted the twin. Merged statistics got worse doing it: R_meas 0.1310 -> 0.1732, CC_half 0.9978 -> 0.9966. Rescale the three bounds by the same factor as the statistic. This is a change of units, not a re-tuning: b_new > 0.808 * 1.78 is the same condition as b_old > 1.78, and three independent derivations agree on 1.44. Verified by running one binary with only these constants changed - the promotion reverts to the deposited group at the original numbers, while an unrelated rescue at order 1 -> 2 is untouched, being structurally immune where the parent b is zero. The census that missed this counted the veto's firings. The same constant drives the rescue, and a rescue leaves no refusal to count - it is recorded as an adoption. When a change moves the scale of a statistic, census the statistic's distribution and every bound compared against it, not the code path edited. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- image_analysis/scale_merge/SearchSpaceGroup.h | 27 +++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/image_analysis/scale_merge/SearchSpaceGroup.h b/image_analysis/scale_merge/SearchSpaceGroup.h index c574ff112..b5f61acd4 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.h +++ b/image_analysis/scale_merge/SearchSpaceGroup.h @@ -259,7 +259,24 @@ struct SearchSpaceGroupOptions { // It is a 4% margin on each side, narrower than the shift a change in how the search merge is // scaled has already been measured to produce on these same four numbers. Re-measure it whenever // that scaling changes rather than trusting it through such a change. - double max_systematic_b_ratio = 1.78; + // + // 1.44, not the 1.78 those four numbers were read in. This bound and max_systematic_b_veto below + // are bounds on a RATIO of two b values, so their scale is set by how merge_systematic_b + // normalises its reduced chi^2 - and that normalisation changed, from the observation count N to + // the fit's degrees of freedom N - G. Every ratio in the corpus fell with it: a measured -23.7% + // median over three disjoint corpora, lower in 42 of 42 promotions, because the low-multiplicity + // parent loses a larger fraction of its dof than the candidate and so gains more b. Measured on + // one trigonal crystal, same merge, only the divisor differing: the parent's b 0.126 -> 0.181 + // (+43%) against the candidate's 0.219 -> 0.254 (+16%), and the ratio 1.74 -> 1.40. + // + // Leaving the bound at 1.78 through that was a change of units without a conversion, and it + // loosened the RESCUE below onto a merohedral twin the old convention refused. Rescaling by the + // same factor restores every verdict exactly - b_new > 0.805 x 1.78 is the same test as + // b_old > 1.78 - which is what the divisor change intended and did not deliver. Derived three + // ways that agree: 1.78 x 0.808; the midpoint of the calibration pair above re-read in the + // corrected convention (genuine 1.71 -> 1.38, twin 1.85 -> 1.49); and a population-median + // derivation that argued 1.38. + double max_systematic_b_ratio = 1.44; // Veto bound. The systematic-b test above is otherwise only a rescue - it can promote a chi^2-borderline // genuine step but never demote a chi^2-passing one. A merohedral twin whose within-orbit scatter looks @@ -279,7 +296,13 @@ struct SearchSpaceGroupOptions { // max_operator_h_ratio). Where both fire the promotion is still refused, and every twin on the battery // and in the synthetic harness fires both. The veto only ever keeps a clearly-ballooned promotion down, // never promotes. - double max_systematic_b_veto = 2.0; + // 1.62 = 2.0 rescaled by the same 0.808 as max_systematic_b_ratio above, and for the same reason: + // it reads the same ratio, whose scale moved under it. This half is the less certain of the two - + // the veto has not been observed to fire on any corpus it has been measured over, so nothing + // measured says where it now sits - but leaving it in the old units while the statistic moved is + // the very mistake being corrected, and it would leave the veto systematically weaker than the + // calibration above intends. + double max_systematic_b_veto = 1.62; // Floor on the parent's systematic-b when forming the veto ratio. On excellent data a genuine merge's // b is near zero (ISa well above 20), so even a small, harmless absolute increase gives a huge b-ratio -- 2.54.0 From d6ccdbdb611828c24f99951ef8c86abaa9ca4666 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 13:40:59 +0200 Subject: [PATCH 33/75] rugnux: the report says what it found before it says how it found it The report was written in the order the pipeline runs, so a user reading one had to reach line 380 before meeting the first evaluative statement, and the warnings were last. Every run was 314 to 407 lines whether it succeeded or failed, about 200 of them fixed prose. The anisotropy section printed 21 keys and announced DETECTED (strong) on 54% of all runs - "strong" is the statistical confidence, which a reader takes as the severity - and reported a censored fit, which is survival-analysis vocabulary for good news. Assemble the report into a document and render from it, rather than streaming it out as the pipeline goes. That is what allows a verdict to sit above the evidence it was drawn from: SUMMARY carries VERDICT, one sentence of plain text, the warnings and ten facts, and it is composed after the sections that produce them. A clean run is 188 lines, a failed one 93, and the verdict is on line 25 in both. Warnings now carry a closed pathology vocabulary alongside their free text, so a consumer can switch on the code and a reader still gets the sentence. Everything removed from the default report is still written under --developer, which also carries the internals worth having when diagnosing the program rather than the crystal: the gate keys behind the anisotropy verdict, the operator and candidate tables, the sweep and spindle internals, and the essays. Nine statements the report made that were not true are fixed here as well. Among them: --mode scale printed a detector tilt of exactly zero on tilted data, which is worse than printing nothing because nothing about it looks wrong; every --mode scale run carried a "No image indexed" warning, because a key that is absent and a key measured to be zero were the same value; and a fitted resolution was asserted past the point where the run's own table shows CC1/2 at zero. REPORT_VERSION is 8. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 8 + docs/RUGNUX_REPORT.md | 53 +- rugnux/CMakeLists.txt | 1 + rugnux/ReportDocument.h | 104 ++ rugnux/ResultReport.cpp | 1840 +++++++++++++++++++++++------------- rugnux/ResultReport.h | 15 + rugnux/Rugnux.cpp | 14 +- rugnux/Rugnux.h | 8 +- rugnux/rugnux_cli.cpp | 24 +- tests/ResultReportTest.cpp | 230 ++++- 10 files changed, 1596 insertions(+), 701 deletions(-) create mode 100644 rugnux/ReportDocument.h diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 49176581a..3ffead23b 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -3,6 +3,14 @@ ### 1.0.0-rc.166 +* The rugnux results report opens with a summary: `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), a sentence saying what the run produced, the ten facts a user reads first, and the warnings - which used to be its last section. +* The rugnux results report carries `PATHOLOGY_FLAGS=`, one closed-vocabulary code per condition that warned, beside the plain-English `WARNING:` lines. +* The rugnux results report warns when the merged data carry no usable signal, and when too little of reciprocal space was measured inside the fitted resolution. +* `rugnux --developer` writes the full results report: the pipeline-internal keys and the long explanations the default report leaves out. +* The rugnux results report is shorter and its sections are renumbered 1-5 with no gaps; `REPORT_VERSION` is 8. +* `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. +* The rugnux results report omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off, instead of asserting a resolution the shell table refutes. + * `rugnux` corrects each reflection for the sensor's quantum efficiency at the angle the diffracted beam meets the detector. * `rugnux` takes the sensor attenuation length from tabulated coefficients rather than a wavelength-cubed approximation, which corrects the parallax term of the spot-width variance on CdTe sensors. * `rugnux --model` treats the model as a hypothesis: it decides the enantiomorph and the indexing only where its R-work beats that of the same model in random orientations, and a model the data reject is still scored, placed and mapped, but leaves the reflection files byte for byte what a run with no model writes. diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 52e5ac65d..725e61d8e 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -73,7 +73,7 @@ keep their numbers. **`SPOT_RESOLUTION_ESTIMATE=`** in section 1 is how far the merged data are expected to reach, read off the found spots alone — no lattice, no integration, no merge — so it is there on a run that never -merges, and on a run that does it can be read against `INCLUDE_RESOLUTION_RANGE` in section 5. It is a +merges, and on a run that does it can be read against `INCLUDE_RESOLUTION_RANGE` in section 3. It is a prediction, good to about 0.2 Å on rotation data; nothing is cut on it. It is **not** limited to what the detector records: where it reads finer than the high-resolution end of `INCLUDE_RESOLUTION_RANGE`, the crystal diffracts past the corner and the run was detector-limited. @@ -95,15 +95,52 @@ grep '^JFJOCH_DATASET_SETTINGS=' out_report.txt | cut -d' ' -f2- > geometry.json **Which pass.** A rotation run integrates twice — once at the geometry in the input file, then again at the post-refined geometry — and can integrate a third time if a guard rejects the second pass. There is **one** report, for the pass that became the canonical output, and `PASS=` / -`PASS_DECISION=` in section 1 say which pass that is and on what evidence, so no number in the file +`PASS_DECISION=` (`--developer`) in section 1 say which pass that is and on what evidence, so no number in the file is ambiguous about which geometry produced it. **Not in the report:** timing, frame rates, thread counts, per-image progress and library banners. Those are process, not result, and stay on stdout. +## The summary, and what the run decided + +The file opens with a **`SUMMARY`** section, above everything it summarises. It exists because the +report used to have no evaluative line anywhere until its last section: a run that produced garbage +and a run that produced a textbook data set read identically for their first three hundred lines. + +- **`VERDICT=`** is a closed vocabulary — `OK`, `WARNINGS`, `UNUSABLE`, `FAILED`. `FAILED` means no + lattice was determined or the run was cancelled; `UNUSABLE` means the data merged but carry no + usable signal; `WARNINGS` means something else needs attention; `OK` means nothing did. It is + decided from the warnings the rest of the report produced, so it introduces no new analysis and + cannot disagree with the sections below it. +- **`VERDICT_TEXT=`** is one to three sentences of free text saying the same thing in English. +- **`PATHOLOGY_FLAGS=`** is the *type* of each condition that fired, from a closed vocabulary, so a + consumer switches on a code rather than parsing a sentence: `NO_LATTICE`, `INDEXING_AMBIGUITY`, + `SYMMETRY_AMBIGUITY`, `CENTERING_UNTESTED`, `UNUSABLE_MERGE`, `LOW_COMPLETENESS`, `SWEEP_GAPS`, + `GONIO_SCALE`, `SPINDLE_CAP`, `ANISOTROPY`, `TWINNING`, `MODEL_HAND`, `MODEL_NOT_VALIDATED`, + `CANCELLED`, `RESOLUTION_FIT`. `NONE` when nothing fired. A code appears if and only if its + warning fired, so the flags and the `WARNING:` lines are two renderings of one list — the closed + type for machinery, the open sentence for a person. +- **`WARNING_COUNT=`** and the **`WARNING:`** lines follow, in the same section. They are what they + always were; they have moved from the bottom of the file to the top. + +Then four sections, numbered contiguously: **1. DATA SET AND GEOMETRY**, **2. CRYSTAL**, +**3. MERGED DATA**, **4. DIAGNOSTICS**, and **5. MODEL VALIDATION** only with `--model`. + +## The developer report + +`--developer` renders the same report in full. The default report carries what a person deciding +*keep or recollect* acts on; `--developer` adds the pipeline's own internals — the anisotropy +detection gate's parameters, the space-group operator and candidate tables, the model-fit null, the +sweep-quality internals, the twinning statistics measured before the search, and the long +explanatory passages — plus advisories about the cut's own behaviour that no user can act on. + +Nothing is computed differently and nothing is lost by leaving the flag off: the report is built +once, in full, and the flag selects how much of it is written. Every key the default report writes, +`--developer` writes too. + ## Sweep quality and the reason vocabulary -Section 8 lists the stretches of the sweep over which the crystal delivered much less than the rest +Section 4 lists the stretches of the sweep over which the crystal delivered much less than the rest of the run — the feedback a beamline control system needs to tell an operator that a crystal should be recentred or recollected. Nothing is excluded on the strength of it; the frames still carry signal, and this is a message for the beamline, not a filter. @@ -145,14 +182,14 @@ source image is `start + ordinal * stride`); `ROTATION` — the width of the ran `SEVERITY` — the fraction of the run's typical diffracting power missing over the range, 0 (as good as the run) to 1 (nothing at all); `SCALE` and `CC` — the range's mean per-image scale and CC-to-merge relative to the run median; `INDEXED` — the fraction of the range's frames that were -scaled at all. Every range also appears as a `WARNING:` sentence in section 11. +scaled at all. Every range also appears as a `WARNING:` sentence in the SUMMARY. The same finding is written **per image** into the `_process.h5` as `/entry/MX/sweepQuality`, when one is written — see [HDF5](HDF5.md#entry-mx-spot-finding-and-indexing-cxi-style). ## Diffraction anisotropy -Section 9 reports how much the fall-off with resolution depends on **direction**, and whether that is +Section 4 also reports how much the fall-off with resolution depends on **direction**, and whether that is established above the data set's own systematic error. It runs automatically on every merging run — there is no flag — and it is a **description only**: no intensity is corrected, no reflection is removed on a directional criterion, and the merged data and the written reflection files do not @@ -197,7 +234,7 @@ says so, since refinement and map interpretation should allow for it. ## Model validation -Section 10 appears only with `--model`. It reports the supplied model against the merged data — +Section 5 appears only with `--model`. It reports the supplied model against the merged data — R-factors, maps, anomalous sites — and, separately, whether the data accepted the model at all. The two are not the same question, and the report keeps them apart. **The R-factors, the maps and the @@ -241,7 +278,7 @@ files are **byte for byte** what a run with no model would have written — same `_unmerged.mtz`. A rejected model is therefore safe to try: it costs the null's compute and changes nothing else. -`SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` and `SPINDLE_SYMMETRY_AXIS_ORDER=` in section 4 say how the crystal +`SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` and `SPINDLE_SYMMETRY_AXIS_ORDER=` (`--developer`) in section 4 say how the crystal sat on the goniometer: the angle between the spindle and the nearest symmetry axis, and that axis's order. They are descriptive: neither convicts nor clears the mounting on its own, because an aligned axis of any order maps the sweep's blind cone onto itself while an axis near perpendicular does the @@ -257,7 +294,7 @@ content. The same number is written to the master file as `/entry/MX/spindleLost pipeline can read it from either output without parsing prose. Written on every rotation run that determined a space group and merged reflections. -`SPACE_GROUP_ENANTIOMORPH=` in section 4 reads **`ASSUMED_FROM_MODEL`** when the hand written in the +`SPACE_GROUP_ENANTIOMORPH=` in section 2 reads **`ASSUMED_FROM_MODEL`** when the hand written in the files is the model's. Assumed, not determined: merged intensities cannot see the hand at all — |F| is invariant under the change of hand — so an accepted model asserts it out of prior chemical knowledge. It is only ever written where `MODEL_FIT= ACCEPTED`, and the anomalous difference map vetoes it diff --git a/rugnux/CMakeLists.txt b/rugnux/CMakeLists.txt index b437e2e67..55529c5d4 100644 --- a/rugnux/CMakeLists.txt +++ b/rugnux/CMakeLists.txt @@ -16,6 +16,7 @@ ADD_LIBRARY(Rugnux STATIC SigmaA.h ResultReport.cpp ResultReport.h + ReportDocument.h SpotWidth.cpp SpotWidth.h SpindleCuspLoss.cpp diff --git a/rugnux/ReportDocument.h b/rugnux/ReportDocument.h new file mode 100644 index 000000000..31d7cced1 --- /dev/null +++ b/rugnux/ReportDocument.h @@ -0,0 +1,104 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include + +// One populated result object, rendered more than once. +// +// The report used to be printed as the writer walked the results, which fixed the order of the file +// to the order of the pipeline and made a summary at the top impossible - the warnings that decide +// the verdict are produced by the sections the verdict has to sit above. BuildReportDocument() +// therefore populates this document, and a renderer walks it afterwards: the text renderer in +// ResultReport.cpp reads ONLY from here and prints nothing on its own. A second rendering is a +// second walk over the same document, never a second hand-maintained emitter. + +// The TYPE of a pathology, from a closed vocabulary a consumer can switch on, carried beside the +// open free-text description that says what actually happened to this crystal. One code per warning +// PRODUCER: a code appears in PATHOLOGY_FLAGS if and only if its warning fired, so lighting an +// indicator is one grep and needs no prose parsing. Add a producer, add a code here. +namespace PathologyCode { + constexpr const char *NO_LATTICE = "NO_LATTICE"; // nothing indexed + constexpr const char *INDEXING_AMBIGUITY = "INDEXING_AMBIGUITY"; // alternative indexing unresolved + constexpr const char *SYMMETRY_AMBIGUITY = "SYMMETRY_AMBIGUITY"; // the data do not pick a point group + constexpr const char *CENTERING_UNTESTED = "CENTERING_UNTESTED"; // centring from the metric only + constexpr const char *UNUSABLE_MERGE = "UNUSABLE_MERGE"; // the merged data carry no signal + constexpr const char *LOW_COMPLETENESS = "LOW_COMPLETENESS"; // too little of reciprocal space + constexpr const char *SWEEP_GAPS = "SWEEP_GAPS"; // ranges that delivered much less + constexpr const char *GONIO_SCALE = "GONIO_SCALE"; // the stage rotation is mis-scaled + constexpr const char *SPINDLE_CAP = "SPINDLE_CAP"; // the mounting cost unique data + constexpr const char *ANISOTROPY = "ANISOTROPY"; // strongly direction-dependent + constexpr const char *TWINNING = "TWINNING"; // twinning indicated + constexpr const char *MODEL_HAND = "MODEL_HAND"; // data and model in opposite hands + constexpr const char *MODEL_NOT_VALIDATED = "MODEL_NOT_VALIDATED"; // a model was given, not usable + constexpr const char *CANCELLED = "CANCELLED"; // the run did not finish + constexpr const char *RESOLUTION_FIT = "RESOLUTION_FIT"; // the CC1/2 fall-off is not a fall-off +} + +// A closed type plus an open description. The type is what machinery switches on; the text is what a +// person reads, and it is free-form on purpose - the vocabulary must not have to grow a code every +// time a sentence gains a number. +struct ReportWarning { + std::string code; + std::string text; + // Advisories about the pipeline's own internals rather than about this crystal. They still fire, + // still count in the developer report, and are kept out of the default one - a warning nobody can + // act on is how users learn to skip the warnings. + bool developer_only = false; +}; + +// A value with its type kept, so a rendering other than text does not have to parse the string back. +// `text` is always the exact spelling the text report prints, because that spelling is the interface. +struct ReportValue { + enum class Type { Text, Integer, Real, Boolean, Enumerated }; + Type type = Type::Text; + std::string text; + double real = 0.0; + int64_t integer = 0; + bool boolean = false; + std::vector domain; // Enumerated only: the closed vocabulary this value comes from +}; + +// Content that is a table in every rendering. The pre-formatted text lines and the typed cells carry +// the same content: the text renderer prints the former, another rendering reads the latter. +struct ReportTable { + std::vector columns; + std::string text_header; + std::vector text_rows; + std::vector> cells; +}; + +struct ReportEntry { + enum class Kind { Key, Prose, Blank, Table }; + Kind kind = Kind::Key; + std::string key; // Key + ReportValue value; // Key + std::string prose; // Prose: a paragraph, already line-wrapped for the text rendering + ReportTable table; // Table + // Written only when the developer report was asked for. The document always holds everything; + // which rendering shows what is the renderer's decision, so nothing is lost by building once. + bool developer_only = false; + // The other half of the same idea: a short paragraph that exists only because the long one it + // summarises was moved out. Printing both would say the same thing twice. + bool default_only = false; +}; + +struct ReportSection { + std::string title; // empty for the header block, which has no banner of its own + std::vector entries; + bool developer_only = false; +}; + +// The whole report as data. Sections in the order they are written; warnings and the verdict live +// here rather than in a section, because the SUMMARY section is composed from them after every other +// section has been built. +struct ReportDocument { + std::vector sections; + std::vector warnings; + std::string verdict; // closed: OK | WARNINGS | UNUSABLE | FAILED + std::string verdict_text; // one to three sentences, free text + std::vector pathology_flags; // PathologyCode::*, deduplicated, in first-seen order +}; diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 15b9c4875..62f397244 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -21,19 +21,12 @@ namespace { // The version of this file format. Bumped when a key is renamed or removed, a table column moves, // or a reason code changes meaning - a consumer can gate on it. - constexpr int REPORT_VERSION = 7; + // 8: SUMMARY section with VERDICT / PATHOLOGY_FLAGS; sections renumbered 1-5 with no holes; + // pipeline-internal keys and the long explanatory blocks moved behind --developer. + constexpr int REPORT_VERSION = 8; const char *BANNER = " ******************************************************************************"; - void Section(std::ostream &os, const std::string &title) { - os << "\n" << BANNER << "\n " << title << "\n" << BANNER << "\n\n"; - } - - // Every number a consumer might want is written as one of these, so it is one grep away. - template void Key(std::ostream &os, const char *key, const T &value) { - os << key << "= " << value << "\n"; - } - std::string CellString(const UnitCell &c) { return fmt::format("{:.3f} {:.3f} {:.3f} {:.3f} {:.3f} {:.3f}", c.a, c.b, c.c, c.alpha, c.beta, c.gamma); @@ -62,564 +55,801 @@ namespace { } return fmt::format("{} (cos {:.2f})", NAME[best], best_cos); } + + // ------------------------------------------------------------------ document builders + // Small constructors, so the body below reads as a list of what the report SAYS rather than as a + // list of how it is punctuated. Each fills the typed field beside the text, so a rendering that + // is not text does not have to parse the string back. + + ReportEntry KeyText(const char *key, std::string value, bool dev = false) { + ReportEntry e; + e.key = key; + e.value.type = ReportValue::Type::Text; + e.value.text = std::move(value); + e.developer_only = dev; + return e; + } + + ReportEntry KeyEnum(const char *key, const std::string &value, + std::vector domain, bool dev = false) { + ReportEntry e = KeyText(key, value, dev); + e.value.type = ReportValue::Type::Enumerated; + e.value.domain = std::move(domain); + return e; + } + + ReportEntry KeyBool(const char *key, bool value, bool dev = false) { + ReportEntry e = KeyEnum(key, value ? "TRUE" : "FALSE", {"TRUE", "FALSE"}, dev); + e.value.type = ReportValue::Type::Boolean; + e.value.boolean = value; + return e; + } + + ReportEntry KeyInt(const char *key, int64_t value, bool dev = false) { + ReportEntry e; + e.key = key; + e.value.type = ReportValue::Type::Integer; + e.value.integer = value; + e.value.text = std::to_string(value); + e.developer_only = dev; + return e; + } + + // The format is passed in because the spelling of a number is part of the interface: a consumer + // that greps R_MEAS gets four decimals whatever a renderer would rather do with it. + ReportEntry KeyReal(const char *key, double value, const char *format, bool dev = false) { + ReportEntry e; + e.key = key; + e.value.type = ReportValue::Type::Real; + e.value.real = value; + e.value.text = fmt::format(fmt::runtime(format), value); + e.developer_only = dev; + return e; + } + + ReportEntry Prose(std::string text, bool dev = false) { + ReportEntry e; + e.kind = ReportEntry::Kind::Prose; + e.prose = std::move(text); + e.developer_only = dev; + return e; + } + + // A paragraph that exists only because the long version of the same point moved to --developer. + ReportEntry ProseDefaultOnly(std::string text) { + ReportEntry e = Prose(std::move(text)); + e.default_only = true; + return e; + } + + ReportEntry Blank() { + ReportEntry e; + e.kind = ReportEntry::Kind::Blank; + return e; + } + + void Add(ReportSection &s, ReportEntry e) { s.entries.push_back(std::move(e)); } + + void Warn(ReportDocument &doc, const char *code, std::string text, bool dev = false) { + doc.warnings.push_back({code, std::move(text), dev}); + } + + // Completeness over the shells the fit says carry signal, which is not the same number as the + // overall completeness: the resolution cut is deliberately generous, so the outer shells dilute + // the denominator with reciprocal space the detector only reaches in its corners. Users compare + // the overall figure against another program's and conclude data were lost. + struct CompletenessInRange { + bool measured = false; + double percent = 0.0; + double d_min = 0.0; + }; + + CompletenessInRange CompletenessTo(const MergeStatistics &ms, double d_min_limit) { + CompletenessInRange out; + int64_t unique = 0, possible = 0; + for (const auto &sh : ms.shells) { + if (sh.d_min < d_min_limit) + continue; + unique += sh.unique_reflections; + possible += sh.possible_unique_reflections; + out.d_min = sh.d_min; + } + if (possible > 0) { + out.measured = true; + out.percent = 100.0 * static_cast(unique) / static_cast(possible); + } + return out; + } + + // The finest shell whose CC1/2 still reaches the cut's own target. A fitted resolution finer than + // this was read off a curve that never behaved like a fall-off, and asserting it states a + // resolution the shell table on the same page refutes. + double FinestSupportedShell(const MergeStatistics &ms, double target) { + double finest = 0.0; + for (const auto &sh : ms.shells) + if (std::isfinite(sh.cc_half) && sh.cc_half >= target && (finest == 0.0 || sh.d_min < finest)) + finest = sh.d_min; + return finest; // 0.0 = no shell reaches the target at all + } } -std::string RenderResultReport(const std::string &output_prefix, - const std::string &input_file, - const DiffractionExperiment &experiment, - const ProcessResult &result, - const RunProvenance &provenance) { - std::ostringstream os; +// ---------------------------------------------------------------------------- the emitter +// Populates the document. Nothing here writes to a stream: the order of the FILE is the renderer's +// business, which is what lets the SUMMARY sit above the sections whose warnings it reports. +ReportDocument BuildReportDocument(const std::string &output_prefix, + const std::string &input_file, + const DiffractionExperiment &experiment, + const ProcessResult &result, + const RunProvenance &provenance) { + ReportDocument doc; const bool rotation = experiment.IsRotationIndexing(); const bool merged = result.has_merge_statistics; - std::vector warnings = result.warnings; + doc.warnings = result.warnings; - os << BANNER << "\n" - << " RUGNUX PROCESSING REPORT\n" - << BANNER << "\n\n" - << " What this run determined, written next to its other output. The `KEY= value` lines and\n" - << " the tables below are a stable interface - a script greps them, and REPORT_VERSION says\n" - << " when that interface last changed. Rates and per-image progress are not here; they are\n" - << " on stdout.\n\n"; - - Key(os, "REPORT_VERSION", REPORT_VERSION); - Key(os, "RUGNUX_VERSION", jfjoch_version()); - if (!jfjoch_git_sha1().empty()) - Key(os, "RUGNUX_GIT", jfjoch_git_sha1().substr(0, 6) + " " + jfjoch_git_date()); - Key(os, "DATE", time_UTC(std::chrono::system_clock::now())); - Key(os, "INPUT_FILE", input_file); - Key(os, "OUTPUT_PREFIX", output_prefix); - // How the result was produced, what it cost and what it ran on, so the report stands on its own - // once the shell history it came from is gone. Absent rather than zero where the caller does not - // know them - the library and the viewer have no command line and no invocation to time. - if (!provenance.command_line.empty()) - Key(os, "COMMAND_LINE", provenance.command_line); - if (provenance.wall_time_s > 0.0) - Key(os, "WALL_TIME", fmt::format("{:.2f}", provenance.wall_time_s)); - if (provenance.gpu_count >= 0) { - Key(os, "GPU_COUNT", provenance.gpu_count); - // GPU_COUNT= 0 with no GPU= line is the CPU-only case, and saying so is the point: whether - // the GPUs were there is the first question about how long the run took. - if (!provenance.gpu_description.empty()) - Key(os, "GPU", provenance.gpu_description); - } - - // ---------------------------------------------------------------- 1. DATA SET - Section(os, "1. DATA SET"); - Key(os, "EXPERIMENT_TYPE", rotation ? "ROTATION" : "STILLS"); - Key(os, "IMAGES_PROCESSED", result.images_processed); - Key(os, "WAVELENGTH", fmt::format("{:.5f}", experiment.GetWavelength_A())); - if (const auto gonio = experiment.GetGoniometer()) { - Key(os, "OSCILLATION_RANGE", fmt::format("{:.4f}", gonio->GetIncrement_deg())); - Key(os, "STARTING_ANGLE", fmt::format("{:.3f}", gonio->GetStart_deg())); - const auto ax = gonio->GetAxis(); - Key(os, "ROTATION_AXIS", fmt::format("{:.6f} {:.6f} {:.6f}", ax.x, ax.y, ax.z)); - } - Key(os, "DETECTOR_DISTANCE", fmt::format("{:.3f}", result.used_distance_mm)); - Key(os, "BEAM_CENTRE", fmt::format("{:.2f} {:.2f}", result.used_beam_x_pxl, result.used_beam_y_pxl)); - // The tilt this run integrated at, and the two points it separates. BEAM_CENTRE is the PONI - the - // foot of the perpendicular from the sample - so on a tilted detector it is NOT where the direct - // beam lands, and the two were previously indistinguishable in this report because only one of - // them was printed. Degrees here; the JSON below carries radians, as the API spells it. + // ------------------------------------------------------------------ header block { - Key(os, "DETECTOR_TILT", fmt::format("{:.4f} {:.4f} {:.4f}", result.used_detector_tilt_deg[0], - result.used_detector_tilt_deg[1], - result.used_detector_tilt_deg[2])); - Key(os, "DIRECT_BEAM", fmt::format("{:.2f} {:.2f}", result.used_direct_beam_x_pxl, - result.used_direct_beam_y_pxl)); + ReportSection h; + Add(h, Prose(" What this run determined, written next to its other output. The `KEY= value` lines\n" + " and the tables below are a stable interface - a script greps them, and REPORT_VERSION\n" + " says when that interface last changed. --developer adds the pipeline internals and the\n" + " long explanations; leaving it off relocates them, it does not lose them.")); + Add(h, Blank()); + Add(h, KeyInt("REPORT_VERSION", REPORT_VERSION)); + Add(h, KeyText("RUGNUX_VERSION", jfjoch_version())); + if (!jfjoch_git_sha1().empty()) + Add(h, KeyText("RUGNUX_GIT", jfjoch_git_sha1().substr(0, 6) + " " + jfjoch_git_date())); + Add(h, KeyText("DATE", time_UTC(std::chrono::system_clock::now()))); + Add(h, KeyText("INPUT_FILE", input_file)); + Add(h, KeyText("OUTPUT_PREFIX", output_prefix)); + // How the result was produced, what it cost and what it ran on, so the report stands on its + // own once the shell history it came from is gone. Absent rather than zero where the caller + // does not know them - the library and the viewer have no command line and no run to time. + if (!provenance.command_line.empty()) + Add(h, KeyText("COMMAND_LINE", provenance.command_line)); + if (provenance.wall_time_s > 0.0) + Add(h, KeyReal("WALL_TIME", provenance.wall_time_s, "{:.2f}")); + if (provenance.gpu_count >= 0) { + // GPU_COUNT= 0 with no GPU= line is the CPU-only case, and saying so is the point: + // whether the GPUs were there is the first question about how long the run took. + Add(h, KeyInt("GPU_COUNT", provenance.gpu_count)); + if (!provenance.gpu_description.empty()) + Add(h, KeyText("GPU", provenance.gpu_description)); + } + doc.sections.push_back(std::move(h)); + } + + // ------------------------------------------------- 1. DATA SET AND GEOMETRY + { + ReportSection s; + s.title = "1. DATA SET AND GEOMETRY"; + Add(s, KeyEnum("EXPERIMENT_TYPE", rotation ? "ROTATION" : "STILLS", {"ROTATION", "STILLS"})); + Add(s, KeyInt("IMAGES_PROCESSED", static_cast(result.images_processed))); + Add(s, KeyReal("WAVELENGTH", experiment.GetWavelength_A(), "{:.5f}")); + if (const auto gonio = experiment.GetGoniometer()) { + Add(s, KeyReal("OSCILLATION_RANGE", gonio->GetIncrement_deg(), "{:.4f}")); + Add(s, KeyReal("STARTING_ANGLE", gonio->GetStart_deg(), "{:.3f}")); + const auto ax = gonio->GetAxis(); + Add(s, KeyText("ROTATION_AXIS", fmt::format("{:.6f} {:.6f} {:.6f}", ax.x, ax.y, ax.z))); + } + Add(s, KeyReal("DETECTOR_DISTANCE", result.used_distance_mm, "{:.3f}")); + Add(s, KeyText("BEAM_CENTRE", fmt::format("{:.2f} {:.2f}", result.used_beam_x_pxl, + result.used_beam_y_pxl))); + Add(s, KeyText("DETECTOR_TILT", fmt::format("{:.4f} {:.4f} {:.4f}", + result.used_detector_tilt_deg[0], + result.used_detector_tilt_deg[1], + result.used_detector_tilt_deg[2]))); + Add(s, KeyText("DIRECT_BEAM", fmt::format("{:.2f} {:.2f}", result.used_direct_beam_x_pxl, + result.used_direct_beam_y_pxl))); if (result.refined_detector_tilt_deg) - Key(os, "REFINED_DETECTOR_TILT", - fmt::format("{:.4f} {:.4f}", (*result.refined_detector_tilt_deg)[0], - (*result.refined_detector_tilt_deg)[1])); - } - os << "\n" - << " The distance and beam centre above are the ones this result was integrated at, which on\n" - << " a rotation run is the post-refined geometry rather than the values in the input file.\n" - << " DETECTOR_TILT is rot1/rot2/rot3 in degrees, as the run INTEGRATED at them - on a rotation\n" - << " run that is a rot1/rot2 rotation indexing fitted, carried over from the first pass, not the\n" - << " value in the input file. REFINED_DETECTOR_TILT below is what the pass named at the end of\n" - << " section 1 fitted for itself, which nothing consumes.\n" - << " BEAM_CENTRE is the PONI and DIRECT_BEAM is where the beam actually lands; they differ by\n" - << " distance*tan(tilt)/pixel and are identical only when the tilt is zero. Quote whichever the\n" - << " program you are feeding expects, and check which one it means.\n"; - if (result.refined_detector_tilt_deg) - os << " REFINED_DETECTOR_TILT is rot1/rot2 in degrees as THIS pass's rotation indexing fitted\n" - << " them, and is not what this pass integrated at: on a two-pass run the tilt above is the\n" - << " FIRST pass's fit, and this is the second pass re-fitting it on its own better geometry.\n" - << " Where the two agree the fit has settled; where they do not, that component is still\n" - << " travelling and neither number is the end of it.\n" - << " How much one sweep determines depends on how far its 2theta reaches. The tilt moves the\n" - << " direct beam exactly as the beam centre does, so at small 2theta what a sweep determines\n" - << " is the DIRECT_BEAM above and not the split between the two; what survives that alias\n" - << " grows as the square of the scattering angle, and on a short-distance long-wavelength\n" - << " sweep it is large enough that the fit reproduces a separately measured tilt to a few\n" - << " hundredths of a degree. Where 2theta is ordinary, compare the MEDIAN of this value over\n" - << " several crystals collected on the same detector rather than one run's: that does track\n" - << " a calibration well enough to show up a placeholder or a stale tilt in the file, and it\n" - << " does not replace the calibration.\n" - << " rot3 is omitted because a rotation about the beam is an exact null of this experiment\n" - << " and the fit cannot move it.\n"; + Add(s, KeyText("REFINED_DETECTOR_TILT", + fmt::format("{:.4f} {:.4f}", (*result.refined_detector_tilt_deg)[0], + (*result.refined_detector_tilt_deg)[1]))); - // The same geometry once more, as the object jfjoch_broker takes it in: the four required - // properties of dataset_settings in broker/jfjoch_api.yaml, spelled the way the API spells them. - // A run that refined the geometry is usually the best measurement of it anyone has, and without - // this the only way back into the instrument is to read two numbers off this report by eye and - // retype them. One line, valid JSON, so a script can lift it with a grep and POST it. - os << "\n"; + // The same geometry once more as the object jfjoch_broker takes it in - the four required + // properties of dataset_settings in broker/jfjoch_api.yaml, spelled the way the API spells + // them. One line, valid JSON, so a script can lift it with a grep and POST it back. + { + constexpr double RAD = PI / 180.0; + const auto &t = result.used_detector_tilt_deg; + std::string rot; + // The rotations belong here whenever they are not zero: a block that omits them describes + // a FLAT detector, which is a different geometry from the one this run used. + if (t[0] != 0.0 || t[1] != 0.0 || t[2] != 0.0) + rot = fmt::format(R"(, "poni_rot1_rad": {:.6f}, "poni_rot2_rad": {:.6f}, )" + R"("poni_rot3_rad": {:.6f})", + t[0] * RAD, t[1] * RAD, t[2] * RAD); + Add(s, KeyText("JFJOCH_DATASET_SETTINGS", + fmt::format(R"({{"beam_x_pxl": {:.2f}, "beam_y_pxl": {:.2f}, )" + R"("detector_distance_mm": {:.3f}, "incident_energy_keV": {:.4f}{}}})", + result.used_beam_x_pxl, result.used_beam_y_pxl, + result.used_distance_mm, + experiment.GetDatasetSettings().GetPhotonEnergy_keV(), rot))); + } + if (result.spot_resolution_estimate_A.has_value()) + Add(s, KeyReal("SPOT_RESOLUTION_ESTIMATE", *result.spot_resolution_estimate_A, "{:.2f}")); + if (result.pass_count > 1) { + Add(s, KeyText("PASS", fmt::format("{} of {}", result.pass_number, result.pass_count))); + Add(s, KeyText("PASS_DECISION", result.pass_decision, true)); + } + + Add(s, Blank()); + // Where the geometry came from. This is the whole provenance story a user needs, because a + // user acts on exactly one geometry: the one the result was produced with. + Add(s, Prose(fmt::format( + " This is the geometry the result was integrated at{}. BEAM_CENTRE is the PONI - the\n" + " foot of the perpendicular from the sample - and DIRECT_BEAM is where the beam actually\n" + " lands; on a tilted detector they are different points. DETECTOR_TILT is rot1/rot2/rot3\n" + " in degrees. JFJOCH_DATASET_SETTINGS is the same geometry as jfjoch_broker takes it,\n" + " ready to POST back for the next collection.", + result.pass_count > 1 ? ", which on a rotation run is the post-refined\n" + " geometry rather than the values in the input file" + : ""))); + if (result.spot_resolution_estimate_A.has_value()) + Add(s, Prose(" SPOT_RESOLUTION_ESTIMATE is how far the data were expected to reach, read off the\n" + " found spots alone - a prediction to about 0.2 A, not what the run achieved.")); + + // The long-form geometry material: correct, and none of it changes what a user does next. + if (result.refined_detector_tilt_deg) + Add(s, Prose(" REFINED_DETECTOR_TILT is rot1/rot2 in degrees as THIS pass's rotation indexing fitted\n" + " them, and is not what this pass integrated at: on a two-pass run the tilt above is the\n" + " FIRST pass's fit, and this is the second pass re-fitting it on its own better geometry.\n" + " Where the two agree the fit has settled; where they do not, that component is still\n" + " travelling and neither number is the end of it.\n" + " How much one sweep determines depends on how far its 2theta reaches. The tilt moves the\n" + " direct beam exactly as the beam centre does, so at small 2theta what a sweep determines\n" + " is the DIRECT_BEAM above and not the split between the two; what survives that alias\n" + " grows as the square of the scattering angle, and on a short-distance long-wavelength\n" + " sweep it is large enough that the fit reproduces a separately measured tilt to a few\n" + " hundredths of a degree. Where 2theta is ordinary, compare the MEDIAN of this value over\n" + " several crystals collected on the same detector rather than one run's: that does track\n" + " a calibration well enough to show up a placeholder or a stale tilt in the file, and it\n" + " does not replace the calibration.\n" + " rot3 is omitted because a rotation about the beam is an exact null of this experiment\n" + " and the fit cannot move it.", true)); + if (result.pass_count > 1) + Add(s, Prose(fmt::format( + " A rotation run integrates twice: once at the geometry in the input file, then again at\n" + " the post-refined geometry. Every number in this report describes the pass named above,\n" + " whose files are {}.*; the header-geometry pass is not written.", output_prefix), true)); + doc.sections.push_back(std::move(s)); + } + + // ------------------------------------------------------------- 2. CRYSTAL { - // The rotations belong here whenever they are not zero: dataset_settings carries - // poni_rot1/2/3_rad, and a block that omits them describes a FLAT detector - a different - // geometry from the one this run used, silently, on every tilted setup. Omitted when zero - // because the API's own default is 0.0, so the shorter block means the same thing. - constexpr double RAD = PI / 180.0; - const auto &t = result.used_detector_tilt_deg; - std::string rot; - if (t[0] != 0.0 || t[1] != 0.0 || t[2] != 0.0) - rot = fmt::format(R"(, "poni_rot1_rad": {:.6f}, "poni_rot2_rad": {:.6f}, )" - R"("poni_rot3_rad": {:.6f})", - t[0] * RAD, t[1] * RAD, t[2] * RAD); - Key(os, "JFJOCH_DATASET_SETTINGS", - fmt::format(R"({{"beam_x_pxl": {:.2f}, "beam_y_pxl": {:.2f}, "detector_distance_mm": {:.3f}, )" - R"("incident_energy_keV": {:.4f}{}}})", - result.used_beam_x_pxl, result.used_beam_y_pxl, result.used_distance_mm, - experiment.GetDatasetSettings().GetPhotonEnergy_keV(), rot)); - } - os << "\n" - << " The geometry above as jfjoch_broker's dataset_settings, to carry a refined beam centre and\n" - << " distance back to the instrument for the next collection. The PONI rotations ride with it\n" - << " when they are non-zero, because without them the block describes a flat detector.\n"; + ReportSection s; + s.title = "2. CRYSTAL"; + if (result.indexing_rate.has_value()) + Add(s, KeyReal("INDEXING_RATE", *result.indexing_rate, "{:.4f}")); + Add(s, KeyBool("LATTICE_FOUND", result.consensus_cell.has_value())); + if (result.consensus_cell.has_value()) + Add(s, KeyText("UNIT_CELL_CONSTANTS", CellString(*result.consensus_cell))); + if (result.space_group.has_value()) { + Add(s, KeyInt("SPACE_GROUP_NUMBER", result.space_group->number)); + // The number alone does not name the setting, and the search can return one that is not + // gemmi's reference setting - P 1 1 2_1 and P 1 2_1 1 are both number 4, different axes. + Add(s, KeyText("SPACE_GROUP_NAME", result.space_group->xhm())); + } + // This fired on every --mode scale run, falsely: that path never fills indexing_rate, because + // the lattice comes from the input file, and a missing optional read as a measured zero. + if (result.indexing_rate.has_value() && *result.indexing_rate <= 0.0f) + Warn(doc, PathologyCode::NO_LATTICE, + "No image indexed - no crystal lattice was determined from this dataset"); - if (result.spot_resolution_estimate_A.has_value()) { - os << "\n"; - Key(os, "SPOT_RESOLUTION_ESTIMATE", fmt::format("{:.2f}", *result.spot_resolution_estimate_A)); - os << "\n" - << " How far the merged data are expected to reach, read off the found spots alone - no\n" - << " lattice, no integration, no merge. It is a prediction, good to about 0.2 A on the\n" - << " rotation data it was calibrated on, and it is not what the run achieved: compare it\n" - << " with INCLUDE_RESOLUTION_RANGE in section 5. It is not limited to what this detector\n" - << " records: where it reads finer than the high-resolution end of that range, the crystal\n" - << " diffracts past the corner and the run is detector-limited.\n"; - } - - if (result.pass_count > 1) { - os << "\n"; - Key(os, "PASS", fmt::format("{} of {}", result.pass_number, result.pass_count)); - Key(os, "PASS_DECISION", result.pass_decision); - os << "\n" - << " A rotation run integrates twice: once at the geometry in the input file, then again at\n" - << " the post-refined geometry. Every number in this report describes the pass named above,\n" - << " whose files are " << output_prefix << ".*; the header-geometry pass is not written.\n"; - } - - // ---------------------------------------------------------------- 2. INDEXING - Section(os, "2. INDEXING"); - if (result.indexing_rate.has_value()) - Key(os, "INDEXING_RATE", fmt::format("{:.4f}", result.indexing_rate.value())); - Key(os, "LATTICE_FOUND", (result.consensus_cell.has_value() ? "TRUE" : "FALSE")); - if (result.consensus_cell.has_value()) - Key(os, "UNIT_CELL_CONSTANTS", CellString(*result.consensus_cell)); - if (result.space_group.has_value()) { - Key(os, "SPACE_GROUP_NUMBER", result.space_group->number); - // The number alone does not name the setting, and the search can now return one that is not - // gemmi's reference setting - P 1 1 2_1 and P 1 2_1 1 are both number 4, on different axes. - Key(os, "SPACE_GROUP_NAME", result.space_group->xhm()); - } - if (result.indexing_rate.value_or(0.0f) <= 0.0f) - warnings.emplace_back("No image indexed - no crystal lattice was determined from this dataset"); - - // ---------------------------------------------- 3. GEOMETRY POST-REFINEMENT - if (result.post_refine.has_value()) { - const auto &pr = *result.post_refine; - Section(os, "3. GEOMETRY POST-REFINEMENT"); - os << " The rotation two-pass fits the crystal (orientation, cell, rotation axis) and the detector\n" - << " (distance, beam centre) together, against the observed spot positions and the observed\n" - << " rocking angles at once. It is committed only if it improves a held-out residual.\n\n"; - Key(os, "POSTREFINE_EVENTS_USED", pr.events_used); - Key(os, "POSTREFINE_OBS_USED", pr.obs_used); - Key(os, "POSTREFINE_CELL_COMMITTED", pr.cell_refined ? "TRUE" : "FALSE"); - Key(os, "POSTREFINE_DETECTOR_COMMITTED", pr.detector_refined ? "TRUE" : "FALSE"); - Key(os, "POSTREFINE_DISTANCE", fmt::format("{:.3f} -> {:.3f}", pr.distance_before_mm, - pr.distance_after_mm)); - Key(os, "POSTREFINE_BEAM_CENTRE", fmt::format("{:.2f} {:.2f} -> {:.2f} {:.2f}", - pr.beam_x_before_px, pr.beam_y_before_px, - pr.beam_x_after_px, pr.beam_y_after_px)); - Key(os, "GONIOMETER_ROTATION_SCALE", fmt::format("{:.5f}", pr.rotation_scale)); - Key(os, "GONIOMETER_ROTATION_SCALE_SUSPECT", pr.rotation_scale_suspect ? "TRUE" : "FALSE"); - os << "\n GONIOMETER_ROTATION_SCALE is the factor by which the stage actually turned relative to\n" - << " the angles stored in the file (which are the commanded ones). 1.0 = they agree. It drives\n" - << " the second integration pass only when SUSPECT is TRUE - both cross-validated and outside\n" - << " the tolerance - since a stage that is in fact well calibrated must be left alone. A\n" - << " manual --rotation-scale replaces it and is applied to both passes.\n"; - if (pr.rotation_scale_suspect) - warnings.emplace_back(fmt::format( - "The goniometer turned by a factor {:.5f} of the angles stored in the file - the " - "stage rotation looks mis-calibrated by {:+.2f}%. The correction was applied to this " - "run, but the fault is in the hardware and should be fixed there", - pr.rotation_scale, 100.0 * (pr.rotation_scale - 1.0))); - } - - // ---------------------------------------------- 4. SPACE GROUP DETERMINATION - Section(os, "4. SPACE GROUP DETERMINATION"); - - // What the data could NOT decide, as keys rather than only as prose. SPACE_GROUP_NAME above is a - // scalar and reads like a determination; where several groups predict the same absences it is one - // of them, chosen by convention. A script that greps only the name records a coin-flip as an - // answer - measured, refining against the deposited model in the wrong enantiomorph gives - // R = 0.549 - so the ambiguity has to survive the same grep. - { + // What the data could NOT decide, beside what they did. SPACE_GROUP_NAME is a scalar and reads + // like a determination; where several groups predict the same absences it is one of them, + // chosen by convention, so the ambiguity has to survive the same grep. std::string alts; if (result.space_group_search.has_value() && result.space_group.has_value()) for (const auto &alt : result.space_group_search->alternatives) if (alt.number != result.space_group->number) alts += (alts.empty() ? "" : " | ") + alt.xhm(); - Key(os, "SPACE_GROUP_ALTERNATIVES", alts.empty() ? "NONE" : alts); + Add(s, KeyText("SPACE_GROUP_ALTERNATIVES", alts.empty() ? "NONE" : alts)); - // The hand is a separate question from the alternatives list, and it is decidable from the - // 22 groups' numbers alone, so it is answered even where the search did not run. Merged - // intensities never decide it: an enantiomorphic pair has the same absences and the same - // Laue class, and only a model, a substructure or an anomalous signal names the hand. - const char *enantiomorph = "NOT_APPLICABLE"; - if (result.space_group.has_value() && result.space_group->is_enantiomorphic()) { + // The hand is a separate question from the alternatives, decidable from the 22 groups' numbers + // alone. Written only where the group HAS an enantiomorphic partner: on every other group the + // answer is NOT_APPLICABLE, which answers a question nobody asked. + const bool enantiomorphic = result.space_group.has_value() + && result.space_group->is_enantiomorphic(); + if (!enantiomorphic) + Add(s, KeyEnum("SPACE_GROUP_ENANTIOMORPH", "NOT_APPLICABLE", + {"NOT_APPLICABLE", "UNDETERMINED", "GIVEN", "ASSUMED_FROM_MODEL"}, true)); + if (enantiomorphic) { + const char *enantiomorph = "UNDETERMINED"; + const auto &mv = result.model_validation; // ASSUMED, not determined: nothing here measured the hand. A model that fits carries prior // chemical knowledge these intensities do not - and cannot, since |Fcalc| is invariant - // under the change of hand - so taking its group is an assertion, not a measurement. It is - // only made where the model was shown to fit (section 10), and it names the hand either by - // the group actually written being the model's or by the model's having been adopted for it. - const auto &mv = result.model_validation; + // under the change of hand - so taking its group is an assertion, not a measurement. if (mv.has_value() && mv->ok && mv->model_fits && (mv->adopted_model_enantiomorph || mv->model_space_group_number == result.space_group->number)) enantiomorph = "ASSUMED_FROM_MODEL"; else if (!result.space_group_search.has_value()) enantiomorph = "GIVEN"; // -S, or a reference MTZ: the user's assertion - else - enantiomorph = "UNDETERMINED"; + Add(s, KeyEnum("SPACE_GROUP_ENANTIOMORPH", enantiomorph, + {"UNDETERMINED", "GIVEN", "ASSUMED_FROM_MODEL"})); } - Key(os, "SPACE_GROUP_ENANTIOMORPH", enantiomorph); - // A higher point group whose operators the intensities confirmed and whose promotion was - // refused. It exists only as prose in the search text below, so nothing can act on it - and - // a refusal is exactly the case where a user might want to try the higher group as well. + // refused - written only when there was one, because a refusal is exactly the case where a + // user might want to try the higher group as well. const bool refused = result.space_group_search.has_value() && !result.space_group_search->refused_point_group_hm.empty(); - Key(os, "SPACE_GROUP_REFUSED_POINT_GROUP", - refused ? result.space_group_search->refused_point_group_hm : std::string("NONE")); - if (refused) - Key(os, "SPACE_GROUP_REFUSED_REASON", result.space_group_search->refused_reason); - // How the crystal sat on the spindle. Until now this reached only stdout unless it crossed the - // warning threshold, so the ordinary "the mounting was fine" case - the one a user needs when - // deciding whether an incomplete cusp is the mounting's fault or the data's - was not greppable. - if (result.spindle_symmetry_axis_deg.has_value()) { - Key(os, "SPINDLE_SYMMETRY_AXIS_ANGLE_DEG", - fmt::format("{:.1f}", *result.spindle_symmetry_axis_deg)); - Key(os, "SPINDLE_SYMMETRY_AXIS_ORDER", std::to_string(result.spindle_symmetry_axis_order)); + if (refused) { + Add(s, KeyText("SPACE_GROUP_REFUSED_POINT_GROUP", + result.space_group_search->refused_point_group_hm)); + Add(s, KeyText("SPACE_GROUP_REFUSED_REASON", result.space_group_search->refused_reason)); + } else { + Add(s, KeyText("SPACE_GROUP_REFUSED_POINT_GROUP", "NONE", true)); } - // The exact verdict on the mounting, which the angle above cannot give on its own: the - // fraction (0-1) of unique reflections, to this run's resolution limit, that the measured - // point group could not recover from the sweep's blind cone. 0.0000 means the mounting - // cost nothing. Machine-readable on purpose - a downstream pipeline decides on this - // number, not on the warning prose. - if (result.spindle_lost_unique_fraction.has_value()) - Key(os, "SPINDLE_LOST_UNIQUE_FRACTION", - fmt::format("{:.4f}", *result.spindle_lost_unique_fraction)); - os << "\n SPACE_GROUP_ALTERNATIVES names every group these data cannot separate from the one\n" - << " adopted (enantiomorphic partners, origin-ambiguous pairs, groups a gap in the data\n" - << " leaves untested); NONE means the absences single the answer out. SPACE_GROUP_ENANTIOMORPH\n" - << " is UNDETERMINED whenever the group is one of the 22 that come in enantiomorphic pairs and\n" - << " nothing outside the merged intensities named the hand - the intensities cannot, so this\n" - << " is the normal outcome, not a failure. SPACE_GROUP_REFUSED_POINT_GROUP is a higher point\n" - << " group the operator correlations supported and the consistency tests would not take.\n"; - } - - if (result.space_group_search.has_value()) { - Key(os, "SPACE_GROUP_SEARCH", "DE_NOVO"); - os << "\n" << SearchSpaceGroupResultToText(*result.space_group_search) << "\n"; - // A centering the data could not test must not read like one they confirmed. The group may - // still be right - the lattice metric says so - but nothing in these intensities backs it, - // and that belongs beside the warnings rather than in a table column alone. - const auto &search = *result.space_group_search; - if (search.best_space_group.has_value()) - for (const auto &c : search.candidates) - if (c.space_group.number == search.best_space_group->number && c.centering_untested) - warnings.emplace_back(fmt::format( - "The {} centering of {} was NOT confirmed from these data: the crystal was " - "indexed and integrated on the primitive sub-cell, so the reflections a " - "{}-centred lattice extinguishes are not in this merge at all. It comes " - "from the lattice metric. The point group is confirmed from the " - "intensities; the centering is not", - search.best_space_group->centring_type(), - search.best_space_group->short_name(), - search.best_space_group->centring_type())); - } else if (result.space_group.has_value()) { - Key(os, "SPACE_GROUP_SEARCH", "FIXED"); - os << "\n The space group was given, not determined here.\n"; - } else { - Key(os, "SPACE_GROUP_SEARCH", "NONE"); - os << "\n No space group was determined.\n"; - } - - // ---------------------------------------------------- 5. SCALING AND MERGING - Section(os, "5. SCALING AND MERGING"); - if (!merged) { - Key(os, "MERGE", "NOT_PERFORMED"); - os << "\n No scaling or merging was performed on this run, so there are no merging statistics, no\n" - << " error model, and no sweep-quality diagnosis below. The integrated reflections are in\n" - << " " << output_prefix << "_process.h5.\n"; - } else { - const auto &o = result.merge_statistics.overall; - Key(os, "MERGE", "PERFORMED"); - Key(os, "INCLUDE_RESOLUTION_RANGE", fmt::format("{:.3f} {:.3f}", o.d_max, o.d_min)); - if (result.resolution_fit_A) - Key(os, "FITTED_RESOLUTION", fmt::format("{:.2f}", *result.resolution_fit_A)); - Key(os, "FRIEDELS_LAW", experiment.GetScalingSettings().GetMergeFriedel() ? "TRUE" : "FALSE"); - Key(os, "UNIQUE_REFLECTIONS", o.unique_reflections); - Key(os, "TOTAL_OBSERVATIONS", o.total_observations); - // Outlier rejection drops observations from the merge AND from R_meas and the CC(1/2) - // half-sets, so a run that rejects too much scores BETTER on every other number in this - // block. Without this key that failure is silent, and it is not hypothetical: a merge whose - // rejection rule was wrong at low multiplicity dropped observations that halved the - // correlation with an external reference while its own R_meas improved. - Key(os, "OBSERVATIONS_REJECTED", result.merge_statistics.n_observations_rejected); - // One rule for every quantity here: a run that did not measure it writes NO key, rather than - // the word "nan" or a zero that reads as a measured absence. SIGANO is the common case - a - // Friedel-merged run splits no Bijvoet pair - and it sat one line from CC_ANOM, which already - // did this, reporting the same missing quantity two different ways. - if (o.possible_unique_reflections > 0) - Key(os, "COMPLETENESS", - fmt::format("{:.1f}", 100.0 * o.unique_reflections / o.possible_unique_reflections)); - if (o.unique_reflections > 0) - Key(os, "MULTIPLICITY", - fmt::format("{:.2f}", static_cast(o.total_observations) / o.unique_reflections)); - if (std::isfinite(o.mean_i_over_sigma)) - Key(os, "I_OVER_SIGMA", fmt::format("{:.2f}", o.mean_i_over_sigma)); - if (std::isfinite(o.r_meas)) - Key(os, "R_MEAS", fmt::format("{:.4f}", o.r_meas)); - if (std::isfinite(o.cc_half)) - Key(os, "CC_HALF", fmt::format("{:.4f}", o.cc_half)); - if (std::isfinite(o.abs_diff_over_sigma_anomalous)) - Key(os, "SIGANO", fmt::format("{:.3f}", o.abs_diff_over_sigma_anomalous)); - if (std::isfinite(o.cc_anom)) - Key(os, "CC_ANOM", fmt::format("{:.4f}", o.cc_anom)); - if (std::isfinite(result.merge_statistics.wilson_b)) - Key(os, "WILSON_B", fmt::format("{:.2f}", result.merge_statistics.wilson_b)); - // The error model in XDS's convention, so the numbers are directly comparable with a CORRECT.LP. - Key(os, "ERROR_MODEL_A", fmt::format("{:.4f}", result.error_model_a)); - Key(os, "ERROR_MODEL_B", fmt::format("{:.4e}", result.error_model_b)); - Key(os, "ISA", fmt::format("{:.2f}", result.error_model_isa)); - if (result.error_model_isa_asymptotic > 0.0) - Key(os, "ISA_ASYMPTOTIC", fmt::format("{:.2f}", result.error_model_isa_asymptotic)); - Key(os, "REFERENCE_DATA_USED", result.has_reference ? "TRUE" : "FALSE"); - // The shell table straight off the statistics rather than result.merge_statistics_text: that - // string also carries the twinning analysis and the advisories, which have sections of their own. - os << "\n"; - if (result.resolution_fit_A) - os << fmt::format( - " INCLUDE_RESOLUTION_RANGE is the range the reflections were WRITTEN to;" - " FITTED_RESOLUTION\n is where the CC1/2 fall-off crosses {:.2f}, and is the number to" - " quote. The data are kept one\n shell past it on purpose: a shell that is included can" - " still be downweighted or dropped by\n refinement, while one that was truncated cannot" - " be put back.\n\n", - experiment.GetScalingSettings().GetResolutionCCTarget()); - os << " CC_ANOM is the anomalous difference measured twice - I(+)-I(-) from one half of the\n" - << " observations against the same difference from the other half - and correlated over the\n" - << " acentric pairs where both hands were measured at least twice. It is what says whether\n" - << " there is an anomalous signal to phase on, and unlike SIGANO it is not a ratio against\n" - << " the error model, so an optimistic sigma cannot inflate it. It agrees with AIMLESS's\n" - << " CCanom and phenix.merging_statistics' cc_anom. XDS's CORRECT.LP has a column named\n" - << " `Anomal Corr` which is NOT this quantity and reads considerably higher at low\n" - << " resolution, so the two are not comparable.\n" - << " A negative value is a measurement, not an error: on data with little anomalous signal\n" - << " and around two observations per Bijvoet mate the statistic is unstable and goes\n" - << " negative, in this program and in the others alike. Where no pair could be split in\n" - << " both hands the quantity does not exist: the key is then absent here and the column is\n" - << " a dash in the table below, which is not the same claim as a signal measured to be\n" - << " zero.\n\n" - << " ERROR_MODEL_A / ERROR_MODEL_B are in XDS's convention, sigma^2 = a*(sigma0^2 + b*I^2),\n" - << " so ISA = 1/sqrt(a*b) means what CORRECT.LP's ISa means. ISA_ASYMPTOTIC, where present,\n" - << " is the strong-reflection tier only.\n\n" - << result.merge_statistics; - } - - // --------------------------------------------------------------- 6. TWINNING - if (merged && result.twinning.l_test_pairs > 0) { - Section(os, "6. TWINNING"); - Key(os, "TWINNING_SUSPECTED", result.twinning.twinning_suspected ? "TRUE" : "FALSE"); - Key(os, "L_TEST_MEAN_ABS_L", fmt::format("{:.4f}", result.twinning.mean_abs_l)); - Key(os, "L_TEST_MEAN_L_SQUARED", fmt::format("{:.4f}", result.twinning.mean_l_squared)); - Key(os, "SECOND_MOMENT_I", fmt::format("{:.4f}", result.twinning.second_moment)); - // The same two statistics measured before the group was chosen. Printed next to the ones above - // rather than instead of them, because they answer different questions and can disagree: these - // are the only ones a promotion cannot have contaminated. - if (result.pre_promotion_twinning) { - Key(os, "L_TEST_MEAN_ABS_L_BEFORE_SEARCH", - fmt::format("{:.4f}", result.pre_promotion_twinning->mean_abs_l)); - Key(os, "SECOND_MOMENT_I_BEFORE_SEARCH", - fmt::format("{:.4f}", result.pre_promotion_twinning->second_moment)); + if (result.post_refine.has_value()) { + const auto &pr = *result.post_refine; + Add(s, KeyInt("POSTREFINE_EVENTS_USED", pr.events_used)); + Add(s, KeyInt("POSTREFINE_OBS_USED", pr.obs_used)); + Add(s, KeyBool("POSTREFINE_CELL_COMMITTED", pr.cell_refined)); + Add(s, KeyBool("POSTREFINE_DETECTOR_COMMITTED", pr.detector_refined)); + Add(s, KeyText("POSTREFINE_DISTANCE", fmt::format("{:.3f} -> {:.3f}", pr.distance_before_mm, + pr.distance_after_mm))); + Add(s, KeyText("POSTREFINE_BEAM_CENTRE", + fmt::format("{:.2f} {:.2f} -> {:.2f} {:.2f}", pr.beam_x_before_px, + pr.beam_y_before_px, pr.beam_x_after_px, pr.beam_y_after_px))); + Add(s, KeyReal("GONIOMETER_ROTATION_SCALE", pr.rotation_scale, "{:.5f}")); + Add(s, KeyBool("GONIOMETER_ROTATION_SCALE_SUSPECT", pr.rotation_scale_suspect)); + if (pr.rotation_scale_suspect) + Warn(doc, PathologyCode::GONIO_SCALE, fmt::format( + "The goniometer turned by a factor {:.5f} of the angles stored in the file - the " + "stage rotation looks mis-calibrated by {:+.2f}%. The correction was applied to " + "this run, but the fault is in the hardware and should be fixed there", + pr.rotation_scale, 100.0 * (pr.rotation_scale - 1.0))); } - Key(os, "ESTIMATED_TWIN_FRACTION", fmt::format("{:.3f}", result.twinning.estimated_twin_fraction)); - os << "\n" << TwinningAnalysisToText(result.twinning) << "\n"; - if (result.pre_promotion_twinning) - os << " The _BEFORE_SEARCH pair was measured on the merge the space-group search was given,\n" - << " before any point group was adopted. That ordering is the whole point: the statistics\n" - << " above are computed in the Laue class this run ADOPTED, so if the search promoted the\n" - << " point group they can only report that no twin law exists inside the class it chose -\n" - << " and a twin is precisely what would have caused that promotion. Where the two pairs\n" - << " disagree, believe the _BEFORE_SEARCH one about whether the crystal is twinned.\n"; - if (result.twinning.twinning_suspected) - warnings.emplace_back(fmt::format( - "Twinning is indicated (<|L|> = {:.3f}, /^2 = {:.3f}, estimated twin " - "fraction {:.2f}) - refine against the merged data with care", - result.twinning.mean_abs_l, result.twinning.second_moment, - result.twinning.estimated_twin_fraction)); + + Add(s, KeyEnum("SPACE_GROUP_SEARCH", + result.space_group_search.has_value() ? "DE_NOVO" + : result.space_group.has_value() ? "FIXED" : "NONE", + {"DE_NOVO", "FIXED", "NONE"})); + + Add(s, Blank()); + if (result.space_group_search.has_value()) { + const auto &search = *result.space_group_search; + // The compact form, composed here from the search result rather than taken from its text + // helper, so the default report says what was decided and on what margin without the + // operator and candidate tables that justify it. + std::string line = fmt::format(" Point group {} from the intensity correlations", + search.point_group_hm); + if (std::isfinite(search.h_ratio)) + line += fmt::format(";\n twin-law H ratio {:.2f} against a bound of {:.2f}", + search.h_ratio, search.h_ratio_bound); + line += ".\n The space group follows from the systematic absences."; + if (!alts.empty()) + line += fmt::format("\n These data cannot separate it from {} - both are consistent with" + " every absence\n measured here.", alts); + Add(s, Prose(line)); + if (enantiomorphic && !alts.empty() && alts.find('|') == std::string::npos) + Add(s, Prose(" The pair differs only in the hand of the screw axis, which merged intensities\n" + " cannot name. Try both in molecular replacement and keep whichever refines -\n" + " that is the normal outcome for a chiral group, not a failure of the run.")); + if (refused) + Add(s, Prose(fmt::format( + " A promotion to {} was refused: {}.\n" + " If the higher symmetry is expected here, it is worth re-running with -S to force\n" + " it and comparing.", search.refused_point_group_hm, search.refused_reason))); + // The full evidence - operator correlations, the ranked candidate table, the screw + // conditions zone by zone - is what a person diagnosing a symmetry call needs and nothing + // a person deciding whether to recollect can act on. + Add(s, Prose("\n" + SearchSpaceGroupResultToText(search), true)); + + // A centering the data could not test must not read like one they confirmed. The group may + // still be right - the lattice metric says so - but nothing in these intensities backs it. + if (search.best_space_group.has_value()) + for (const auto &c : search.candidates) + if (c.space_group.number == search.best_space_group->number && c.centering_untested) + Warn(doc, PathologyCode::CENTERING_UNTESTED, fmt::format( + "The {} centering of {} was NOT confirmed from these data: the crystal was " + "indexed and integrated on the primitive sub-cell, so the reflections a " + "{}-centred lattice extinguishes are not in this merge at all. It comes " + "from the lattice metric. The point group is confirmed from the " + "intensities; the centering is not", + search.best_space_group->centring_type(), + search.best_space_group->short_name(), + search.best_space_group->centring_type())); + } else if (result.space_group.has_value()) { + Add(s, Prose(" The space group was given, not determined here.")); + } else { + Add(s, Prose(" No space group was determined.")); + } + doc.sections.push_back(std::move(s)); } - // ------------------------------------------------------- 7. RADIATION DAMAGE - if (!result.radiation_damage_text.empty()) { - Section(os, "7. RADIATION DAMAGE"); - // A number, or a word saying why there is none: NOT_A_TREND where the per-batch curve was measured - // but no straight line describes it (damage is progressive, so that curve is not dose), NOT_MEASURED - // where the monitor could not run at all. - const double db = result.merge_statistics.radiation_damage_delta_b; - Key(os, "RADIATION_DAMAGE_RELATIVE_B", - std::isfinite(db) ? fmt::format("{:.2f}", db) - : result.merge_statistics.radiation_damage_b_batch.empty() ? std::string("NOT_MEASURED") - : std::string("NOT_A_TREND")); - os << "\n" << result.radiation_damage_text << "\n"; - } - - // ------------------------------------------------------ 8. SWEEP QUALITY - const auto &sq = result.merge_statistics.sweep_quality; - Section(os, "8. SWEEP QUALITY"); - os << " Stretches of the sweep over which the crystal delivered much less than the rest of the run.\n" - << " REASON comes from a closed vocabulary, listed below so a consumer can tell an unknown code\n" - << " from a missing one. SEVERITY is the fraction of the run's typical diffracting power missing\n" - << " over the range (0 = as good as the run, 1 = nothing at all); SCALE and CC are the range's\n" - << " mean per-image scale and CC-to-merge relative to the run median; INDEXED is the fraction of\n" - << " the range's frames that were scaled at all. Nothing is excluded on the strength of this.\n\n"; - Key(os, "SWEEP_QUALITY_STATUS", sq.measured ? "COMPUTED" : "NOT_COMPUTED"); - Key(os, "SWEEP_QUALITY_COUNT", sq.ranges.size()); + // -------------------------------------------------------- 3. MERGED DATA + // Facts the SUMMARY needs, measured while this section is built. + bool unusable_merge = false; + bool completeness_narrower = false; // the signal range is coarser than what was written + CompletenessInRange completeness_fit; + std::optional fitted_resolution; { - std::string codes; - for (int r = 0; r <= static_cast(SweepQualityReason::RadiationDamage); ++r) - codes += (codes.empty() ? "" : " ") - + std::string(SweepQualityReasonCode(static_cast(r))); - Key(os, "SWEEP_QUALITY_REASONS", codes); - } - if (sq.measured) { - Key(os, "SWEEP_ROTATION", fmt::format("{:.1f}", sq.sweep_deg)); - Key(os, "FLUX_PEAK_TO_TROUGH", fmt::format("{:.2f}", sq.flux_peak_to_trough)); - Key(os, "SCALE_MODULATION_PEAK_TO_TROUGH", fmt::format("{:.2f}", sq.modulation_peak_to_trough)); - } - os << "\n" - << " FIRST_IMAGE LAST_IMAGE N_IMAGES ROTATION REASON SEVERITY SCALE CC INDEXED\n" - << " ----------- ----------- --------- -------- -------------------- -------- ------ ------ --------\n"; - for (const auto &r : sq.ranges) { - os << fmt::format(" {:11d} {:11d} {:9d} {:8.1f} {:<20} {:8.2f} {:6.2f} {:6.2f} {:8.2f}\n", - r.first_image, r.last_image, r.last_image - r.first_image + 1, r.rotation_deg, - SweepQualityReasonCode(r.reason), r.severity, r.mean_relative_scale, - r.mean_relative_cc, r.indexed_fraction); - warnings.push_back(fmt::format( - "Frames {}-{} {} ({:.1f} deg, scale {:.2f} and CC {:.2f} of the run, {:.0f}% scaled)", - r.first_image, r.last_image, SweepQualityReasonText(r.reason), r.rotation_deg, - r.mean_relative_scale, r.mean_relative_cc, 100.0 * r.indexed_fraction)); - } - os << " ----------- ----------- --------- -------- -------------------- -------- ------ ------ --------\n"; + ReportSection s; + s.title = "3. MERGED DATA"; + if (!merged) { + Add(s, KeyEnum("MERGE", "NOT_PERFORMED", {"PERFORMED", "NOT_PERFORMED"})); + Add(s, Blank()); + Add(s, Prose(fmt::format( + " No scaling or merging was performed on this run, so there are no merging statistics,\n" + " no error model and no sweep-quality diagnosis. The integrated reflections are in\n" + " {}_process.h5.", output_prefix))); + } else { + const auto &ms = result.merge_statistics; + const auto &o = ms.overall; + const double cc_target = experiment.GetScalingSettings().GetResolutionCCTarget(); - // ---------------------------------------------------------- 9. DIFFRACTION ANISOTROPY - const auto &an = result.merge_statistics.anisotropy; - if (merged && an.n_reflections > 0) { - Section(os, "9. DIFFRACTION ANISOTROPY"); - os << " How much the fall-off depends on direction, and whether that is established above this\n" - << " data set's own systematic error. Nothing here corrects an intensity or removes a\n" - << " reflection: the merged data and the written files do not depend on direction at all.\n" - << " ANISOTROPY_DELTA_B is the range of the principal components of the anisotropy tensor,\n" - << " on the ordinary crystallographic B scale (the same scale as phenix.xtriage's B_cart and\n" - << " ctruncate's anisotropic B), fitted on intensities with nothing dropped;\n" - << " ANISOTROPY_SIGNIFICANCE gates ANISOTROPY_DELTA_B_LINEAR, the part of it that follows\n" - << " exp(-1/2 s^T B s), which is not the same number. A 1 in ANISOTROPY_D_MIN_CENSORED marks\n" - << " a direction whose limit is the edge of the measured data rather than the crystal's own.\n\n"; - Key(os, "ANISOTROPY_VERDICT", AnisotropyVerdictCode(an.verdict)); - Key(os, "ANISOTROPY_FREE_DIRECTIONS", an.n_free_parameters); - Key(os, "ANISOTROPY_DELTA_B", fmt::format("{:.2f}", an.delta_b)); - Key(os, "ANISOTROPY_DELTA_B_LINEAR", fmt::format("{:.2f}", an.delta_b_linear)); - Key(os, "ANISOTROPY_PRINCIPAL_B", fmt::format("{:.2f} {:.2f} {:.2f}", - an.eigenvalue[0] - an.eigenvalue[2], - an.eigenvalue[1] - an.eigenvalue[2], 0.0)); - Key(os, "ANISOTROPY_FOLD_WEAKENING", fmt::format("{:.1f}", an.fold_weakening)); - Key(os, "ANISOTROPY_D_MIN_PRINCIPAL", fmt::format("{:.2f} {:.2f} {:.2f}", an.d_min_axis[0], - an.d_min_axis[1], an.d_min_axis[2])); - Key(os, "ANISOTROPY_D_MIN_CENSORED", fmt::format("{} {} {}", an.d_min_censored[0] ? 1 : 0, - an.d_min_censored[1] ? 1 : 0, - an.d_min_censored[2] ? 1 : 0)); - Key(os, "ANISOTROPY_D_MIN_SPREAD", fmt::format("{:.2f}", an.d_min_spread)); - // The finest of the three principal limits, on its own line so it can be grepped. The cut this - // run applied is isotropic, so on an anisotropic crystal the outer shells are complete in count - // and empty in signal along the weak directions; this says how far the crystal actually reaches - // where it reaches furthest. - // Skipping the non-finite entries, as the warning block below does: a cone too sparse to - // cross the threshold leaves its limit NaN, and every comparison against NaN is false, so a - // bare min_element returns element 0 and prints nan even where the other two are measured. - const double *best_axis = nullptr; - for (const double *p = an.d_min_axis; p != an.d_min_axis + 3; ++p) - if (std::isfinite(*p) && (!best_axis || *p < *best_axis)) - best_axis = p; - if (best_axis) - Key(os, "ANISOTROPY_D_MIN_BEST", fmt::format("{:.2f}", *best_axis)); - Key(os, "ANISOTROPY_SHAPE", AnisotropyShapeCode(an.shape)); - Key(os, "ANISOTROPY_SHAPE_INTERCEPT", fmt::format("{:.3f}", an.shape_intercept)); - Key(os, "ANISOTROPY_SHAPE_INTERCEPT_Z", fmt::format("{:.1f}", an.shape_intercept_z)); - Key(os, "ANISOTROPY_SHAPE_SLOPE", fmt::format("{:.2f}", an.shape_slope)); - Key(os, "ANISOTROPY_SHAPE_RESIDUAL", fmt::format("{:.1f}", an.shape_residual)); - Key(os, "ANISOTROPY_N_OBSERVATIONS", an.n_observations); - Key(os, "ANISOTROPY_SIGMA_SYSTEMATIC", fmt::format("{:.3f}", an.sigma_systematic)); - Key(os, "ANISOTROPY_FORBIDDEN_Z", fmt::format("{:.1f}", an.forbidden_z)); - Key(os, "ANISOTROPY_FLOOR", fmt::format("{:.3f}", an.floor)); - Key(os, "ANISOTROPY_SIGNIFICANCE", fmt::format("{:.2f}", an.significance)); - Key(os, "ANISOTROPY_DETECTION_LIMIT", fmt::format("{:.2f}", an.detection_limit)); - os << "\n" << AnisotropyToText(an) << "\n"; - // Three different resolutions appear in this report and they answer three different questions. - // Say so here rather than let a reader assume one of them is "the" resolution. - if (result.resolution_fit_A) - os << fmt::format( - " Three resolutions, three questions. INCLUDE_RESOLUTION_RANGE ({:.2f} A) is what was\n" - " WRITTEN, deliberately one shell past the fit. FITTED_RESOLUTION ({:.2f} A) is where\n" - " the isotropic CC1/2 fall-off crosses its target - the overall measurement.\n" - " ANISOTROPY_D_MIN_BEST ({:.2f} A) is how far the crystal reaches along its strongest\n" - " direction. It is read off the fitted tensor, not off the cut, so it can land either\n" - " side of the other two - and where ANISOTROPY_D_MIN_CENSORED flags that direction it\n" - " is the edge of the measured data rather than the crystal's limit. None of the three\n" - " is wrong; quoting one without saying which it is, is.\n\n", - result.merge_statistics.overall.d_min, *result.resolution_fit_A, - *std::min_element(an.d_min_axis, an.d_min_axis + 3)); - // CC1/2 that falls and then climbs again is not a fall-off, so the single number the cut is read - // off does not describe these data. Reported as an observation, with no advice to re-cut: the - // generous cut is deliberate and refinement can downweight what it does not want. - { - // A climb only counts while the shell still carries signal: past CC1/2 ~ 0.3 the data are - // beyond any cut this run would take and the number is oscillating in noise, which says - // nothing about the fall-off. 0.05 is well above that surviving noise. - constexpr double CC_HALF_RISE = 0.05, CC_HALF_ALIVE = 0.30; - const auto &sh = result.merge_statistics.shells; - size_t fell = 0, rose_after = 0; - for (size_t i = 1; i < sh.size(); ++i) { - if (sh[i].cc_half < sh[i - 1].cc_half) fell = i; - else if (fell > 0 && sh[i].cc_half > sh[i - 1].cc_half + CC_HALF_RISE - && sh[i].cc_half > CC_HALF_ALIVE) rose_after = i; + Add(s, KeyEnum("MERGE", "PERFORMED", {"PERFORMED", "NOT_PERFORMED"})); + Add(s, KeyText("INCLUDE_RESOLUTION_RANGE", fmt::format("{:.3f} {:.3f}", o.d_max, o.d_min))); + + // FITTED_RESOLUTION is presented by this report as "the number to quote", so it must not + // be quotable when the curve it was read off never behaved like a fall-off. A fit finer + // than the finest shell that still reaches the target asserts a resolution the shell + // table on this very page refutes. + const double finest_supported = FinestSupportedShell(ms, cc_target); + std::string fit_suppressed; + // Only where there are shells to judge the fit against; with no table there is no + // evidence either way, and suppressing on no evidence is its own false claim. + if (result.resolution_fit_A && !ms.shells.empty()) { + if (finest_supported == 0.0) + fit_suppressed = fmt::format( + "CC1/2 never reaches the cut's target of {:.2f} in any shell, so there is no" + " fall-off to fit", cc_target); + else if (*result.resolution_fit_A < finest_supported - 1e-6) + fit_suppressed = fmt::format( + "the fit reads {:.2f} A, finer than the finest shell whose CC1/2 still reaches" + " {:.2f} ({:.2f} A), so the curve it was read off is not a fall-off", + *result.resolution_fit_A, cc_target, finest_supported); + else + fitted_resolution = *result.resolution_fit_A; } - if (rose_after > 0) - warnings.emplace_back(fmt::format( - "CC1/2 is not monotone with resolution (it climbs again at {:.2f}-{:.2f} A) - the " - "fall-off the resolution cut is read off does not describe these data", - sh[rose_after].d_max, sh[rose_after].d_min)); + if (fitted_resolution) + Add(s, KeyReal("FITTED_RESOLUTION", *fitted_resolution, "{:.2f}")); + + Add(s, KeyBool("FRIEDELS_LAW", experiment.GetScalingSettings().GetMergeFriedel())); + Add(s, KeyInt("UNIQUE_REFLECTIONS", o.unique_reflections)); + Add(s, KeyInt("TOTAL_OBSERVATIONS", o.total_observations)); + // Outlier rejection drops observations from the merge AND from R_meas and the CC1/2 + // half-sets, so a run that rejects too much scores BETTER on every other number here. + Add(s, KeyInt("OBSERVATIONS_REJECTED", ms.n_observations_rejected)); + // One rule for every quantity: a run that did not measure it writes NO key, rather than + // the word "nan" or a zero that reads as a measured absence. + if (o.possible_unique_reflections > 0) + Add(s, KeyReal("COMPLETENESS", + 100.0 * o.unique_reflections / o.possible_unique_reflections, "{:.1f}")); + if (o.unique_reflections > 0) + Add(s, KeyReal("MULTIPLICITY", + static_cast(o.total_observations) / o.unique_reflections, "{:.2f}")); + if (std::isfinite(o.mean_i_over_sigma)) + Add(s, KeyReal("I_OVER_SIGMA", o.mean_i_over_sigma, "{:.2f}")); + if (std::isfinite(o.r_meas)) + Add(s, KeyReal("R_MEAS", o.r_meas, "{:.4f}")); + if (std::isfinite(o.cc_half)) + Add(s, KeyReal("CC_HALF", o.cc_half, "{:.4f}")); + if (std::isfinite(o.abs_diff_over_sigma_anomalous)) + Add(s, KeyReal("SIGANO", o.abs_diff_over_sigma_anomalous, "{:.3f}")); + if (std::isfinite(o.cc_anom)) + Add(s, KeyReal("CC_ANOM", o.cc_anom, "{:.4f}")); + if (std::isfinite(ms.wilson_b)) + Add(s, KeyReal("WILSON_B", ms.wilson_b, "{:.2f}")); + // The error model in XDS's convention, so the numbers are directly comparable with a + // CORRECT.LP. Kept in the default report because beamline monitoring scripts read them. + Add(s, KeyReal("ERROR_MODEL_A", result.error_model_a, "{:.4f}")); + Add(s, KeyReal("ERROR_MODEL_B", result.error_model_b, "{:.4e}")); + Add(s, KeyReal("ISA", result.error_model_isa, "{:.2f}")); + if (result.error_model_isa_asymptotic > 0.0) + Add(s, KeyReal("ISA_ASYMPTOTIC", result.error_model_isa_asymptotic, "{:.2f}", true)); + // Only when TRUE: FALSE is the ordinary case and says nothing. + if (result.has_reference) + Add(s, KeyBool("REFERENCE_DATA_USED", true)); + else + Add(s, KeyBool("REFERENCE_DATA_USED", false, true)); + + const double signal_limit = fitted_resolution.value_or( + finest_supported > 0.0 ? finest_supported : o.d_min); + completeness_fit = CompletenessTo(ms, signal_limit); + // Worth PRINTING twice only when the two ranges are actually different. On a + // detector-limited crystal every shell still carries signal, so the range where the + // signal is IS the written range, and "82.4 % to 1.22 A; 82.4 % over the full written + // range" says one thing twice. The warning below is a different question and must + // still fire on a thin data set whose signal reaches the edge of what was written. + completeness_narrower = completeness_fit.measured && signal_limit > o.d_min + 1e-3; + + Add(s, Blank()); + if (!fit_suppressed.empty()) + Add(s, Prose(fmt::format( + " No FITTED_RESOLUTION is given: {}.\n" + " INCLUDE_RESOLUTION_RANGE says what was WRITTEN, and the shell table below says\n" + " where the signal actually stops.", fit_suppressed))); + else if (fitted_resolution) + Add(s, Prose(fmt::format( + " INCLUDE_RESOLUTION_RANGE is the range the reflections were WRITTEN to;" + " FITTED_RESOLUTION\n is where the CC1/2 fall-off crosses {:.2f}, and is the number" + " to quote. The data are kept\n one shell past it on purpose: an included shell can" + " still be downweighted by refinement,\n while a truncated one cannot be put back.", + cc_target))); + if (completeness_narrower && o.possible_unique_reflections > 0) + Add(s, Prose(fmt::format( + " COMPLETENESS is over that whole written range. Within the fitted resolution it is\n" + " {:.1f}% to {:.2f} A - the lower overall figure is the generous cut diluting the\n" + " denominator with reciprocal space the detector only reaches in its corners.", + completeness_fit.percent, completeness_fit.d_min))); + if (std::isfinite(o.cc_anom)) + Add(s, ProseDefaultOnly( + " CC_ANOM is the anomalous difference measured twice and correlated between the two\n" + " halves; it is what says whether there is a signal to phase on. A negative value is\n" + " a measurement, not an error. XDS's `Anomal Corr` is NOT this quantity.")); + + Add(s, Prose(" CC_ANOM is the anomalous difference measured twice - I(+)-I(-) from one half of the\n" + " observations against the same difference from the other half - and correlated over the\n" + " acentric pairs where both hands were measured at least twice. It is what says whether\n" + " there is an anomalous signal to phase on, and unlike SIGANO it is not a ratio against\n" + " the error model, so an optimistic sigma cannot inflate it. It agrees with AIMLESS's\n" + " CCanom and phenix.merging_statistics' cc_anom. XDS's CORRECT.LP has a column named\n" + " `Anomal Corr` which is NOT this quantity and reads considerably higher at low\n" + " resolution, so the two are not comparable.\n" + " A negative value is a measurement, not an error: on data with little anomalous signal\n" + " and around two observations per Bijvoet mate the statistic is unstable and goes\n" + " negative, in this program and in the others alike. Where no pair could be split in\n" + " both hands the quantity does not exist: the key is then absent here and the column is\n" + " a dash in the table below, which is not the same claim as a signal measured to be\n" + " zero.\n\n" + " ERROR_MODEL_A / ERROR_MODEL_B are in XDS's convention, sigma^2 = a*(sigma0^2 + b*I^2),\n" + " so ISA = 1/sqrt(a*b) means what CORRECT.LP's ISa means. ISA_ASYMPTOTIC, where present,\n" + " is the strong-reflection tier only.", true)); + + { + std::ostringstream shells; + shells << ms; + Add(s, Blank()); + Add(s, Prose(shells.str())); + } + + // A merge with no signal in it. Neither of these is caught anywhere else: CC1/2 near zero + // beside a healthy I/sigma is the signature of observations of DIFFERENT reflections being + // averaged together (mixed indexing hands, a wrong point group), and a rejection count + // above the kept count is a scaling that threw the data away. Both leave every other + // number in this section looking ordinary. + const bool no_correlation = std::isfinite(o.cc_half) && o.cc_half < 0.5 + && std::isfinite(o.mean_i_over_sigma) && o.mean_i_over_sigma > 3.0; + const bool over_rejected = ms.n_observations_rejected > o.total_observations; + if (no_correlation || over_rejected) { + unusable_merge = true; + const std::string why = no_correlation + ? fmt::format("CC1/2 is {:.3f} while I/sigma is {:.1f} - the observations correlate" + " with each other far worse than their own errors allow, which means" + " reflections that are not equivalent were averaged together", + o.cc_half, o.mean_i_over_sigma) + : fmt::format("{} observations were rejected against {} kept - scaling threw away" + " more than it merged", ms.n_observations_rejected, + o.total_observations); + Warn(doc, PathologyCode::UNUSABLE_MERGE, + "These merged data are not usable as they stand: " + why + + ". Do not refine against the written reflections without establishing why"); + } + // Completeness measured where the signal is, not over the generous cut - so a + // corner-limited detector on good data does not fire it and a genuinely thin dataset does. + constexpr double MIN_COMPLETENESS_PERCENT = 80.0; + if (completeness_fit.measured && completeness_fit.percent < MIN_COMPLETENESS_PERCENT) + Warn(doc, PathologyCode::LOW_COMPLETENESS, fmt::format( + "Only {:.1f}% of the unique reflections to {:.2f} A were measured - the merge is " + "missing a large part of reciprocal space, which no amount of multiplicity in what " + "was measured makes up for. Collect a wider sweep, or a second one about a " + "different axis", completeness_fit.percent, completeness_fit.d_min)); + if (!fit_suppressed.empty() && !unusable_merge) + Warn(doc, PathologyCode::RESOLUTION_FIT, + "No resolution could be fitted: " + fit_suppressed + + ". Read the shell table rather than quoting a single number"); } - if (an.verdict == AnisotropyVerdict::Detected && an.d_min_spread > 0.5) { - // Say WHICH direction each limit belongs to. Without it the warning states that the crystal - // is anisotropic and leaves the reader no way to act on it; the eigenvectors are measured - // here and reach the mmCIF, so the name costs nothing. - // Skipping the non-finite entries: a cone that never crosses the threshold leaves its limit - // NaN, and every comparison against NaN is false - so max_element and min_element would both - // return that entry and the warning would name one direction twice, with nan for both limits. + doc.sections.push_back(std::move(s)); + } + + // -------------------------------------------------------- 4. DIAGNOSTICS + { + ReportSection s; + s.title = "4. DIAGNOSTICS"; + + // ---- twinning + if (merged && result.twinning.l_test_pairs > 0) { + const auto &tw = result.twinning; + Add(s, KeyBool("TWINNING_SUSPECTED", tw.twinning_suspected)); + Add(s, KeyReal("L_TEST_MEAN_ABS_L", tw.mean_abs_l, "{:.4f}")); + Add(s, KeyReal("SECOND_MOMENT_I", tw.second_moment, "{:.4f}")); + Add(s, KeyReal("ESTIMATED_TWIN_FRACTION", tw.estimated_twin_fraction, "{:.3f}")); + // Which statistic the fraction came from. The L-test in the same block can imply a much + // larger fraction, and printing the two side by side with no label made the headline key + // look like a reconciliation of both when it is derived from one of them. + Add(s, KeyEnum("ESTIMATED_TWIN_FRACTION_FROM", "SECOND_MOMENT", + {"SECOND_MOMENT", "L_TEST"})); + Add(s, KeyReal("L_TEST_MEAN_L_SQUARED", tw.mean_l_squared, "{:.4f}", true)); + if (result.pre_promotion_twinning) { + Add(s, KeyReal("L_TEST_MEAN_ABS_L_BEFORE_SEARCH", + result.pre_promotion_twinning->mean_abs_l, "{:.4f}", true)); + Add(s, KeyReal("SECOND_MOMENT_I_BEFORE_SEARCH", + result.pre_promotion_twinning->second_moment, "{:.4f}", true)); + } + Add(s, Blank()); + // One calm sentence, in the same terms in both branches. The old prose opened "Cannot rule + // out twinning from these numbers" on runs whose own key said FALSE - a property of the + // symmetry stated as if it were a finding about these data. + if (tw.twinning_suspected) + Add(s, Prose(fmt::format( + " Twinning: INDICATED. <|L|> is {:.3f}, where an untwinned crystal reads about 0.5 and\n" + " a perfect twin about 0.375. The estimated fraction {:.2f} is derived from the second\n" + " moment alone and the L-test may imply a larger one. Refine with a twin law.", + tw.mean_abs_l, tw.estimated_twin_fraction))); + else + Add(s, Prose(fmt::format( + " Twinning: no indication (<|L|> {:.3f}, against about 0.5 for an untwinned crystal\n" + " and 0.375 for a perfect twin).", tw.mean_abs_l))); + Add(s, Prose("\n" + TwinningAnalysisToText(tw), true)); + if (result.pre_promotion_twinning) + Add(s, Prose(" The _BEFORE_SEARCH pair was measured on the merge the space-group search was given,\n" + " before any point group was adopted. That ordering is the whole point: the statistics\n" + " above are computed in the Laue class this run ADOPTED, so if the search promoted the\n" + " point group they can only report that no twin law exists inside the class it chose -\n" + " and a twin is precisely what would have caused that promotion. Where the two pairs\n" + " disagree, believe the _BEFORE_SEARCH one about whether the crystal is twinned.", true)); + if (tw.twinning_suspected) + Warn(doc, PathologyCode::TWINNING, fmt::format( + "Twinning is indicated (<|L|> = {:.3f}, /^2 = {:.3f}, estimated twin " + "fraction {:.2f} from the second moment) - refine against the merged data with care", + tw.mean_abs_l, tw.second_moment, tw.estimated_twin_fraction)); + } + + // ---- radiation damage + if (!result.radiation_damage_text.empty()) { + const double db = result.merge_statistics.radiation_damage_delta_b; + // A number, or a word saying why there is none: NOT_A_TREND where the per-batch curve was + // measured but no straight line describes it, NOT_MEASURED where the monitor could not run. + if (std::isfinite(db)) + Add(s, KeyReal("RADIATION_DAMAGE_RELATIVE_B", db, "{:.2f}")); + else + Add(s, KeyEnum("RADIATION_DAMAGE_RELATIVE_B", + result.merge_statistics.radiation_damage_b_batch.empty() + ? "NOT_MEASURED" : "NOT_A_TREND", + {"NOT_MEASURED", "NOT_A_TREND"})); + Add(s, Blank()); + if (std::isfinite(db)) + Add(s, Prose(fmt::format( + " Radiation damage: the relative B rises by {:.2f} A^2 over the sweep{}.", + db, db < 2.0 ? ", which is negligible" : ""))); + Add(s, Prose("\n" + result.radiation_damage_text, true)); + } + + // ---- sweep quality + { + const auto &sq = result.merge_statistics.sweep_quality; + Add(s, KeyInt("SWEEP_QUALITY_COUNT", static_cast(sq.ranges.size()))); + Add(s, KeyEnum("SWEEP_QUALITY_STATUS", sq.measured ? "COMPUTED" : "NOT_COMPUTED", + {"COMPUTED", "NOT_COMPUTED"}, true)); + { + std::string codes; + for (int r = 0; r <= static_cast(SweepQualityReason::RadiationDamage); ++r) + codes += (codes.empty() ? "" : " ") + + std::string(SweepQualityReasonCode(static_cast(r))); + Add(s, KeyText("SWEEP_QUALITY_REASONS", codes, true)); + } + if (sq.measured) { + Add(s, KeyReal("SWEEP_ROTATION", sq.sweep_deg, "{:.1f}", true)); + Add(s, KeyReal("FLUX_PEAK_TO_TROUGH", sq.flux_peak_to_trough, "{:.2f}", true)); + Add(s, KeyReal("SCALE_MODULATION_PEAK_TO_TROUGH", sq.modulation_peak_to_trough, + "{:.2f}", true)); + } + Add(s, Blank()); + if (sq.ranges.empty()) { + // The empty table used to print its header and two rules around nothing on every + // clean run - a table that says "no problem" by being blank. + Add(s, Prose(sq.measured + ? fmt::format(" Sweep: no degraded ranges over {:.1f} deg - the crystal delivered evenly.", + sq.sweep_deg) + : std::string(" Sweep quality was not measured on this run."))); + } else { + Add(s, Prose(" Stretches of the sweep over which the crystal delivered much less than the rest of\n" + " the run. SEVERITY is the fraction of the run's typical diffracting power missing over\n" + " the range; SCALE and CC are relative to the run median; INDEXED is the fraction of the\n" + " range's frames that were scaled at all. Nothing is excluded on the strength of this.\n")); + ReportEntry e; + e.kind = ReportEntry::Kind::Table; + e.table.columns = {"FIRST_IMAGE", "LAST_IMAGE", "N_IMAGES", "ROTATION", "REASON", + "SEVERITY", "SCALE", "CC", "INDEXED"}; + e.table.text_header = + " FIRST_IMAGE LAST_IMAGE N_IMAGES ROTATION REASON SEVERITY SCALE CC INDEXED\n" + " ----------- ----------- --------- -------- -------------------- -------- ------ ------ --------"; + for (const auto &r : sq.ranges) { + e.table.text_rows.push_back(fmt::format( + " {:11d} {:11d} {:9d} {:8.1f} {:<20} {:8.2f} {:6.2f} {:6.2f} {:8.2f}", + r.first_image, r.last_image, r.last_image - r.first_image + 1, + r.rotation_deg, SweepQualityReasonCode(r.reason), r.severity, + r.mean_relative_scale, r.mean_relative_cc, r.indexed_fraction)); + std::vector row; + row.push_back(KeyInt("", r.first_image).value); + row.push_back(KeyInt("", r.last_image).value); + row.push_back(KeyInt("", r.last_image - r.first_image + 1).value); + row.push_back(KeyReal("", r.rotation_deg, "{:.1f}").value); + row.push_back(KeyText("", SweepQualityReasonCode(r.reason)).value); + row.push_back(KeyReal("", r.severity, "{:.2f}").value); + row.push_back(KeyReal("", r.mean_relative_scale, "{:.2f}").value); + row.push_back(KeyReal("", r.mean_relative_cc, "{:.2f}").value); + row.push_back(KeyReal("", r.indexed_fraction, "{:.2f}").value); + e.table.cells.push_back(std::move(row)); + Warn(doc, PathologyCode::SWEEP_GAPS, fmt::format( + "Frames {}-{} {} ({:.1f} deg, scale {:.2f} and CC {:.2f} of the run, {:.0f}% scaled)", + r.first_image, r.last_image, SweepQualityReasonText(r.reason), + r.rotation_deg, r.mean_relative_scale, r.mean_relative_cc, + 100.0 * r.indexed_fraction)); + } + Add(s, std::move(e)); + } + } + + // ---- how the crystal sat on the spindle + if (result.spindle_lost_unique_fraction.has_value()) + Add(s, KeyReal("SPINDLE_LOST_UNIQUE_FRACTION", *result.spindle_lost_unique_fraction, "{:.4f}")); + if (result.spindle_symmetry_axis_deg.has_value()) { + Add(s, KeyReal("SPINDLE_SYMMETRY_AXIS_ANGLE_DEG", *result.spindle_symmetry_axis_deg, + "{:.1f}", true)); + Add(s, KeyInt("SPINDLE_SYMMETRY_AXIS_ORDER", result.spindle_symmetry_axis_order, true)); + } + if (result.spindle_lost_unique_fraction.has_value()) { + Add(s, Blank()); + Add(s, Prose(*result.spindle_lost_unique_fraction > 0.0 + ? fmt::format(" Mounting: the sweep's blind cone about the spindle costs {:.1f}% of the unique\n" + " reflections, which the measured point group cannot recover however long the\n" + " run.", 100.0 * *result.spindle_lost_unique_fraction) + : std::string(" Mounting: the crystal's symmetry axis was far enough off the spindle that the\n" + " blind cone costs no unique reflections."))); + } + + // ---- anisotropy + const auto &an = result.merge_statistics.anisotropy; + if (merged && an.n_reflections > 0) { + Add(s, KeyEnum("ANISOTROPY_VERDICT", AnisotropyVerdictCode(an.verdict), + {"DETECTED", "NOT_DETECTED", "CANNOT_DETERMINE"})); + Add(s, KeyReal("ANISOTROPY_DELTA_B", an.delta_b, "{:.2f}")); + Add(s, KeyText("ANISOTROPY_D_MIN_PRINCIPAL", + fmt::format("{:.2f} {:.2f} {:.2f}", an.d_min_axis[0], an.d_min_axis[1], + an.d_min_axis[2]))); + Add(s, KeyReal("ANISOTROPY_D_MIN_SPREAD", an.d_min_spread, "{:.2f}")); + + // The gate's internals. Correct, and none of it changes what a user does next; the verdict + // key above already carries the decision they were computed to make. + Add(s, KeyInt("ANISOTROPY_FREE_DIRECTIONS", an.n_free_parameters, true)); + Add(s, KeyReal("ANISOTROPY_DELTA_B_LINEAR", an.delta_b_linear, "{:.2f}", true)); + Add(s, KeyText("ANISOTROPY_PRINCIPAL_B", + fmt::format("{:.2f} {:.2f} {:.2f}", an.eigenvalue[0] - an.eigenvalue[2], + an.eigenvalue[1] - an.eigenvalue[2], 0.0), true)); + // Capped, because the uncapped value is the Debye-Waller factor of the fitted deltaB + // evaluated at the GLOBAL d_min - far beyond where the weak direction has any data at all + // - and printing six figures of it states something unphysical. + Add(s, KeyText("ANISOTROPY_FOLD_WEAKENING", + an.fold_weakening > 1000.0 ? std::string("> 1000") + : fmt::format("{:.1f}", an.fold_weakening), true)); + Add(s, KeyText("ANISOTROPY_D_MIN_CENSORED", + fmt::format("{} {} {}", an.d_min_censored[0] ? 1 : 0, + an.d_min_censored[1] ? 1 : 0, an.d_min_censored[2] ? 1 : 0), true)); + // Skipping the non-finite entries: a cone too sparse to cross the threshold leaves its + // limit NaN, and every comparison against NaN is false, so a bare min_element returns + // element 0 and prints nan even where the other two are measured. + const double *best_axis = nullptr; + for (const double *p = an.d_min_axis; p != an.d_min_axis + 3; ++p) + if (std::isfinite(*p) && (!best_axis || *p < *best_axis)) + best_axis = p; + if (best_axis) + Add(s, KeyReal("ANISOTROPY_D_MIN_BEST", *best_axis, "{:.2f}", true)); + Add(s, KeyText("ANISOTROPY_SHAPE", AnisotropyShapeCode(an.shape), true)); + Add(s, KeyReal("ANISOTROPY_SHAPE_INTERCEPT", an.shape_intercept, "{:.3f}", true)); + Add(s, KeyReal("ANISOTROPY_SHAPE_INTERCEPT_Z", an.shape_intercept_z, "{:.1f}", true)); + Add(s, KeyReal("ANISOTROPY_SHAPE_SLOPE", an.shape_slope, "{:.2f}", true)); + Add(s, KeyReal("ANISOTROPY_SHAPE_RESIDUAL", an.shape_residual, "{:.1f}", true)); + Add(s, KeyInt("ANISOTROPY_N_OBSERVATIONS", an.n_observations, true)); + Add(s, KeyReal("ANISOTROPY_SIGMA_SYSTEMATIC", an.sigma_systematic, "{:.3f}", true)); + Add(s, KeyReal("ANISOTROPY_FORBIDDEN_Z", an.forbidden_z, "{:.1f}", true)); + Add(s, KeyReal("ANISOTROPY_FLOOR", an.floor, "{:.3f}", true)); + Add(s, KeyReal("ANISOTROPY_SIGNIFICANCE", an.significance, "{:.2f}", true)); + Add(s, KeyReal("ANISOTROPY_DETECTION_LIMIT", an.detection_limit, "{:.2f}", true)); + const double *worst = nullptr, *best = nullptr; for (const double *p = an.d_min_axis; p != an.d_min_axis + 3; ++p) { if (!std::isfinite(*p)) @@ -633,186 +863,454 @@ std::string RenderResultReport(const std::string &output_prefix, ? ReciprocalAxisLabel(*result.consensus_cell, an.eigenvector[n]) : fmt::format("principal direction {}", n + 1); }; - if (worst && best && worst != best) - warnings.emplace_back(fmt::format( - "Diffraction is anisotropic (deltaB {:.1f} A^2; the diffraction limit is {:.2f} A " - "along {} and {:.2f} A along {}) - refinement and map interpretation should allow " - "for it; no intensity has been corrected for it here", - an.delta_b, *worst, along(worst), *best, along(best))); - else - warnings.emplace_back(fmt::format( - "Diffraction is anisotropic (deltaB {:.1f} A^2) - refinement and map " - "interpretation should allow for it; no intensity has been corrected for it here", - an.delta_b)); + // Severity in the one unit a user feels - how far apart the diffraction limits are - and + // never in the confidence of the detection. "DETECTED (strong)" meant statistically + // established and was read as severely anisotropic on every second dataset in the corpus. + const char *severity = an.d_min_spread < 0.25 ? "measurable but small" + : an.d_min_spread < 0.5 ? "noticeable" + : "STRONG"; + Add(s, Blank()); + if (an.verdict == AnisotropyVerdict::Detected && worst && best && worst != best) { + std::string p = fmt::format( + " Anisotropy: {}. Diffraction reaches {:.2f} A along {} and {:.2f} A along {}\n" + " (deltaB {:.1f} A^2).", severity, *best, along(best), *worst, along(worst), + an.delta_b); + // "Censored" is survival-analysis jargon for good news, and it read as an accusation. + // The whole content of the flag, in plain words: + const int worst_i = static_cast(worst - an.d_min_axis); + if (an.d_min_censored[worst_i]) + p += fmt::format("\n Along {} the crystal reaches AT LEAST {:.2f} A - the measured data" + " end before\n the signal does.", along(worst), *worst); + p += "\n Nothing was corrected or removed: the merged data and the written files do not\n" + " depend on direction at all."; + Add(s, Prose(p)); + } else if (an.verdict == AnisotropyVerdict::Detected) { + Add(s, Prose(fmt::format( + " Anisotropy: {} (deltaB {:.1f} A^2). Nothing was corrected or removed: the merged\n" + " data do not depend on direction.", severity, an.delta_b))); + } else if (an.verdict == AnisotropyVerdict::CannotDetermine) { + Add(s, Prose(" Anisotropy: cannot be determined in this Laue class - there is no\n" + " symmetry-forbidden direction to calibrate the fit's own noise against.")); + } else { + Add(s, Prose(" Anisotropy: below this data set's own noise floor. No directional statement can\n" + " be made, and the merged data do not depend on direction.")); + } + Add(s, Prose("\n How much the fall-off depends on direction, and whether that is established above this\n" + " data set's own systematic error. ANISOTROPY_DELTA_B is the range of the principal\n" + " components of the anisotropy tensor, on the ordinary crystallographic B scale (the same\n" + " scale as phenix.xtriage's B_cart and ctruncate's anisotropic B), fitted on intensities\n" + " with nothing dropped; ANISOTROPY_SIGNIFICANCE gates ANISOTROPY_DELTA_B_LINEAR, the part\n" + " of it that follows exp(-1/2 s^T B s), which is not the same number. A 1 in\n" + " ANISOTROPY_D_MIN_CENSORED marks a direction whose limit is the edge of the measured\n" + " data rather than the crystal's own. ANISOTROPY_FOLD_WEAKENING is evaluated at the\n" + " GLOBAL d_min, which extrapolates past the weak direction's own limit, and is capped\n" + " at 1000 here for that reason.", true)); + if (fitted_resolution && best_axis) + Add(s, Prose(fmt::format( + " Three resolutions, three questions. INCLUDE_RESOLUTION_RANGE ({:.2f} A) is what was\n" + " WRITTEN, deliberately one shell past the fit. FITTED_RESOLUTION ({:.2f} A) is where the\n" + " isotropic CC1/2 fall-off crosses its target. ANISOTROPY_D_MIN_BEST ({:.2f} A) is how far\n" + " the crystal reaches along its strongest direction. None of the three is wrong; quoting\n" + " one without saying which it is, is.", + result.merge_statistics.overall.d_min, *fitted_resolution, *best_axis), true)); + + if (an.verdict == AnisotropyVerdict::Detected && an.d_min_spread > 0.5) { + if (worst && best && worst != best) + Warn(doc, PathologyCode::ANISOTROPY, fmt::format( + "Diffraction is anisotropic (deltaB {:.1f} A^2; the diffraction limit is {:.2f} A " + "along {} and {:.2f} A along {}) - refinement and map interpretation should allow " + "for it; no intensity has been corrected for it here", + an.delta_b, *worst, along(worst), *best, along(best))); + else + Warn(doc, PathologyCode::ANISOTROPY, fmt::format( + "Diffraction is anisotropic (deltaB {:.1f} A^2) - refinement and map " + "interpretation should allow for it; no intensity has been corrected for it here", + an.delta_b)); + } + + // CC1/2 that falls and then climbs again is not a fall-off. An observation about the + // resolution cut's own internals, which no user can act on: it belongs in the developer + // report, because a warning nobody can act on teaches people to skip the section. + { + // A climb only counts while the shell still carries signal: past CC1/2 ~ 0.3 the data + // are beyond any cut this run would take and the number is oscillating in noise. + constexpr double CC_HALF_RISE = 0.05, CC_HALF_ALIVE = 0.30; + const auto &sh = result.merge_statistics.shells; + size_t fell = 0, rose_after = 0; + for (size_t i = 1; i < sh.size(); ++i) { + if (sh[i].cc_half < sh[i - 1].cc_half) fell = i; + else if (fell > 0 && sh[i].cc_half > sh[i - 1].cc_half + CC_HALF_RISE + && sh[i].cc_half > CC_HALF_ALIVE) rose_after = i; + } + if (rose_after > 0) + Warn(doc, PathologyCode::RESOLUTION_FIT, fmt::format( + "CC1/2 is not monotone with resolution (it climbs again at {:.2f}-{:.2f} A) - the " + "fall-off the resolution cut is read off does not describe these data", + sh[rose_after].d_max, sh[rose_after].d_min), true); + } } + doc.sections.push_back(std::move(s)); } - // ------------------------------------------------------- 10. MODEL VALIDATION + // -------------------------------------------------- 5. MODEL VALIDATION if (result.model_validation.has_value()) { const auto &mv = *result.model_validation; - Section(os, "10. MODEL VALIDATION"); - Key(os, "MODEL_FILE", mv.model_path); + ReportSection s; + s.title = "5. MODEL VALIDATION"; + Add(s, KeyText("MODEL_FILE", mv.model_path)); if (!mv.ok) { - Key(os, "MODEL_VALIDATION", "NOT_PERFORMED"); - Key(os, "MODEL_VALIDATION_REASON", mv.failure_reason); - os << "\n A model was given but could not be used, so there are no R-factors and no maps.\n" - << " Everything else in this report is unaffected: the merge does not depend on the model.\n"; + Add(s, KeyEnum("MODEL_VALIDATION", "NOT_PERFORMED", {"PERFORMED", "NOT_PERFORMED"})); + Add(s, KeyText("MODEL_VALIDATION_REASON", mv.failure_reason)); + Add(s, Blank()); + Add(s, Prose(" A model was given but could not be used, so there are no R-factors and no maps.\n" + " Everything else in this report is unaffected: the merge does not depend on the model.")); } else { - // The counterpart of NOT_PERFORMED above, so a consumer can grep one key for either answer - // instead of having to infer success from the absence of a failure line. - Key(os, "MODEL_VALIDATION", "PERFORMED"); - Key(os, "MODEL_SPACE_GROUP_NUMBER", mv.model_space_group_number); - Key(os, "R_WORK", fmt::format("{:.4f}", mv.r_work)); - Key(os, "R_FREE", fmt::format("{:.4f}", mv.r_free)); - Key(os, "R_WORK_REFLECTIONS", mv.n_work); - Key(os, "R_FREE_REFLECTIONS", mv.n_free); - Key(os, "BULK_SOLVENT_K_SOL", fmt::format("{:.3f}", mv.k_sol)); - Key(os, "BULK_SOLVENT_B_SOL", fmt::format("{:.1f}", mv.b_sol)); - Key(os, "SCALE_OVERALL", fmt::format("{:.4f}", mv.k_overall)); - // Does this model describe these data? Not a threshold on R - the R a model that explains - // nothing reaches depends on the model as much as on the data - but the same model refitted - // from random orientations, which is the only null that fits both. NOT_TESTED is a third - // answer and not a missing one: the model claimed nothing that needed arbitrating, so the - // null was not built, and a consumer must not read that as a model the data refused. - Key(os, "MODEL_FIT", - !mv.fit_tested ? "NOT_TESTED" : (mv.model_fits ? "ACCEPTED" : "REJECTED")); + Add(s, KeyEnum("MODEL_VALIDATION", "PERFORMED", {"PERFORMED", "NOT_PERFORMED"})); + Add(s, KeyInt("MODEL_SPACE_GROUP_NUMBER", mv.model_space_group_number)); + Add(s, KeyReal("R_WORK", mv.r_work, "{:.4f}")); + Add(s, KeyReal("R_FREE", mv.r_free, "{:.4f}")); + Add(s, KeyInt("R_WORK_REFLECTIONS", mv.n_work)); + Add(s, KeyInt("R_FREE_REFLECTIONS", mv.n_free)); + Add(s, KeyReal("BULK_SOLVENT_K_SOL", mv.k_sol, "{:.3f}", true)); + Add(s, KeyReal("BULK_SOLVENT_B_SOL", mv.b_sol, "{:.1f}", true)); + Add(s, KeyReal("SCALE_OVERALL", mv.k_overall, "{:.4f}", true)); + // NOT_TESTED is a third answer and not a missing one: the model claimed nothing that + // needed arbitrating, so the null was not built. + Add(s, KeyEnum("MODEL_FIT", + !mv.fit_tested ? "NOT_TESTED" : (mv.model_fits ? "ACCEPTED" : "REJECTED"), + {"ACCEPTED", "REJECTED", "NOT_TESTED"})); if (mv.fit_tested) { - Key(os, "MODEL_FIT_STATISTIC", "R_WORK"); - Key(os, "MODEL_FIT_VALUE", fmt::format("{:.4f}", mv.r_work)); - Key(os, "MODEL_FIT_NULL_MEAN", fmt::format("{:.4f}", mv.null_r_work_mean)); - Key(os, "MODEL_FIT_NULL_SD", fmt::format("{:.4f}", mv.null_r_work_sd)); - Key(os, "MODEL_FIT_NULL_REPLICATES", mv.null_replicates); - Key(os, "MODEL_FIT_SIGMA", fmt::format("{:+.2f}", mv.r_work_sigma)); + Add(s, KeyText("MODEL_FIT_STATISTIC", "R_WORK", true)); + Add(s, KeyReal("MODEL_FIT_VALUE", mv.r_work, "{:.4f}", true)); + Add(s, KeyReal("MODEL_FIT_NULL_MEAN", mv.null_r_work_mean, "{:.4f}", true)); + Add(s, KeyReal("MODEL_FIT_NULL_SD", mv.null_r_work_sd, "{:.4f}", true)); + Add(s, KeyInt("MODEL_FIT_NULL_REPLICATES", mv.null_replicates, true)); + Add(s, KeyReal("MODEL_FIT_SIGMA", mv.r_work_sigma, "{:+.2f}")); } - // The model is placed against the data as one rigid body, and only where the free - // reflections say it helped - so R_FREE_BEFORE_RIGID_BODY says what the placement bought. - Key(os, "RIGID_BODY", mv.rigid_body_applied ? "APPLIED" : "NOT_APPLIED"); + Add(s, KeyEnum("RIGID_BODY", mv.rigid_body_applied ? "APPLIED" : "NOT_APPLIED", + {"APPLIED", "NOT_APPLIED"})); if (mv.rigid_body_applied) { - Key(os, "RIGID_BODY_ROTATION_DEG", fmt::format("{:.3f}", mv.rigid_body_angle_deg)); - Key(os, "RIGID_BODY_SHIFT_A", fmt::format("{:.3f}", mv.rigid_body_shift_A)); - Key(os, "R_FREE_BEFORE_RIGID_BODY", fmt::format("{:.4f}", mv.r_free_before_rigid_body)); + Add(s, KeyReal("RIGID_BODY_ROTATION_DEG", mv.rigid_body_angle_deg, "{:.3f}")); + Add(s, KeyReal("RIGID_BODY_SHIFT_A", mv.rigid_body_shift_A, "{:.3f}")); + Add(s, KeyReal("R_FREE_BEFORE_RIGID_BODY", mv.r_free_before_rigid_body, "{:.4f}")); } - Key(os, "MAP_COEFFICIENTS", "2mFo-DFc / mFo-DFc"); - Key(os, "MAP_SIGMA_A_SHELLS", mv.sigma_a_shells); - Key(os, "MAP_MEAN_FOM", fmt::format("{:.3f}", mv.mean_fom)); - Key(os, "MEAN_ATOM_DENSITY_SIGMA", fmt::format("{:.2f}", mv.mean_atom_density_sigma)); - // The anomalous scatterers the data themselves found, named by the model's atoms. + Add(s, KeyText("MAP_COEFFICIENTS", "2mFo-DFc / mFo-DFc")); + Add(s, KeyInt("MAP_SIGMA_A_SHELLS", mv.sigma_a_shells, true)); + Add(s, KeyReal("MAP_MEAN_FOM", mv.mean_fom, "{:.3f}")); + Add(s, KeyReal("MEAN_ATOM_DENSITY_SIGMA", mv.mean_atom_density_sigma, "{:.2f}")); if (!mv.anomalous_sites.empty()) { - Key(os, "ANOMALOUS_BIJVOET_PAIRS", mv.anomalous_pairs); + Add(s, KeyInt("ANOMALOUS_BIJVOET_PAIRS", mv.anomalous_pairs)); for (size_t i = 0; i < mv.anomalous_sites.size(); i++) // Two digits so the ten keys are the same width and the values line up. - Key(os, fmt::format("ANOMALOUS_SITE_{:02}", i + 1).c_str(), - fmt::format("{:<18} {:6.2f} sigma", mv.anomalous_sites[i].label, - mv.anomalous_sites[i].sigma)); + Add(s, KeyText(fmt::format("ANOMALOUS_SITE_{:02}", i + 1).c_str(), + fmt::format("{:<18} {:6.2f} sigma", mv.anomalous_sites[i].label, + mv.anomalous_sites[i].sigma))); } - // What was applied to the written reflections, so a reader can tell whether the file is - // in the indexing it was merged in or in the model's. The enantiomorph is a label only. const bool took_indexing = !(mv.indexing_op == gemmi::Op::identity()); - Key(os, "MODEL_DECISIONS_TAKEN", - mv.adopted_model_enantiomorph - ? (took_indexing ? "ENANTIOMORPH+INDEXING" : "ENANTIOMORPH") - : (took_indexing ? "INDEXING" : "NONE")); - Key(os, "MODEL_ENANTIOMORPH_ADOPTED", mv.adopted_model_enantiomorph ? "TRUE" : "FALSE"); - Key(os, "MODEL_INDEXING_OPERATOR", mv.indexing_op.triplet()); - // The merohedral choice beside what the same choice looks like when it is made by a model - // in a random orientation - which also picks a winner, and by a comparable lead. + Add(s, KeyEnum("MODEL_DECISIONS_TAKEN", + mv.adopted_model_enantiomorph + ? (took_indexing ? "ENANTIOMORPH+INDEXING" : "ENANTIOMORPH") + : (took_indexing ? "INDEXING" : "NONE"), + {"NONE", "INDEXING", "ENANTIOMORPH", "ENANTIOMORPH+INDEXING"})); + Add(s, KeyBool("MODEL_ENANTIOMORPH_ADOPTED", mv.adopted_model_enantiomorph)); + Add(s, KeyText("MODEL_INDEXING_OPERATOR", mv.indexing_op.triplet())); if (mv.indexing_probed) { - Key(os, "MODEL_INDEXING_MARGIN", fmt::format("{:.4f}", mv.indexing_margin)); - Key(os, "MODEL_INDEXING_MARGIN_NULL", - fmt::format("{:.4f} +- {:.4f}", mv.indexing_margin_null_mean, - mv.indexing_margin_null_sd)); - Key(os, "MODEL_INDEXING_MARGIN_SIGMA", - fmt::format("{:+.2f}", mv.indexing_margin_sigma)); + Add(s, KeyReal("MODEL_INDEXING_MARGIN", mv.indexing_margin, "{:.4f}", true)); + Add(s, KeyText("MODEL_INDEXING_MARGIN_NULL", + fmt::format("{:.4f} +- {:.4f}", mv.indexing_margin_null_mean, + mv.indexing_margin_null_sd), true)); + Add(s, KeyReal("MODEL_INDEXING_MARGIN_SIGMA", mv.indexing_margin_sigma, "{:+.2f}", true)); } if (!mv.maps_prefix.empty()) - Key(os, "MAPS_PREFIX", mv.maps_prefix); - os << "\n R-free here measures the merged intensities against an external structure, which is\n" - << " what CC1/2 and R_meas cannot do - they only measure the data against themselves. The\n" - << " model is not refined: it is scaled to the data with a flat bulk solvent and an overall\n" - << " anisotropic B, so these R-factors are higher than a refined structure's and are a\n" - << " data-quality reading, not a refinement result.\n"; + Add(s, KeyText("MAPS_PREFIX", mv.maps_prefix)); + + Add(s, Blank()); + Add(s, Prose(" R-free here measures the merged intensities against an external structure, which is what\n" + " CC1/2 and R_meas cannot do - they only measure the data against themselves. The model is\n" + " not refined, only scaled and placed, so these R-factors are higher than a refined\n" + " structure's and are a data-quality reading, not a refinement result.")); if (mv.fit_tested) - os << "\n The model is a hypothesis, and MODEL_FIT says whether these data accept it. There is no\n" - << " value of R that settles that on its own - what a model which explains nothing reaches\n" - << " depends on its atom count and B-factors as much as on the data - so the same model was\n" - << " refitted, and re-placed, from " << mv.null_replicates << " random orientations about its own centroid, and\n" - << " MODEL_FIT_SIGMA is how far the real fit sits above that null. R-work carries the\n" - << " decision because the null was placed the same way the real fit was, so what those six\n" - << " placement parameters buy is bought on both sides and cancels - and it has far more\n" - << " reflections than R-free. MODEL_DECISIONS_TAKEN names what the model was allowed to change about the\n" - << " written reflections; R-factors, maps and the rigid-body placement are reported either\n" - << " way, because they describe the model, not the data.\n"; + Add(s, Prose(fmt::format( + " MODEL_FIT says whether these data accept the model: the same model was refitted from\n" + " {} random orientations, and MODEL_FIT_SIGMA ({:+.2f}) is how far the real fit sits\n" + " above that null.", mv.null_replicates, mv.r_work_sigma))); else - os << "\n MODEL_FIT= NOT_TESTED, which is not a failed test. A model can change only two things\n" - << " about the written reflections - the space-group label, where it asserts the other\n" - << " enantiomorph, and the indexing - and this one asserted neither: it names no other\n" - << " hand for these data, and prefers the indexing they were merged in. With nothing to\n" - << " arbitrate there was nothing to arbitrate it against, so the null the other two answers\n" - << " are measured against was not built and the run did not pay for it. The R-factors, the\n" - << " maps and the rigid-body placement above are exactly what they would have been.\n"; + Add(s, Prose(" MODEL_FIT= NOT_TESTED is not a failed test: the model asserted neither another hand\n" + " nor another indexing, so there was nothing to arbitrate and the null was not built.\n" + " The R-factors, the maps and the rigid-body placement are exactly what they would\n" + " have been.")); if (mv.fit_tested && !mv.model_fits) - os << "\n The model was tried and REJECTED. It was scaled and placed against these data exactly\n" - << " as an accepted one would have been, and reached R-work " << fmt::format("{:.4f}", mv.r_work) << " where the same\n" - << " model in " << mv.null_replicates << " random orientations reached " - << fmt::format("{:.4f} +- {:.4f}", mv.null_r_work_mean, mv.null_r_work_sd) << " - " - << fmt::format("{:+.2f}", mv.r_work_sigma) << " sigma, which is no\n" - << " better than chance. Nothing downstream moved: the space group is the one the data\n" - << " were merged in, the indexing is the one they were merged in, and the reflection\n" - << " files are byte for byte what a run with no model would have written. The R-factors\n" - << " and the maps above still describe this model against these data - they are the\n" - << " negative result, not a failure of the run.\n"; + Add(s, Prose(fmt::format( + " The model was tried and REJECTED - R-work {:.4f} against {:.4f} +- {:.4f} for the same\n" + " model in random orientations, which is no better than chance. Nothing downstream moved:\n" + " the reflection files are byte for byte what a run with no model would have written.", + mv.r_work, mv.null_r_work_mean, mv.null_r_work_sd))); if (!mv.anomalous_sites.empty()) - os << "\n The anomalous sites are the highest peaks of the anomalous difference map -\n" - << " F(+)-F(-) on the model phase turned back by 90 degrees - read at the model's own\n" - << " atom centres, so each one is named rather than left as a coordinate. A dataset with\n" - << " no anomalous signal still lists ten sites: it is their height, a few sigma at most,\n" - << " that says so. The map itself is written as _anom.ccp4, where a scatterer the model\n" - << " does not contain would show up as a peak on nothing.\n"; - if (!(mv.indexing_op == gemmi::Op::identity())) - os << "\n The written reflections were reindexed into the model's frame - the operator above\n" - << " says how - so the reflection files, the R-factors and the maps all describe one\n" - << " indexing.\n"; + Add(s, Prose(" The anomalous sites are the highest peaks of the anomalous difference map, read at\n" + " the model's own atom centres. A dataset with no anomalous signal still lists ten\n" + " sites; it is their height, a few sigma at most, that says so.")); + if (took_indexing) + Add(s, Prose(" The written reflections were reindexed into the model's frame - the operator above\n" + " says how - so the reflection files, the R-factors and the maps all describe one\n" + " indexing.")); if (mv.adopted_model_enantiomorph) - os << "\n The written reflections carry the model's enantiomorph as their space group. That is a\n" - << " change of label and nothing else: the two groups of an enantiomorphic pair have the same\n" - << " rotation operations, so no reflection moved. Reindexing by the change-of-hand operator\n" - << " would have swapped I(+) with I(-) - flipping the anomalous differences, not correcting\n" - << " them - on the strength of a label the space-group search reports as undetermined.\n"; + Add(s, Prose(" The written reflections carry the model's enantiomorph as their space group. That is\n" + " a change of label and nothing else: the two groups of an enantiomorphic pair have\n" + " the same rotation operations, so no reflection moved.")); else if (mv.model_enantiomorph_candidate) - os << "\n The model asserts the other enantiomorph of the written space group, and that assertion\n" - << " was NOT taken up. Adopting it would rewrite the group of every reflection written here on\n" - << " the strength of a model these data did not accept, and merged intensities cannot check it:\n" - << " |Fcalc| is invariant under the change of hand, so R-free reads the same for both. The\n" - << " anomalous difference map is the only measurement here that is sensitive to the hand, and\n" - << " where it says the two disagree it vetoes the adoption outright.\n"; + Add(s, Prose(" The model asserts the other enantiomorph of the written space group, and that\n" + " assertion was NOT taken up: merged intensities cannot check it, and the anomalous\n" + " difference map is the only measurement here sensitive to the hand.")); if (mv.anomalous_hands_disagree) - warnings.emplace_back(fmt::format( + Warn(doc, PathologyCode::MODEL_HAND, fmt::format( "The anomalous density at the model's atoms is inverted ({} reads {:.1f} sigma, deeper " "than the highest peak) - the data and the model are in opposite hands. Either the model " "is the wrong enantiomorph for this crystal or the data were indexed in the wrong hand; " "the reflections have not been reindexed, which would have hidden which of the two it is", mv.anomalous_deepest_site, mv.anomalous_deepest_sigma)); } + doc.sections.push_back(std::move(s)); } - // --------------------------------------------------------------- 11. WARNINGS if (result.cancelled) - warnings.emplace_back(fmt::format("Processing was cancelled after {} images - this report " - "describes an incomplete run", result.images_processed)); - Section(os, "11. WARNINGS"); - os << " Everything that needs a person's attention, one line each, marked so a script can find\n" - << " them with a single grep for \"WARNING:\".\n\n"; - Key(os, "WARNING_COUNT", warnings.size()); - os << "\n"; - for (const auto &w : warnings) - os << "WARNING: " << w << "\n"; - if (warnings.empty()) - os << " (none)\n"; + Warn(doc, PathologyCode::CANCELLED, + fmt::format("Processing was cancelled after {} images - this report describes an " + "incomplete run", result.images_processed)); + + // ---------------------------------------------------------------- SUMMARY + // Composed last and written first. Everything it says was decided by the sections above; this is + // the decide-late step a writer that printed as it went could not take at all. + { + for (const auto &w : doc.warnings) + if (!w.developer_only + && std::find(doc.pathology_flags.begin(), doc.pathology_flags.end(), w.code) + == doc.pathology_flags.end()) + doc.pathology_flags.push_back(w.code); + + const bool user_warnings = std::any_of(doc.warnings.begin(), doc.warnings.end(), + [](const ReportWarning &w) { return !w.developer_only; }); + const bool no_lattice = !result.consensus_cell.has_value(); + if (no_lattice || result.cancelled) + doc.verdict = "FAILED"; + else if (unusable_merge) + doc.verdict = "UNUSABLE"; + else if (user_warnings) + doc.verdict = "WARNINGS"; + else + doc.verdict = "OK"; + + const auto &o = result.merge_statistics.overall; + const std::string group = result.space_group ? result.space_group->xhm() : std::string("P 1"); + if (no_lattice) + doc.verdict_text = fmt::format( + "No crystal lattice was determined from {} images; nothing was integrated or merged.", + result.images_processed); + else if (result.cancelled) + doc.verdict_text = fmt::format( + "The run was cancelled after {} images - these results describe an incomplete sweep.", + result.images_processed); + else if (!merged) + doc.verdict_text = fmt::format( + "Integrated in {}; no merge was performed, so the reflections are in {}_process.h5.", + group, output_prefix); + else { + doc.verdict_text = fmt::format("Merged to {:.2f} A in {}", o.d_min, group); + std::string alt; + if (result.space_group_search.has_value() && result.space_group.has_value()) + for (const auto &a : result.space_group_search->alternatives) + if (a.number != result.space_group->number) + alt += (alt.empty() ? "" : " or ") + a.xhm(); + if (!alt.empty()) + doc.verdict_text += fmt::format(" (or {}, which these data cannot separate from it)", alt); + doc.verdict_text += "."; + if (unusable_merge) + doc.verdict_text += " The merged data carry no usable signal - see the warnings before" + " using any of the written files."; + else if (!doc.pathology_flags.empty()) { + std::string f; + for (const auto &c : doc.pathology_flags) + f += (f.empty() ? "" : ", ") + c; + doc.verdict_text += fmt::format(" {} condition(s) need attention: {}.", + doc.pathology_flags.size(), f); + } else { + doc.verdict_text += " No warnings."; + } + } + + ReportSection s; + s.title = "SUMMARY"; + Add(s, KeyEnum("VERDICT", doc.verdict, {"OK", "WARNINGS", "UNUSABLE", "FAILED"})); + Add(s, KeyText("VERDICT_TEXT", doc.verdict_text)); + Add(s, Blank()); + + // The facts a reader needs before deciding whether to read further, one line each. + { + std::string f; + const auto row = [&](const char *label, const std::string &value) { + if (!value.empty()) + f += fmt::format(" {:<18} {}\n", label, value); + }; + if (result.space_group) + row("Space group", group); + if (result.consensus_cell) + row("Unit cell", CellString(*result.consensus_cell)); + if (merged) { + std::string res = fmt::format("{:.2f} A written", o.d_min); + if (fitted_resolution) + res += fmt::format("; the CC1/2 fall-off is fitted at {:.2f} A", *fitted_resolution); + row("Resolution", res); + if (o.possible_unique_reflections > 0) { + std::string c = fmt::format("{:.1f} % over the full written range", + 100.0 * o.unique_reflections / o.possible_unique_reflections); + if (completeness_narrower) + c = fmt::format("{:.1f} % to {:.2f} A; {}", completeness_fit.percent, + completeness_fit.d_min, c); + row("Completeness", c); + } + std::string sig; + if (std::isfinite(o.mean_i_over_sigma)) + sig += fmt::format("I/sigma {:.1f}", o.mean_i_over_sigma); + if (std::isfinite(o.r_meas)) + sig += fmt::format(" R_meas {:.1f} %", 100.0 * o.r_meas); + if (std::isfinite(o.cc_half)) + sig += fmt::format(" CC1/2 {:.3f}", o.cc_half); + if (result.error_model_isa > 0.0) + sig += fmt::format(" ISa {:.1f}", result.error_model_isa); + row("Signal", sig); + if (std::isfinite(o.cc_anom)) + row("Anomalous", o.cc_anom > 0.15 + ? fmt::format("signal present (CC_anom {:.2f})", o.cc_anom) + : fmt::format("no usable signal (CC_anom {:.2f})", o.cc_anom)); + const auto &an = result.merge_statistics.anisotropy; + if (an.n_reflections > 0) { + if (an.verdict != AnisotropyVerdict::Detected) + row("Anisotropy", "not established above this data set's own noise"); + else { + const double *lo = nullptr, *hi = nullptr; + for (const double *p = an.d_min_axis; p != an.d_min_axis + 3; ++p) { + if (!std::isfinite(*p)) + continue; + if (!lo || *p < *lo) lo = p; + if (!hi || *p > *hi) hi = p; + } + const char *sev = an.d_min_spread < 0.25 ? "small" + : an.d_min_spread < 0.5 ? "noticeable" : "STRONG"; + row("Anisotropy", lo && hi + ? fmt::format("{}: deltaB {:.1f} A^2, d_min {:.2f}-{:.2f} A by direction", + sev, an.delta_b, *lo, *hi) + : fmt::format("{}: deltaB {:.1f} A^2", sev, an.delta_b)); + } + } + if (result.twinning.l_test_pairs > 0) + row("Twinning", result.twinning.twinning_suspected + ? fmt::format("INDICATED (<|L|> {:.3f})", result.twinning.mean_abs_l) + : std::string("no indication")); + const double db = result.merge_statistics.radiation_damage_delta_b; + if (std::isfinite(db)) + row("Radiation damage", fmt::format("relative B {:+.2f} A^2 over the sweep", db)); + const auto &sq = result.merge_statistics.sweep_quality; + if (sq.measured) + row("Sweep", sq.ranges.empty() + ? fmt::format("no degraded ranges in {:.1f} deg", sq.sweep_deg) + : fmt::format("{} degraded range(s) in {:.1f} deg", sq.ranges.size(), + sq.sweep_deg)); + } + if (!f.empty()) { + f.pop_back(); + Add(s, Prose(f)); + Add(s, Blank()); + } + } + + Add(s, KeyInt("WARNING_COUNT", static_cast( + std::count_if(doc.warnings.begin(), doc.warnings.end(), + [](const ReportWarning &w) { return !w.developer_only; })))); + { + std::string flags; + for (const auto &c : doc.pathology_flags) + flags += (flags.empty() ? "" : " ") + c; + Add(s, KeyText("PATHOLOGY_FLAGS", flags.empty() ? "NONE" : flags)); + } + Add(s, Blank()); + Add(s, Prose(" Everything that needs a person's attention, one line each, marked so a script can find\n" + " them with a single grep for \"WARNING:\". PATHOLOGY_FLAGS is the same information as a\n" + " closed vocabulary, one code per condition, for a consumer that switches on it.")); + doc.sections.insert(doc.sections.begin() + 1, std::move(s)); + } + + return doc; +} + +// -------------------------------------------------------------------------- the text rendering +// Reads only from the document. Nothing here decides what the report SAYS; it decides only how the +// text file looks, which is what makes a second rendering a second function rather than a fork. +std::string RenderReportText(const ReportDocument &doc, bool developer) { + std::ostringstream os; + os << BANNER << "\n RUGNUX PROCESSING REPORT\n" << BANNER << "\n\n"; + + for (const auto §ion : doc.sections) { + if (section.developer_only && !developer) + continue; + // A section whose every entry is developer-only would otherwise print an empty banner. + const bool any = std::any_of(section.entries.begin(), section.entries.end(), + [&](const ReportEntry &e) { + return developer ? !e.default_only : !e.developer_only; + }); + if (!any) + continue; + if (!section.title.empty()) + os << "\n" << BANNER << "\n " << section.title << "\n" << BANNER << "\n\n"; + + for (const auto &e : section.entries) { + if ((e.developer_only && !developer) || (e.default_only && developer)) + continue; + switch (e.kind) { + case ReportEntry::Kind::Key: + os << e.key << "= " << e.value.text << "\n"; + break; + case ReportEntry::Kind::Prose: + os << e.prose << "\n"; + break; + case ReportEntry::Kind::Blank: + os << "\n"; + break; + case ReportEntry::Kind::Table: + os << e.table.text_header << "\n"; + for (const auto &r : e.table.text_rows) + os << r << "\n"; + break; + } + } + + // The warnings belong to the SUMMARY, which is where a reader looks first. + if (section.title == "SUMMARY") { + os << "\n"; + size_t shown = 0; + for (const auto &w : doc.warnings) { + if (w.developer_only && !developer) + continue; + os << "WARNING: " << w.text << "\n"; + ++shown; + } + if (shown == 0) + os << " (none)\n"; + } + } os << "\n" << BANNER << "\n END OF REPORT\n" << BANNER << "\n"; return os.str(); } +std::string RenderResultReport(const std::string &output_prefix, + const std::string &input_file, + const DiffractionExperiment &experiment, + const ProcessResult &result, + const RunProvenance &provenance) { + return RenderReportText( + BuildReportDocument(output_prefix, input_file, experiment, result, provenance), + provenance.developer); +} + void WriteResultReport(const std::string &output_prefix, const std::string &input_file, const DiffractionExperiment &experiment, diff --git a/rugnux/ResultReport.h b/rugnux/ResultReport.h index 119e7c57d..f99cb6a7d 100644 --- a/rugnux/ResultReport.h +++ b/rugnux/ResultReport.h @@ -7,6 +7,7 @@ #include "../common/DiffractionExperiment.h" #include "../common/Logger.h" +#include "ReportDocument.h" #include "Rugnux.h" // _report.txt - what the run DETERMINED, written next to the .mtz/.cif/.hkl on every run @@ -28,8 +29,22 @@ struct RunProvenance { double wall_time_s = 0.0; // the whole invocation; 0 = not measured int32_t gpu_count = -1; // -1 = not reported; 0 is a statement, not an omission std::string gpu_description; // "4x NVIDIA A100-SXM4-80GB"; empty when there is no GPU + // --developer: render the pipeline internals and the long explanations as well. The document is + // built the same either way; this only selects how much of it the text rendering prints. + bool developer = false; }; +// The emitter. Populates the result object every rendering reads from; it writes no text itself. +ReportDocument BuildReportDocument(const std::string &output_prefix, + const std::string &input_file, + const DiffractionExperiment &experiment, + const ProcessResult &result, + const RunProvenance &provenance = {}); + +// The text rendering of that document - XDS-style prose, tables and `KEY= value` lines. Reads only +// from the document, so a second rendering is a second function of the same input. +std::string RenderReportText(const ReportDocument &doc, bool developer); + // Renders the report. Exposed for testing; RunPipeline results are the only input. std::string RenderResultReport(const std::string &output_prefix, const std::string &input_file, diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 9e37ca379..5b99ae89b 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -1735,7 +1735,10 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { // Say so rather than printing a zero that reads as a measurement. const auto compl_text = [&] { return compl_measured ? fmt::format("completeness {:.1f}% vs {:.1f}%", compl2, compl1) - : std::string("completeness not measured"); + // "not compared", not "not measured": completeness IS measured + // and reported; what did not happen is its use in arbitrating + // between the two passes. + : std::string("completeness not compared"); }; const double cc1 = pass1.search_merge_cc_half; const double cc2 = pass2.search_merge_cc_half; @@ -4214,7 +4217,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b a, b, alt.point_group_hm, sg_search.point_group_hm, a, b); logger.Warning("{}", msg); stats_text << " !! " << msg << "\n\n"; - result.warnings.push_back(msg); + result.warnings.push_back({PathologyCode::SYMMETRY_AMBIGUITY, msg}); } if (order_of(alt) > order_of(sg_search)) logger.Info("Space-group search: all-observation merge supports {} where the " @@ -4825,7 +4828,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b 100.0 * loss->lost_unique_fraction); logger.Warning("{}", msg); stats_text << " !! " << msg << "\n\n"; - result.warnings.push_back(msg); + result.warnings.push_back({PathologyCode::SPINDLE_CAP, msg}); } } else { stats_text << "\n"; @@ -4865,7 +4868,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b experiment_.IsRotationIndexing() ? "--model" : "--model, which needs -C and -S here"); logger.Warning("{}", msg); stats_text << " !! " << msg << "\n\n"; - result.warnings.push_back(msg); + result.warnings.push_back({PathologyCode::INDEXING_AMBIGUITY, msg}); } else if (config_.reference_data.empty()) { // Said before the model has been read, so it says what will be tried, not what // came of it - a model that turns out to be unreadable is reported where it is @@ -5069,7 +5072,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // report - indistinguishable from a run that was never given --model at all. result.model_validation = validation; if (!validation.failure_reason.empty()) - result.warnings.push_back("Model validation did not run: " + validation.failure_reason); + result.warnings.push_back({PathologyCode::MODEL_NOT_VALIDATED, + "Model validation did not run: " + validation.failure_reason}); if (data_sg.has_value()) { const gemmi::SpaceGroup *adopted = AdoptModelFrame(validation, sm.merged, *data_sg, diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 2f460af2c..6c2840e19 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -19,6 +19,7 @@ #include "../image_analysis/scale_merge/Merge.h" // MergeStatistics #include "../image_analysis/scale_merge/TwinningAnalysis.h" // TwinningAnalysisResult #include "ModelValidation.h" // ModelValidationResult +#include "ReportDocument.h" // ReportWarning, PathologyCode #include "../image_analysis/scale_merge/SearchSpaceGroup.h" // SearchSpaceGroupResult #include "../image_analysis/geom_refinement/PostRefine.h" // PostRefineResult #include "../image_analysis/geom_refinement/BeamCenterFromBackground.h" // BeamCenterEstimate @@ -315,9 +316,10 @@ struct ProcessResult { // The radiation-damage report (rotation), the same text the log prints. Empty when not measured. std::string radiation_damage_text; - // Conditions that need a person's attention, one plain sentence each (an ambiguous space group, a - // symmetry axis on the spindle, an indexing ambiguity). The same messages the log warns about. - std::vector warnings; + // Conditions that need a person's attention (an ambiguous space group, a symmetry axis on the + // spindle, an indexing ambiguity). The same messages the log warns about, each carrying a closed + // PathologyCode for machinery beside the open sentence for a person. + std::vector warnings; // How the crystal sat on the spindle: the angle to the nearest symmetry axis and that axis's // order. A property of the mounting rather than of the data, and the one number that says whether diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index f37c27b2a..cb377e355 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -149,6 +149,7 @@ void print_usage() { std::cout << " --no-scale-fulls Disable the rot3d scale-fulls refit (it is on by default for rot3d)" << std::endl; std::cout << " --write-process-h5 Also write the (large) _process.h5 when merging (default: only .mtz/.cif when merging)" << std::endl; std::cout << " --finalist-ledger Report the full-resolution evidence for each space group the search considered, not only the one it adopted (report-only; the decision is unchanged)" << std::endl; + std::cout << " --developer Write the full _report.txt: the pipeline-internal keys (the anisotropy gate, the space-group candidate and operator tables, the model-fit null, the sweep internals) and the long explanations, which the default report leaves out. Nothing is computed differently - the same report, rendered in full" << std::endl; std::cout << " --smooth-g[=deg] rot3d: smooth per-frame scale G over a deg-degree rotation range (XDS DELPHI-like) before the combine (default: 5 for rot3d; 0 = off)" << std::endl; std::cout << " --relative-b[=deg] rot3d: fit a per-batch relative-B (beyond the single decay slope) over deg-degree batches; cross-validated (default: 10 deg when bare; off otherwise)" << std::endl; std::cout << " --no-scaling-corrections rot3d: disable the (default-on) decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls" << std::endl; @@ -281,6 +282,7 @@ enum { OPT_NO_SCALE_FULLS, OPT_WRITE_PROCESS_H5, OPT_FINALIST_LEDGER, + OPT_DEVELOPER, OPT_FORCE_STILL, OPT_AZIM_MIN_Q, OPT_AZIM_MAX_Q, @@ -335,6 +337,7 @@ static option long_options[] = { {"no-scale-fulls", no_argument, nullptr, OPT_NO_SCALE_FULLS}, {"write-process-h5", no_argument, nullptr, OPT_WRITE_PROCESS_H5}, {"finalist-ledger", no_argument, nullptr, OPT_FINALIST_LEDGER}, + {"developer", no_argument, nullptr, OPT_DEVELOPER}, {"smooth-g", optional_argument, nullptr, OPT_SMOOTH_G}, {"relative-b", optional_argument, nullptr, OPT_RELATIVE_B}, {"no-scaling-corrections", no_argument, nullptr, OPT_NO_SCALING_CORRECTIONS}, @@ -1130,6 +1133,9 @@ static int RunRugnux(int argc, char **argv) { case OPT_FINALIST_LEDGER: finalist_ledger_flag = true; break; + case OPT_DEVELOPER: + provenance.developer = true; + break; case OPT_SMOOTH_G: smooth_g_deg_arg = optarg ? parse_double_arg(optarg, "--smooth-g", logger) : SMOOTH_G_DEFAULT_DEG; break; @@ -1876,6 +1882,21 @@ static int RunRugnux(int argc, char **argv) { scale_result.used_beam_x_pxl = experiment.GetBeamX_pxl(); scale_result.used_beam_y_pxl = experiment.GetBeamY_pxl(); scale_result.used_distance_mm = experiment.GetDetectorDistance_mm(); + // The tilt and the direct beam belong with them: the reader restores the poni rotations from + // the input file's transformations chain, so they are known here. Left unset they stayed at + // their zero-initialised default and the report asserted a FLAT detector on tilted data - a + // measurement that was never made, in a block a user is invited to POST back to the + // instrument. Same fill as the end of the per-image pass in Rugnux.cpp, from the same + // experiment, so the four numbers describe ONE geometry. + { + const auto &g = experiment.GetDiffractionGeometry(); + constexpr double DEG = 180.0 / PI; + scale_result.used_detector_tilt_deg = {g.GetPoniRot1_rad() * DEG, g.GetPoniRot2_rad() * DEG, + g.GetPoniRot3_rad() * DEG}; + const auto direct = g.GetDirectBeam_pxl(); + scale_result.used_direct_beam_x_pxl = direct.first; + scale_result.used_direct_beam_y_pxl = direct.second; + } scale_result.has_merge_statistics = true; scale_result.merge_statistics = merged_statistics; { std::ostringstream s; s << merged_statistics; scale_result.merge_statistics_text = s.str(); } @@ -1888,7 +1909,8 @@ static int RunRugnux(int argc, char **argv) { scale_result.twinning = twinning; scale_result.model_validation = model_validation; if (!model_validation_failure.empty()) - scale_result.warnings.push_back("Model validation did not run: " + model_validation_failure); + scale_result.warnings.push_back({PathologyCode::MODEL_NOT_VALIDATED, + "Model validation did not run: " + model_validation_failure}); provenance.wall_time_s = std::chrono::duration( std::chrono::steady_clock::now() - invocation_start).count(); std::cout << fmt::format("Total wall time: {:.2f} s", provenance.wall_time_s) << std::endl; diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index d63348ec1..2432c16aa 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -70,7 +70,7 @@ TEST_CASE("ResultReport_Render", "[Diagnostics]") { // The stable keys a consumer greps for. The version is pinned on purpose: a key added to the // report is a contract change, and this line is where it has to be acknowledged. - CHECK(text.find("\nREPORT_VERSION= 7\n") != std::string::npos); + CHECK(text.find("\nREPORT_VERSION= 8\n") != std::string::npos); CHECK(text.find("\nOUTPUT_PREFIX= prefix\n") != std::string::npos); CHECK(text.find("\nINDEXING_RATE= 0.8700\n") != std::string::npos); CHECK(text.find("\nSPACE_GROUP_NUMBER= 96\n") != std::string::npos); @@ -78,11 +78,19 @@ TEST_CASE("ResultReport_Render", "[Diagnostics]") { // than searched for, and it is one of the 22 that come in enantiomorphic pairs. CHECK(text.find("\nSPACE_GROUP_ALTERNATIVES= NONE\n") != std::string::npos); CHECK(text.find("\nSPACE_GROUP_ENANTIOMORPH= GIVEN\n") != std::string::npos); - CHECK(text.find("\nSPACE_GROUP_REFUSED_POINT_GROUP= NONE\n") != std::string::npos); - CHECK(text.find("\nSWEEP_QUALITY_STATUS= COMPUTED\n") != std::string::npos); + // Written only where there WAS a refusal: "NONE" on every other run answers a question nobody + // asked. The vocabulary and the status live in the developer report now, and are pinned there. + CHECK(text.find("SPACE_GROUP_REFUSED_POINT_GROUP=") == std::string::npos); CHECK(text.find("\nSWEEP_QUALITY_COUNT= 2\n") != std::string::npos); - CHECK(text.find("\nSWEEP_QUALITY_REASONS= no_diffraction crystal_out_of_beam weak_diffraction " - "loss_of_centring radiation_damage\n") != std::string::npos); + CHECK(text.find("SWEEP_QUALITY_STATUS=") == std::string::npos); + CHECK(text.find("SWEEP_QUALITY_REASONS=") == std::string::npos); + + RunProvenance dev; + dev.developer = true; + const auto dev_text = RenderResultReport("prefix", "in.h5", x, result, dev); + CHECK(dev_text.find("\nSWEEP_QUALITY_STATUS= COMPUTED\n") != std::string::npos); + CHECK(dev_text.find("\nSWEEP_QUALITY_REASONS= no_diffraction crystal_out_of_beam weak_diffraction " + "loss_of_centring radiation_damage\n") != std::string::npos); // One table row per range, with the reason code verbatim. CHECK(text.find(" 100 149 50 5.0 crystal_out_of_beam ") @@ -90,8 +98,12 @@ TEST_CASE("ResultReport_Render", "[Diagnostics]") { CHECK(text.find(" 400 499 100 10.0 radiation_damage ") != std::string::npos); - // ... and one plain-English WARNING line per range, greppable by the marker alone. + // ... and one plain-English WARNING line per range, greppable by the marker alone. They sit in + // the SUMMARY at the top of the file now, which is what the verdict is composed from. CHECK(text.find("\nWARNING_COUNT= 2\n") != std::string::npos); + CHECK(text.find("\nVERDICT= WARNINGS\n") != std::string::npos); + CHECK(text.find("\nPATHOLOGY_FLAGS= SWEEP_GAPS\n") != std::string::npos); + CHECK(text.find("VERDICT=") < text.find("SWEEP_QUALITY_COUNT=")); CHECK(text.find("\nWARNING: Frames 100-149 out of beam (5.0 deg,") != std::string::npos); CHECK(text.find("\nWARNING: Frames 400-499 radiation damage (10.0 deg,") != std::string::npos); } @@ -104,16 +116,25 @@ TEST_CASE("ResultReport_RenderEmpty", "[Diagnostics]") { ProcessResult clean; clean.has_merge_statistics = true; clean.merge_statistics.sweep_quality.measured = true; + RunProvenance dev; + dev.developer = true; + const auto clean_text = RenderResultReport("p", "in.h5", x, clean); - CHECK(clean_text.find("\nSWEEP_QUALITY_STATUS= COMPUTED\n") != std::string::npos); CHECK(clean_text.find("\nSWEEP_QUALITY_COUNT= 0\n") != std::string::npos); - CHECK(clean_text.find("FIRST_IMAGE LAST_IMAGE") != std::string::npos); + CHECK(RenderResultReport("p", "in.h5", x, clean, dev) + .find("\nSWEEP_QUALITY_STATUS= COMPUTED\n") != std::string::npos); + // An empty table used to print its header and two rules around nothing on every clean run. + CHECK(clean_text.find("FIRST_IMAGE LAST_IMAGE") == std::string::npos); + CHECK(clean_text.find("no degraded ranges") != std::string::npos); ProcessResult not_merged; const auto not_merged_text = RenderResultReport("p", "in.h5", x, not_merged); - CHECK(not_merged_text.find("\nSWEEP_QUALITY_STATUS= NOT_COMPUTED\n") != std::string::npos); + CHECK(RenderResultReport("p", "in.h5", x, not_merged, dev) + .find("\nSWEEP_QUALITY_STATUS= NOT_COMPUTED\n") != std::string::npos); CHECK(not_merged_text.find("\nMERGE= NOT_PERFORMED\n") != std::string::npos); - CHECK(not_merged_text.find("FIRST_IMAGE LAST_IMAGE") != std::string::npos); + CHECK(not_merged_text.find("FIRST_IMAGE LAST_IMAGE") == std::string::npos); + // No lattice was found, and that is the FIRST thing the report says. + CHECK(not_merged_text.find("\nVERDICT= FAILED\n") != std::string::npos); } TEST_CASE("ResultReport_RadiationDamage", "[Diagnostics]") { @@ -264,15 +285,24 @@ TEST_CASE("ResultReport_ModelValidationSection", "[Diagnostics]") { result.space_group = *gemmi::find_spacegroup_by_number(96); const auto text = RenderResultReport("p", "in.h5", x, result); - CHECK(text.find("10. MODEL VALIDATION") != std::string::npos); + CHECK(text.find("5. MODEL VALIDATION") != std::string::npos); CHECK(text.find("\nMODEL_FILE= model.cif\n") != std::string::npos); CHECK(text.find("\nR_FREE= 0.2145\n") != std::string::npos); CHECK(text.find("\nR_FREE_REFLECTIONS= 928\n") != std::string::npos); CHECK(text.find("\nMODEL_VALIDATION= PERFORMED\n") != std::string::npos); CHECK(text.find("MODEL_VALIDATION= NOT_PERFORMED") == std::string::npos); CHECK(text.find("\nMODEL_FIT= ACCEPTED\n") != std::string::npos); - CHECK(text.find("\nMODEL_FIT_STATISTIC= R_WORK\n") != std::string::npos); - CHECK(text.find("\nMODEL_FIT_NULL_REPLICATES= 5\n") != std::string::npos); + // The null's internals are the gate's own workings; MODEL_FIT and MODEL_FIT_SIGMA carry the + // answer they were computed for and stay in the default report. + CHECK(text.find("MODEL_FIT_STATISTIC=") == std::string::npos); + CHECK(text.find("MODEL_FIT_NULL_REPLICATES=") == std::string::npos); + { + RunProvenance dev; + dev.developer = true; + const auto dev_text = RenderResultReport("p", "in.h5", x, result, dev); + CHECK(dev_text.find("\nMODEL_FIT_STATISTIC= R_WORK\n") != std::string::npos); + CHECK(dev_text.find("\nMODEL_FIT_NULL_REPLICATES= 5\n") != std::string::npos); + } CHECK(text.find("\nMODEL_FIT_SIGMA= +4.12\n") != std::string::npos); CHECK(text.find("\nMODEL_DECISIONS_TAKEN= ENANTIOMORPH\n") != std::string::npos); // The hand is ASSUMED from the model, never determined: merged intensities cannot see it. @@ -332,3 +362,177 @@ TEST_CASE("ResultReport_GpuDescription", "[Diagnostics]") { else CHECK(description.find(names.front()) != std::string::npos); } + +// The verdict is the first evaluative line in the file, and it is composed from the warnings the +// sections produced - which is only possible because the report is built as a document and rendered +// afterwards. Its vocabulary is closed, so a consumer switches on it. +TEST_CASE("ResultReport_Verdict", "[Diagnostics]") { + DiffractionExperiment x(DetJF(1)); + + // No lattice at all: the run failed, and the report says so on the first screen rather than in + // its last line. + ProcessResult nothing; + nothing.images_processed = 100; + nothing.indexing_rate = 0.0f; + const auto failed = RenderResultReport("p", "in.h5", x, nothing); + CHECK(failed.find("\nVERDICT= FAILED\n") != std::string::npos); + CHECK(failed.find("\nPATHOLOGY_FLAGS= NO_LATTICE\n") != std::string::npos); + CHECK(failed.find("WARNING: No image indexed") != std::string::npos); + // The verdict comes before the section that produced the warning behind it. + CHECK(failed.find("VERDICT=") < failed.find("LATTICE_FOUND=")); + + // A --mode scale run has a lattice from its input file and never fills indexing_rate. The + // "No image indexed" warning used to fire on every one of them, on reports whose own + // LATTICE_FOUND said TRUE - a warning that is false on a whole run mode is how users learn to + // skip the section. + ProcessResult rescaled; + rescaled.consensus_cell = UnitCell{.a = 79.0f, .b = 79.0f, .c = 38.0f, + .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + const auto scale_text = RenderResultReport("p", "in.h5", x, rescaled); + CHECK(scale_text.find("\nLATTICE_FOUND= TRUE\n") != std::string::npos); + CHECK(scale_text.find("No image indexed") == std::string::npos); + CHECK(scale_text.find("INDEXING_RATE=") == std::string::npos); +} + +// Two conditions that left every other number in the report looking ordinary: a merge whose +// observations do not correlate, and a merge that measured too little of reciprocal space. +TEST_CASE("ResultReport_UnusableMergeAndCompleteness", "[Diagnostics]") { + DiffractionExperiment x(DetJF(1)); + + ProcessResult result; + result.consensus_cell = UnitCell{.a = 79.0f, .b = 79.0f, .c = 38.0f, + .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + result.has_merge_statistics = true; + auto &o = result.merge_statistics.overall; + o.d_max = 50.0f; + o.d_min = 2.00f; + o.unique_reflections = 1000; + o.possible_unique_reflections = 1100; + o.total_observations = 4000; + o.mean_i_over_sigma = 12.0; // strong... + o.cc_half = -0.005; // ...and uncorrelated: not equivalent reflections + o.r_meas = 1.09; + + const auto text = RenderResultReport("p", "in.h5", x, result); + CHECK(text.find("\nVERDICT= UNUSABLE\n") != std::string::npos); + CHECK(text.find("WARNING: These merged data are not usable") != std::string::npos); + CHECK(text.find("UNUSABLE_MERGE") != std::string::npos); + + // Rejecting more observations than were kept is the other way a merge throws the data away, and + // it improves every other statistic in the block while it does so. + ProcessResult rejected = result; + rejected.merge_statistics.overall.cc_half = 0.98; + rejected.merge_statistics.n_observations_rejected = 5000; + CHECK(RenderResultReport("p", "in.h5", x, rejected).find("threw away") != std::string::npos); + + // Completeness is judged inside the fitted resolution, not over the deliberately generous cut, + // so a corner-limited detector on good data does not fire it. + ProcessResult thin = result; + thin.merge_statistics.overall.cc_half = 0.98; + thin.resolution_fit_A = 2.0; + MergeStatisticsShell shell; + shell.d_max = 50.0f; + shell.d_min = 2.0f; + shell.cc_half = 0.98; + shell.unique_reflections = 300; + shell.possible_unique_reflections = 2000; + thin.merge_statistics.shells.push_back(shell); + const auto thin_text = RenderResultReport("p", "in.h5", x, thin); + CHECK(thin_text.find("LOW_COMPLETENESS") != std::string::npos); + CHECK(thin_text.find("\nVERDICT= WARNINGS\n") != std::string::npos); +} + +// FITTED_RESOLUTION is presented as "the number to quote", so it must not be quotable when the CC1/2 +// curve it was read off never behaved like a fall-off. +TEST_CASE("ResultReport_FittedResolutionSuppressed", "[Diagnostics]") { + DiffractionExperiment x(DetJF(1)); + + ProcessResult result; + result.consensus_cell = UnitCell{.a = 7.0f, .b = 7.0f, .c = 7.0f, + .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + result.has_merge_statistics = true; + result.merge_statistics.overall.d_max = 6.0f; + result.merge_statistics.overall.d_min = 0.80f; + result.merge_statistics.overall.unique_reflections = 500; + result.merge_statistics.overall.possible_unique_reflections = 520; + result.merge_statistics.overall.total_observations = 2000; + result.merge_statistics.overall.cc_half = 0.90; + result.merge_statistics.overall.mean_i_over_sigma = 8.0; + // Signal to 0.90 A and nothing beyond it... + MergeStatisticsShell inner; + inner.d_max = 6.0f; + inner.d_min = 0.90f; + inner.cc_half = 0.95; + inner.unique_reflections = 250; + inner.possible_unique_reflections = 260; + MergeStatisticsShell outer = inner; + outer.d_max = 0.90f; + outer.d_min = 0.80f; + outer.cc_half = 0.02; + result.merge_statistics.shells.push_back(inner); + result.merge_statistics.shells.push_back(outer); + // ...and a fit that claims 0.70 A, finer than any shell that still reaches the target. + result.resolution_fit_A = 0.70; + + const auto text = RenderResultReport("p", "in.h5", x, result); + CHECK(text.find("FITTED_RESOLUTION=") == std::string::npos); + CHECK(text.find("No FITTED_RESOLUTION is given") != std::string::npos); + CHECK(text.find("RESOLUTION_FIT") != std::string::npos); + + // A fit the shells DO support is written as before. + result.resolution_fit_A = 0.95; + CHECK(RenderResultReport("p", "in.h5", x, result).find("\nFITTED_RESOLUTION= 0.95\n") + != std::string::npos); +} + +// The completeness of the range where the signal is, beside the completeness of the range that was +// written. Users compare the single overall figure against another program's and conclude the run +// lost data, when it is the deliberately generous cut diluting the denominator - so the report says +// both. But only when they are different numbers: on a detector-limited crystal every shell still +// carries signal, and printing "82.4 % to 1.22 A; 82.4 % over the full written range" says one +// thing twice. +TEST_CASE("ResultReport_CompletenessTwoRanges", "[Diagnostics]") { + DiffractionExperiment x(DetJF(1)); + + ProcessResult result; + result.consensus_cell = UnitCell{.a = 79.0f, .b = 79.0f, .c = 38.0f, + .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + result.has_merge_statistics = true; + auto &o = result.merge_statistics.overall; + o.d_max = 50.0f; + o.d_min = 1.20f; + o.unique_reflections = 900; + o.possible_unique_reflections = 1200; + o.total_observations = 9000; + o.cc_half = 0.95; + o.mean_i_over_sigma = 10.0; + + MergeStatisticsShell inner; // signal here + inner.d_max = 50.0f; + inner.d_min = 1.50f; + inner.cc_half = 0.95; + inner.unique_reflections = 700; + inner.possible_unique_reflections = 720; + MergeStatisticsShell outer = inner; // written, but past the fall-off + outer.d_max = 1.50f; + outer.d_min = 1.20f; + outer.cc_half = 0.02; + outer.unique_reflections = 200; + outer.possible_unique_reflections = 480; + result.merge_statistics.shells.push_back(inner); + result.merge_statistics.shells.push_back(outer); + + // The signal stops at 1.50 A and the reflections were written to 1.20 A: two different ranges, + // so both numbers are reported and the outer shells are visibly what dilutes the overall one. + const auto narrower = RenderResultReport("p", "in.h5", x, result); + CHECK(narrower.find("% to 1.50 A;") != std::string::npos); + CHECK(narrower.find("\nCOMPLETENESS= 75.0\n") != std::string::npos); + + // Detector-limited: every shell still reaches the target, so the range where the signal is IS + // the written range and the second number would repeat the first. + result.merge_statistics.shells[1].cc_half = 0.95; + result.merge_statistics.shells[1].possible_unique_reflections = 200; + const auto same = RenderResultReport("p", "in.h5", x, result); + CHECK(same.find(" % to ") == std::string::npos); + CHECK(same.find("over the full written range") != std::string::npos); +} -- 2.54.0 From db9cc9106fb871f8e120274de58b569765aeb2fe Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 13:41:16 +0200 Subject: [PATCH 34/75] rugnux: the per-reflection correction factor is named for what it is, not for what it once held The factor multiplied into each integrated intensity was called rlp, for reciprocal Lorentz-polarization, and until this week that is all it held. It now also carries the sensor efficiency at the angle the beam arrives, and on the stills path it holds that efficiency and the polarization with no Lorentz term at all - correctly, since the Lorentz factor of a still is one. Three different products under one name that promises exactly one of them, in code where the neighbouring member is the total correction. Rename it prescaling_corr: multiplicative, applied before scaling, therefore not a scale, and silent about its contents - which is the point, since the contents have now grown twice. It is also what DIALS calls the same product. The stills refinement member spelled "1 / rlp" becomes inv_corr, and the comments and usage text that promised "the Lorentz-polarization factor and nothing else" now say what is actually there. The Lorentz term keeps its own name where it is computed, because that name is correct. The two external spellings are untouched: the CBOR key and the reflection dataset are a published format, and a reader that meets an unknown key would take the factor as zero, which both the merge key and the ingest treat as a reflection to drop - so every reflection would vanish and the run would still exit zero. No output changes: the merged and unmerged files of two full runs are byte for byte what the previous binary wrote, four stored files from before the efficiency correction still re-scale identically, and the reflection datasets of the process file are unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- common/Reflection.h | 11 ++++- docs/CBOR.md | 2 +- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 6 +-- docs/HDF5.md | 2 +- docs/RUGNUX_ADVANCED.md | 2 +- docs/RUGNUX_INTEGRATION.md | 8 ++-- frame_serialize/CBORStream2Deserializer.cpp | 2 +- frame_serialize/CBORStream2Serializer.cpp | 2 +- image_analysis/UpdateReflectionResolution.cpp | 2 +- image_analysis/WriteReflections.cpp | 48 +++++++++++-------- image_analysis/WriteReflections.h | 6 ++- .../BraggIntegrationEngine.cpp | 4 +- .../bragg_prediction/BraggPrediction.cpp | 2 +- .../bragg_prediction/BraggPredictionGPU.cu | 2 +- .../bragg_prediction/BraggPredictionRot.cpp | 2 +- .../bragg_prediction/BraggPredictionRotGPU.cu | 2 +- .../scale_merge/AnisotropyAnalysis.cpp | 8 ++-- image_analysis/scale_merge/HKLKey.cpp | 4 +- .../scale_merge/ReindexAmbiguity.cpp | 2 +- .../scale_merge/RotationScaleMerge.cpp | 31 ++++++------ .../scale_merge/RotationScaleMerge.h | 8 ++-- .../scale_merge/RotationScaleMergeGPU.cu | 28 +++++------ .../scale_merge/RotationScaleMergeGPU.h | 2 +- image_analysis/scale_merge/ScaleOnTheFly.cpp | 6 +-- .../scale_merge/StillsPartialityRefine.cpp | 23 ++++----- reader/HDF5MetadataSource.cpp | 2 +- rugnux/rugnux_cli.cpp | 2 +- tests/AnisotropyAnalysisTest.cpp | 2 +- ...ggIntegrationEngineCompressedImageTest.cpp | 2 +- tests/BraggIntegrationEngineGPUTest.cpp | 2 +- tests/CBORTest.cpp | 10 ++-- tests/JFJochReaderTest.cpp | 4 +- tests/MergeScaleTest.cpp | 2 +- tools/jfjoch_extract_hkl.cpp | 2 +- .../JFJochViewerReflectionListWindow.cpp | 2 +- writer/HDF5DataFilePluginReflection.cpp | 2 +- 36 files changed, 134 insertions(+), 113 deletions(-) diff --git a/common/Reflection.h b/common/Reflection.h index 1bba51809..e3e86ec17 100644 --- a/common/Reflection.h +++ b/common/Reflection.h @@ -25,10 +25,17 @@ struct Reflection { float var_bkg; // non-signal (background) part of sigma^2, carried to the merge float sigma; float dist_ewald; - float rlp; + // Everything known a priori that multiplies the raw integrated count on its way to a quantity + // proportional to |F|^2: the reciprocal Lorentz factor, the reciprocal polarization factor and + // the sensor's angle-dependent efficiency, whichever of them the path that filled it applies. + // It is deliberately NOT an enumeration of those terms - it is the product of every deterministic + // per-reflection correction, and that set has grown before and will grow again. What it is not is + // a scale: the fitted per-image scale and the partiality stay out of it and are divided in + // separately below. (Named after DIALS's prescaling_correction, which holds the same product.) + float prescaling_corr; float partiality; // fraction of the reflection recorded in the sampled (rocking) slice float zeta; - float image_scale_corr; // I_true = scaling_correction * I; scaling_correction = rlp / (partiality * image_scale) + float image_scale_corr; // I_true = image_scale_corr * I; = prescaling_corr / (partiality * image_scale) bool observed = false; bool on_ice_ring = false; // sits on a hexagonal-ice powder ring: excluded from scaling, kept for merging }; diff --git a/docs/CBOR.md b/docs/CBOR.md index 536a11803..8be543079 100644 --- a/docs/CBOR.md +++ b/docs/CBOR.md @@ -167,7 +167,7 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s | - sigma | float | standard deviation, estimated from counting statistics (photons) | | | | - image | float | image number (present for each spot) | | | | - rp | float | Distance to Ewald sphere \[Angstrom^-1\] | | | -| - rlp | float | Reciprocal Lorentz and polarization corrections | | | +| - rlp | float | Prescaling correction: the product of every deterministic per-reflection correction (reciprocal Lorentz, reciprocal polarization, sensor efficiency). The key name is historical and kept for compatibility | | | | - partiality | float | Partiality of the reflection | | | | - phi | float | phi angle from XDS: difference from middle of current frame, not absolute \[deg\] | | | | - zeta | float | Lorentz zeta factor (reciprocal-space geometry term) | | | diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 5c3d0ca14..70a16cee8 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -159,9 +159,9 @@ where $c$ is the pixel value and the de-biased variance $v$ (background plus mod The integrator is selected by `--integrator boxsum|gaussian|empirical` (default `gaussian`). -### 9.4 Lorentz–polarization factor handling +### 9.4 The prescaling correction -For integrated reflections, polarization correction can be applied as a multiplicative correction to the reflection scale via the geometry-based polarization term (§2.2). A Lorentz-like factor is carried as `rlp` in predictions, and used during scaling/merging (§10). +Every deterministic per-reflection correction is carried as one multiplicative factor, `prescaling_corr`, which the prediction fills and the integrator adds to. It currently holds the reciprocal Lorentz factor (rotation only), the reciprocal polarization factor from the geometry-based term (§2.2), and the sensor's angle-dependent efficiency (§9.6); it is a product, not a fixed list, and terms have been added to it before. Scaling and merging (§10) use it as the numerator of the reflection's total correction. It is not a scale — the fitted per-image scale and the partiality are separate. --- @@ -204,7 +204,7 @@ $ where: - $G_i$ is the image scale factor, -- $L_{ij}$ is a Lorentz-like / geometry factor; predictions carry its **reciprocal** as `rlp`, so $L = 1/\texttt{rlp}$ and the correction below is applied as a multiplication by `rlp`, +- $L_{ij}$ is the prescaling correction of §9.4; predictions carry its **reciprocal** as `prescaling_corr`, so $L = 1/\texttt{prescaling\_corr}$ and the correction below is applied as a multiplication by `prescaling_corr`, - $P_{ij}$ is a partiality term (model-dependent), - $I_h$ is the merged (true) intensity parameter for that unique reflection. diff --git a/docs/HDF5.md b/docs/HDF5.md index 52a46f29c..2b5c4c7ea 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -435,7 +435,7 @@ mostly onto the standard | `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` | | `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) | | `observed_frame` | | standard | image number of the reflection | -| `lp` | | standard | Lorentz–polarization factor (stored as `1/rlp`) | +| `lp` | | standard | reciprocal of the prescaling correction (stored as `1/prescaling_corr`). The NXreflections name is `lp`, but the stored product also carries the sensor efficiency term | | `partiality` | | standard | recorded fraction of the reflection | | `delta_phi` | deg | **extension** | XDS Δφ: offset from the centre of the current frame | | `zeta` | | **extension** | Lorentz ζ factor (reciprocal-space geometry term) | diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 24daa9691..8649a450e 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -335,7 +335,7 @@ Scaling and merging: | `--reference-column