From e715d3999e4659774a8db622e958bf7e80d880d1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 09:48:12 +0200 Subject: [PATCH 01/28] Twinning: normalised L-test, and one verdict read in every Laue class The L-test now divides each intensity by its resolution-shell mean before forming pairs, over the shells whose reaches 1 (the floor the second moment already used). Two index steps are not the same resolution, and on a small cell with a steep fall-off the raw pairs read <|L|> up to 0.08 high: an untwinned crystal 0.568 and a partial twin 0.427 where phenix.xtriage reads 0.486 and 0.360. Normalised, the four reference merges (6toc, 6iu9, 5j23, a lysozyme) agree with xtriage within 0.02 (0.487, 0.376, 0.374, 0.480). Selection stays by shell, never by the reflection's own I/sigma, which biases <|L|> down. Equal-count shells were tried for the second moment and widen its gap to xtriage, so the shells are kept. The L-test is no longer switched off in holohedral Laue classes. Merging I(h) with I(Th) under a false operator gives (I(h)+I(Th))/2 for every twin fraction, the perfect-twin distribution, so <|L|> < 0.42 there is reported as an adopted operator averaging unequal intensities (promotion suspect, or a twin law absorbed into the point group) - a warning, not a veto. Over 42 holohedral merges of the corpus the genuine ones read 0.436-0.513, the over-promoted H3 twin 0.374, and one ~490 A-axis crystal 0.365. The verdict is one line (TwinningVerdictLine) used by the stats text, the report prose, the summary row, the warning and the viewer; a new TWINNING_VERDICT key names it. The twin fraction comes from the statistic that carries the verdict (the L-test unless the call rests on the second moment alone), is not quoted from a second moment under a detected pseudo-translation, and is not quoted at all on a merge under a suspect operator. The pre-search numbers (measured on the P1 search merge) are user-visible, are measured with that merge's own pseudo-translation declared, and leave out the reflections the lattice centring extinguishes - an R lattice in its hexagonal cell otherwise pairs present with absent reflections (5j23: 0.633 with them, 0.40 without). Report-only: no space-group decision, merge or intensity changes. Co-Authored-By: Claude Opus 5 (1M context) --- docs/CPU_DATA_ANALYSIS_DECISIONS.md | 2 +- .../scale_merge/TwinningAnalysis.cpp | 380 ++++++++++++------ image_analysis/scale_merge/TwinningAnalysis.h | 37 +- rugnux/ResultReport.cpp | 110 ++--- rugnux/Rugnux.cpp | 15 +- rugnux/Rugnux.h | 3 + tests/TwinningAnalysisTest.cpp | 100 ++++- viewer/windows/JFJochMergeStatsWindow.cpp | 6 +- 8 files changed, 459 insertions(+), 194 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index 27181699b..7ab356908 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -52,7 +52,7 @@ Both search merges also drop the **frames whose fitted per-frame scale came out ### 13.2 Twinning check, and translational pseudo-symmetry -A Padilla–Yeates $L$-test ($\langle|L|\rangle$, $\langle L^2\rangle$ — 0.500 and 0.333 untwinned, 0.375 and 0.200 for a perfect twin) and the second moment $\langle I^2\rangle/\langle I\rangle^2$ (2.0 for untwinned acentric data, 1.5 for a perfect twin; taken per resolution shell with noise-only shells skipped and Wilson outliers rejected, so a single strong reflection in a collapsed-mean shell cannot skew it) are written to the merged mmCIF as a twinning diagnostic. A merohedral twin law exists only where the Laue class is a proper subgroup of the lattice holohedry, so in the high-symmetry holohedral classes ($4/mmm$, $6/mmm$, $m\bar{3}m$, and $\bar{3}m$ on a rhombohedral lattice) twinning is never flagged, and a low $\langle|L|\rangle$ there is reported as a statistical artefact instead. The low-symmetry holohedral classes ($\bar{1}$, $2/m$, $mmm$) also admit no strictly merohedral law, but they stay eligible for the flag on purpose: *pseudo*-merohedral twinning through an accidentally special metric cannot be ruled out from the symmetry alone, and those are the classes it happens in. +A Padilla–Yeates $L$-test ($\langle|L|\rangle$, $\langle L^2\rangle$ — 0.500 and 0.333 untwinned, 0.375 and 0.200 for a perfect twin) and the second moment $\langle I^2\rangle/\langle I\rangle^2$ (2.0 for untwinned acentric data, 1.5 for a perfect twin) are written to the merged mmCIF as a twinning diagnostic. Both are taken on intensities divided by their resolution-shell mean, over the shells whose $\langle I/\sigma\rangle$ reaches 1, with Wilson outliers rejected; the $L$-test pairs are therefore compared on one scale even where two index steps span a steep fall-off, as in phenix.xtriage and ctruncate, and the selection is by shell, never by the individual reflection's $I/\sigma$, which would cut the weak tail and bias $\langle|L|\rangle$ down. The twin fraction is quoted from the statistic that carries the verdict — the $L$-test unless the call rests on the second moment alone — and a second moment is not turned into a fraction under a detected pseudo-translation, which inflates it. A merohedral twin law exists only where the Laue class is a proper subgroup of the lattice holohedry, so in the high-symmetry holohedral classes ($4/mmm$, $6/mmm$, $m\bar{3}m$, and $\bar{3}m$ on a rhombohedral lattice) no twin is called. The $L$-test is still read there, for a different question: merging $I(h)$ with $I(Th)$ under a false operator $T$ gives $(I(h)+I(Th))/2$ whatever the twin fraction, which has the perfect-twin distribution, so $\langle|L|\rangle$ below 0.42 in a holohedral class is reported as **an adopted operator averaging unequal intensities** — the space group is too high, or a twin law was absorbed into the point group — a warning, never a change to the space group. A genuine operator leaves the untwinned 0.5. Reflections that overlap along a very long axis narrow the distribution the same way, which is one reason it stays a warning. The same numbers measured on the P1 merge the space-group search was given, before any point group was adopted, are reported beside them (`_BEFORE_SEARCH`), with that merge's own pseudo-translation declared and the reflections its lattice centring extinguishes left out. The low-symmetry holohedral classes ($\bar{1}$, $2/m$, $mmm$) also admit no strictly merohedral law, but they stay eligible for the twin call on purpose: *pseudo*-merohedral twinning through an accidentally special metric cannot be ruled out from the symmetry alone, and those are the classes it happens in. **Translational pseudo-symmetry** — two copies of the contents of the asymmetric unit related by a pure translation that is not a lattice vector — is looked for on every merging run, because it is the classic predictor of a failed molecular replacement, and because it raises the second moment where twinning lowers it, so each can mask the other's test. The detection is the native-Patterson route of Read, Adams & McCoy (see the [references](CPU_DATA_ANALYSIS.md#references)): the largest off-origin peak of a Patterson computed from the merged intensities, as a fraction of the origin peak, with the peak vector then refined against the intensity modulation it should produce — the ratio of the strongest to the weakest bin mean of $\langle E^2\rangle$ over the phase $\mathrm{frac}(\mathbf{h}\cdot\mathbf{u})$. Both halves are scored against a null computed for the crystal at hand rather than against a fixed bound — both against the same intensities **permuted within resolution shells** — the peak against that map, the modulation against the same measurement made on the shuffled intensities and started where the measurement starts — because the noise floor of the peak statistic spans an order of magnitude across data sets, so no fixed percentage means the same thing twice. The shuffle leaves the reflection count, the $E^2$ distribution and the phase-bin populations untouched and removes only the correlation between a reflection's intensity and $\mathbf{h}\cdot\mathbf{u}$, which is the thing the modulation claims to see. Re-running the greedy search from random starting vectors over the *same* intensities does not work as a control: a greedy search started anywhere walks into a real modulation's basin, so it measures the search **and** whatever modulation the crystal carries — measured over 110 merged datasets, single draws of that control reached 199× and 6162× on crystals whose own modulation reads 4.0× and 67×, and on 28 of the 110 it came out at or above the signal it is subtracted from. Requiring both halves is what keeps the false-positive rate down; either alone over-calls by about a factor of two. A translation the merged data are **exactly** invariant under is reported as an undeclared lattice translation instead — a translation the data are exactly invariant under is a lattice vector by definition, so the centring or the cell is wrong, not the packing — and the pseudo-symmetry search continues underneath it, so a real pseudo-translation sitting under an undeclared centring is still found. The finding itself is report-only (the `TNCS_*` keys of `_report.txt`): it gates nothing and changes no reflection, no scale and no group — but the axial-absence test (§13.1) and the $L$-test here both correct themselves against the modulation it measures. diff --git a/image_analysis/scale_merge/TwinningAnalysis.cpp b/image_analysis/scale_merge/TwinningAnalysis.cpp index 0a16c3cde..6f96e9ab2 100644 --- a/image_analysis/scale_merge/TwinningAnalysis.cpp +++ b/image_analysis/scale_merge/TwinningAnalysis.cpp @@ -29,6 +29,33 @@ namespace { // admit no such operation, so twinning is geometrically impossible and the intensity statistics // cannot be indicating it. Low-symmetry classes stay eligible because pseudo-merohedral twinning // through an accidental metric specialisation cannot be excluded from the symmetry alone. + // <|L|> of a partial twin with fraction a: I = (1-a) X + a Y for independent Wilson X, Y, and the + // expectation of |I1 - I2|/(I1 + I2) over two independent such intensities, integrated in closed + // form (0.500 at a = 0, 0.440 at 0.1, 0.408 at 0.2, 0.375 at 0.5; checked by simulation, and + // the relation ctruncate tabulates). + double MeanAbsLForTwinFraction(double a) { + const double p = 1.0 - a, q = a; + const double A = 1.0 / q, B = 1.0 / p - 1.0 / q; + auto F = [&](double t) { return 2.0 / (B * B) * std::log(A + B * t) + (2.0 * A / B + 1.0) / (B * (A + B * t)); }; + const double J = F(1.0) + F(0.0) - 2.0 * F(0.5); + return ((p * p + q * q) / 2.0 - 2.0 * J) / ((p - q) * (p - q)); + } + + // The inverse, by bisection: the relation is monotonic, and flat near a = 0.5, where <|L|> can no + // longer tell 0.4 from 0.5. + double TwinFractionFromMeanAbsL(double mean_abs_l) { + double lo = 1e-4, hi = 0.4999; + if (mean_abs_l >= MeanAbsLForTwinFraction(lo)) + return 0.0; + if (mean_abs_l <= MeanAbsLForTwinFraction(hi)) + return 0.5; + for (int i = 0; i < 50; ++i) { + const double mid = 0.5 * (lo + hi); + (MeanAbsLForTwinFraction(mid) > mean_abs_l ? lo : hi) = mid; + } + return 0.5 * (lo + hi); + } + bool MerohedralTwinningPossible(const gemmi::SpaceGroup* sg) { if (!sg) return true; // P1 / unknown symmetry: cannot rule twinning out @@ -47,7 +74,7 @@ namespace { TwinningAnalysisResult AnalyzeTwinning(const std::vector& merged, const gemmi::SpaceGroup* space_group, int resolution_shells, - const std::array* tncs_vector) { + const std::array* tncs_vector, char lattice_centring) { TwinningAnalysisResult result; if (merged.empty()) return result; @@ -58,6 +85,108 @@ TwinningAnalysisResult AnalyzeTwinning(const std::vector& merg auto acentric = [&](const MergedReflection& r) { return !space_group || !gops.is_reflection_centric(gemmi::Op::Miller{{r.h, r.k, r.l}}); }; + // A merge in P1 of a centred lattice still holds the reflections the centring extinguishes (an + // R lattice indexed in its hexagonal cell keeps the two thirds with -h+k+l != 3n). They are + // noise, not intensities, and an L-test partner step that crosses into them pairs a real + // reflection with an absent one. So they are left out of both statistics. + gemmi::GroupOps centring; + centring.sym_ops = {gemmi::Op::identity()}; + centring.cen_ops = gemmi::centring_vectors(lattice_centring); + auto used = [&](const MergedReflection& r) { + return UsableIntensity(r) && acentric(r) + && !centring.is_systematically_absent(gemmi::Op::Miller{{r.h, r.k, r.l}}); + }; + + // --- Resolution shells: the normalising mean of both statistics --- + // Binning by 1/d^2 removes the resolution fall-off, so the second moment is 2.0 (untwinned) or 1.5 + // (perfect twin) regardless of the overall B-factor, and an L-test pair compares two intensities + // on the same scale. The moment divides by the *square* of the shell-mean intensity, so it is not + // robust: on weak or mis-integrated data a shell mean can collapse to the noise floor and one + // outlier reflection then dominates (I/mean)^2 (a single I=158 in a mean~1 shell contributed 78% of + // a whole dataset's value). To keep these twinning indicators rather than data-quality artefacts - + // as phenix.xtriage does - we skip noise-only shells ( below 1) and reject Wilson outliers + // (E^2 above 8, ~exp(-8) upper tail) with one shell-mean re-iteration so the outlier does not + // corrupt the normalising mean either. + // + // The shells are equal in width in 1/d^2. Equal-COUNT shells were measured as the alternative + // (the first equal-width shell spans a wide range of d, and its fall-off inflates the moment): + // they move the moment the wrong way on the same data, widening its gap to phenix.xtriage by + // 0.01-0.04 rather than closing it, so the shells were kept as they are. + constexpr double min_shell_isig = 1.0; // shells below this are noise, not signal + constexpr double wilson_outlier_e2 = 8.0; // reject improbably strong reflections (P ~ e^-8) + const int n_shells = std::max(1, resolution_shells); + double min_s = std::numeric_limits::infinity(); + double max_s = -std::numeric_limits::infinity(); + for (const auto& r : merged) { + if (!used(r)) + continue; + const double s = 1.0 / (r.d * r.d); + min_s = std::min(min_s, s); + max_s = std::max(max_s, s); + } + if (!std::isfinite(min_s) || !(max_s > min_s)) + return result; + auto shell_of = [&](double d) { + const double t = (1.0 / (d * d) - min_s) / (max_s - min_s); + return std::min(n_shells - 1, std::max(0, static_cast(t * n_shells))); + }; + // Group intensities by shell, and accumulate to gauge each shell's signal. + std::vector> shell_I(n_shells); + std::vector shell_isig_sum(n_shells, 0.0); + std::vector shell_isig_n(n_shells, 0); + for (const auto& r : merged) { + if (!used(r)) + continue; + const int b = shell_of(r.d); + shell_I[b].push_back(r.I); + if (std::isfinite(r.sigma) && r.sigma > 0.0) { + shell_isig_sum[b] += r.I / r.sigma; + shell_isig_n[b] += 1; + } + } + // The mean intensity of each shell that carries signal; 0 marks a shell that does not. + std::vector shell_mean(n_shells, 0.0); + for (int b = 0; b < n_shells; ++b) { + const auto& intensities = shell_I[b]; + if (intensities.empty() || shell_isig_n[b] == 0 + || shell_isig_sum[b] / shell_isig_n[b] < min_shell_isig) + continue; + double sum = 0.0; + for (double I : intensities) + sum += I; + const double mean = sum / intensities.size(); + if (mean <= 0.0) + continue; + // Re-fit the mean over the reflections that pass the outlier cut, so the outlier does not + // inflate the very mean it is measured against. + sum = 0.0; + int n_kept = 0; + for (double I : intensities) + if (I / mean <= wilson_outlier_e2) { + sum += I; + ++n_kept; + } + if (n_kept > 0 && sum > 0.0) + shell_mean[b] = sum / n_kept; + } + + // --- Second moment /^2 of acentric intensities, normalised per resolution shell --- + double sum_e4 = 0.0; + int n_moment = 0; + for (int b = 0; b < n_shells; ++b) { + if (shell_mean[b] <= 0.0) + continue; + for (double I : shell_I[b]) { + const double e2 = I / shell_mean[b]; + if (e2 > wilson_outlier_e2) + continue; + sum_e4 += e2 * e2; + ++n_moment; + } + } + result.moment_reflections = n_moment; + if (n_moment > 0) + result.second_moment = sum_e4 / n_moment; // --- L-test --- // Following Padilla & Yeates (2003) Acta Cryst. D59, 1124-1130 @@ -66,11 +195,24 @@ TwinningAnalysisResult AnalyzeTwinning(const std::vector& merg // merged reflections are unique in the asymmetric unit, so any other merged reflection is // genuinely non-equivalent - exactly the pairing the L-test wants. Only acentric reflections // with positive intensity enter, which also keeps L = (I1-I2)/(I1+I2) bounded in [-1, 1]. + // + // The test assumes the two members of a pair have the same EXPECTED intensity, and two index + // steps are not the same resolution: on a small cell with a large B the pair's expected + // intensities differ by up to a factor ~3 at 2 A, which pushes <|L|> up by as much as 0.08 (an + // untwinned crystal read 0.568, a partial twin 0.427 where phenix.xtriage reads 0.486 and 0.360). + // So each intensity is divided by its shell mean first, as xtriage and ctruncate do, and only the + // shells the second moment reads enter. The selection stays on the SHELL: cutting individual + // reflections on their own I/sigma removes the weak tail of every shell and biases <|L|> down + // (the same untwinned crystal reads 0.423 with I/sigma > 2). std::unordered_map intensity; intensity.reserve(merged.size() * 2); - for (const auto& r : merged) - if (UsableIntensity(r) && r.I > 0.0 && acentric(r)) - intensity.emplace(PackHKL(r.h, r.k, r.l), r.I); + for (const auto& r : merged) { + if (!used(r) || r.I <= 0.0) + continue; + const double mean = shell_mean[shell_of(r.d)]; + if (mean > 0.0) + intensity.emplace(PackHKL(r.h, r.k, r.l), r.I / mean); + } // Step to a nearby, symmetry-independent partner. The axis step of 2 is load-bearing for TWO // reasons, and only the first was ever written down. @@ -165,98 +307,20 @@ TwinningAnalysisResult AnalyzeTwinning(const std::vector& merg run_l_test(all_steps); } - // --- Second moment /^2 of acentric intensities, normalised per resolution shell --- - // Binning by 1/d^2 removes the resolution fall-off, so the moment is 2.0 (untwinned) or 1.5 - // (perfect twin) regardless of the overall B-factor. The moment divides by the *square* of the - // shell-mean intensity, so it is not robust: on weak or mis-integrated data a shell mean can - // collapse to the noise floor and one outlier reflection then dominates (I/mean)^2 (a single - // I=158 in a mean~1 shell contributed 78% of a whole dataset's value). To keep this a twinning - // indicator rather than a data-quality artefact - as phenix.xtriage does - we skip noise-only - // shells ( below 1) and reject Wilson outliers (E^2 above 8, ~exp(-8) upper tail) with - // one shell-mean re-iteration so the outlier does not corrupt the normalising mean either. - constexpr double min_shell_isig = 1.0; // shells below this are noise, not signal - constexpr double wilson_outlier_e2 = 8.0; // reject improbably strong reflections (P ~ e^-8) - int n_shells = std::max(1, resolution_shells); - double min_s = std::numeric_limits::infinity(); - double max_s = -std::numeric_limits::infinity(); - for (const auto& r : merged) { - if (!UsableIntensity(r) || !acentric(r)) - continue; - const double s = 1.0 / (r.d * r.d); - min_s = std::min(min_s, s); - max_s = std::max(max_s, s); - } - if (std::isfinite(min_s) && max_s > min_s) { - auto shell_of = [&](double d) { - const double t = (1.0 / (d * d) - min_s) / (max_s - min_s); - return std::min(n_shells - 1, std::max(0, static_cast(t * n_shells))); - }; - // Group acentric intensities by shell, and accumulate to gauge each shell's signal. - std::vector> shell_I(n_shells); - std::vector shell_isig_sum(n_shells, 0.0); - std::vector shell_isig_n(n_shells, 0); - for (const auto& r : merged) { - if (!UsableIntensity(r) || !acentric(r)) - continue; - const int b = shell_of(r.d); - shell_I[b].push_back(r.I); - if (std::isfinite(r.sigma) && r.sigma > 0.0) { - shell_isig_sum[b] += r.I / r.sigma; - shell_isig_n[b] += 1; - } - } - - double sum_e4 = 0.0; - int n_moment = 0; - for (int b = 0; b < n_shells; ++b) { - const auto& intensities = shell_I[b]; - if (intensities.empty() || shell_isig_n[b] == 0 - || shell_isig_sum[b] / shell_isig_n[b] < min_shell_isig) - continue; - - double sum = 0.0; - for (double I : intensities) - sum += I; - double mean = sum / intensities.size(); - if (mean <= 0.0) - continue; - // Re-fit the mean over the reflections that pass the outlier cut, so the outlier does not - // inflate the very mean it is measured against. - sum = 0.0; - int n_kept = 0; - for (double I : intensities) - if (I / mean <= wilson_outlier_e2) { - sum += I; - ++n_kept; - } - if (n_kept == 0) - continue; - mean = sum / n_kept; - if (mean <= 0.0) - continue; - - for (double I : intensities) { - const double e2 = I / mean; - if (e2 > wilson_outlier_e2) - continue; - sum_e4 += e2 * e2; - ++n_moment; - } - } - result.moment_reflections = n_moment; - if (n_moment > 0) - result.second_moment = sum_e4 / n_moment; - } - - // Twin fraction from the second moment M = 2(1 - a + a^2): a = (1 - sqrt(2M-3))/2. - if (result.second_moment > 0.0) { - const double m = std::clamp(result.second_moment, 1.5, 2.0); - result.estimated_twin_fraction = (1.0 - std::sqrt(std::max(0.0, 2.0 * m - 3.0))) / 2.0; - } - // Either indicator dropping clearly below its untwinned value is suspicious - but only where a twin - // law can actually exist. In a holohedral Laue class (e.g. 422) no merohedral twinning is - // possible, so a low <|L|> is a statistical artefact (correlated near-neighbours) rather than a - // twin, and must not be flagged. + // Either indicator dropping clearly below its untwinned value is suspicious - in every Laue + // class, holohedral ones included. A holohedral class admits no twin law WITHIN it, so there a low + // reading does not mean "twinned in this group"; it means an operator of the group averages + // intensities that are not equal. Merging I(h) with I(Th) under a false operator T gives + // (I(h) + I(Th))/2 for every twin fraction - the fraction cancels - so a merge under a false + // operator has the perfect-twin distribution by construction, whether the crystal is an untwinned + // pseudo-symmetric one, a partial twin or a perfect twin, and only a genuine operator leaves it + // at the untwinned 0.5. Noise cannot fake it (with the I > 0 selection it pushes <|L|> towards + // ~0.43, not 0.375) and a pseudo-translation pushes the second moment UP. Measured over 42 + // holohedral merges of the corpus, the genuine ones read <|L|> 0.436-0.513 and the one + // over-promotion (an H3 twin merged in R32) 0.374. One more reads low, 0.365: a crystal with a + // ~490 A axis, whose neighbouring reflections overlap and share counts - averaging of another + // kind, which narrows the distribution the same way. That is why this is a warning and never a + // veto on the space group. result.merohedral_twinning_possible = MerohedralTwinningPossible(space_group); // The two indicators can only move one way under twinning: <|L|> down from 0.500 towards 0.375, // the second moment down from 2.0 towards 1.5. A narrow intensity distribution sitting next to an @@ -282,9 +346,39 @@ TwinningAnalysisResult AnalyzeTwinning(const std::vector& merg // alone, and the text says the L-test could not be read. This REMOVES a veto; it never adds one. const bool l_test_usable = result.l_test_pairs > 0 && !result.l_test_contaminated_by_tncs; const bool l_test_contradicts_twin = l_test_usable && result.mean_abs_l >= 0.50; - result.twinning_suspected = result.merohedral_twinning_possible && !l_test_contradicts_twin && - ((l_test_usable && result.mean_abs_l < 0.44) || - (result.moment_reflections > 0 && result.second_moment < 1.85)); + const bool l_test_low = l_test_usable && result.mean_abs_l < 0.44; + const bool moment_low = result.moment_reflections > 0 && result.second_moment < 1.85; + if (result.merohedral_twinning_possible) + result.twinning_suspected = !l_test_contradicts_twin && (l_test_low || moment_low); + else { + // In a holohedral class the alternative is not a partial twin anywhere between 0.5 and 0.375 + // but a false operator, which puts <|L|> at the perfect-twin 0.375 itself - so the bar sits + // nearer that value. Measured on the corpus (normalised L-test): the weakest genuine + // holohedral crystal reads 0.436, the over-promoted twin 0.374. The second moment does not + // enter here: it is the statistic a pseudo-translation inflates (that same twin read 2.08, + // the untwinned value), and it is the one a pseudo-cubic lattice in a 4/mmm group, whose twin + // law lies OUTSIDE the Laue class, pulls down (1.66 beside an untwinned <|L|>). + result.adopted_operators_suspect = l_test_usable && result.mean_abs_l < 0.42; + result.twinning_suspected = result.adopted_operators_suspect; + } + + // The twin fraction, from the statistic that carries the verdict. Merged under a false operator + // the fraction is lost (see above), so none is quoted there. Otherwise the L-test, which is local + // and so indifferent to anisotropy and to a pseudo-translation it has been repaired for, unless + // the call rests on the second moment alone. The second moment is not quoted under a detected + // pseudo-translation, which inflates it: an H3 twin with tNCS read 2.08, the untwinned value, + // beside a perfect-twin <|L|>. + if (!result.adopted_operators_suspect) { + if (l_test_usable && (l_test_low || !moment_low)) { + result.estimated_twin_fraction = TwinFractionFromMeanAbsL(result.mean_abs_l); + result.twin_fraction_source = TwinFractionSource::LTest; + } else if (result.moment_reflections > 0 && tncs_vector == nullptr) { + // M = 2(1 - a + a^2): a = (1 - sqrt(2M-3))/2. + const double m = std::clamp(result.second_moment, 1.5, 2.0); + result.estimated_twin_fraction = (1.0 - std::sqrt(std::max(0.0, 2.0 * m - 3.0))) / 2.0; + result.twin_fraction_source = TwinFractionSource::SecondMoment; + } + } return result; } @@ -310,34 +404,46 @@ std::string TwinningAnalysisToText(const TwinningAnalysisResult& result) { if (result.moment_reflections > 0) os << " Second moment /^2 = " << result.second_moment << " [untwinned 2.00, perfect twin 1.50]\n"; - if (result.twinning_suspected && result.estimated_twin_fraction > 0.01) - os << " => Twinning suspected (estimated twin fraction ~" - << result.estimated_twin_fraction << "). Statistics flag the presence of twinning, not\n" - << " the twin law; the law itself is a question for a dedicated twin-law analysis.\n"; - else if (result.twinning_suspected) - // The fraction is derived from the second moment alone, so a twin called by the L-test on a - // crystal whose second moment sits at or above its untwinned value has no fraction to quote - - // printing the 0.000 the formula returns would contradict the sentence it sits in. The - // disagreement is worth stating, and where a pseudo-symmetry is present it has a named cause. - os << " => Twinning suspected on the L-test. The second moment does not agree (" - << result.second_moment << ", at or\n" - << " above its untwinned 2.00), so no twin fraction is quoted" - << (result.l_test_tncs_step_restricted - ? " - and this\n" - " crystal's translational pseudo-symmetry is a known reason for the second\n" - " moment to run high, so the L-test carries the verdict here.\n" - : ".\n") - << " Statistics flag the presence of twinning, not the twin law; the law itself is\n" - << " a question for a dedicated twin-law analysis.\n"; - else if (!result.merohedral_twinning_possible && result.laue_class_was_chosen_by_promotion) - os << " => Cannot rule out twinning from these numbers: the Laue class is holohedral, so no\n" - << " merohedral twin law exists WITHIN it - but this Laue class was chosen by the\n" - << " space-group search itself, and promoting into a twin's holohedry is precisely what\n" - << " a merohedral twin looks like. The subgroup statistics reported by the search,\n" - << " not these numbers, are where the twinning is decided.\n"; + const bool l_test_read = result.l_test_pairs > 0 && !result.l_test_contaminated_by_tncs; + if (result.adopted_operators_suspect) + os << " => An adopted operator relates UNEQUAL intensities. The Laue class is holohedral, so no\n" + << " twin law exists within it - yet the intensities read like a twin's, and merging under\n" + << " a false operator produces exactly that distribution, whatever the twin fraction.\n" + << (result.laue_class_was_chosen_by_promotion + ? " This class was chosen by the space-group search: the promotion is suspect.\n" + : " The space group in use is higher than these intensities support.\n") + << " Either the crystal has lower symmetry, or it is a twin whose law the point group has\n" + << " absorbed; either way it has to be refined in a subgroup. The twin fraction cannot be\n" + << " measured on this merge.\n"; + else if (result.twinning_suspected) { + os << " => Twinning suspected"; + if (result.twin_fraction_source == TwinFractionSource::LTest) + os << " (estimated twin fraction ~" << result.estimated_twin_fraction << ", from the L-test)"; + else if (result.twin_fraction_source == TwinFractionSource::SecondMoment) + os << " (estimated twin fraction ~" << result.estimated_twin_fraction + << ", from the second moment)"; + else + os << " on the second moment; no twin fraction is quoted, because this crystal's\n" + << " pseudo-translation inflates the second moment and <|L|> does not agree"; + os << ".\n"; + if (l_test_read && result.mean_abs_l < 0.44 && result.moment_reflections > 0 + && result.second_moment >= 1.85) + os << " The second moment does not agree (" << result.second_moment << ")" + << (result.l_test_tncs_step_restricted + ? " - this crystal's translational pseudo-symmetry is a known reason for it\n" + " to run high, so the L-test carries the verdict here.\n" + : ".\n"); + os << " Statistics flag the presence of twinning, not the twin law; the law itself is a\n" + << " question for a dedicated twin-law analysis.\n"; + } else if (!result.merohedral_twinning_possible && result.laue_class_was_chosen_by_promotion) + os << " => No twinning indicated. The Laue class is holohedral, so no twin law exists within it,\n" + << " and it was chosen by the space-group search - but its operators read as genuine: a\n" + << " merge under a false operator has the perfect-twin distribution, not the untwinned one\n" + << " measured here. An operator that is only APPROXIMATE (non-crystallographic symmetry\n" + << " along it) also passes; the subgroup statistics reported by the search speak to that.\n"; else if (!result.merohedral_twinning_possible) - os << " => No twinning: the Laue class is holohedral, so no merohedral twin law exists\n" - << " (any <|L|> below 0.5 here is a statistical artefact, not twinning).\n"; + os << " => No twinning: the Laue class is holohedral, so no merohedral twin law exists, and no\n" + << " operator of it averages unequal intensities.\n"; else if (result.l_test_contaminated_by_tncs) os << " => No twinning indicated by the second moment. The L-test, which is normally the\n" << " stronger of the two, could not be read on this crystal (see above), so this is a\n" @@ -347,3 +453,25 @@ std::string TwinningAnalysisToText(const TwinningAnalysisResult& result) { os << " => No twinning indicated.\n"; return os.str(); } + +std::string TwinningVerdictLine(const TwinningAnalysisResult& result) { + std::ostringstream os; + os << std::fixed << std::setprecision(3); + const bool l_test_read = result.l_test_pairs > 0 && !result.l_test_contaminated_by_tncs; + if (result.adopted_operators_suspect) + os << "SYMMETRY SUSPECT (<|L|> " << result.mean_abs_l << ", second moment " + << std::setprecision(2) << result.second_moment + << ": an adopted operator averages unequal intensities)"; + else if (result.twinning_suspected) { + os << "INDICATED (<|L|> " << result.mean_abs_l; + if (result.twin_fraction_source != TwinFractionSource::None) + os << std::setprecision(2) << ", twin fraction ~" << result.estimated_twin_fraction << " from the " + << (result.twin_fraction_source == TwinFractionSource::LTest ? "L-test" : "second moment"); + os << ")"; + } else if (l_test_read) + os << "no indication (<|L|> " << result.mean_abs_l << ")"; + else + os << "no indication (second moment " << std::setprecision(2) << result.second_moment + << "; <|L|> not readable)"; + return os.str(); +} diff --git a/image_analysis/scale_merge/TwinningAnalysis.h b/image_analysis/scale_merge/TwinningAnalysis.h index 538f9a1ff..965c4da89 100644 --- a/image_analysis/scale_merge/TwinningAnalysis.h +++ b/image_analysis/scale_merge/TwinningAnalysis.h @@ -10,18 +10,23 @@ #include "../../common/Reflection.h" #include "gemmi/symmetry.hpp" -// Intensity-statistics test for merohedral twinning, in the spirit of phenix.xtriage. Two robust, -// normalisation-free indicators on the merged intensities: +// Intensity-statistics test for merohedral twinning, in the spirit of phenix.xtriage. Two +// indicators on the merged intensities, both normalised per resolution shell: // // - Padilla & Yeates L-test (Acta Cryst D59, 2003): for pairs of nearby but symmetry-independent // reflections, L = (I1 - I2)/(I1 + I2). Untwinned acentric data give <|L|> = 0.5, = 1/3; // a perfect merohedral twin narrows this to <|L|> = 0.375, = 0.2. Insensitive to -// anisotropy and overall scale because it is a local ratio. +// anisotropy and overall scale because it is a local ratio of shell-normalised intensities. // // - Second moment /^2 of acentric intensities, normalised per resolution shell so it is // 2.0 for untwinned data and 1.5 for a perfect twin (= 2(1 - a + a^2) for twin fraction a). // -// These flag the *presence* of twinning; they do not by themselves identify the twin law. +// These flag the *presence* of twinning; they do not by themselves identify the twin law. In a +// holohedral Laue class, where no twin law exists, the same low reading says instead that an operator +// of the group averages unequal intensities - the group is too high, or a twin law was absorbed into +// it (adopted_operators_suspect). + +enum class TwinFractionSource { None, LTest, SecondMoment }; struct TwinningAnalysisResult { int l_test_pairs = 0; @@ -31,14 +36,25 @@ struct TwinningAnalysisResult { int moment_reflections = 0; double second_moment = 0.0; // /^2 (shell-normalised): 2.0 untwinned, 1.5 perfect twin - // Twin fraction estimated from the second moment, a = (1 - sqrt(2M-3))/2, clamped to [0, 0.5]. + // Twin fraction in [0, 0.5], from the statistic named by twin_fraction_source: the L-test unless + // the verdict rests on the second moment alone, a = (1 - sqrt(2M-3))/2. None where it cannot be + // measured - merged under a false operator, or a second moment under a pseudo-translation. double estimated_twin_fraction = 0.0; + TwinFractionSource twin_fraction_source = TwinFractionSource::None; + + // An indicator reads twin-like (<|L|> < 0.44 or second moment < 1.85, and <|L|> not at or above + // 0.5). Set in every Laue class; what it means depends on the next two fields. bool twinning_suspected = false; // False when the Laue class is holohedral (4/mmm, 6/mmm, m-3m, rhombohedral -3m): no merohedral - // twin law can exist, so a low <|L|> / second moment there is a statistical artefact, not twinning. + // twin law can exist WITHIN it. bool merohedral_twinning_possible = true; + // twinning_suspected in a holohedral class: an operator of the group averages intensities that + // are not equal, so the group is too high or it has absorbed a twin law. A warning about the + // symmetry, not a twin call - and the twin fraction is not measurable on such a merge. + bool adopted_operators_suspect = false; + // How the L-test coped with a translational pseudo-symmetry, when one was declared (see the // partner-step note in the .cpp). Exactly one of these can be set: // - `l_test_tncs_step_restricted`: partners were taken only along steps that preserve the @@ -58,6 +74,8 @@ struct TwinningAnalysisResult { // The space group (when known) is used to drop centric reflections, which follow different // statistics (<|L|>, /^2 = 3 not 2); pass nullptr for P1 data, where none are centric. +// `lattice_centring` drops the reflections the lattice centring extinguishes, for a P1 merge of a +// centred lattice (whose space_group is nullptr and so says nothing about them). // `tncs_vector` is the fractional pseudo-translation of a DETECTED translational pseudo-symmetry on // this same merge (TranslationalNCSVectorForLTest), or nullptr. It changes which partner reflections // the L-test pairs with, so it has to be an argument rather than a flag set on the result afterwards: @@ -65,6 +83,11 @@ struct TwinningAnalysisResult { TwinningAnalysisResult AnalyzeTwinning(const std::vector& merged, const gemmi::SpaceGroup* space_group = nullptr, int resolution_shells = 20, - const std::array* tncs_vector = nullptr); + const std::array* tncs_vector = nullptr, + char lattice_centring = 'P'); std::string TwinningAnalysisToText(const TwinningAnalysisResult& result); + +// The verdict in one line - "INDICATED (...)", "SYMMETRY SUSPECT (...)" or "no indication (...)" - +// so the report's summary row, its prose and the viewer all say the same thing. +std::string TwinningVerdictLine(const TwinningAnalysisResult& result); diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index f3d6b25e4..bb48a625c 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -1054,59 +1054,79 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, : "UNAFFECTED", {"UNAFFECTED", "REPAIRED", "UNREADABLE"})); Add(s, KeyReal("SECOND_MOMENT_I", tw.second_moment, "{:.4f}")); - Add(s, KeyReal("ESTIMATED_TWIN_FRACTION", tw.estimated_twin_fraction, "{:.3f}")); - // Which statistic the fraction came from. The L-test in the same block can imply a much - // larger fraction, and printing the two side by side with no label made the headline key - // look like a reconciliation of both when it is derived from one of them. - Add(s, KeyEnum("ESTIMATED_TWIN_FRACTION_FROM", "SECOND_MOMENT", - {"SECOND_MOMENT", "L_TEST"})); + // One verdict, in the words every other place uses (TwinningVerdictLine): INDICATED is a twin + // in a class that admits one; SYMMETRY_SUSPECT is the same low reading in a holohedral class, + // where it says an adopted operator averages unequal intensities. + Add(s, KeyEnum("TWINNING_VERDICT", + tw.adopted_operators_suspect ? "SYMMETRY_SUSPECT" + : tw.twinning_suspected ? "INDICATED" + : "NO_INDICATION", + {"NO_INDICATION", "INDICATED", "SYMMETRY_SUSPECT"})); + // The fraction comes from the statistic that carries the verdict, and the key beside it says + // which. None is quoted where it cannot be measured: merged under a false operator, or from a + // second moment a pseudo-translation inflates. + if (tw.twin_fraction_source != TwinFractionSource::None) + Add(s, KeyReal("ESTIMATED_TWIN_FRACTION", tw.estimated_twin_fraction, "{:.3f}")); + Add(s, KeyEnum("ESTIMATED_TWIN_FRACTION_FROM", + tw.twin_fraction_source == TwinFractionSource::LTest ? "L_TEST" + : tw.twin_fraction_source == TwinFractionSource::SecondMoment ? "SECOND_MOMENT" + : "NONE", + {"SECOND_MOMENT", "L_TEST", "NONE"})); Add(s, KeyReal("L_TEST_MEAN_L_SQUARED", tw.mean_l_squared, "{:.4f}", true)); if (result.pre_promotion_twinning) { - Add(s, KeyReal("L_TEST_MEAN_ABS_L_BEFORE_SEARCH", - result.pre_promotion_twinning->mean_abs_l, "{:.4f}", true)); - Add(s, KeyReal("SECOND_MOMENT_I_BEFORE_SEARCH", - result.pre_promotion_twinning->second_moment, "{:.4f}", true)); + const auto &pre = *result.pre_promotion_twinning; + Add(s, KeyReal("L_TEST_MEAN_ABS_L_BEFORE_SEARCH", pre.mean_abs_l, "{:.4f}")); + Add(s, KeyReal("SECOND_MOMENT_I_BEFORE_SEARCH", pre.second_moment, "{:.4f}")); + Add(s, KeyEnum("L_TEST_BEFORE_SEARCH_VS_TNCS", + pre.l_test_contaminated_by_tncs ? "UNREADABLE" + : pre.l_test_tncs_step_restricted ? "REPAIRED" + : result.pre_promotion_tncs_measured ? "UNAFFECTED" + : "NOT_MEASURED", + {"UNAFFECTED", "REPAIRED", "UNREADABLE", "NOT_MEASURED"})); } Add(s, Blank()); - // One calm sentence, in the same terms in both branches. The old prose opened "Cannot rule - // out twinning from these numbers" on runs whose own key said FALSE - a property of the - // symmetry stated as if it were a finding about these data. - if (tw.twinning_suspected) + Add(s, Prose(" Twinning: " + TwinningVerdictLine(tw) + ".")); + if (tw.adopted_operators_suspect) Add(s, Prose(fmt::format( - " Twinning: INDICATED. <|L|> is {:.3f}, where an untwinned crystal reads about 0.5 and\n" - " a perfect twin about 0.375. The estimated fraction {:.2f} is derived from the second\n" - " moment alone and the L-test may imply a larger one. Refinement against these data\n" - " needs a twin law.", - tw.mean_abs_l, tw.estimated_twin_fraction))); + " The Laue class is holohedral, so no twin law exists within it - but merging under a\n" + " false operator gives exactly this distribution, whatever the twin fraction. {} Either\n" + " the crystal has lower symmetry or it is a twin whose law the point group absorbed;\n" + " refinement belongs in a subgroup.", + tw.laue_class_was_chosen_by_promotion + ? "The class was chosen by the space-group search, so that promotion is suspect." + : "The space group in use is higher than these intensities support."))); + else if (tw.twinning_suspected) + Add(s, Prose(" Refinement against these data needs a twin law.")); else if (tw.l_test_contaminated_by_tncs) // <|L|> is the number this sentence would otherwise rest on, and on this crystal it is - // biased by the pseudo-translation above. Saying "no indication (<|L|> 0.529)" here - // would be the same silent veto in a different place. - Add(s, Prose(fmt::format( - " Twinning: no indication from the second moment ({:.3f}, against 2.00 for an untwinned\n" - " crystal and 1.50 for a perfect twin). The L-test, normally the stronger of the two, could\n" - " not be read here: this crystal's translational pseudo-symmetry biases <|L|> and no partner\n" - " reflection available to the test escapes that bias. A weaker statement than usual.", - tw.second_moment))); - else - Add(s, Prose(fmt::format( - " Twinning: no indication (<|L|> {:.3f}, against about 0.5 for an untwinned crystal\n" - " and 0.375 for a perfect twin).", tw.mean_abs_l))); + // biased by the pseudo-translation above. + Add(s, Prose(" The L-test, normally the stronger of the two statistics, could not be read here:\n" + " this crystal's translational pseudo-symmetry biases <|L|> and no partner reflection\n" + " available to the test escapes that bias. A weaker statement than usual.")); Add(s, Prose("\n" + TwinningAnalysisToText(tw), true)); - if (result.pre_promotion_twinning) + if (result.pre_promotion_twinning) { + const auto &pre = *result.pre_promotion_twinning; + Add(s, Prose(fmt::format( + " Before the space-group search (P1 merge{}): {}.", + result.pre_promotion_tncs_measured ? "" : ", not corrected for a pseudo-translation", + TwinningVerdictLine(pre)))); Add(s, Prose(" The _BEFORE_SEARCH pair was measured on the merge the space-group search was given,\n" - " before any point group was adopted. That ordering is the whole point: the statistics\n" - " above are computed in the Laue class this run ADOPTED, so if the search promoted the\n" - " point group they can only report that no twin law exists inside the class it chose -\n" - " and a twin is precisely what would have caused that promotion. Where the two pairs\n" - " disagree, the _BEFORE_SEARCH pair is the one that answers whether the crystal is\n" - " twinned.", true)); - if (tw.twinning_suspected) + " before any point group was adopted. The statistics above are computed in the Laue\n" + " class this run ADOPTED; where the search promoted the point group, a twin law it\n" + " absorbed can only show there as an operator averaging unequal intensities. These\n" + " numbers are not circular in that way.", true)); + } + if (tw.adopted_operators_suspect) Warn(doc, PathologyCode::TWINNING, fmt::format( - "Twinning is indicated (<|L|> = {:.3f}, /^2 = {:.3f}, estimated twin " - "fraction {:.2f} from the second moment) - refinement against the merged data " - "needs a twin law", - tw.mean_abs_l, tw.second_moment, tw.estimated_twin_fraction)); + "An adopted symmetry operator relates unequal intensities (<|L|> = {:.3f} in a holohedral " + "Laue class, where a genuine operator reads about 0.5 and a false one 0.375): the " + "space group may be too high, or the crystal is a twin whose law the point group " + "absorbed - refine in a subgroup", + tw.mean_abs_l)); + else if (tw.twinning_suspected) + Warn(doc, PathologyCode::TWINNING, fmt::format( + "Twinning is indicated ({}) - refinement against the merged data needs a twin law", + TwinningVerdictLine(tw))); } // ---- radiation damage @@ -1725,9 +1745,7 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, result.tncs.peak_percent, result.tncs.vector_length_A) : std::string("no indication")); if (result.twinning.l_test_pairs > 0) - row("Twinning", result.twinning.twinning_suspected - ? fmt::format("INDICATED (<|L|> {:.3f})", result.twinning.mean_abs_l) - : std::string("no indication")); + row("Twinning", TwinningVerdictLine(result.twinning)); const double db = result.merge_statistics.radiation_damage_delta_b; if (std::isfinite(db)) row("Radiation damage", fmt::format("relative B {:+.2f} A^2 over the sweep", db)); diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index baa964214..60942321b 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -5383,7 +5383,20 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // given. Reported alongside the post-adoption analysis: once a higher Laue class has been // adopted, its own twinning test can only say "no twin law exists within this class", which // is circular - the promotion is what a twin would have caused. These numbers are not. - const auto pre_promotion_twinning = AnalyzeTwinning(sm.merged, nullptr); + // + // Measured as the post-adoption test is: with the pseudo-translation of this same merge + // declared, so the L-test can be repaired for it, and - the merge being in P1 - without the + // reflections the lattice centring extinguishes, which an R lattice in its hexagonal cell + // still carries (two thirds of them, and every axis step lands on one). + TranslationalNCSResult pre_tncs; + if (result.consensus_cell) { + pre_tncs = AnalyzeTranslationalNCS(sm.merged, gemmi::UnitCell(*result.consensus_cell)); + result.pre_promotion_tncs_measured = pre_tncs.measurable; + } + const char lattice_centring = end_msg.rotation_lattice_type.has_value() + ? end_msg.rotation_lattice_type->centering : 'P'; + const auto pre_promotion_twinning = AnalyzeTwinning( + sm.merged, nullptr, 20, TranslationalNCSVectorForLTest(pre_tncs), lattice_centring); result.pre_promotion_twinning = pre_promotion_twinning; if (pre_promotion_twinning.twinning_suspected) logger.Warning("Twinning indicators BEFORE the space-group decision (subgroup merge): " diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index e992ffa3b..54ab34074 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -313,6 +313,9 @@ struct ProcessResult { // class is adopted, its own twinning test can say no more than "no twin law exists inside this // class", and the promotion is what a twin would have caused. Empty when no de-novo search ran. std::optional pre_promotion_twinning; + // Whether a pseudo-translation was looked for on that same merge, so its L-test could be repaired + // for one. False when there was no cell to run the Patterson on. + bool pre_promotion_tncs_measured = false; // What --model found, when one was given. Set even when the validation could not run - its // failure_reason then says why - so the report can report the failure rather than leave a run diff --git a/tests/TwinningAnalysisTest.cpp b/tests/TwinningAnalysisTest.cpp index e4999781b..006f9a2b6 100644 --- a/tests/TwinningAnalysisTest.cpp +++ b/tests/TwinningAnalysisTest.cpp @@ -5,6 +5,7 @@ #include #include +#include #include #include @@ -118,17 +119,100 @@ TEST_CASE("A pseudo-translation can hide a twin, and declaring it restores the c CHECK(declared.twinning_suspected); // and restored } -// Standing behaviour that had no test at all. A holohedral Laue class admits no merohedral twin law, -// so whatever the intensity statistics do there they must not be reported as twinning. -TEST_CASE("No twin is called in a holohedral Laue class", "[twinning]") { +namespace { + // The merge of a synthetic crystal in `group`: one row per reflection of that group's asymmetric + // unit, the average of the P1 intensities of its orbit - which is what merging under the group + // does, whether or not its operators are real. + std::vector MergedIn(const std::vector &p1, const char *group) { + const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name(group); + const gemmi::GroupOps gops = sg->operations(); + const gemmi::ReciprocalAsu asu(sg); + std::map, std::pair> sum; + for (const auto &r : p1) { + const auto key = asu.to_asu(gemmi::Op::Miller{{r.h, r.k, r.l}}, gops).first; + auto [it, fresh] = sum.try_emplace({key[0], key[1], key[2]}, r, 0); + if (!fresh) + it->second.first.I += r.I; + it->second.second += 1; + } + std::vector out; + for (auto &[key, v] : sum) { + MergedReflection r = v.first; + r.h = key[0]; + r.k = key[1]; + r.l = key[2]; + r.I /= static_cast(v.second); + out.push_back(r); + } + return out; + } + + std::vector Tetragonal(const char *true_group, double twin_fraction) { + jfjoch_test::SyntheticMergeParams p; + p.true_space_group = true_group; + p.twin_supergroup = "P 4 2 2"; + p.twin_fraction = twin_fraction; + p.d_min_A = 2.5; + return jfjoch_test::GenerateSyntheticMerged(p); + } +} + +// The L-test compares two intensities as if they had the same expected value, and two index steps are +// not the same resolution: on a small cell with a steep fall-off the raw pair differs systematically, +// which reads as an untwinned crystal being "more untwinned" than 0.5. The shell normalisation removes +// that, and the fraction quoted is the L-test's own. +TEST_CASE("L-test on shell-normalised intensities", "[twinning]") { jfjoch_test::SyntheticMergeParams p; - p.true_space_group = "P 4 21 2"; - p.twin_supergroup = "P 4 21 2"; + p.true_space_group = "P 1"; + p.twin_supergroup = "P 1"; + p.wilson_b_A2 = 60.0; + p.d_min_A = 2.5; + const auto r = AnalyzeTwinning(jfjoch_test::GenerateSyntheticMerged(p), nullptr); + INFO("<|L|> " << r.mean_abs_l); + CHECK(r.mean_abs_l == Catch::Approx(0.5).margin(0.02)); + CHECK_FALSE(r.twinning_suspected); + + const auto twin = AnalyzeTwinning(Crystal(0.2), nullptr); + CHECK(twin.twin_fraction_source == TwinFractionSource::LTest); + CHECK(twin.estimated_twin_fraction == Catch::Approx(0.2).margin(0.05)); +} + +// A P1 merge of a centred lattice holds the reflections the centring extinguishes. Told the centring, +// the test leaves them out and reads the crystal; not told, it pairs present with absent ones. +TEST_CASE("L-test on a P1 merge of a centred lattice", "[twinning]") { + jfjoch_test::SyntheticMergeParams p; + p.true_space_group = "R 3 :H"; + p.twin_supergroup = "R 3 :H"; p.d_min_A = 3.0; const auto merged = jfjoch_test::GenerateSyntheticMerged(p); - const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 4 21 2"); - REQUIRE(sg != nullptr); - const auto r = AnalyzeTwinning(merged, sg); + const auto told = AnalyzeTwinning(merged, nullptr, 20, nullptr, 'R'); + CHECK(told.mean_abs_l == Catch::Approx(0.5).margin(0.02)); + const auto not_told = AnalyzeTwinning(merged, nullptr); + CHECK(std::fabs(not_told.mean_abs_l - 0.5) > 0.05); +} + +// A genuine holohedral crystal: no twin law exists, and its operators read as real. +TEST_CASE("No twin is called in a genuine holohedral Laue class", "[twinning]") { + const auto merged = MergedIn(Tetragonal("P 4 2 2", 0.0), "P 4 2 2"); + const auto r = AnalyzeTwinning(merged, gemmi::find_spacegroup_by_name("P 4 2 2")); CHECK_FALSE(r.merohedral_twinning_possible); + CHECK(r.mean_abs_l == Catch::Approx(0.5).margin(0.03)); CHECK_FALSE(r.twinning_suspected); + CHECK_FALSE(r.adopted_operators_suspect); +} + +// Merging under a false operator averages I(h) with I(Th), and (I(h) + I(Th))/2 has the perfect-twin +// distribution whatever the twin fraction - so a subgroup crystal merged in its lattice's holohedry +// reads <|L|> ~0.375 at every fraction, and the holohedral class must say so rather than "no twin +// law exists". +TEST_CASE("A holohedral merge under a false operator is called suspect", "[twinning]") { + const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 4 2 2"); + for (double alpha : {0.0, 0.2, 0.5}) { + const auto r = AnalyzeTwinning(MergedIn(Tetragonal("P 4", alpha), "P 4 2 2"), sg); + CAPTURE(alpha); + CHECK(r.mean_abs_l == Catch::Approx(0.375).margin(0.02)); + CHECK(r.adopted_operators_suspect); + CHECK(r.twin_fraction_source == TwinFractionSource::None); // lost in the merge + CHECK(TwinningVerdictLine(r).rfind("SYMMETRY SUSPECT", 0) == 0); + } } diff --git a/viewer/windows/JFJochMergeStatsWindow.cpp b/viewer/windows/JFJochMergeStatsWindow.cpp index c6260e622..d04de09a0 100644 --- a/viewer/windows/JFJochMergeStatsWindow.cpp +++ b/viewer/windows/JFJochMergeStatsWindow.cpp @@ -165,11 +165,7 @@ JFJochMergeStatsWindow::JFJochMergeStatsWindow(const QString &title, const Merge QString text = QString("Twinning: ⟨|L|⟩ = %1 ⟨I²⟩/⟨I⟩² = %2") .arg(twinning.mean_abs_l, 0, 'f', 3) .arg(twinning.second_moment, 0, 'f', 2); - if (twinning.twinning_suspected) - text += QString(" — twinning suspected (estimated fraction ≈ %1)") - .arg(twinning.estimated_twin_fraction, 0, 'f', 2); - else - text += " — no twinning indicated"; + text += " — " + QString::fromStdString(TwinningVerdictLine(twinning)); auto *banner = new QLabel(text, this); banner->setContentsMargins(10, 6, 10, 6); From 7ae1da277ad5ecedfb3c8c9944fa64a8e86f6338 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 09:48:22 +0200 Subject: [PATCH 02/28] Space-group search: log the refusal the two-merge rule overrides When the all-observation and Lorentz-filtered search merges disagree, the higher point group is adopted. The losing arm's answer was dropped, so a promotion one arm refused on its twin gates (H ratio, added-operator R) went through with nothing in the output saying a gate had fired - and the report printed the adopted group's failing H ratio ("2.02 against a bound of 1.85") as if it were a margin. Seen on an H3 partial twin (5j23), refused on both twin gates by the all-observation merge and promoted to R32 by the filtered one. Now the losing arm's refusal (group and reason), or the fact that it never confirmed the operators, is logged as a warning and written to the report (SPACE_GROUP_ARM_OVERRIDE). A bound the adopted group exceeds is printed as "EXCEEDS its bound - that gate fired and was overridden", for the H ratio and for the best-operator R. The comment that claimed both arms keep their vetoes now describes the rule as implemented. The rule itself is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- rugnux/ResultReport.cpp | 21 +++++++++++++++++++-- rugnux/Rugnux.cpp | 31 +++++++++++++++++++++++++++---- rugnux/Rugnux.h | 4 ++++ 3 files changed, 50 insertions(+), 6 deletions(-) diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index bb48a625c..4106b22e4 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -528,9 +528,21 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, // operator and candidate tables that justify it. std::string line = fmt::format(" Point group {} from the intensity correlations", search.point_group_hm); + // A bound the adopted group EXCEEDS is a gate that fired and was overridden - the point group + // was taken from the other search arm - and must not read as the margin of a pass. if (std::isfinite(search.h_ratio)) - line += fmt::format(";\n twin-law H ratio {:.2f} against a bound of {:.2f}", - search.h_ratio, search.h_ratio_bound); + line += search.h_ratio > search.h_ratio_bound + ? fmt::format(";\n twin-law H ratio {:.2f} EXCEEDS its bound of {:.2f} - that gate fired " + "and was overridden", search.h_ratio, search.h_ratio_bound) + : fmt::format(";\n twin-law H ratio {:.2f} against a bound of {:.2f}", + search.h_ratio, search.h_ratio_bound); + // The best-operator R gate is the one that applies once two operators are confirmed, i.e. + // above order 2 (the first step out of P1 is judged on the noise floor instead). + if (search.point_group_order > 2 && std::isfinite(search.r_over_best) + && search.r_over_best_bound > 0.0 && search.r_over_best > search.r_over_best_bound) + line += fmt::format(";\n added-operator R {:.2f}x the best operator EXCEEDS its bound of {:.2f}x - " + "that gate fired and was overridden", search.r_over_best, + search.r_over_best_bound); line += ".\n The space group follows from the systematic absences."; if (!alts.empty()) line += fmt::format("\n These data cannot separate it from {} - both are consistent with" @@ -541,6 +553,11 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, " cannot name. Molecular replacement settles it: one of the two refines and the\n" " other does not - the normal outcome for a chiral group, not a failure of the\n" " run.")); + if (!result.space_group_arm_override.empty()) { + Add(s, KeyText("SPACE_GROUP_ARM_OVERRIDE", result.space_group_arm_override)); + Add(s, Prose(fmt::format(" The two search merges disagreed and the higher point group was taken:\n {}.", + result.space_group_arm_override))); + } if (refused) Add(s, Prose(fmt::format( " A promotion to {} was refused: {}.\n" diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 60942321b..20d4d0063 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -5280,9 +5280,12 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // rotations are real, because a promotion is refused by the most damning statistic it can // be shown and only one arm has to be starved for that to happen, while the ABSENCES are // always judged on the all-observation merge, which is the only one holding the weak - // reflections a systematic absence is made of. Both searches keep their vetoes, so a merohedral twin - // stays refused in each (verified on a real twin at the three frame ranges where it - // over-promotes: refused in both arms every time). + // reflections a systematic absence is made of. The vetoes are NOT kept per arm: a promotion + // one arm refused on its twin gates (H ratio, added-operator R) is adopted when the other arm + // confirms it, because the higher order wins. That held on one real twin at three frame + // ranges, where both arms refused, but an H3 partial twin has been seen refused by the + // all-observation arm on both twin gates and promoted to R32 by the filtered one. So the + // losing arm's refusal is logged and carried to the report below; the rule is unchanged. if (rsm && rsm->GetSearchMinZeta() > 0.0 && !sg_search.point_group_hm.empty()) { const double zeta = rsm->GetSearchMinZeta(); rsm->SetSearchMinZeta(0.0); @@ -5341,10 +5344,29 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b stats_text << " !! " << msg << "\n\n"; result.warnings.push_back({PathologyCode::SYMMETRY_AMBIGUITY, msg}); } - if (order_of(alt) > order_of(sg_search)) + // The arm whose lower answer is about to be overridden, and why it stopped there. A + // refusal by a consistency gate is the case that matters: the higher group is adopted + // anyway, and without this nothing in the output says a gate ever fired. + const auto overridden = [&](const SearchSpaceGroupResult &lower, const char *lower_name, + const char *higher_name) { + std::string msg; + if (!lower.refused_point_group_hm.empty()) + msg = fmt::format("the {} merge REFUSED {} ({}); overridden, because the {} merge " + "confirmed the higher point group", lower_name, + lower.refused_point_group_hm, lower.refused_reason, higher_name); + else + msg = fmt::format("the {} merge did not confirm the operators of the higher point " + "group (no gate refused it); the {} merge did", lower_name, + higher_name); + logger.Warning("Space-group search: {}", msg); + result.space_group_arm_override = msg; + }; + if (order_of(alt) > order_of(sg_search)) { logger.Info("Space-group search: all-observation merge supports {} where the " "Lorentz-filtered one supports {} - taking the higher symmetry", alt.point_group_hm, sg_search.point_group_hm); + overridden(sg_search, "Lorentz-filtered", "all-observation"); + } if (order_of(alt) < order_of(sg_search)) { // The filtered arm found symmetry the all-observation arm did not. Discarding it // was costing real point groups: over the corpus the two arms disagree on eight @@ -5359,6 +5381,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // that found it and the absences are re-judged here, on every observation. That // is what fixed_point_group is for: Stage A still runs and still reports what it // would have refused, only the choice is overridden. + overridden(alt, "all-observation", "Lorentz-filtered"); sg_opts.fixed_point_group = sg_search.point_group_representative; auto split = SearchSpaceGroup(sm_all.merged, sg_opts); sg_opts.fixed_point_group.reset(); diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 54ab34074..b6e8d67bc 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -334,6 +334,10 @@ struct ProcessResult { // populated only when a search ran (no space group was fixed). The library no longer renders it to // text - the CLI formats it for stdout, the viewer draws it as a table. std::optional space_group_search; + // When the two search arms (all observations / Lorentz-filtered) disagreed and the higher point + // group was taken from one of them: what the other arm said - which gate refused the promotion, + // or that it never confirmed the operators. Empty when the arms agreed. + std::string space_group_arm_override; // ---- Material for the results report (_report.txt, see rugnux/ResultReport.h) ---- From c657204a72aaf0478c1cd76da80c6c6486b058ef Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 10:06:52 +0200 Subject: [PATCH 03/28] R-free flags keyed on the lattice holohedry The free-set hash was keyed on the Laue ASU of the merging group, so a twin law - a lattice symmetry the crystal lacks - put nearly every free reflection's twin mate in the working set (measured: 97-98% of the free-touching twin pairs mixed, on the main P3121 output of 6iu9 and on every law of the 6toc P1 cross-check), and each file of one crystal carried a different free set (P1 vs merged agreed on 88-92% of reflections). With the cell given, the key is now the reflection's orbit, Friedel mate included, under the lattice holohedry: the metric point group of the cell (gemmi Le Page two-folds, 3 deg obliquity, lattice of the cell's own basis vectors so every file gets the same group; this contains the merging group). Twin mates share a flag, and the merged MTZ/mmCIF, the P1 cross-check and any subgroup re-merge carry one free set (nested where the small-data floor lifts the fraction of one file more than another). This is phenix.refine's default (use_lattice_symmetry). Where the cell does not carry the merging group, the merging group's key is used as before. The small-data floor still counts reflections of the merging group, so the free fraction is unchanged. Reference free sets are untouched. On 6toc, 6iu9, 5j23: mixed twin pairs 0 for every law; P1 and merged flags agree on 100% (6iu9, 5j23) and nested on 6toc; free fraction 0.050-0.051 as before (6toc merged 0.086, floor unchanged). Intensities and space groups identical to rc171. Co-Authored-By: Claude Opus 5 (1M context) --- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 3 +- image_analysis/scale_merge/Merge.cpp | 2 +- image_analysis/scale_merge/RfreeFlags.cpp | 59 +++++++++++++++++-- image_analysis/scale_merge/RfreeFlags.h | 13 +++- .../scale_merge/RotationScaleMerge.cpp | 2 +- tests/RfreeFlagsTest.cpp | 32 ++++++++++ 6 files changed, 101 insertions(+), 10 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 1d8bd3450..61b4bf3a1 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -382,9 +382,10 @@ Each batch's $B$ is fitted on **resolution-shell means**, not on single observat ### 10.7 R-free test-set flags -A fraction of the unique reflections (`rfree_fraction`, default 0.05) is flagged as a **free (test) set**, written to the output (MTZ `FreeR_flag`, mmCIF `_refln.status` = `f`, a text-HKL column) for model validation (§14) and for downstream refinement. The flag is a pure function of the reflection's **Friedel-merged (Laue) ASU index**, which gives three properties: +A fraction of the unique reflections (`rfree_fraction`, default 0.05) is flagged as a **free (test) set**, written to the output (MTZ `FreeR_flag`, mmCIF `_refln.status` = `f`, a text-HKL column) for model validation (§14) and for downstream refinement. The flag is a pure function of the reflection's orbit under the **lattice holohedry** — the point group of the cell's metric, found as twin laws are (Le Page two-folds within 3° obliquity, taking the lattice of the cell's own basis vectors), which contains the merging group — together with its Friedel mate. Where the cell does not carry the merging group (a space group forced on a metric without it), the Friedel-merged (Laue) ASU index of the merging group is used instead. That gives four properties: - all symmetry- and Friedel-equivalent reflections share one flag — in particular a Bijvoet pair $I(+)/I(-)$, kept as two separate merged rows in anomalous mode, is **never split** across the work and free sets (which would bias R-free); +- **twin-law mates share one flag too**, since a twin law is a lattice symmetry the crystal lacks. Keyed on the merging group instead, nearly every free reflection's twin mate lands in the working set, and in twin refinement the free reflection's calculated intensity then carries the working-set fit. The same set is what phenix.refine generates by default (`use_lattice_symmetry`). It also makes the free set independent of the space group a file is merged in: the merged file, the P1 cross-check and a re-merge in any subgroup carry one free set (nested, where the small-data floor below lifts the fraction of one of them more than another's); - the free/work decision is a deterministic hash of that key, so the same reflection always lands in the same set — reproducible run-to-run and independent of the order in which observations were merged; - the hash depends only on the reflection index, **not** on this dataset's resolution range or which reflections it happens to contain, so a uniform draw takes ~`rfree_fraction` of the distinct reflections free and — crucially — **every dataset of one crystal form gets the same free set**. That cross-dataset consistency is what a multi-dataset campaign (ensemble refinement, PanDDA) requires; a per-shell stratification tied to each dataset's own $d_\mathrm{min}$ would break it. diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index ce7b87f2e..2945ce2f7 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -389,7 +389,7 @@ std::vector MergeOnTheFly::ExportReflections() { out.emplace_back(mr); } - AssignRfreeFlags(out, space_group, scaling_settings.GetRfreeFraction()); + AssignRfreeFlags(out, space_group, scaling_settings.GetRfreeFraction(), 500, reference_cell); ApplyFrenchWilson(out, space_group); return out; } diff --git a/image_analysis/scale_merge/RfreeFlags.cpp b/image_analysis/scale_merge/RfreeFlags.cpp index 6b7b17cd2..cfc099d6e 100644 --- a/image_analysis/scale_merge/RfreeFlags.cpp +++ b/image_analysis/scale_merge/RfreeFlags.cpp @@ -8,6 +8,7 @@ #include #include "HKLKey.h" +#include "gemmi/twin.hpp" namespace { // splitmix64 bit-mix of a key -> uniform double in [0, 1). Same key -> same value, so all @@ -20,10 +21,36 @@ namespace { z = z ^ (z >> 31); return static_cast(z >> 11) * (1.0 / 9007199254740992.0); } + + + // The rotations of the lattice's own point group - its holohedry - found from the metric as for + // twin laws (Le Page two-folds, with phenix.xtriage's default obliquity of 3 degrees, so a + // pseudo-merohedral lattice counts as its higher metric; grouping too much only clusters the + // free set, grouping too little splits twin mates). The centring is taken as P: the lattice of + // the cell's own basis vectors, the same whatever group a merge is in, so the merged file, the P1 + // cross-check and any re-merge get one holohedry. On an R lattice in its hexagonal cell that is + // 6/mmm rather than -3m, which groups each present reflection with exactly its -3m mates, the + // extra ones being centring-absent. Empty when the cell does not carry the merging group itself + // (a space group forced on a metric that does not have it): there is no holohedry to key on. + std::vector LatticeHolohedry(const UnitCell &cell, const gemmi::SpaceGroup &space_group) { + std::vector rots; + for (const auto &op : gemmi::find_lattice_symmetry(static_cast(cell), 'P', 3.0).sym_ops) + rots.push_back(gemmi::Op{op.rot, {0, 0, 0}, 'x'}); + for (const auto &op : space_group.operations().sym_ops) + if (std::none_of(rots.begin(), rots.end(), [&](const gemmi::Op &r) { return r.rot == op.rot; })) + return {}; + return rots; + } + + uint64_t PackIndex(const gemmi::Op::Miller &h) { + constexpr int64_t bias = 1 << 20; + return (static_cast(h[0] + bias) << 42) | (static_cast(h[1] + bias) << 21) + | static_cast(h[2] + bias); + } } void AssignRfreeFlags(std::vector &merged, const gemmi::SpaceGroup &space_group, - double rfree_fraction, int min_free_reflections) { + double rfree_fraction, int min_free_reflections, const std::optional &cell) { for (auto &r : merged) r.rfree_flag = false; if (rfree_fraction <= 0.0 || merged.empty()) @@ -36,10 +63,34 @@ void AssignRfreeFlags(std::vector &merged, const gemmi::SpaceG // dataset of one crystal form gets the SAME free set, which is what a multi-dataset campaign // (ensemble refinement, PanDDA) needs. A uniform hash draws ~rfree_fraction of the distinct // reflections free; a stratified per-shell draw would be tied to the dataset and break that. + // + // The key is the orbit of the reflection under the LATTICE HOLOHEDRY, not under the merging group. + // A twin law is a lattice symmetry that the crystal lacks, so keying on the merging group puts a + // free reflection's twin mate in the working set almost every time (measured: 97-98% of the free + // reflections that have a mate), and a twin-refined R-free then reads the working set through + // I_calc. Keyed on the holohedry, twin mates share a flag, and the free set no longer depends on + // the space group a file is merged in - the merged MTZ, the P1 cross-check and a re-merge in any + // subgroup carry one free set (where the small-data floor below lifts the fraction differently in + // two of them, the smaller set is contained in the larger: each orbit has one draw). As + // phenix.refine does by default (use_lattice_symmetry). const HKLKeyGenerator laue_key(/*merge_friedel=*/true, space_group); + const std::vector holohedry = cell ? LatticeHolohedry(*cell, space_group) : std::vector{}; + auto key_of = [&](const MergedReflection &r) -> uint64_t { + if (holohedry.empty()) + return laue_key(r).pack(); + // The largest index of the orbit, Friedel mates included, names it. + gemmi::Op::Miller best{{r.h, r.k, r.l}}; + for (const auto &op : holohedry) { + const gemmi::Op::Miller h = op.apply_to_hkl({{r.h, r.k, r.l}}); + best = std::max({best, h, gemmi::Op::Miller{{-h[0], -h[1], -h[2]}}}); + } + return PackIndex(best); + }; - // Count the distinct test-eligible reflections (distinct Laue-ASU keys; mates collapse to one) so - // the fraction can be floored to a usable test-set size on small data. + // Count the distinct test-eligible reflections (distinct Laue-ASU keys of the merging group; mates + // collapse to one) so the fraction can be floored to a usable test-set size on small data. Counted + // in the merging group, not in holohedral orbits: R-free is a sum over the reflections of this file, + // and counting orbits would lift the fraction on every dataset below its holohedry. std::unordered_set distinct; distinct.reserve(merged.size()); for (const auto &r : merged) @@ -54,7 +105,7 @@ void AssignRfreeFlags(std::vector &merged, const gemmi::SpaceG const double eff_fraction = std::max(rfree_fraction, floor_fraction); for (auto &r : merged) - r.rfree_flag = UniformFromKey(laue_key(r).pack()) < eff_fraction; + r.rfree_flag = UniformFromKey(key_of(r)) < eff_fraction; } size_t ApplyReferenceFreeFlags(std::vector &merged, const gemmi::SpaceGroup &space_group, diff --git a/image_analysis/scale_merge/RfreeFlags.h b/image_analysis/scale_merge/RfreeFlags.h index fd0402577..6b00f6f7d 100644 --- a/image_analysis/scale_merge/RfreeFlags.h +++ b/image_analysis/scale_merge/RfreeFlags.h @@ -5,15 +5,21 @@ #include #include +#include #include #include "../../common/Reflection.h" +#include "../../common/UnitCell.h" #include "gemmi/symmetry.hpp" -// Assign R-free (test-set) flags to merged reflections. The flag is a pure function of the -// Friedel-merged (Laue) ASU key, which gives three properties: +// Assign R-free (test-set) flags to merged reflections. With the cell given, the flag is a pure +// function of the reflection's orbit under the LATTICE HOLOHEDRY (the metric symmetry of the cell, +// which contains the merging group); without one, or when the cell does not carry the merging group, +// of its Friedel-merged (Laue) ASU key in space_group. That gives four properties: // - all symmetry- and Friedel-equivalent reflections share one flag, so a Bijvoet pair I(+)/I(-) // is never split across the work and free sets; +// - with the cell, twin-law mates share one too, and every file of one crystal - whatever space +// group it is merged in - carries the same free set; // - the assignment is reproducible run-to-run and independent of merge/call order; // - it depends only on the reflection index, not on the dataset's resolution range or contents, so // every dataset of one crystal form gets the SAME free set (needed for multi-dataset campaigns: @@ -28,7 +34,8 @@ // matters more than cross-dataset identity (and a shared reference/ApplyReferenceFreeFlags is the way // to keep exact identity there). Pass 0 to disable the floor. void AssignRfreeFlags(std::vector &merged, const gemmi::SpaceGroup &space_group, - double rfree_fraction, int min_free_reflections = 500); + double rfree_fraction, int min_free_reflections = 500, + const std::optional &cell = std::nullopt); // Override the R-free flags on `merged` from a reference set (e.g. a reference MTZ's FreeR_flag // column), so an entire campaign shares one test set. A merged reflection whose Friedel-merged diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 39db29683..e651313f3 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -4077,7 +4077,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // is not written needs neither. French-Wilson is deferred until after the anomalous accumulator // below has attached I(+)/I(-), so the two hands get their amplitudes in one pass. if (full_stats) - AssignRfreeFlags(result.merged, x.GetSpaceGroupOrP1(), rfree_fraction); + AssignRfreeFlags(result.merged, x.GetSpaceGroupOrP1(), rfree_fraction, 500, reference_cell); if (reject_count > 0) logger.Info("Merge outlier rejection: dropped {} observations", reject_count); diff --git a/tests/RfreeFlagsTest.cpp b/tests/RfreeFlagsTest.cpp index bb78a8f8b..b267034e2 100644 --- a/tests/RfreeFlagsTest.cpp +++ b/tests/RfreeFlagsTest.cpp @@ -182,3 +182,35 @@ TEST_CASE("R-free fraction of zero flags nothing", "[rfree]") { AssignRfreeFlags(v, SG(1), 0.0); for (const auto& r : v) CHECK(!r.rfree_flag); } + +TEST_CASE("R-free flags keyed on the lattice holohedry keep twin mates together", "[rfree]") { + // A P4 crystal on a tetragonal cell: the lattice is 4/mmm, and (h,k,l) -> (k,h,-l) is a twin law. + // Keyed on the holohedry, a reflection and its twin mate share a flag, and the P1 merge of the same + // crystal flags every reflection exactly as the P4 merge does. Synthetic cell. + const std::optional cell = UnitCell{47.0f, 47.0f, 63.0f, 90.0f, 90.0f, 90.0f}; + std::vector p4, p1; + for (int h = -10; h <= 10; ++h) + for (int k = -10; k <= 10; ++k) + for (int l = -10; l <= 10; ++l) { + if (h == 0 && k == 0 && l == 0) continue; + const float d = 60.0f / (1 + h * h + k * k + l * l); + p1.push_back(Refl(h, k, l, d)); + p4.push_back(Refl(h, k, l, d)); + } + AssignRfreeFlags(p4, SG(75), 0.05, /*min_free=*/0, cell); + AssignRfreeFlags(p1, SG(1), 0.05, /*min_free=*/0, cell); + + std::map, bool> flag; + for (const auto& r : p4) flag[{r.h, r.k, r.l}] = r.rfree_flag; + int twin_pairs = 0; + for (const auto& r : p4) { + CHECK(flag[{r.k, r.h, -r.l}] == r.rfree_flag); // twin mate + CHECK(flag[{-r.k, r.h, r.l}] == r.rfree_flag); // P4 mate + ++twin_pairs; + } + CHECK(twin_pairs > 0); + for (const auto& r : p1) + CHECK(flag[{r.h, r.k, r.l}] == r.rfree_flag); // one free set whatever the merging group + CHECK(FreeFraction(p4) > 0.03); + CHECK(FreeFraction(p4) < 0.08); +} From 49541101ad1e9a8986e0316c1f42ffb671426a2a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 10:07:01 +0200 Subject: [PATCH 04/28] Twin-immune zone evidence for the operators a promotion adds (report-only) Under a twin law T a reflection whose twin mate is itself (up to the true group and Friedel) is untouched at every twin fraction. For each index-2 subgroup H of the adopted point group, those are the reflections centric in the group and acentric in H: centric if the operators the group adds over H are real, acentric if they are a twin law or a pseudo-symmetry - the one intensity statistic that still separates the two at fraction 0.5, where every operator statistic reads "real". Read on the P1 cross-check merge: epsilon-1 reflections in shells with >= 5 (noise inflates every class towards centric), each class normalised against its own mean in bins of ~100 reflections and within the two phase classes of a detected pseudo-translation, Wilson outliers above E^2 = 20 dropped. Reported per subgroup: the added operators, n, <|E^2-1|> +- SE read absolutely against 0.968 / 0.736, and the centric-over- acentric Wilson log-likelihood in nats with both densities convolved with each reflection's error (flooring E^2 at its sigma instead read a genuine 1.2 A lysozyme zone as acentric), with the acentric control beside it. Log, report prose and TWIN_ZONE_n keys. Nothing reads it back; no decision changes. On the reference sets: 6toc P4222, all three subgroups centric (1.04-1.07, +253 to +534 nats); 6iu9 P3121 over P31 acentric (0.789, -294 nats); 5j23 R32 over R3 acentric (0.805, -304 nats, tNCS-class normalised); lysozyme P41212 centric (0.90, +476 to +1297); a P21 myoglobin centric (0.930, +135). Co-Authored-By: Claude Opus 5 (1M context) --- docs/CPU_DATA_ANALYSIS_DECISIONS.md | 2 + .../scale_merge/TwinningAnalysis.cpp | 238 ++++++++++++++++++ image_analysis/scale_merge/TwinningAnalysis.h | 38 +++ rugnux/ResultReport.cpp | 15 ++ rugnux/Rugnux.cpp | 10 + rugnux/Rugnux.h | 4 + tests/TwinningAnalysisTest.cpp | 72 ++++++ 7 files changed, 379 insertions(+) diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index 7ab356908..8718102f0 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -54,6 +54,8 @@ Both search merges also drop the **frames whose fitted per-frame scale came out A Padilla–Yeates $L$-test ($\langle|L|\rangle$, $\langle L^2\rangle$ — 0.500 and 0.333 untwinned, 0.375 and 0.200 for a perfect twin) and the second moment $\langle I^2\rangle/\langle I\rangle^2$ (2.0 for untwinned acentric data, 1.5 for a perfect twin) are written to the merged mmCIF as a twinning diagnostic. Both are taken on intensities divided by their resolution-shell mean, over the shells whose $\langle I/\sigma\rangle$ reaches 1, with Wilson outliers rejected; the $L$-test pairs are therefore compared on one scale even where two index steps span a steep fall-off, as in phenix.xtriage and ctruncate, and the selection is by shell, never by the individual reflection's $I/\sigma$, which would cut the weak tail and bias $\langle|L|\rangle$ down. The twin fraction is quoted from the statistic that carries the verdict — the $L$-test unless the call rests on the second moment alone — and a second moment is not turned into a fraction under a detected pseudo-translation, which inflates it. A merohedral twin law exists only where the Laue class is a proper subgroup of the lattice holohedry, so in the high-symmetry holohedral classes ($4/mmm$, $6/mmm$, $m\bar{3}m$, and $\bar{3}m$ on a rhombohedral lattice) no twin is called. The $L$-test is still read there, for a different question: merging $I(h)$ with $I(Th)$ under a false operator $T$ gives $(I(h)+I(Th))/2$ whatever the twin fraction, which has the perfect-twin distribution, so $\langle|L|\rangle$ below 0.42 in a holohedral class is reported as **an adopted operator averaging unequal intensities** — the space group is too high, or a twin law was absorbed into the point group — a warning, never a change to the space group. A genuine operator leaves the untwinned 0.5. Reflections that overlap along a very long axis narrow the distribution the same way, which is one reason it stays a warning. The same numbers measured on the P1 merge the space-group search was given, before any point group was adopted, are reported beside them (`_BEFORE_SEARCH`), with that merge's own pseudo-translation declared and the reflections its lattice centring extinguishes left out. The low-symmetry holohedral classes ($\bar{1}$, $2/m$, $mmm$) also admit no strictly merohedral law, but they stay eligible for the twin call on purpose: *pseudo*-merohedral twinning through an accidentally special metric cannot be ruled out from the symmetry alone, and those are the classes it happens in. +Beside them, on rotation data, the run reports **twin-immune zone evidence** for the operators the adopted point group adds over each of its index-2 subgroups, read on the P1 cross-check merge. Under a twin law $T$, $I_\mathrm{obs}(h)=(1-\alpha)I(h)+\alpha I(Th)$; a reflection whose twin mate is itself up to the subgroup and Friedel is untouched at every $\alpha$, and those are exactly the reflections centric in the group but acentric in the subgroup. They read centric ($\langle|E^2-1|\rangle = 0.968$) if the added operators are real and acentric (0.736) if they are a twin law or a pseudo-symmetry — the one intensity statistic that still separates the two at $\alpha = 0.5$, where every operator statistic reads "real". Each zone is normalised against its own mean in resolution bins (and within the two phase classes of a detected pseudo-translation), read only in shells with $\langle I/\sigma\rangle \ge 5$ because noise inflates every class towards centric, and reported with $n$, a standard error and the centric-over-acentric Wilson log-likelihood ratio in nats (both densities convolved with each reflection's measurement error), beside the acentric control. It is read absolutely, never as the difference to the control: a perfect twin's control (0.541) plus noise inflation would otherwise look like true symmetry. It is **report-only** — nothing in the space-group decision reads it. It is an in-house method: published practice (Yeates' $H$-test, xtriage) excludes these reflections as uninformative about the twin fraction, and no published test was found that uses them positively; the one precedent here is a trigonal crystal whose twin-immune zone read centric and whose higher group was then confirmed by refinement. + **Translational pseudo-symmetry** — two copies of the contents of the asymmetric unit related by a pure translation that is not a lattice vector — is looked for on every merging run, because it is the classic predictor of a failed molecular replacement, and because it raises the second moment where twinning lowers it, so each can mask the other's test. The detection is the native-Patterson route of Read, Adams & McCoy (see the [references](CPU_DATA_ANALYSIS.md#references)): the largest off-origin peak of a Patterson computed from the merged intensities, as a fraction of the origin peak, with the peak vector then refined against the intensity modulation it should produce — the ratio of the strongest to the weakest bin mean of $\langle E^2\rangle$ over the phase $\mathrm{frac}(\mathbf{h}\cdot\mathbf{u})$. Both halves are scored against a null computed for the crystal at hand rather than against a fixed bound — both against the same intensities **permuted within resolution shells** — the peak against that map, the modulation against the same measurement made on the shuffled intensities and started where the measurement starts — because the noise floor of the peak statistic spans an order of magnitude across data sets, so no fixed percentage means the same thing twice. The shuffle leaves the reflection count, the $E^2$ distribution and the phase-bin populations untouched and removes only the correlation between a reflection's intensity and $\mathbf{h}\cdot\mathbf{u}$, which is the thing the modulation claims to see. Re-running the greedy search from random starting vectors over the *same* intensities does not work as a control: a greedy search started anywhere walks into a real modulation's basin, so it measures the search **and** whatever modulation the crystal carries — measured over 110 merged datasets, single draws of that control reached 199× and 6162× on crystals whose own modulation reads 4.0× and 67×, and on 28 of the 110 it came out at or above the signal it is subtracted from. Requiring both halves is what keeps the false-positive rate down; either alone over-calls by about a factor of two. A translation the merged data are **exactly** invariant under is reported as an undeclared lattice translation instead — a translation the data are exactly invariant under is a lattice vector by definition, so the centring or the cell is wrong, not the packing — and the pseudo-symmetry search continues underneath it, so a real pseudo-translation sitting under an undeclared centring is still found. The finding itself is report-only (the `TNCS_*` keys of `_report.txt`): it gates nothing and changes no reflection, no scale and no group — but the axial-absence test (§13.1) and the $L$-test here both correct themselves against the modulation it measures. **The $L$-test partners are chosen so a pseudo-translation cannot silence it.** The test compares each reflection with a partner a fixed index step away, and a pseudo-translation biases $\langle|L|\rangle$ upwards unless the partner shares its modulation class. The ordinary step of 2 preserves the class of a half-integer translation — the pseudo-centering the statistic's authors call it robust to — but not of one of a third, which inflates $\langle|L|\rangle$ past the bound that is read as evidence *against* twinning and silently loses the twin call on a crystal that has one. Where a pseudo-translation is detected, the partner steps are restricted to those that preserve its class; where no step does, the statistic is dropped from the twin verdict **in both directions** — it can no longer indicate a twin and can no longer be read as proof that there is none — and the second moment decides alone. `L_TEST_VS_TNCS=` in the report says which of the three happened. diff --git a/image_analysis/scale_merge/TwinningAnalysis.cpp b/image_analysis/scale_merge/TwinningAnalysis.cpp index 6f96e9ab2..488470f84 100644 --- a/image_analysis/scale_merge/TwinningAnalysis.cpp +++ b/image_analysis/scale_merge/TwinningAnalysis.cpp @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -475,3 +476,240 @@ std::string TwinningVerdictLine(const TwinningAnalysisResult& result) { << "; <|L|> not readable)"; return os.str(); } + +namespace { + // Measurement noise inflates <|E^2-1|> towards centric in every class (by +0.44 at a shell + // of 2.5 and ~0 at 16, measured), so only shells above this are read. At 5 the acentric + // control of four reference merges reads 0.65-0.74. + constexpr double ZONE_MIN_SHELL_ISIG = 5.0; + // Each class is normalised against its own mean in resolution bins of this many reflections, so a + // zone that is a plane through reciprocal space is not judged on another direction's fall-off. + constexpr size_t ZONE_BIN = 100; + // A reflection this far above its bin mean is not a Wilson draw (centric P ~ 1e-5): one such + // reflection read 2.33 for a whole zone. + constexpr double ZONE_OUTLIER_E2 = 20.0; + + // ln p_centric(E^2) - ln p_acentric(E^2) for a measured E^2 with error s: both Wilson densities + // convolved with the measurement error. Without the convolution the centric density, infinite at + // zero, reads every noisy weak reflection as decisive; flooring E^2 at s instead turns the centric + // excess of weak reflections into moderate values and read a genuine centric zone of a 1.2 A + // crystal as acentric. Integrated over u = sqrt(E^2_true), where neither density is singular, + // across +-6 s of the measurement (trapezoid). + double CentricOverAcentric(double e2, double s) { + constexpr int n = 200; + const double u_lo = std::sqrt(std::max(0.0, e2 - 6.0 * s)); + const double u_hi = std::sqrt(std::max(0.0, e2 + 6.0 * s)); + if (!(u_hi > u_lo)) + return 0.0; + const double du = (u_hi - u_lo) / n; + double p_centric = 0.0, p_acentric = 0.0; + for (int i = 0; i <= n; ++i) { + const double u = u_lo + i * du; + const double w = (i == 0 || i == n ? 0.5 : 1.0) * std::exp(-0.5 * std::pow((e2 - u * u) / s, 2)); + p_centric += w * std::sqrt(2.0 / std::numbers::pi) * std::exp(-0.5 * u * u); + p_acentric += w * 2.0 * u * std::exp(-u * u); + } + return p_centric > 0.0 && p_acentric > 0.0 ? std::log(p_centric / p_acentric) : 0.0; + } + + TwinImmuneZone ReadZone(std::vector refl, const std::array* tncs) { + TwinImmuneZone z; + // A pseudo-translation u multiplies the intensity by 1 + f cos(2 pi h.u): normalising within + // each sign of the cosine keeps the two populations from reading as one broad distribution. + std::vector classes[2]; + for (const auto* r : refl) { + const double c = tncs == nullptr ? 1.0 + : std::cos(2.0 * std::numbers::pi * (r->h * (*tncs)[0] + r->k * (*tncs)[1] + r->l * (*tncs)[2])); + classes[c >= 0.0 ? 0 : 1].push_back(r); + } + double sum = 0.0, sum2 = 0.0; + for (auto& cls : classes) { + std::sort(cls.begin(), cls.end(), [](const auto* a, const auto* b) { return a->d > b->d; }); + const size_t n_bins = std::max(1, cls.size() / ZONE_BIN); + for (size_t b = 0; b < n_bins; ++b) { + const size_t first = b * cls.size() / n_bins, last = (b + 1) * cls.size() / n_bins; + double mean = 0.0; + for (size_t i = first; i < last; ++i) + mean += cls[i]->I; + mean /= static_cast(last - first); + if (mean <= 0.0) + continue; + double kept = 0.0; + int n_kept = 0; + for (size_t i = first; i < last; ++i) + if (cls[i]->I / mean <= ZONE_OUTLIER_E2) { + kept += cls[i]->I; + ++n_kept; + } + if (n_kept == 0 || kept <= 0.0) + continue; + mean = kept / n_kept; + for (size_t i = first; i < last; ++i) { + const double e2 = cls[i]->I / mean; + if (e2 > ZONE_OUTLIER_E2) + continue; + const double v = std::fabs(e2 - 1.0); + sum += v; + sum2 += v * v; + ++z.n; + z.evidence_nats += CentricOverAcentric(e2, cls[i]->sigma / mean); + } + } + } + if (z.n > 0) { + z.mean_abs_e2_minus_1 = sum / z.n; + const double var = std::max(0.0, sum2 / z.n - z.mean_abs_e2_minus_1 * z.mean_abs_e2_minus_1); + z.standard_error = std::sqrt(var / z.n); + } + return z; + } +} + +TwinImmuneZoneResult AnalyzeTwinImmuneZones(const std::vector& p1_merged, + const gemmi::SpaceGroup& group, + const std::array* tncs_vector) { + TwinImmuneZoneResult result; + result.tncs_normalised = tncs_vector != nullptr; + const gemmi::GroupOps gops = group.operations(); + + // The index-2 subgroups H of the group's rotations: each is a hypothesis "the crystal is H, and the + // rest of the group (G minus H, one coset) is a twin law or a pseudo-symmetry". H holds every square + // of G, so it is the subgroup the squares generate, or that plus one more rotation; for a proper + // point group that finds them all (G over its squares is at most a four-group). + auto same = [](const gemmi::Op& a, const gemmi::Op& b) { return a.rot == b.rot; }; + auto contains = [&](const std::vector& set, const gemmi::Op& op) { + return std::any_of(set.begin(), set.end(), [&](const gemmi::Op& o) { return same(o, op); }); + }; + auto closure = [&](const std::vector& generators) { + std::vector ops; + for (const auto& g : generators) + if (!contains(ops, g)) + ops.push_back(g); + for (size_t i = 0; i < ops.size(); ++i) + for (size_t j = 0; j <= i; ++j) + for (const gemmi::Op& p : {ops[i] * ops[j], ops[j] * ops[i]}) + if (!contains(ops, p)) + ops.push_back(p); + return ops; + }; + std::vector rotations; + for (const auto& op : gops.derive_symmorphic().sym_ops) + if (op.det_rot() > 0) + rotations.push_back(gemmi::Op{op.rot, {0, 0, 0}, 'x'}); + std::vector squares{gemmi::Op::identity()}; + for (const auto& g : rotations) + squares.push_back(g * g); + const auto square_group = closure(squares); + std::vector> subgroups; + auto add_subgroup = [&](const std::vector& h) { + if (2 * h.size() != rotations.size()) + return; + for (const auto& known : subgroups) + if (known.size() == h.size() + && std::all_of(h.begin(), h.end(), [&](const gemmi::Op& o) { return contains(known, o); })) + return; + subgroups.push_back(h); + }; + add_subgroup(square_group); + for (const auto& g : rotations) { + auto gen = square_group; + gen.push_back(g); + add_subgroup(closure(gen)); + } + if (subgroups.empty()) + return result; + + // Reflections present in the group and in general position (epsilon 1): an axial row is fixed by + // more than one operator and follows neither distribution. + std::vector usable; + for (const auto& r : p1_merged) { + const gemmi::Op::Miller h{{r.h, r.k, r.l}}; + if (UsableIntensity(r) && std::isfinite(r.sigma) && r.sigma > 0.0 + && !gops.is_systematically_absent(h) && gops.epsilon_factor_without_centering(h) == 1) + usable.push_back(&r); + } + // The shells with signal enough to read (ZONE_MIN_SHELL_ISIG), 20 equal in 1/d^2. + constexpr int n_shells = 20; + double min_s = std::numeric_limits::infinity(), max_s = 0.0; + for (const auto* r : usable) { + min_s = std::min(min_s, 1.0 / (r->d * r->d)); + max_s = std::max(max_s, 1.0 / (r->d * r->d)); + } + if (!(max_s > min_s)) + return result; + auto shell_of = [&](const MergedReflection* r) { + const double t = (1.0 / (r->d * r->d) - min_s) / (max_s - min_s); + return std::min(n_shells - 1, std::max(0, static_cast(t * n_shells))); + }; + std::vector isig_sum(n_shells, 0.0); + std::vector isig_n(n_shells, 0); + for (const auto* r : usable) { + isig_sum[shell_of(r)] += r->I / r->sigma; + isig_n[shell_of(r)] += 1; + } + std::vector strong; + for (const auto* r : usable) { + const int b = shell_of(r); + if (isig_n[b] > 0 && isig_sum[b] / isig_n[b] >= ZONE_MIN_SHELL_ISIG) + strong.push_back(r); + } + if (strong.empty()) + return result; + result.d_max_A = 0.0; + result.d_min_A = std::numeric_limits::infinity(); + for (const auto* r : strong) { + result.d_max_A = std::max(result.d_max_A, r->d); + result.d_min_A = std::min(result.d_min_A, r->d); + } + + std::vector control; + for (const auto* r : strong) + if (!gops.is_reflection_centric(gemmi::Op::Miller{{r->h, r->k, r->l}})) + control.push_back(r); + result.control = ReadZone(control, tncs_vector); + result.control.operators = "acentric"; + + // Per hypothesis H, the zone: centric in the group, acentric in H - the reflections some added + // operator sends to -h, i.e. the self-mates of that coset read as a twin law. + for (const auto& h_ops : subgroups) { + std::vector zone; + for (const auto* r : strong) { + const gemmi::Op::Miller h{{r->h, r->k, r->l}}; + const gemmi::Op::Miller minus{{-gemmi::Op::DEN * r->h, -gemmi::Op::DEN * r->k, -gemmi::Op::DEN * r->l}}; + const bool centric_in_h = std::any_of(h_ops.begin(), h_ops.end(), [&](const gemmi::Op& op) { + return op.apply_to_hkl_without_division(h) == minus; + }); + if (gops.is_reflection_centric(h) && !centric_in_h) + zone.push_back(r); + } + TwinImmuneZone z = ReadZone(zone, tncs_vector); + for (const auto& op : rotations) + if (!contains(h_ops, op)) + z.operators += (z.operators.empty() ? "" : " ") + op.as_hkl().triplet(); + result.zones.push_back(z); + } + return result; +} + +std::string TwinImmuneZonesToText(const TwinImmuneZoneResult& result) { + std::ostringstream os; + os << std::fixed; + os << "Twin-immune zones of the operators the adopted group adds over each index-2 subgroup (P1 merge, " << std::setprecision(2) << result.d_max_A + << "-" << result.d_min_A << " A, shells with >= " << std::setprecision(0) + << ZONE_MIN_SHELL_ISIG << (result.tncs_normalised ? ", normalised per pseudo-translation class" : "") + << ")\n" + << " Reflections centric in the group but not in the subgroup are their own twin mates: centric if\n" + << " the added operators are real, acentric if they are a twin law or a pseudo-symmetry. <|E^2-1|>\n" + << " reads 0.968 centric, 0.736 acentric; evidence is the centric/acentric log-likelihood ratio\n" + << " (> 0: the added operators are real).\n"; + auto row = [&](const TwinImmuneZone& z) { + os << " " << std::left << std::setw(34) << z.operators << std::right << " n " << std::setw(6) << z.n + << " <|E^2-1|> " << std::setprecision(3) << z.mean_abs_e2_minus_1 << " +- " << z.standard_error + << " evidence " << std::showpos << std::setprecision(1) << z.evidence_nats << std::noshowpos + << " nats\n"; + }; + for (const auto& z : result.zones) + row(z); + row(result.control); + return os.str(); +} diff --git a/image_analysis/scale_merge/TwinningAnalysis.h b/image_analysis/scale_merge/TwinningAnalysis.h index 965c4da89..259d89cdb 100644 --- a/image_analysis/scale_merge/TwinningAnalysis.h +++ b/image_analysis/scale_merge/TwinningAnalysis.h @@ -91,3 +91,41 @@ std::string TwinningAnalysisToText(const TwinningAnalysisResult& result); // The verdict in one line - "INDICATED (...)", "SYMMETRY SUSPECT (...)" or "no indication (...)" - // so the report's summary row, its prose and the viewer all say the same thing. std::string TwinningVerdictLine(const TwinningAnalysisResult& result); + +// Twin-immune zone evidence for the operators a point group adds over each of its index-2 subgroups, +// read on a P1 merge. REPORT-ONLY. +// +// Under a twin law T, I_obs(h) = (1-a) I(h) + a I(Th); a reflection whose twin mate Th is h itself, up +// to the true group and Friedel, is untouched by twinning at every fraction a. For an index-2 subgroup +// H of the adopted group G those are the reflections centric in G but acentric in H: centric if the +// operators G adds over H are real, acentric if they are a twin law or a pseudo-symmetry the promotion +// took for real. So their intensity distribution says which, and does so where every operator +// statistic is blind - at a = 0.5, where I(h) = I(Th) exactly. Read ABSOLUTELY, <|E^2-1|> against +// 0.968 (centric) and 0.736 (acentric); never as the difference to the control, because noise, a +// pseudo-translation and anisotropy all inflate every class towards centric, and a perfect twin's +// control (0.541) plus that inflation looks like true symmetry's. +struct TwinImmuneZone { + std::string operators; // the operators G adds over H, as h,k,l triplets + int n = 0; + double mean_abs_e2_minus_1 = 0.0; // 0.968 centric (operator real), 0.736 acentric (not) + double standard_error = 0.0; + // Log-likelihood ratio, centric over acentric Wilson (each convolved with the reflection's error), + // summed over the zone: > 0 favours the operators being real symmetry, < 0 a twin law or + // pseudo-symmetry. About +0.2 nats per reflection for a centric zone and -0.13 for an acentric one + // on error-free data; less either way as the errors grow. + double evidence_nats = 0.0; +}; + +struct TwinImmuneZoneResult { + std::vector zones; // one per index-2 subgroup of the group + TwinImmuneZone control; // the reflections acentric in the group + double d_max_A = 0.0; // resolution range of the shells read + double d_min_A = 0.0; + bool tncs_normalised = false; // normalised within the pseudo-translation's two phase classes +}; + +TwinImmuneZoneResult AnalyzeTwinImmuneZones(const std::vector& p1_merged, + const gemmi::SpaceGroup& group, + const std::array* tncs_vector = nullptr); + +std::string TwinImmuneZonesToText(const TwinImmuneZoneResult& result); diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 4106b22e4..552b7380f 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -1133,6 +1133,21 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, " absorbed can only show there as an operator averaging unequal intensities. These\n" " numbers are not circular in that way.", true)); } + if (result.twin_immune_zones && !result.twin_immune_zones->zones.empty()) { + // Report-only evidence, one row per class of 2-folds of the adopted group. Not a decision: + // the space group above was chosen without it. + const auto &tz = *result.twin_immune_zones; + for (size_t i = 0; i < tz.zones.size(); ++i) { + const auto &z = tz.zones[i]; + Add(s, KeyText(fmt::format("TWIN_ZONE_{}", i + 1).c_str(), + fmt::format("{}; n={} <|E2-1|>={:.3f}+-{:.3f} evidence={:+.1f} nats", z.operators, + z.n, z.mean_abs_e2_minus_1, z.standard_error, z.evidence_nats))); + } + Add(s, KeyText("TWIN_ZONE_CONTROL", + fmt::format("n={} <|E2-1|>={:.3f}+-{:.3f}", tz.control.n, + tz.control.mean_abs_e2_minus_1, tz.control.standard_error))); + Add(s, Prose("\n" + TwinImmuneZonesToText(tz))); + } if (tw.adopted_operators_suspect) Warn(doc, PathologyCode::TWINNING, fmt::format( "An adopted symmetry operator relates unequal intensities (<|L|> = {:.3f} in a holohedral " diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 20d4d0063..44a16b013 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -6784,6 +6784,16 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b experiment_.SpaceGroupNumber(1); const auto p1 = scale_and_merge("P1 cross-check", false); + // Twin-immune zone evidence for each 2-fold the adopted group holds, read on this merge + // because it is folded under no symmetry, at full resolution and production-scaled. + // Report-only: nothing reads it back. + if (determined != nullptr && determined->number > 1) { + result.twin_immune_zones = AnalyzeTwinImmuneZones( + p1.merged, *determined, TranslationalNCSVectorForLTest(result.tncs)); + if (!result.twin_immune_zones->zones.empty()) + logger.Info("{}", TwinImmuneZonesToText(*result.twin_immune_zones)); + } + // FINALIST LEDGER (--finalist-ledger), report-only and deliberately behind a flag: these // numbers are an instrument, and reading them as a decision before they are calibrated // is the one way this can do harm. diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index b6e8d67bc..2dc87f29f 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -317,6 +317,10 @@ struct ProcessResult { // for one. False when there was no cell to run the Patterson on. bool pre_promotion_tncs_measured = false; + // Twin-immune zone evidence for the 2-folds of the adopted group, on the P1 cross-check merge + // (AnalyzeTwinImmuneZones). Report-only. Unset when no P1 cross-check merge was built. + std::optional twin_immune_zones; + // What --model found, when one was given. Set even when the validation could not run - its // failure_reason then says why - so the report can report the failure rather than leave a run // that was asked to validate looking like one that was not. Unset when no model was asked for. diff --git a/tests/TwinningAnalysisTest.cpp b/tests/TwinningAnalysisTest.cpp index 006f9a2b6..06c6b8d60 100644 --- a/tests/TwinningAnalysisTest.cpp +++ b/tests/TwinningAnalysisTest.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -216,3 +217,74 @@ TEST_CASE("A holohedral merge under a false operator is called suspect", "[twinn CHECK(TwinningVerdictLine(r).rfind("SYMMETRY SUSPECT", 0) == 0); } } + +namespace { + // A P1 merge whose centric reflections are centric: SyntheticMergedReflections draws every + // intensity from the acentric distribution, which is all the searches it was written for need, but + // the zone test reads exactly the difference. E^2 is chi^2 with one degree of freedom for a + // reflection centric in the true group (a squared Box-Muller normal), exponential otherwise, keyed + // on the asymmetric-unit index so the true group's symmetry is exact. + std::vector WilsonTetragonal(const char *true_group, double twin_fraction) { + const gemmi::SpaceGroup &sub = gemmi::get_spacegroup_by_name(true_group); + const gemmi::SpaceGroup &super = gemmi::get_spacegroup_by_name("P 4 2 2"); + const gemmi::Op twin = jfjoch_test::TwinLaw(sub, super); + const gemmi::GroupOps gops = sub.operations(); + const gemmi::ReciprocalAsu rasu(&sub); + const gemmi::UnitCell cell(47, 47, 63, 90, 90, 90); + auto true_intensity = [&](const gemmi::Op::Miller &hkl) { + const auto asu = rasu.to_asu(hkl, gops).first; + const double u1 = jfjoch_test::detail::UniformFromHkl(asu); + const double u2 = jfjoch_test::detail::UniformFromHkl({{asu[0] + 300, asu[1] + 300, asu[2] + 300}}); + const double c = std::cos(2.0 * std::numbers::pi * u2); + const double e2 = gops.is_reflection_centric(hkl) ? -2.0 * std::log(u1) * c * c : -std::log(u1); + return 1000.0 * e2 * std::exp(-20.0 / (2.0 * std::pow(cell.calculate_d(hkl), 2))); + }; + std::vector out; + for (int h = -19; h <= 19; ++h) + for (int k = -19; k <= 19; ++k) + for (int l = -26; l <= 26; ++l) { + if (std::make_tuple(h, k, l) <= std::make_tuple(-h, -k, -l)) + continue; + const gemmi::Op::Miller hkl{{h, k, l}}; + const double d = cell.calculate_d(hkl); + if (d < 2.5) + continue; + MergedReflection r; + r.h = h; + r.k = k; + r.l = l; + r.d = static_cast(d); + r.I = static_cast((1.0 - twin_fraction) * true_intensity(hkl) + + twin_fraction * true_intensity(twin.apply_to_hkl(hkl))); + r.sigma = static_cast(0.01 * r.I + 1.0); + out.push_back(r); + } + return out; + } +} + +// Reflections centric in the adopted group but acentric in a subgroup are their own twin mates under +// the operators the group adds. A perfect twin makes every operator statistic read "real", but these +// stay acentric under it and are centric only where the added operators are real. +TEST_CASE("Twin-immune zones tell a perfect twin from real symmetry", "[twinning]") { + const gemmi::SpaceGroup &p422 = gemmi::get_spacegroup_by_name("P 4 2 2"); + + const auto real = AnalyzeTwinImmuneZones(WilsonTetragonal("P 4 2 2", 0.0), p422); + REQUIRE(real.zones.size() == 3); // over 4, over 222, over the diagonal 222 + for (const auto &z : real.zones) { + CAPTURE(z.operators); + CHECK(z.mean_abs_e2_minus_1 == Catch::Approx(0.968).margin(0.08)); + CHECK(z.evidence_nats > 0.0); + } + CHECK(real.control.mean_abs_e2_minus_1 == Catch::Approx(0.736).margin(0.05)); + + // A perfect P4 twin: whichever subgroup is taken as the parent, the operators added over it are + // not all real, and the zone reads acentric. + const auto twin = AnalyzeTwinImmuneZones(WilsonTetragonal("P 4", 0.5), p422); + REQUIRE(twin.zones.size() == 3); + for (const auto &z : twin.zones) { + CAPTURE(z.operators); + CHECK(z.mean_abs_e2_minus_1 == Catch::Approx(0.736).margin(0.08)); + CHECK(z.evidence_nats < 0.0); + } +} From 34834d00ba4008bab45d229a71c90f95da403fd7 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 10:06:05 +0200 Subject: [PATCH 05/28] rugnux: the report's shell check compares the fit with the shell the crossing lies in FITTED_RESOLUTION was withheld whenever the fit was finer than the d_min of the last shell whose CC1/2 still reached the target. A shell's CC1/2 is an average over the shell, so on every ordinary fall-off the crossing lies inside the first shell whose average is below the target - 0.01-0.1 A finer than the number the check allowed - and the report said "no resolution could be fitted" on data whose fall-off is textbook: 68 of 100 battery datasets, with the fit 0.01-0.02 A inside the first sub-target shell. The comparison is now against that shell's d_min, and the RESOLUTION_FIT flag fires only where the fit is finer than the shell the crossing can lie in. Co-Authored-By: Claude Fable 5.1 --- rugnux/ResultReport.cpp | 35 +++++++++++++++++++++++++++-------- tests/ResultReportTest.cpp | 7 +++++++ 2 files changed, 34 insertions(+), 8 deletions(-) diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 552b7380f..f2f76ab82 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -204,9 +204,8 @@ namespace { return o.str(); } - // The finest shell whose CC1/2 still reaches the cut's own target. A fitted resolution finer than - // this was read off a curve that never behaved like a fall-off, and asserting it states a - // resolution the shell table on the same page refutes. + // The finest shell whose CC1/2 still reaches the cut's own target: where the table says the signal + // is, for the completeness quoted beside the written range. double FinestSupportedShell(const MergeStatistics &ms, double target) { double finest = 0.0; for (const auto &sh : ms.shells) @@ -215,6 +214,25 @@ namespace { return finest; // 0.0 = no shell reaches the target at all } + // The finest resolution the shell table supports a fitted crossing at. A shell's CC1/2 is an + // average over the shell, so the crossing sits INSIDE the first shell whose average falls below the + // target, between its d_max and its d_min: a fit anywhere down to that shell's d_min is what the + // table shows, and only a fit finer than it is one the table refutes. Comparing against the last + // shell ABOVE the target instead withheld the number on 68 of 100 battery datasets whose fit sat + // 0.01-0.02 A inside the first sub-target shell. 0.0 when no shell reaches the target; the finest + // shell when none falls below it. + double CrossingShellLimit(const MergeStatistics &ms, double target) { + double finest = 0.0; + for (const auto &sh : ms.shells) { + if (!std::isfinite(sh.cc_half)) + continue; + if (sh.cc_half < target) + return finest == 0.0 ? 0.0 : sh.d_min; + finest = sh.d_min; + } + return finest; + } + // Where CC1/2 climbs back above the cut's target after it has already fallen below it. Below the // target a shell carries no signal the cut would keep, so a shell past that point which // correlates again is noise correlating with noise: the curve is not a fall-off, and no single @@ -619,6 +637,7 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, // than the finest shell that still reaches the target asserts a resolution the shell // table on this very page refutes. const double finest_supported = FinestSupportedShell(ms, cc_target); + const double crossing_limit = CrossingShellLimit(ms, cc_target); const double climbs_back = CCHalfClimbsBackAboveTarget(ms, cc_target); std::string fit_suppressed; // Only where there are shells to judge the fit against; with no table there is no @@ -630,15 +649,15 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, " at {:.2f} A, so this curve is not a fall-off and no one resolution can be" " read off it", cc_target, climbs_back); else if (result.resolution_fit_A) { - if (finest_supported == 0.0) + if (crossing_limit == 0.0) fit_suppressed = fmt::format( "CC1/2 never reaches the cut's target of {:.2f} in any shell, so there is" " no fall-off to fit", cc_target); - else if (*result.resolution_fit_A < finest_supported - 1e-6) + else if (*result.resolution_fit_A < crossing_limit - 1e-6) fit_suppressed = fmt::format( - "the fit reads {:.2f} A, finer than the finest shell whose CC1/2 still" - " reaches {:.2f} ({:.2f} A), so the curve it was read off is not a" - " fall-off", *result.resolution_fit_A, cc_target, finest_supported); + "the fit reads {:.2f} A, finer than the shell where CC1/2 falls below" + " {:.2f} ({:.2f} A), so the curve it was read off is not a fall-off", + *result.resolution_fit_A, cc_target, crossing_limit); else fitted_resolution = *result.resolution_fit_A; } diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index 5bafc0525..9a246edb8 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -628,6 +628,13 @@ TEST_CASE("ResultReport_FittedResolutionSuppressed", "[Diagnostics]") { CHECK(RenderResultReport("p", "in.h5", x, result).find("\nFITTED_RESOLUTION= 0.95\n") != std::string::npos); + // A shell's CC1/2 is its average, so the crossing lies inside the first shell below the target: + // a fit there is what the table shows, not something it refutes. + result.resolution_fit_A = 0.85; + CHECK(RenderResultReport("p", "in.h5", x, result).find("\nFITTED_RESOLUTION= 0.85\n") + != std::string::npos); + result.resolution_fit_A = 0.95; + // CC1/2 that drops through the target and climbs back above it is not a fall-off either, and the // finest shell reaching the target is then noise correlating with noise - so a fit the test above // accepts must still be suppressed, and the user told so in the default report. From aff64921198f92e4f5d734e1c825823972d193e4 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 08:39:27 +0200 Subject: [PATCH 06/28] rugnux: weighted median for the rot3d merge outlier test The per-reflection median the merge rejects outliers against was an unweighted median of the scaled intensities. On a sweep where the crystal barely diffracts over a long stretch, observations from those frames are scaled up by 1/G together with their sigmas, and in P1 at multiplicity ~3 two of them outvote one well-measured observation: the median becomes noise and the well-measured observation is rejected against its own small sigma. On one P1 sweep with ~44% such frames this rejected 52,574 observations (2,131 once those frames are dropped) and took the correlation of the merged intensities with an external reference from 0.92 to 0.68 at low resolution. The median is now weighted by 1/(sigma*corr)^2, which leaves it unchanged where the observations are comparably precise: rejections 3,932, correlation 0.92. Co-Authored-By: Claude Opus 5 (1M context) --- .../scale_merge/RotationScaleMerge.cpp | 24 +++++++++++++++---- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index e651313f3..7bb72e504 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -3752,21 +3752,35 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool std::vector start(n_groups + 1, 0); for (int g = 0; g < n_groups; ++g) start[g + 1] = start[g] + (cnt[g] >= 3 ? cnt[g] : 0); - std::vector iv(start[n_groups]); + // Inverse-variance WEIGHTED median: a frame the crystal barely diffracted on is scaled up by + // 1/G together with its sigma, so a plain median lets two such observations outvote one + // well-measured one - and the test below then rejects the well-measured one against its + // own small sigma. + std::vector> iv(start[n_groups]); // (I*corr, 1/(sigma*corr)^2) { std::vector fill(start.begin(), start.end() - 1); for (int i = 0; i < n_full; ++i) { const int g = mf.group[i]; - if (g >= 0 && cnt[g] >= 3) iv[fill[g]++] = mf.I[i] * mf.corr[i]; + if (g >= 0 && cnt[g] >= 3) { + const float sc = mf.sigma[i] * mf.corr[i]; + iv[fill[g]++] = {mf.I[i] * mf.corr[i], sc > 0.0f ? 1.0f / (sc * sc) : 0.0f}; + } } } ParallelChunks(n_groups, ThreadsForWork(iv.size(), nthreads), [&](int glo, int ghi) { for (int g = glo; g < ghi; ++g) { const int lo = start[g], hi = start[g + 1]; if (lo == hi) continue; - const auto mid = iv.begin() + lo + (hi - lo) / 2; - std::nth_element(iv.begin() + lo, mid, iv.begin() + hi); - reject_median[g] = *mid; + std::sort(iv.begin() + lo, iv.begin() + hi); + double total = 0.0; + for (int j = lo; j < hi; ++j) total += iv[j].second; + double run = 0.0; + int j = lo; + for (; j < hi - 1; ++j) { + run += iv[j].second; + if (run >= 0.5 * total) break; + } + reject_median[g] = iv[j].first; } }); } From 0a342fcb134876347dff1d59780e12b4fe788ae0 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 10:34:48 +0200 Subject: [PATCH 07/28] rugnux: weight every CC1/2 by the information a reflection carries The resolution cutoff, the shell table and the overall CC1/2 counted every unique reflection equally. The merge itself is inverse-variance weighted, so an observation from a frame the crystal barely diffracted on enters it at 1/G^2 of a good one - honestly, with its sigma - but a reflection measured only on such frames is scaled-up noise that then counts as much as a well-measured pair in every Pearson CC1/2 read off the merge. On a sweep where half the frames are weak the curve collapses at every resolution: the cut lands at 3.7 A on a P1 crystal whose good frames reach 1.5 A, and the UNUSABLE verdict fires (CC1/2 0.40 beside I/sigma 10). Each merged reflection now carries cc_weight: the precision its half-sets would have had with every observation at the run's typical frame scale, over the precision they have. G_ref = sum G^3 / sum G^2 over the usable observations is the precision-weighted typical scale, which the dead frames cannot drag down however many there are; the factor per observation is max(1, (G_ref/G)^2), with G the frame's total scale (partial scale, flux and the fulls' own G) taken before the correction surfaces and before collapsed frames are dropped, so nothing intensity- or resolution-dependent enters it. On a sweep without a weak stretch every weight is 1 and the CC1/2 is the plain Pearson it was. The cutoff fit, the shell table and the overall CC1/2 (and so the UNUSABLE verdict and the report's shell checks) all read the same weighted statistic. The two extra per-group sums are accumulated on both merge paths, the host loop and MergeAccumKernel, from one per-frame factor array; a host recompute of the device sums agrees to 1e-15 relative on every merge after the corrected corr is uploaded, and a host-merge run gives the same cut, space group, CC1/2 and ISa on five sets. Weighting by the half-set error variance alone (1/(v0+v1)) is not this: v grows with the intensity, so it weights the weak end of the intensity distribution and biases homogeneous data coarser. Measured (written resolution, together with the weighted outlier median): a P1 sweep with a long weak stretch 3.73 -> 1.40 A against a 1.63 A XDS reference, UNUSABLE withdrawn (overall CC1/2 0.40 -> 0.97); a second 3.61 -> 3.00 A; one with most of the sweep out of beam 7.35 -> 5.20 A. Lysozyme, thaumatin and two insulin sets unchanged to 0.01 A (one lysozyme sweep with a weak wedge 1.13 -> 1.16 A, from the median), same space groups throughout. Co-Authored-By: Claude Opus 5 (1M context) --- common/CorrelationCoefficient.cpp | 17 +++--- common/CorrelationCoefficient.h | 5 +- common/Reflection.h | 5 ++ .../scale_merge/ResolutionCutoff.cpp | 22 +++++++- image_analysis/scale_merge/ResolutionCutoff.h | 8 +++ .../scale_merge/RotationScaleMerge.cpp | 39 ++++++++++++- .../scale_merge/RotationScaleMerge.h | 10 +++- .../scale_merge/RotationScaleMergeGPU.cu | 31 +++++++---- .../scale_merge/RotationScaleMergeGPU.h | 8 ++- tests/MergeScaleTest.cpp | 55 ++++++++++++++++++- 10 files changed, 168 insertions(+), 32 deletions(-) diff --git a/common/CorrelationCoefficient.cpp b/common/CorrelationCoefficient.cpp index 24925571c..7c21dbb26 100644 --- a/common/CorrelationCoefficient.cpp +++ b/common/CorrelationCoefficient.cpp @@ -6,19 +6,20 @@ #include "CorrelationCoefficient.h" -void CorrelationCoefficient::Add(double x, double y) { - sum_x += x; - sum_y += y; - sum_x2 += x * x; - sum_y2 += y * y; - sum_xy += x * y; +void CorrelationCoefficient::Add(double x, double y, double w) { + sum_x += w * x; + sum_y += w * y; + sum_x2 += w * x * x; + sum_y2 += w * y * y; + sum_xy += w * x * y; + sum_w += w; n_cc++; } double CorrelationCoefficient::GetCC() const { - if (n_cc < 2) + if (n_cc < 2 || !(sum_w > 0.0)) return NAN; - const double n = static_cast(n_cc); + const double n = sum_w; const double mean_x = sum_x / n; const double mean_y = sum_y / n; diff --git a/common/CorrelationCoefficient.h b/common/CorrelationCoefficient.h index 711284226..943a3936a 100644 --- a/common/CorrelationCoefficient.h +++ b/common/CorrelationCoefficient.h @@ -9,9 +9,12 @@ class CorrelationCoefficient { double sum_x2 = 0.0; double sum_y2 = 0.0; double sum_xy = 0.0; + double sum_w = 0.0; int n_cc = 0; public: - void Add(double x, double y); + void Add(double x, double y) { Add(x, y, 1.0); } + // Weighted pair: a plain Pearson when every weight is the same. + void Add(double x, double y, double w); [[nodiscard]] double GetCC() const; }; diff --git a/common/Reflection.h b/common/Reflection.h index 5e4f86321..2e304cd79 100644 --- a/common/Reflection.h +++ b/common/Reflection.h @@ -64,6 +64,11 @@ struct MergedReflection { float sigma = NAN; float I_half[2] = {NAN, NAN}; float sigma_half[2] = {NAN, NAN}; + // Weight of this reflection in a CC1/2: the precision its half-sets would have had with every + // observation at the run's typical frame scale, over the precision they have. 1 on a sweep + // without a weak stretch; small for a reflection measured only where the crystal barely + // diffracted, whose scaled-up noise would otherwise count as much as a well-measured pair. + float cc_weight = 1.0f; float d = 0.0; // Any observation of this reflection sat on an ice ring. The intensity is still merged and // written - the ring contaminates it, it does not make it absent - and this only marks it so diff --git a/image_analysis/scale_merge/ResolutionCutoff.cpp b/image_analysis/scale_merge/ResolutionCutoff.cpp index 071a32b56..236a0323c 100644 --- a/image_analysis/scale_merge/ResolutionCutoff.cpp +++ b/image_analysis/scale_merge/ResolutionCutoff.cpp @@ -33,6 +33,26 @@ namespace { } } +std::vector CCHalfFrameFactors(const std::vector &frame_scale, + const std::vector &n_obs) { + auto g_of = [&](size_t f) { + return std::isfinite(frame_scale[f]) && frame_scale[f] > 0.0 ? frame_scale[f] : 1.0; + }; + double s3 = 0.0, s2 = 0.0; + for (size_t f = 0; f < frame_scale.size(); ++f) { + const double g = g_of(f); + s3 += n_obs[f] * g * g * g; + s2 += n_obs[f] * g * g; + } + const double g_ref = s2 > 0.0 ? s3 / s2 : 1.0; + std::vector factor(frame_scale.size()); + for (size_t f = 0; f < frame_scale.size(); ++f) { + const double r = g_ref / g_of(f); + factor[f] = std::max(1.0, r * r); + } + return factor; +} + ResolutionCutoffResult ComputeCCHalfLogisticCutoff(const std::vector &merged, double cc_target, Logger &logger) { ResolutionCutoffResult result; @@ -82,7 +102,7 @@ ResolutionCutoffResult ComputeCCHalfLogisticCutoff(const std::vector(m.d) * m.d); int b = static_cast((s - s_lo) / bin_w); b = std::clamp(b, 0, N_FIT_BINS - 1); - bin_cc[b].Add(m.I_half[0], m.I_half[1]); + bin_cc[b].Add(m.I_half[0], m.I_half[1], m.cc_weight); ++bin_n[b]; } diff --git a/image_analysis/scale_merge/ResolutionCutoff.h b/image_analysis/scale_merge/ResolutionCutoff.h index b6913aea3..730013d97 100644 --- a/image_analysis/scale_merge/ResolutionCutoff.h +++ b/image_analysis/scale_merge/ResolutionCutoff.h @@ -34,6 +34,14 @@ struct ResolutionCutoffResult { std::string note; // human-readable description of the decision (for logging) }; +// The per-frame factor of the CC1/2 weight (MergedReflection::cc_weight): max(1, (G_ref/G)^2), the +// variance an observation from a frame at scale G carries over the same observation at the typical +// scale G_ref = sum G^3 / sum G^2 over the observations (n_obs[f] of them on frame f). Precision- +// weighted, so frames the crystal barely diffracted on cannot drag the typical scale down however many +// of them there are. A frame without a finite positive scale counts as G = 1. +std::vector CCHalfFrameFactors(const std::vector &frame_scale, + const std::vector &n_obs); + ResolutionCutoffResult ComputeCCHalfLogisticCutoff(const std::vector &merged, double cc_target, Logger &logger); diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 7bb72e504..7457c5c05 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -3819,6 +3819,22 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // above, on all threads, so a second serial walk over the fat records only rebuilds it. AssignHalvesByRank(obs_key, mf.group, n_groups, merge_half, nthreads); + // ---- CC1/2 weight (MergedReflection::cc_weight): the precision each half-set WOULD have had with + // every observation at the run's typical frame scale, over the precision it has. An observation + // from a frame the crystal barely diffracted on was scaled up by G_ref/G, so it carries + // (G_ref/G)^2 more counting variance than the same observation at the typical scale; that + // factor, and nothing intensity-dependent, is what the pair is discounted by. G_ref = + // sum G^3 / sum G^2 over the usable observations: the precision-weighted typical scale, which + // the dead frames cannot drag down however many of them there are. A frame brighter than + // typical earns nothing (the factor is floored at 1), so on a sweep without a weak stretch + // every weight is 1 and the CC1/2 is the plain Pearson it was. G is the frame's scale taken + // before the correction surfaces (frame_scale_fulls), which are per-observation and + // resolution-dependent and are not a property of the frame. ---- + std::vector n_obs_frame(frame_scale_fulls.size(), 0); + for (int i = 0; i < n_full; ++i) + if (mf.group[i] >= 0) ++n_obs_frame[fulls[i].frame]; + const std::vector frame_cc_factor = CCHalfFrameFactors(frame_scale_fulls, n_obs_frame); + // ---- Merge: per-group inverse-variance sums with corrected sigma + deterministic half sets, then // export. In a lambda because the automatic resolution cutoff below re-runs it once the error // model has been refitted on the reflections that survive the cut. ---- @@ -3852,8 +3868,10 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool ms.Resize(n_groups); gpu_->MergeAccum(error_model_a, error_model_b, error_model_active, reject_outliers, reject_nsigma, reject_median.data(), merge_half.data(), + frame_cc_factor.data(), ms.swI.data(), ms.sw.data(), ms.swIh0.data(), ms.swIh1.data(), - ms.swh0.data(), ms.swh1.data(), ms.nh0.data(), ms.nh1.data(), + ms.swh0.data(), ms.swh1.data(), ms.swh_typ0.data(), ms.swh_typ1.data(), + ms.nh0.data(), ms.nh1.data(), ms.d.data(), ms.rej.data(), ms.on_ice.data(), rejected_obs.data()); // Per-group and independent, so the unpack splits; the rejected count is an integer sum and // does not care in which order the chunks add theirs in. @@ -3864,6 +3882,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool Accum &a = acc[g]; a.swI = ms.swI[g]; a.sw = ms.sw[g]; a.swIh[0] = ms.swIh0[g]; a.swIh[1] = ms.swIh1[g]; a.swh[0] = ms.swh0[g]; a.swh[1] = ms.swh1[g]; + a.swh_typ[0] = ms.swh_typ0[g]; a.swh_typ[1] = ms.swh_typ1[g]; a.nh[0] = static_cast(ms.nh0[g]); a.nh[1] = static_cast(ms.nh1[g]); a.d = static_cast(ms.d[g]); a.on_ice = ms.on_ice[g] != 0; @@ -3893,6 +3912,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool const int half = merge_half[&o - fulls.data()]; a.swI += wI; a.sw += w; a.swIh[half] += wI; a.swh[half] += w; a.nh[half]++; + a.swh_typ[half] += w * frame_cc_factor[o.frame]; // Any contributing observation on an ice ring marks the merged reflection (as // MergeOnTheFly does on the stills path): the intensity is still merged and written, // the flag only lets a consumer that must not read ring signal as crystal signal - @@ -3922,6 +3942,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool mr.I_half[i] = static_cast(a.swIh[i] / a.swh[i]); mr.sigma_half[i] = static_cast(1.0 / std::sqrt(a.swh[i])); } + mr.cc_weight = static_cast((1.0 / a.swh_typ[0] + 1.0 / a.swh_typ[1]) + / (1.0 / a.swh[0] + 1.0 / a.swh[1])); } if (!std::isfinite(a.d) || a.d <= 0.0f) continue; merged_I[g] = mr.I; @@ -4142,8 +4164,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool s.unique++; s.sum_i_over_sigma += m.I / m.sigma; s.n_i_over_sigma++; if (std::isfinite(m.I_half[0]) && std::isfinite(m.I_half[1])) { - s.cc_half.Add(m.I_half[0], m.I_half[1]); - cc_half_overall.Add(m.I_half[0], m.I_half[1]); + s.cc_half.Add(m.I_half[0], m.I_half[1], m.cc_weight); + cc_half_overall.Add(m.I_half[0], m.I_half[1], m.cc_weight); } } } @@ -4792,6 +4814,17 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st } logger.Info("Scaled fulls (XDS order, Unity model)"); } + // The total scale each frame's fulls were amplified by - partial scale, incident flux and the fulls' + // own G (corr = 1/G here) - taken BEFORE the correction surfaces fold per-observation terms into + // corr, and before a collapsed frame's fulls are zeroed: the resident fulls only get that zero with + // the corrected corr, so a merge on the device before then still sees them, and must weigh them by + // their own scale. Read by MergeAndStats for the CC1/2 weight. + frame_scale_fulls.assign(n_frames, NAN); + for (const auto &o : fulls) + if (std::isfinite(o.corr) && o.corr > 0.0f && !std::isfinite(frame_scale_fulls[o.frame])) { + const double gp = g_partial[o.frame] * frame_flux[o.frame]; + frame_scale_fulls[o.frame] = (std::isfinite(gp) && gp > 0.0 ? gp : 1.0) / o.corr; + } const bool rejected_full_scales = scale_fulls && DropCollapsedFullScales(combined_on_gpu && scaled_fulls_on_gpu); diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index 24a1fed00..3855f1a0f 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -273,6 +273,7 @@ private: std::vector fulls; // combined fulls (rebuilt each Run), sorted by frame std::vector fulls_frame_start, fulls_frame_count; // CSR ranges of `fulls` per frame std::vector g_full; // per-frame scale on the fulls + std::vector frame_scale_fulls; // per-frame total scale of the fulls before the surfaces (cc_weight) // The five fields the host-side walks over the fulls actually read, pulled out of the 80-byte record // once per merge (see MergeAndStats). `group` carries the usability decision: -1 means the full is @@ -306,15 +307,18 @@ private: // The merge accumulators (see MergeAndStats' run_merge): one entry per ASU group, plus the arrays // the device kernel fills that the host unpacks into them. Members for the same reason as // FullsStaging - a merge runs several times per run and this is a hundred megabytes between them. - struct Accum { double swI = 0, sw = 0, swIh[2] = {0, 0}, swh[2] = {0, 0}; size_t nh[2] = {0, 0}; float d = NAN; - bool on_ice = false; }; + // swh_typ: each half-set's precision had every observation been at the run's typical frame scale + // (see MergeAndStats, cc_weight). + struct Accum { double swI = 0, sw = 0, swIh[2] = {0, 0}, swh[2] = {0, 0}, swh_typ[2] = {0, 0}; + size_t nh[2] = {0, 0}; float d = NAN; bool on_ice = false; }; std::vector merge_acc; struct MergeAccumStaging { - std::vector swI, sw, swIh0, swIh1, swh0, swh1, d; + std::vector swI, sw, swIh0, swIh1, swh0, swh1, swh_typ0, swh_typ1, d; std::vector nh0, nh1, rej; std::vector on_ice; void Resize(int n) { swI.resize(n); sw.resize(n); swIh0.resize(n); swIh1.resize(n); swh0.resize(n); swh1.resize(n); + swh_typ0.resize(n); swh_typ1.resize(n); d.resize(n); nh0.resize(n); nh1.resize(n); rej.resize(n); on_ice.resize(n); } }; diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.cu b/image_analysis/scale_merge/RotationScaleMergeGPU.cu index 40e146dc5..2f51def78 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.cu +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.cu @@ -464,13 +464,13 @@ namespace { const int32_t *group, *frame; const uint8_t *on_ice, *frame_cell_ok, *half; const int32_t *gperm, *gstart, *gcount; - const double *em_mean, *merged_I; + const double *em_mean, *merged_I, *cc_factor; // outputs double *sw, *swI, *em_mean_out; int32_t *cnt; double *s2, *I2, *dev2; uint8_t *valid; - double *a_swI, *a_sw, *a_swIh0, *a_swIh1, *a_swh0, *a_swh1, *a_d; + double *a_swI, *a_sw, *a_swIh0, *a_swIh1, *a_swh0, *a_swh1, *a_swht0, *a_swht1, *a_d; int32_t *a_nh0, *a_nh1, *a_rejected; uint8_t *a_on_ice; double *r_absdev, *r_sumI; @@ -544,7 +544,7 @@ namespace { __global__ void MergeAccumKernel(MergeParams p) { for (int g = blockIdx.x * blockDim.x + threadIdx.x; g < p.n_groups; g += gridDim.x * blockDim.x) { const int lo = p.gstart[g], hi = lo + p.gcount[g]; - double swI = 0, sw = 0, swIh0 = 0, swIh1 = 0, swh0 = 0, swh1 = 0, dd = NAN; + double swI = 0, sw = 0, swIh0 = 0, swIh1 = 0, swh0 = 0, swh1 = 0, swht0 = 0, swht1 = 0, dd = NAN; int nh0 = 0, nh1 = 0, rejected = 0, onice = 0; const float rmed = p.reject_median[g]; for (int q = lo; q < hi; ++q) { @@ -574,13 +574,16 @@ namespace { // a rank within the reflection cannot be read off a running count, and computing it // in one place is what makes this kernel and the host merge loop agree exactly. const int half = p.half[i]; + const double wt = w * p.cc_factor[p.frame[i]]; // see the host merge loop (cc_weight) swI += wI; sw += w; - if (half) { swIh1 += wI; swh1 += w; ++nh1; } else { swIh0 += wI; swh0 += w; ++nh0; } + if (half) { swIh1 += wI; swh1 += w; swht1 += wt; ++nh1; } + else { swIh0 += wI; swh0 += w; swht0 += wt; ++nh0; } if (p.on_ice[i]) onice = 1; // see the host merge loop if (!isfinite(dd) && isfinite(p.d[i]) && p.d[i] > 0.0f) dd = p.d[i]; } p.a_swI[g] = swI; p.a_sw[g] = sw; p.a_swIh0[g] = swIh0; p.a_swIh1[g] = swIh1; - p.a_swh0[g] = swh0; p.a_swh1[g] = swh1; p.a_nh0[g] = nh0; p.a_nh1[g] = nh1; p.a_d[g] = dd; + p.a_swh0[g] = swh0; p.a_swh1[g] = swh1; p.a_swht0[g] = swht0; p.a_swht1[g] = swht1; + p.a_nh0[g] = nh0; p.a_nh1[g] = nh1; p.a_d[g] = dd; p.a_rejected[g] = rejected; p.a_on_ice[g] = uint8_t(onice); } } @@ -657,7 +660,8 @@ struct RotationScaleMergeGPU::Impl { CudaDevicePtr m_valid; CudaDevicePtr m_rejected; // per-full outlier-rejected flag (MergeAccum -> MergeRmeas) CudaDevicePtr m_half; // per-full CC1/2 half-set, assigned on the host - CudaDevicePtr a_swI, a_sw, a_swIh0, a_swIh1, a_swh0, a_swh1, a_d; // merge accum per group + CudaDevicePtr a_swI, a_sw, a_swIh0, a_swIh1, a_swh0, a_swh1, a_swht0, a_swht1, a_d; // merge accum per group + CudaDevicePtr cc_factor; // per-frame CC1/2 weight factor (length n_frames) CudaDevicePtr a_nh0, a_nh1, a_rejected; CudaDevicePtr a_on_ice; // per group: any contributing full on an ice ring CudaDevicePtr reject_median; @@ -902,9 +906,10 @@ void RotationScaleMergeGPU::MergeEmSamples(bool for_search, double min_partialit void RotationScaleMergeGPU::MergeAccum(double error_model_a, double error_model_b, bool error_model_active, bool reject_outliers, double reject_nsigma, const float *reject_median, - const uint8_t *half, + const uint8_t *half, const double *frame_cc_factor, double *swI, double *sw, double *swIh0, double *swIh1, - double *swh0, double *swh1, int32_t *nh0, int32_t *nh1, double *d_out, + double *swh0, double *swh1, double *swh_typ0, double *swh_typ1, + int32_t *nh0, int32_t *nh1, double *d_out, int32_t *rejected, uint8_t *on_ice_out, uint8_t *rejected_obs) { DeviceGuard guard(impl_->device, impl_->available); auto &d = *impl_; @@ -912,6 +917,7 @@ void RotationScaleMergeGPU::MergeAccum(double error_model_a, double error_model_ d.a_swI = CudaDevicePtr(std::max(1, ng)); d.a_sw = CudaDevicePtr(std::max(1, ng)); d.a_swIh0 = CudaDevicePtr(std::max(1, ng)); d.a_swIh1 = CudaDevicePtr(std::max(1, ng)); d.a_swh0 = CudaDevicePtr(std::max(1, ng)); d.a_swh1 = CudaDevicePtr(std::max(1, ng)); + d.a_swht0 = CudaDevicePtr(std::max(1, ng)); d.a_swht1 = CudaDevicePtr(std::max(1, ng)); d.a_nh0 = CudaDevicePtr(std::max(1, ng)); d.a_nh1 = CudaDevicePtr(std::max(1, ng)); d.a_d = CudaDevicePtr(std::max(1, ng)); d.a_rejected = CudaDevicePtr(std::max(1, ng)); d.a_on_ice = CudaDevicePtr(std::max(1, ng)); @@ -920,6 +926,7 @@ void RotationScaleMergeGPU::MergeAccum(double error_model_a, double error_model_ "zero m_rejected"); Upload(d.reject_median, reject_median, ng); Upload(d.m_half, half, d.n_fulls); + Upload(d.cc_factor, frame_cc_factor, d.n_frames); MergeParams p{}; p.n_groups = ng; p.min_partiality = d.merge_min_part; @@ -934,9 +941,10 @@ void RotationScaleMergeGPU::MergeAccum(double error_model_a, double error_model_ p.half = d.m_half.get(); p.var_bkg = d.f_var_bkg.get(); p.var_per_I = d.f_var_per_I.get(); p.gperm = d.f_gperm.get(); p.gstart = d.f_gstart.get(); p.gcount = d.f_gcount.get(); - p.em_mean = d.m_em_mean.get(); + p.em_mean = d.m_em_mean.get(); p.cc_factor = d.cc_factor.get(); p.a_swI = d.a_swI.get(); p.a_sw = d.a_sw.get(); p.a_swIh0 = d.a_swIh0.get(); p.a_swIh1 = d.a_swIh1.get(); - p.a_swh0 = d.a_swh0.get(); p.a_swh1 = d.a_swh1.get(); p.a_nh0 = d.a_nh0.get(); p.a_nh1 = d.a_nh1.get(); + p.a_swh0 = d.a_swh0.get(); p.a_swh1 = d.a_swh1.get(); + p.a_swht0 = d.a_swht0.get(); p.a_swht1 = d.a_swht1.get(); p.a_nh0 = d.a_nh0.get(); p.a_nh1 = d.a_nh1.get(); p.a_d = d.a_d.get(); p.a_rejected = d.a_rejected.get(); p.a_on_ice = d.a_on_ice.get(); p.rejected_obs = d.m_rejected.get(); @@ -947,7 +955,8 @@ void RotationScaleMergeGPU::MergeAccum(double error_model_a, double error_model_ auto dl = [&](void *h, const CudaDevicePtr &s) { CudaCheck(cudaMemcpy(h, s.get(), size_t(ng) * sizeof(double), cudaMemcpyDeviceToHost), "dl accum"); }; dl(swI, d.a_swI); dl(sw, d.a_sw); dl(swIh0, d.a_swIh0); dl(swIh1, d.a_swIh1); - dl(swh0, d.a_swh0); dl(swh1, d.a_swh1); dl(d_out, d.a_d); + dl(swh0, d.a_swh0); dl(swh1, d.a_swh1); dl(swh_typ0, d.a_swht0); dl(swh_typ1, d.a_swht1); + dl(d_out, d.a_d); CudaCheck(cudaMemcpy(nh0, d.a_nh0.get(), size_t(ng) * sizeof(int32_t), cudaMemcpyDeviceToHost), "dl nh0"); CudaCheck(cudaMemcpy(nh1, d.a_nh1.get(), size_t(ng) * sizeof(int32_t), cudaMemcpyDeviceToHost), "dl nh1"); CudaCheck(cudaMemcpy(rejected, d.a_rejected.get(), size_t(ng) * sizeof(int32_t), cudaMemcpyDeviceToHost), "dl rej"); diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.h b/image_analysis/scale_merge/RotationScaleMergeGPU.h index cd4ac9464..fb853b016 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.h +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.h @@ -93,13 +93,15 @@ public: // kernel and the host merge loop cannot disagree about it. // Outputs length n_groups; rejected[g] counts outliers dropped (reject_median uploaded, NAN // where none). rejected_obs is the per-full flag (length n_fulls): the host needs it for the reductions - // it still does itself, above all the anomalous I(+)/I(-) split. + // it still does itself, above all the anomalous I(+)/I(-) split. frame_cc_factor (length n_frames) is + // each frame's (G_ref/G)^2 floored at 1; swh_typ0/1 are the half-set weights multiplied by it. // Requires MergeEmSamples first (em_mean resident). void MergeAccum(double error_model_a, double error_model_b, bool error_model_active, bool reject_outliers, double reject_nsigma, const float *reject_median, - const uint8_t *half, + const uint8_t *half, const double *frame_cc_factor, double *swI, double *sw, double *swIh0, double *swIh1, - double *swh0, double *swh1, int32_t *nh0, int32_t *nh1, double *d_out, + double *swh0, double *swh1, double *swh_typ0, double *swh_typ1, + int32_t *nh0, int32_t *nh1, double *d_out, int32_t *rejected, uint8_t *on_ice_out, uint8_t *rejected_obs); // Per-group R_meas accumulators (sum|I_corr-merged_I|, sum_I, n, and the count this looser walk diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index 31dd7676a..4794b6a7e 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -306,14 +306,17 @@ namespace { // other (signal) or drawn independently (noise, CC1/2 ~ 0), so a whole CC1/2 curve can be built // band by band. void AddBand(std::vector &v, std::mt19937 &rng, - double s_from, double s_to, int n, bool correlated) { - std::normal_distribution g(0.0, 1.0); + double s_from, double s_to, int n, bool correlated, double sigma = 1.0) { + std::normal_distribution g(0.0, sigma); for (int j = 0; j < n; ++j) { MergedReflection m; m.d = static_cast(1.0 / std::sqrt(s_from + (j + 0.5) * (s_to - s_from) / n)); const double a = g(rng), b = g(rng); m.I_half[0] = static_cast(a); m.I_half[1] = static_cast(correlated ? a : b); + // The pair's weight in a CC1/2: the precision it would have had at the typical frame scale + // over the precision it has - 1/sigma^2 for a band scaled up by sigma from dead frames. + m.cc_weight = static_cast(1.0 / (sigma * sigma)); v.push_back(m); } } @@ -336,6 +339,54 @@ TEST_CASE("ResolutionCutoff_CleanFallOff") { CHECK(*rc.d_cut == Catch::Approx(1.92).margin(0.15)); } +// A sweep with a long stretch where the crystal barely diffracted: the reflections measured only there +// are noise scaled up by 1/G, sigmas with them, and there are many of them at every resolution. Counted +// by the information they carry they must not hide where the well-measured reflections stop. +TEST_CASE("ResolutionCutoff_ScaledUpNoiseDoesNotHideTheFallOff") { + Logger logger("test"); + std::mt19937 rng(12345); + std::vector merged; + AddBand(merged, rng, 0.01, 0.25, 480, true); + AddBand(merged, rng, 0.25, 0.51, 520, false); + AddBand(merged, rng, 0.01, 0.51, 300, false, 50.0); + + const auto rc = ComputeCCHalfLogisticCutoff(merged, 0.30, logger); + REQUIRE(rc.d_fit); + CHECK(*rc.d_fit == Catch::Approx(2.0).margin(0.15)); + + // Counted as equals, the same reflections hide it: the curve reads noise from the first bin on. + for (auto &m : merged) m.cc_weight = 1.0f; + const auto unweighted = ComputeCCHalfLogisticCutoff(merged, 0.30, logger); + CHECK((!unweighted.d_fit || *unweighted.d_fit > 3.0)); +} + +// The frame factor of the CC1/2 weight is 1 on a sweep whose frames all sit at one scale, however that +// scale is spread over the frames, and 1 for any frame brighter than typical. +TEST_CASE("CCHalfFrameFactors_UniformScaleIsOne") { + const auto f = CCHalfFrameFactors({1.7, 1.7, 1.7, 1.7}, {100, 3, 250, 0}); + for (double x : f) CHECK(x == 1.0); + const auto g = CCHalfFrameFactors({1.0, 2.0}, {100, 100}); + CHECK(g[1] == 1.0); + CHECK(g[0] == Catch::Approx(9.0 / 5.0 * 9.0 / 5.0)); // G_ref = (1 + 8) / (1 + 4) +} + +// Most of the sweep at 2% of the good frames' scale: the typical scale is still the good frames', not +// the run median, and an observation from a dead frame carries 2500x the variance of a good one. +TEST_CASE("CCHalfFrameFactors_DeadStretchDoesNotSetTheTypicalScale") { + std::vector scale; + std::vector n; + for (int f = 0; f < 100; ++f) { + scale.push_back(f < 60 ? 0.02 : 1.0); + n.push_back(50); + } + scale.push_back(NAN); // a frame without a scale counts as G = 1 + n.push_back(50); + const auto factor = CCHalfFrameFactors(scale, n); + CHECK(factor[0] == Catch::Approx(2500.0).epsilon(0.01)); + CHECK(factor[99] == Catch::Approx(1.0).epsilon(1e-3)); + CHECK(factor[100] == Catch::Approx(1.0).epsilon(1e-3)); +} + // A fall-off region a logistic cannot follow: CC1/2 drops through the target and comes straight back // up. The fitted crossing is then an extrapolation far past the bins it was made over, and reading // the cut off it writes the data deep into the noise; the crossing the bins themselves show is where From d3cd834f014eea75b042af7b26761bbd7fce0296 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 10:36:09 +0200 Subject: [PATCH 08/28] rugnux: print ISa as undetermined when the error model's b is not resolved from zero The (a, b) error-model fit already refuses to report ISa when its top intensity bin has no leverage on b. It still printed 1/b when b had leverage but came out at zero within its own error - the strongest bins scatter no more than counting statistics say (the fitted b^2 is negative and clamps to 0), or there are too few samples for the bin medians to mean anything. ISa then reports the noise in b as an I/sigma: on one low-resolution sweep consecutive merges of the same data gave ISa 0, 28, 46, 113 and 130. The standard error of b^2 is taken from the 16 bins' own scatter about the fitted line, and when b^2 is less than two standard errors above zero the merge result carries isa_resolved = false. Only what is printed follows it: the report's ISA key reads "undetermined", the summary line says so, the asymptote is not printed and the mmCIF carries "?". The fitted ISa itself is unchanged and is still what the space-group search's present-reflection cut and the refused-point-group arbitration read, so no decision moves. Co-Authored-By: Claude Opus 5 (1M context) --- .../scale_merge/RotationScaleMerge.cpp | 29 +++++++++++++++++-- .../scale_merge/RotationScaleMerge.h | 4 +++ rugnux/ResultReport.cpp | 12 ++++++-- rugnux/Rugnux.cpp | 12 ++++++-- rugnux/Rugnux.h | 1 + rugnux/rugnux_cli.cpp | 10 +++++-- tests/ResultReportTest.cpp | 24 +++++++++++++++ 7 files changed, 81 insertions(+), 11 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 7457c5c05..bfde3cd48 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -3548,6 +3548,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool const std::vector *chi2_pool = nullptr; double chi2_a = 0.0, chi2_b2 = 0.0; bool error_model_b_unmeasured = false; // b had no leverage; ISa is not reported + bool error_model_b_resolved = true; // b^2 stood clear of its own standard error std::vector &samples = em_samples; samples.clear(); constexpr int n_bins = 16; @@ -3659,10 +3660,29 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool chi2_pool = ∈ chi2_a = error_model_a; chi2_b2 = 0.0; } } else if (det > 1e-10 * Ass * AII) { - error_model_a = std::clamp((Bs * AII - BI * AsI) / det, 0.25, 100.0); - const double b2 = std::max((Ass * BI - AsI * Bs) / det, 0.0); + const double a_fit = (Bs * AII - BI * AsI) / det; + const double b2_fit = (Ass * BI - AsI * Bs) / det; + error_model_a = std::clamp(a_fit, 0.25, 100.0); + const double b2 = std::max(b2_fit, 0.0); error_model_b = std::sqrt(b2); error_model_b_unmeasured = false; + // Leverage is not enough: b^2 can have it and still come out at zero within its own error - + // the strong bins scatter no more than counting says, or there are too few samples for a + // bin median to mean anything. 1/b then reads the noise in b as an I/sigma, any value from + // ~30 to infinity between merges of the same data. The standard error of b^2 is taken from + // the bins' own scatter about the fitted line; the fit and the sigmas are left as they are, + // and so is the fitted ISa every decision reads - this only decides what is printed. + // Measured: z = 13.6 and 22 on two crystals whose b is plainly visible, 0.1-0.5 on a + // 3000-sample low-multiplicity set and -1.2 to -6 on one whose strongest bins scatter + // LESS than counting says (both printed ISa 96-130). + double chi = 0.0; + for (int bin = 0; bin < n_bins; ++bin) { + const double d2w = std::max(bd2[bin], dev2_floor); + const double r = bd2[bin] - a_fit * bs2[bin] - b2_fit * bI2[bin]; + chi += r * r / (d2w * d2w); + } + const double b2_se = std::sqrt(chi / (n_bins - 2) * Ass / det); + error_model_b_resolved = b2_fit >= 2.0 * b2_se; error_model_active = true; chi2_pool = ∈ chi2_a = error_model_a; chi2_b2 = b2; } @@ -4069,6 +4089,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool { const auto em = ToXdsErrorModel(error_model_a, error_model_b); result.isa = em.isa; + result.isa_resolved = error_model_b_resolved; result.error_model_a = em.a; result.error_model_b = em.b; } @@ -4099,6 +4120,10 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool "not reach far enough for a systematic error to be seen, not that there is " "none; a resolution range matched to the signal would measure it", em.a, error_model_chi2); + else if (!error_model_b_resolved) + logger.Info("Error model (XDS convention): a={:.3f} b={:.3e} - b is not resolved from zero " + "(under two standard errors), so ISa is undetermined (1/b = {:.1f})", + em.a, em.b, em.isa); else if (!full_stats) // Neither the asymptote nor the chi2 median was measured on this merge, so neither is // reported for it. diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index 3855f1a0f..533242ad3 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -49,6 +49,10 @@ public: // only ever be the more optimistic of the two. Both 0 if the model stayed at identity. double isa = 0.0; double isa_asymptotic = 0.0; + // b^2 stood at least two standard errors clear of zero. When it did not, `isa` is still the + // fitted 1/b and every decision reads it as before; only what is PRINTED says "undetermined", + // since 1/b of a b that is zero within its error is the noise in b, not an I/sigma. + bool isa_resolved = true; double error_model_a = 0.0; // XDS convention: sigma^2 = a*(sigma0^2 + b*I^2) double error_model_b = 0.0; // The overall CC1/2 as it stood BEFORE the correction surfaces were folded in. That is what the diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index f2f76ab82..cbb2ae8ba 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -695,8 +695,12 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, // CORRECT.LP. Kept in the default report because beamline monitoring scripts read them. Add(s, KeyReal("ERROR_MODEL_A", result.error_model_a, "{:.4f}")); Add(s, KeyReal("ERROR_MODEL_B", result.error_model_b, "{:.4e}")); - Add(s, KeyReal("ISA", result.error_model_isa, "{:.2f}")); - if (result.error_model_isa_asymptotic > 0.0) + // A b that is zero within its error gives a 1/b that is the noise in b, not an I/sigma. + if (result.error_model_isa_resolved) + Add(s, KeyReal("ISA", result.error_model_isa, "{:.2f}")); + else + Add(s, KeyText("ISA", "undetermined")); + if (result.error_model_isa_resolved && result.error_model_isa_asymptotic > 0.0) Add(s, KeyReal("ISA_ASYMPTOTIC", result.error_model_isa_asymptotic, "{:.2f}", true)); // Only when TRUE: FALSE is the ordinary case and says nothing. if (result.has_reference) @@ -1763,7 +1767,9 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, sig += fmt::format(" R_meas {:.1f} %", 100.0 * o.r_meas); if (std::isfinite(o.cc_half)) sig += fmt::format(" CC1/2 {:.3f}", o.cc_half); - if (result.error_model_isa > 0.0) + if (!result.error_model_isa_resolved) + sig += " ISa undetermined"; + else if (result.error_model_isa > 0.0) sig += fmt::format(" ISa {:.1f}", result.error_model_isa); row("Signal", sig); if (std::isfinite(o.cc_anom)) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 44a16b013..4d97df765 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -4916,6 +4916,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b /*measure_cc_before_corrections=*/true); result.error_model_isa = r.isa; result.error_model_isa_asymptotic = r.isa_asymptotic; + result.error_model_isa_resolved = r.isa_resolved; if (r.resolution_fit_A) result.resolution_fit_A = r.resolution_fit_A; result.error_model_a = r.error_model_a; result.error_model_b = r.error_model_b; @@ -6135,6 +6136,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // comparison merges are not the run's answer, so it goes back if they lose. const double em_isa = result.error_model_isa; const double em_isa_asymptotic = result.error_model_isa_asymptotic; + const bool em_isa_resolved = result.error_model_isa_resolved; const double em_a = result.error_model_a; const double em_b = result.error_model_b; const auto res_fit = result.resolution_fit_A; @@ -6201,6 +6203,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b experiment_.SetSpaceGroup(sg); result.error_model_isa = em_isa; result.error_model_isa_asymptotic = em_isa_asymptotic; + result.error_model_isa_resolved = em_isa_resolved; result.error_model_a = em_a; result.error_model_b = em_b; result.resolution_fit_A = res_fit; @@ -6697,9 +6700,10 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b && !superseded) { phase("Writing reflections"); const ErrorModelReport em_report{ - result.error_model_isa > 0 ? fmt::format("{:.2f}", result.error_model_isa) : "?", - result.error_model_isa_asymptotic > 0 ? fmt::format("{:.2f}", result.error_model_isa_asymptotic) - : std::string(), + result.error_model_isa > 0 && result.error_model_isa_resolved + ? fmt::format("{:.2f}", result.error_model_isa) : "?", + result.error_model_isa_asymptotic > 0 && result.error_model_isa_resolved + ? fmt::format("{:.2f}", result.error_model_isa_asymptotic) : std::string(), result.error_model_a > 0 ? fmt::format("{:.3f}", result.error_model_a) : std::string(), result.error_model_b > 0 ? fmt::format("{:.4e}", result.error_model_b) : std::string()}; WriteReflections(sm.merged, *result.consensus_cell, experiment_, sm.statistics, @@ -6776,6 +6780,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // multiplicity, so its fall-off is not this crystal's. const double em_isa = result.error_model_isa; const double em_isa_asymptotic = result.error_model_isa_asymptotic; + const bool em_isa_resolved = result.error_model_isa_resolved; const double em_a = result.error_model_a; const double em_b = result.error_model_b; const auto res_fit = result.resolution_fit_A; @@ -6825,6 +6830,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b experiment_.SetSpaceGroup(determined_group); result.error_model_isa = em_isa; result.error_model_isa_asymptotic = em_isa_asymptotic; + result.error_model_isa_resolved = em_isa_resolved; result.error_model_a = em_a; result.error_model_b = em_b; result.resolution_fit_A = res_fit; diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 2dc87f29f..d56907983 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -239,6 +239,7 @@ struct ProcessResult { MergeStatistics merge_statistics; double error_model_isa = 0.0; // whole-range 1/sqrt(a*b), XDS's meaning double error_model_isa_asymptotic = 0.0; // strong-reflection tier; rotation path only, 0 on stills + bool error_model_isa_resolved = true; // false: b is zero within its error, ISa printed "undetermined" double error_model_a = 0.0; // XDS convention: sigma^2 = a*(sigma0^2 + b*I^2) double error_model_b = 0.0; bool has_reference = false; diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index ecd6e07cc..6ffbc469d 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -1727,6 +1727,7 @@ static int RunRugnux(int argc, char **argv) { MergeStatistics merged_statistics; double error_model_isa = 0.0; double error_model_isa_asymptotic = 0.0; + bool error_model_isa_resolved = true; std::optional resolution_fit_A; // CC1/2 crossing, without the one-shell extension double error_model_a = 0.0; double error_model_b = 0.0; @@ -1755,6 +1756,7 @@ static int RunRugnux(int argc, char **argv) { merged_statistics = std::move(r.statistics); error_model_isa = r.isa; error_model_isa_asymptotic = r.isa_asymptotic; + error_model_isa_resolved = r.isa_resolved; error_model_a = r.error_model_a; error_model_b = r.error_model_b; resolution_fit_A = r.resolution_fit_A; @@ -1952,9 +1954,10 @@ static int RunRugnux(int argc, char **argv) { if (!output_prefix.empty()) { const ErrorModelReport em_report{ - error_model_isa > 0 ? fmt::format("{:.2f}", error_model_isa) : "?", - error_model_isa_asymptotic > 0 ? fmt::format("{:.2f}", error_model_isa_asymptotic) - : std::string(), + error_model_isa > 0 && error_model_isa_resolved + ? fmt::format("{:.2f}", error_model_isa) : "?", + error_model_isa_asymptotic > 0 && error_model_isa_resolved + ? fmt::format("{:.2f}", error_model_isa_asymptotic) : std::string(), error_model_a > 0 ? fmt::format("{:.3f}", error_model_a) : std::string(), error_model_b > 0 ? fmt::format("{:.4e}", error_model_b) : std::string()}; WriteReflections(merged_reflections, *experiment.GetUnitCell(), experiment, merged_statistics, @@ -1990,6 +1993,7 @@ static int RunRugnux(int argc, char **argv) { { std::ostringstream s; s << merged_statistics; scale_result.merge_statistics_text = s.str(); } scale_result.error_model_isa = error_model_isa; scale_result.error_model_isa_asymptotic = error_model_isa_asymptotic; + scale_result.error_model_isa_resolved = error_model_isa_resolved; scale_result.error_model_a = error_model_a; scale_result.error_model_b = error_model_b; scale_result.resolution_fit_A = resolution_fit_A; diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index 9a246edb8..594306c74 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -651,6 +651,30 @@ TEST_CASE("ResultReport_FittedResolutionSuppressed", "[Diagnostics]") { CHECK(climbs_text.find("RESOLUTION_FIT") != std::string::npos); } +// ISa of a b that is zero within its error is the noise in b; the report says undetermined, not a number. +TEST_CASE("ResultReport_IsaUndetermined", "[Diagnostics]") { + DiffractionExperiment x(DetJF(1)); + ProcessResult result; + result.consensus_cell = UnitCell{.a = 79.0f, .b = 79.0f, .c = 38.0f, + .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + result.has_merge_statistics = true; + result.merge_statistics.overall.d_max = 50.0f; + result.merge_statistics.overall.d_min = 1.50f; + result.merge_statistics.overall.cc_half = 0.99; + result.error_model_a = 1.1; + result.error_model_b = 1e-4; + result.error_model_isa = 95.0; + result.error_model_isa_asymptotic = 120.0; + + const auto resolved = RenderResultReport("p", "in.h5", x, result); + CHECK(resolved.find("\nISA= 95.00\n") != std::string::npos); + + result.error_model_isa_resolved = false; + const auto text = RenderResultReport("p", "in.h5", x, result); + CHECK(text.find("\nISA= undetermined\n") != std::string::npos); + CHECK(text.find("ISA_ASYMPTOTIC") == std::string::npos); +} + // The completeness of the range where the signal is, beside the completeness of the range that was // written. Users compare the single overall figure against another program's and conclude the run // lost data, when it is the deliberately generous cut diluting the denominator - so the report says From bcdaac3b14e1fdfab24168b5ed9b17f18c3a5085 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 09:59:58 +0200 Subject: [PATCH 09/28] rugnux: integrate a sparse rotation lattice the pooled spots accept A weak crystal inside a powder of its own microcrystals plus ice puts only ~5% of each frame's spots on its lattice. The pooled first-pass test accepted that lattice (5.5% of validation spots against 0.6% at a wrong spindle angle), but then: - every frame failed the 20% per-frame floor (LATTICE_MIN_INDEXED_FRACTION), so nothing was integrated and the merge was skipped (indexing rate 0); - the background-measured beam centre, ~3 px from the file's and correct, was not adopted because the arbiter counts validation frames, which are 0/60 at every centre. Changes: - When fewer than 1/6 of the validation frames clear the per-frame floor AND the sweep's pooled on-lattice fraction is itself below that floor, integrate every frame from the sweep's lattice (as XDS/DIALS do) and leave frame selection to scaling. Sweeps sparse only in spots per frame keep the floor. - Merge when a rotation lattice was found even if no frame "indexed" on its own. - When neither centre indexes a validation frame, adopt the measured centre if its pooled excess over chance beats the file's by 3.29 sigma. On the case above: P2 at the XDS cell, merged to 1.77 A, CC1/2 0.95, ISa 3.4 (XDS with the same lattice: ISa 3.3-5.5). Three healthy/partially-indexing rotation sets are bit-identical; a two-wavelength CBF set that currently merges 42 frames on a wrong cell moves (beam centre adopted, 755 frames). Co-Authored-By: Claude Opus 5 (1M context) --- image_analysis/IndexAndRefine.cpp | 2 +- image_analysis/IndexAndRefine.h | 4 ++ image_analysis/indexing/AnalyzeIndexing.cpp | 6 +- image_analysis/indexing/AnalyzeIndexing.h | 6 +- rugnux/Rugnux.cpp | 66 +++++++++++++++++++-- 5 files changed, 74 insertions(+), 10 deletions(-) diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index 39e42da2b..6d48a4b82 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -632,7 +632,7 @@ IndexAndRefine::DetermineRefineAnalyze(DataMessage &msg, const SpotFindingSettin // AnalyzeIndexing answers "is this frame worth integrating"; msg.indexing_result carries the // stricter "does this frame index on its own", which on rotation is not the same question. if (!AnalyzeIndexing(msg, outcome.experiment, *outcome.lattice_candidate, outcome.extra_lattice_candidates, - spots_on_lattice)) + spots_on_lattice, integrate_every_frame_)) return std::nullopt; { diff --git a/image_analysis/IndexAndRefine.h b/image_analysis/IndexAndRefine.h index ec901ea27..0fb0d1fdd 100644 --- a/image_analysis/IndexAndRefine.h +++ b/image_analysis/IndexAndRefine.h @@ -85,6 +85,8 @@ class IndexAndRefine { // the retained outcome instead - and it is a copy of every reflection of every image, so a caller // that writes no per-image file switches it off. bool keep_reflections_in_message_ = true; + // Integrate every rotation frame from the sweep's lattice, whatever fraction of its spots lie on it. + bool integrate_every_frame_ = false; std::vector scale_cc; std::vector > unit_cells; @@ -129,6 +131,8 @@ public: char GetPredictionCentring() const { return prediction_centring_; } // Whether the outgoing message keeps its own copy of the reflections - see the member. void KeepReflectionsInMessage(bool on) { keep_reflections_in_message_ = on; } + // See the member. + void IntegrateEveryFrame(bool on) { integrate_every_frame_ = on; } // Returns whether the frame indexed (a lattice was found and refined). Integration, when it runs, // is a further step gated on quick_integration. diff --git a/image_analysis/indexing/AnalyzeIndexing.cpp b/image_analysis/indexing/AnalyzeIndexing.cpp index 5b39c192e..0272e97c4 100644 --- a/image_analysis/indexing/AnalyzeIndexing.cpp +++ b/image_analysis/indexing/AnalyzeIndexing.cpp @@ -342,7 +342,8 @@ bool AnalyzeIndexing(DataMessage &message, const DiffractionExperiment &experiment, const CrystalLattice &latt, const std::vector &extra_lattices, - int64_t *spots_on_lattice) { + int64_t *spots_on_lattice, + bool integrate_every_frame) { auto start_time = std::chrono::steady_clock::now(); std::vector indexed_spots(message.spots.size()); @@ -441,7 +442,8 @@ bool AnalyzeIndexing(DataMessage &message, // why XDS's INTEGRATE and dials.integrate predict on every image of the sweep. A frame whose spots // largely MISS the lattice is a different matter and is still refused. const bool frame_indexes = lattice_fits && nspots_indexed >= viable_cell_min_spots; - const bool integrate_frame = experiment.IsRotationIndexing() ? lattice_fits : frame_indexes; + const bool integrate_frame = experiment.IsRotationIndexing() ? (lattice_fits || integrate_every_frame) + : frame_indexes; if (integrate_frame) { auto uc = latt.GetUnitCell(); if (ok(uc.a) && ok(uc.b) && ok(uc.c) && ok(uc.alpha) && ok(uc.beta) && ok(uc.gamma)) { diff --git a/image_analysis/indexing/AnalyzeIndexing.h b/image_analysis/indexing/AnalyzeIndexing.h index e20b05f6d..345df657f 100644 --- a/image_analysis/indexing/AnalyzeIndexing.h +++ b/image_analysis/indexing/AnalyzeIndexing.h @@ -46,6 +46,9 @@ constexpr float SPOT_BUDGET_SIGNIFICANCE_Z = 3.29f; // 2(1 - Phi(z)) = 0.001 // same rate all the way down, and the case a bare argmax gets wrong. int64_t SpotBudgetFromEvidence(const std::vector &indexed, const std::vector &counted); +// integrate_every_frame: on rotation, integrate the frame from the sweep's lattice even where fewer than +// LATTICE_MIN_INDEXED_FRACTION of its spots lie on it - for a sweep whose lattice was accepted on the pooled +// spots although its frames, one at a time, cannot clear that floor (see Rugnux). // spots_on_lattice, when given, receives how many of this frame's non-ice spots lie on latt - the // numerator of the frame gate below, reported whatever the gate then decides. The rotation first // pass's acceptance test pools that count over its validation frames, so it needs it from the sparse @@ -54,6 +57,7 @@ bool AnalyzeIndexing(DataMessage &message, const DiffractionExperiment &experiment, const CrystalLattice &latt, const std::vector &extra_lattices = {}, - int64_t *spots_on_lattice = nullptr); + int64_t *spots_on_lattice = nullptr, + bool integrate_every_frame = false); diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 4d97df765..4e6d247b0 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -3589,12 +3589,50 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // around the value the run started with rather than around a fit that has just // been tested and found no better - and so a run that fails for some other // reason fails at the geometry it was given. + // + // Unless the frame counts are both at the floor and the POOLED spots do decide: + // a crystal whose spots are a minority of every frame indexes 0/60 at any centre, + // and then only the pooled excess over chance - each lattice measured against its + // own wrong-spindle null, so a larger cell is not credited for fitting more by + // chance - says which centre sees it. + PooledEvidence at_measured, at_header; + if (alt.result.has_value()) + at_measured = pooled_evidence(*indexer, *alt.result); restore_beam_center(header_x, header_y); - logger.Info("Beam centre check: neither the file's centre ({}/{} frames) nor " - "the measured one {:.2f} px away ({}/{}) indexes this run - the " - "file's centre is kept and this is not a beam-centre failure", - best.score, static_cast(validation.size()), moved, alt.score, - static_cast(validation.size())); + if (best.result.has_value()) + at_header = pooled_evidence(*indexer, *best.result); + auto excess = [](const PooledEvidence &e) { + return static_cast(e.on_lattice - e.by_chance) + / static_cast(std::max(1, e.spots)); + }; + auto rate_var = [](const PooledEvidence &e) { + const double n = static_cast(std::max(1, e.spots)); + const double p = static_cast(e.on_lattice) / n; + return p * (1.0 - p) / n; + }; + const double margin = excess(at_measured) - excess(at_header); + const double margin_sigma = std::sqrt(rate_var(at_measured) + rate_var(at_header)); + if (alt.result.has_value() && beats_chance(at_measured) + && margin > SPOT_BUDGET_SIGNIFICANCE_Z * margin_sigma) { + try_beam_center(measured_x, measured_y); + logger.Warning("Beam centre from the background: neither centre indexes a " + "validation frame on its own, but the pooled spots put " + "{:.1f}% on the lattice at the measured ({:.2f},{:.2f}) against " + "{:.1f}% at the file's (chance {:.1f}% and {:.1f}%) - the run " + "adopts the measured centre", + 100.0 * at_measured.on_lattice / std::max(1, at_measured.spots), + measured_x, measured_y, + 100.0 * at_header.on_lattice / std::max(1, at_header.spots), + 100.0 * at_measured.by_chance / std::max(1, at_measured.spots), + 100.0 * at_header.by_chance / std::max(1, at_header.spots)); + best = alt; + } else { + logger.Info("Beam centre check: neither the file's centre ({}/{} frames) nor " + "the measured one {:.2f} px away ({}/{}) indexes this run - the " + "file's centre is kept and this is not a beam-centre failure", + best.score, static_cast(validation.size()), moved, alt.score, + static_cast(validation.size())); + } } } else { restore_beam_center(header_x, header_y); @@ -4196,6 +4234,22 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b } } indexer->ForceRotationIndexerResult(*best.result); + // A lattice the pooled spots accept while hardly a validation frame clears the per-frame + // floor (LATTICE_MIN_INDEXED_FRACTION) on its own, because the sweep as a whole puts fewer of + // its spots on the lattice than that floor asks of each frame: the crystal is real but its + // spots are a minority of every frame - a weak crystal inside a powder of its own + // microcrystals and ice. The floor cannot tell its frames from anything else, so integrate + // every frame from the sweep's lattice, as XDS and DIALS do, and leave the frames to the + // scaling. A sweep that is sparse only in spots per frame (a small molecule) keeps the floor. + const bool sparse_lattice = best.score < static_cast(validation.size()) / 6 + && pooled < LATTICE_MIN_INDEXED_FRACTION; + indexer->IntegrateEveryFrame(sparse_lattice); + if (sparse_lattice) + logger.Warning("Sparse lattice: {}/{} validation frames clear the per-frame floor, but the " + "pooled spots put {:.1f}% on the lattice against {:.1f}% by chance - every " + "frame is integrated from the sweep's lattice", + best.score, static_cast(validation.size()), 100.0 * pooled, + 100.0 * pooled_chance); logger.Info("First-pass spot finding: {} frames in {} batches, {:.2f} s", spot_cache.size(), prefetch_batches, prefetch_time_s); logger.Info("Two-pass rotation indexing found lattice (scheme '{}': {}/{} validation frames, " @@ -4638,7 +4692,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b } // Scaling and merging (full analysis only). - if (full && !cancelled_ && result.indexing_rate.has_value() && result.indexing_rate > 0 + if (full && !cancelled_ && (result.indexing_rate.value_or(0.0f) > 0 || result.rotation_lattice_found) && (config_.run_scaling || !config_.reference_data.empty())) { // Scaling/merging is a long post-pass; report each sub-step as a phase so the GUI progress // bar reflects what is happening instead of freezing on one label. From 0a7308c6049f584019cdcb6a5c2b75a4801222c4 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 11:32:01 +0200 Subject: [PATCH 10/28] rugnux: the device merge before the correction surfaces sees the collapsed-scale drop DropCollapsedFullScales zeroes the corr of fulls whose frame scale collapsed on the host copy of the fulls; on the resident-fulls (GPU) path that zero reached the device only with the corrected corr pushed after the surfaces. The merge that measures CC1/2 before the correction surfaces runs in between, so on the GPU it still merged the dropped frames' fulls while the host path excluded them - the two builds measured different observations. The corr is now pushed to the device right after the drop, and the later push covers only the surfaces and the frame rejection. This changes the input of the two-pass quality guard (cc_half_before_corrections) on GPU runs where a full scale collapsed; the final merge is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../scale_merge/RotationScaleMerge.cpp | 35 ++++++++++++------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index bfde3cd48..d3597ca03 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -4841,9 +4841,8 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st } // The total scale each frame's fulls were amplified by - partial scale, incident flux and the fulls' // own G (corr = 1/G here) - taken BEFORE the correction surfaces fold per-observation terms into - // corr, and before a collapsed frame's fulls are zeroed: the resident fulls only get that zero with - // the corrected corr, so a merge on the device before then still sees them, and must weigh them by - // their own scale. Read by MergeAndStats for the CC1/2 weight. + // corr, and before a collapsed frame's fulls are zeroed (a zeroed corr says nothing about the + // frame's scale). Read by MergeAndStats for the CC1/2 weight. frame_scale_fulls.assign(n_frames, NAN); for (const auto &o : fulls) if (std::isfinite(o.corr) && o.corr > 0.0f && !std::isfinite(frame_scale_fulls[o.frame])) { @@ -4852,6 +4851,22 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st } const bool rejected_full_scales = scale_fulls && DropCollapsedFullScales(combined_on_gpu && scaled_fulls_on_gpu); +#ifdef JFJOCH_USE_CUDA + // The host fulls' corr is the truth from here on; a merge on the resident (GPU) fulls reads the + // device copy, so every change to it is pushed back before the next device merge. + auto upload_fulls_corr = [&]() { + std::vector fcorr(fulls.size()); + ParallelChunks(static_cast(fulls.size()), ThreadsForWork(fulls.size(), nthreads), + [&](int lo, int hi) { + for (int i = lo; i < hi; ++i) fcorr[i] = fulls[i].corr; + }); + gpu_->SetFullsCorr(fcorr.data()); + }; + // Right away, not with the corrections below: the merge just before the surfaces must see the + // collapsed frames' fulls gone on the device exactly as the host merge does. + if (rejected_full_scales && combined_on_gpu && scaled_fulls_on_gpu) + upload_fulls_corr(); +#endif // --- 4b. Optional correction surfaces (decay = resolution x time; absorption = goniometer-frame // diffracted-beam direction), each an alternating multiplicative fit of the fulls' corr against @@ -4917,16 +4932,10 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st CountSweepDisposition(); } #ifdef JFJOCH_USE_CUDA - // The corrections mutate the host fulls' corr; when the merge runs on the resident (GPU) fulls, push - // the corrected corr back to the device so the merge reads it. - if ((corrections || rejected_full_scales || rejected_frames) && combined_on_gpu && scaled_fulls_on_gpu) { - std::vector fcorr(fulls.size()); - ParallelChunks(static_cast(fulls.size()), ThreadsForWork(fulls.size(), nthreads), - [&](int lo, int hi) { - for (int i = lo; i < hi; ++i) fcorr[i] = fulls[i].corr; - }); - gpu_->SetFullsCorr(fcorr.data()); - } + // The corrections and the frame rejection mutate the host fulls' corr; push it to the device so the + // merge reads it. (The collapsed-scale drop was pushed where it happened.) + if ((corrections || rejected_frames) && combined_on_gpu && scaled_fulls_on_gpu) + upload_fulls_corr(); #endif // --- 5. Error model + merge + statistics. --- From c6f97460dda8214ea63dae40cc0aa7c6a079bfe0 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 11:37:55 +0200 Subject: [PATCH 11/28] PostRefine: report the realised held-out residual, its noise and the starting cell Each post-refinement now records, beside what it commits, what the integration it was handed realises at the geometry it started from: the held-out residual at nominal (the same value the log prints on the left of "held-out ... ->"), its standard error over the held-out residual values, and the cell the pass's own indexing refined. Nothing here changes a decision. Two helpers read them: HeldOutResidualFell (a pass at a new geometry realises a lower residual than another by more than the standard error of the difference) and ReindexPushesCellBack (re-indexing at a committed geometry returns, on the length the fit moved most, a cell on the side the fit moved away from). The geometry walk in RunAllPasses uses both in the next commit. Co-Authored-By: Claude Opus 5 (1M context) --- image_analysis/geom_refinement/PostRefine.cpp | 57 +++++++++++++++++-- image_analysis/geom_refinement/PostRefine.h | 23 ++++++++ tests/CMakeLists.txt | 1 + tests/PostRefineTest.cpp | 53 +++++++++++++++++ 4 files changed, 129 insertions(+), 5 deletions(-) create mode 100644 tests/PostRefineTest.cpp diff --git a/image_analysis/geom_refinement/PostRefine.cpp b/image_analysis/geom_refinement/PostRefine.cpp index 82a8d4027..65a6e905c 100644 --- a/image_analysis/geom_refinement/PostRefine.cpp +++ b/image_analysis/geom_refinement/PostRefine.cpp @@ -497,10 +497,12 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, // Cost of a whole geometry over one subset, the two residual families kept apart so the log // can say which of them moved. Each is a mean per residual VALUE, and the number they are // summed over below is the same weighting the solver itself applies. + // se_out, where asked for: the standard error of the returned mean over the residual values + // it averages. auto joint_cost = [&](Subset s, const double bm[2], const double ds[1], const double rv[3], const double q0[3], const double q1[3], const double q2[3], - double &pos_out, double &exc_out) { - double cp = 0.0, ce = 0.0; + double &pos_out, double &exc_out, double *se_out = nullptr) { + double cp = 0.0, ce = 0.0, c4 = 0.0; size_t np = 0, ne = 0; for (size_t oi = 0; oi < n_obs; ++oi) { const Partial *pp = obs[oi]; @@ -509,7 +511,10 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, angle_rad(pp->img), pp->h, pp->k, pp->l, sys, orientation); double res[3] = {0, 0, 0}; r(bm, ds, det_rot, rv, q0, q1, q2, res); - cp += res[0] * res[0] + res[1] * res[1] + res[2] * res[2]; + for (int j = 0; j < 3; ++j) { + cp += res[j] * res[j]; + c4 += res[j] * res[j] * res[j] * res[j]; + } np += 3; } for (const int32_t i : selected) { @@ -520,11 +525,17 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, double res = 0.0; r(rv, q0, q1, q2, &res); ce += res * res; + c4 += res * res * res * res; ++ne; } pos_out = np ? cp / np : 0.0; exc_out = ne ? ce / ne : 0.0; - return (np + ne) ? (cp + ce) / static_cast(np + ne) : 0.0; + const double n = static_cast(np + ne); + const double mean = (np + ne) ? (cp + ce) / n : 0.0; + if (se_out) + *se_out = (np + ne) > 1 ? std::sqrt(std::max(0.0, c4 / n - mean * mean) / (n - 1.0)) + : std::numeric_limits::quiet_NaN(); + return mean; }; // One problem, both residual families, every block seeded at the geometry it is handed - @@ -596,7 +607,11 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, bool commit = false; if (n_obs >= static_cast(settings.min_events)) { double pos_nom = 0.0, exc_nom = 0.0, pos_ref = 0.0, exc_ref = 0.0; - const double cv_nom = joint_cost(VAL, beam, dist, axv, p0, p1, p2, pos_nom, exc_nom); + double cv_nom_se = 0.0; + const double cv_nom = joint_cost(VAL, beam, dist, axv, p0, p1, p2, pos_nom, exc_nom, + &cv_nom_se); + result.held_out_before = cv_nom; + result.held_out_before_se = cv_nom_se; // How far a geometry has moved from nominal, in units of one step. const auto beyond_a_step = [&](const double ds[1], const double q1c[3]) { @@ -747,6 +762,11 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, double len_nom[3], ang_nom[3], len_fit[3], ang_fit[3]; EffectiveCellFromParams(sys, p1_0, p2_0, len_nom, ang_nom); EffectiveCellFromParams(sys, q1_log, q2_log, len_fit, ang_fit); + result.cell_before = UnitCell{static_cast(len_nom[0]), static_cast(len_nom[1]), + static_cast(len_nom[2]), + static_cast(ang_nom[0] * 180.0 / PI), + static_cast(ang_nom[1] * 180.0 / PI), + static_cast(ang_nom[2] * 180.0 / PI)}; logger.Info("Post-refine GEOM (joint crystal + detector): dist {:.3f} -> {:.3f} mm, beam " "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), cell {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} " "{:.2f} -> {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} {:.2f}, held-out positional " @@ -1016,3 +1036,30 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, } } + +bool HeldOutResidualFell(const PostRefineResult &before, const PostRefineResult &after) { + const double noise = std::hypot(before.held_out_before_se, after.held_out_before_se); + return std::isfinite(noise) && after.held_out_before < before.held_out_before - noise; +} + +int ReindexPushesCellBack(const PostRefineResult &fit, const PostRefineResult &next) { + if (!fit.cell_refined) + return 0; + const double from[3] = {fit.cell_before.a, fit.cell_before.b, fit.cell_before.c}; + const double to[3] = {fit.cell.a, fit.cell.b, fit.cell.c}; + const double again[3] = {next.cell_before.a, next.cell_before.b, next.cell_before.c}; + int axis = -1; + double largest = 0.0; + for (int j = 0; j < 3; ++j) + if (from[j] > 0.0 && std::fabs(to[j] - from[j]) / from[j] > largest) { + largest = std::fabs(to[j] - from[j]) / from[j]; + axis = j; + } + if (axis < 0 || !(again[axis] > 0.0)) + return 0; + const double move = to[axis] - from[axis]; + const double back = again[axis] - to[axis]; + if (!(move * back < 0.0)) + return 0; + return move > 0.0 ? axis + 1 : -(axis + 1); +} diff --git a/image_analysis/geom_refinement/PostRefine.h b/image_analysis/geom_refinement/PostRefine.h index e1f3b7101..b6f20833c 100644 --- a/image_analysis/geom_refinement/PostRefine.h +++ b/image_analysis/geom_refinement/PostRefine.h @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -48,6 +49,17 @@ struct PostRefineResult { // from. Such a move is not refused for its size - it is RATIFIED by re-integrating and re-indexing // at it, which a second lattice does not survive and a wrong header does (see RunAllPasses). bool large_move = false; + // The held-out residual at the geometry the fit STARTED from - what the integration this fit was + // handed actually realises there, before the fit moves anything - and its standard error over the + // held-out values. Two passes at two geometries are compared on this, not on the fit's own + // prediction of where it is going (see the geometry walk in RunAllPasses). NaN where the fit had + // too few observations to measure it. + double held_out_before = std::numeric_limits::quiet_NaN(); + double held_out_before_se = std::numeric_limits::quiet_NaN(); + // The cell the fit started from: the one this pass's own indexing refined. Beside `cell` (what the + // fit committed), and against the next pass's cell_before, it says whether re-indexing at the + // committed geometry ratifies the committed cell. + UnitCell cell_before{}; // Which test refused the fit and what it wanted, where it was refused; empty where it committed. // Nothing else says it - the geometry that comes out of a refusal is the nominal one whatever // refused it, so a silent refusal cannot be told from a fit that never wanted to move. @@ -119,3 +131,14 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations obs, const CrystalLattice &reference_latt, const PostRefineSettings &settings, Logger &logger); + +// Whether the held-out residual `after` realises at its geometry is below the one `before` realises at +// its own by more than the standard error of the difference - the cross-validation noise of the two +// measurements. False where either was not measured. +bool HeldOutResidualFell(const PostRefineResult &before, const PostRefineResult &after); + +// Whether re-indexing at the geometry `fit` committed pushed the committed cell back: on the cell length +// the fit moved most, the cell `next` (the fit of the pass run at that geometry) started from lies on the +// side the fit moved away from. Returns +-(axis + 1), signed as the fit's move, where it did; 0 where the +// re-indexing followed the fit, or the fit committed nothing. +int ReindexPushesCellBack(const PostRefineResult &fit, const PostRefineResult &next); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 45510abb0..1b9ef6040 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -54,6 +54,7 @@ ADD_EXECUTABLE(jfjoch_test ParallelForTest.cpp RugnuxTest.cpp ResultReportTest.cpp + PostRefineTest.cpp RugnuxLargeTest.cpp TestData.h MovingAverageTest.cpp diff --git a/tests/PostRefineTest.cpp b/tests/PostRefineTest.cpp new file mode 100644 index 000000000..eec9cb593 --- /dev/null +++ b/tests/PostRefineTest.cpp @@ -0,0 +1,53 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include + +#include "../image_analysis/geom_refinement/PostRefine.h" + +namespace { + PostRefineResult Measured(double residual, double se) { + PostRefineResult r; + r.held_out_before = residual; + r.held_out_before_se = se; + return r; + } + + PostRefineResult Fit(UnitCell before, UnitCell after) { + PostRefineResult r; + r.cell_refined = true; + r.cell_before = before; + r.cell = after; + return r; + } + + UnitCell Cell(float a, float b, float c) { + return UnitCell{.a = a, .b = b, .c = c, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + } +} + +TEST_CASE("PostRefine_HeldOutResidualFell", "[PostRefine]") { + // Down by more than the standard error of the difference, sqrt(3^2 + 4^2) = 5. + CHECK(HeldOutResidualFell(Measured(100.0, 3.0), Measured(94.0, 4.0))); + // Down, but within that noise. + CHECK_FALSE(HeldOutResidualFell(Measured(100.0, 3.0), Measured(96.0, 4.0))); + // Up. + CHECK_FALSE(HeldOutResidualFell(Measured(100.0, 3.0), Measured(110.0, 4.0))); + // Not measured on either side. + CHECK_FALSE(HeldOutResidualFell(PostRefineResult{}, Measured(10.0, 1.0))); + CHECK_FALSE(HeldOutResidualFell(Measured(100.0, 3.0), PostRefineResult{})); +} + +TEST_CASE("PostRefine_ReindexPushesCellBack", "[PostRefine]") { + // The fit shortens b the most; re-indexing at its geometry returns a longer b: pushed back. + const auto fit = Fit(Cell(96.9f, 107.9f, 112.9f), Cell(96.8f, 106.6f, 112.9f)); + CHECK(ReindexPushesCellBack(fit, Fit(Cell(96.7f, 107.6f, 112.6f), Cell(96.6f, 106.4f, 112.8f))) == -2); + // A walk that re-indexing follows (and overshoots a little) is not pushed back. + const auto walk = Fit(Cell(60.48f, 60.48f, 196.7f), Cell(60.05f, 60.05f, 195.2f)); + CHECK(ReindexPushesCellBack(walk, Fit(Cell(60.00f, 60.00f, 195.1f), Cell(59.8f, 59.8f, 194.2f))) == 0); + // Nothing committed, or no cell measured by the next pass: no evidence either way. + PostRefineResult refused = fit; + refused.cell_refined = false; + CHECK(ReindexPushesCellBack(refused, Fit(Cell(96.7f, 107.6f, 112.6f), Cell(96.6f, 106.4f, 112.8f))) == 0); + CHECK(ReindexPushesCellBack(fit, PostRefineResult{}) == 0); +} From 848de4b7dc1ac629cc8e5d8a7cc09aa619948f33 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 11:37:55 +0200 Subject: [PATCH 12/28] rugnux: the geometry walk continues only while re-integration realises a gain The rotation geometry walk kept re-integrating while the post-refinement at the adopted geometry still committed a move, up to four rounds. It never ended on its own: every walking run hit the cap, and with the cap lifted to twelve every one hit twelve. Each round's fit predicted a held-out gain the re-integration at its geometry then did not realise - the realised residual stayed flat from the first round, or passed a minimum and rose again while the distance crept on along the distance/cell-scale degeneracy - so the cap decided where on the drift the run stopped. A round is now kept only when the held-out residual measured by the pass re-integrated at its geometry falls below the best round's by more than the standard error of the difference. The first round that does not ends the walk, and the run goes back to the best round (one more pass, since the canonical files were written by the last). The walk also stops when re-indexing pushes the fitted cell back on the same length in the same direction on two rounds running: the fit keeps asking for a cell nothing ratifies. The cap is 8; a walk still paying at the cap keeps its last (= best) round, sets geometry_not_converged, and the report flags GEOMETRY_NOT_CONVERGED. Either way the kept round is judged against the header-geometry pass by RefinedPassIsWorse, as before. PASS_DECISION lists distance and realised residual per round and why the walk ended. Runs whose post-refinement does not commit a move of more than one step do not walk and are unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- docs/RUGNUX_REPORT.md | 2 +- rugnux/ReportDocument.h | 1 + rugnux/ResultReport.cpp | 5 ++ rugnux/Rugnux.cpp | 133 +++++++++++++++++++++++++++++-------- rugnux/Rugnux.h | 3 + tests/ResultReportTest.cpp | 19 ++++++ 6 files changed, 133 insertions(+), 30 deletions(-) diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 264bf1cb9..325be8583 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -139,7 +139,7 @@ and a run that produced a textbook data set read identically for their first thr `SYMMETRY_AMBIGUITY`, `CENTERING_UNTESTED`, `UNUSABLE_MERGE`, `LOW_COMPLETENESS`, `SWEEP_GAPS`, `GONIO_SCALE`, `SPINDLE_CAP`, `ANISOTROPY`, `TWINNING`, `PSEUDO_TRANSLATION`, `LATTICE_TRANSLATION`, `MODEL_HAND`, `MODEL_NOT_VALIDATED`, `CANCELLED`, `RESOLUTION_FIT`, - `FLIGHT_PATH`. `NONE` when nothing fired. A code appears if and only if its + `FLIGHT_PATH`, `GEOMETRY_NOT_CONVERGED`. `NONE` when nothing fired. A code appears if and only if its warning fired, so the flags and the `WARNING:` lines are two renderings of one list — the closed type for machinery, the open sentence for a person. - **`WARNING_COUNT=`** and the **`WARNING:`** lines follow, in the same section. They are what they diff --git a/rugnux/ReportDocument.h b/rugnux/ReportDocument.h index 48b42c917..bc587c577 100644 --- a/rugnux/ReportDocument.h +++ b/rugnux/ReportDocument.h @@ -43,6 +43,7 @@ namespace PathologyCode { // in how exact it is. ANISOTROPY and RESOLUTION_FIT are already shared this way. constexpr const char *LATTICE_TRANSLATION = "LATTICE_TRANSLATION"; // a translation the cell does not declare constexpr const char *HARMONIC_CONTAMINATION = "HARMONIC_CONTAMINATION"; // the beam carries a higher harmonic + constexpr const char *GEOMETRY_NOT_CONVERGED = "GEOMETRY_NOT_CONVERGED"; // the geometry walk ran out of rounds } // A closed type plus an open description. The type is what machinery switches on; the text is what a diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index cbb2ae8ba..0a976d95d 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -385,6 +385,11 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, if (result.pass_count > 1) { Add(s, KeyText("PASS", fmt::format("{} of {}", result.pass_number, result.pass_count))); Add(s, KeyText("PASS_DECISION", result.pass_decision, true)); + if (result.geometry_not_converged) + Warn(doc, PathologyCode::GEOMETRY_NOT_CONVERGED, + "the detector geometry post-refinement was still moving when the run stopped walking " + "it: every round lowered the held-out residual, so the geometry this result was " + "integrated at is the best one reached, not a settled one (see PASS_DECISION)"); } Add(s, Blank()); diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 4e6d247b0..a6da5bbb2 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -1987,50 +1987,124 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { // What starts the walk is a move of more than one step: that is the fit saying the geometry is // somewhere else entirely rather than a fraction of a percent away, and it leaves every run // whose fit settles beside its header at the two passes it always had. Once the run IS walking - // it keeps walking while the fit still commits, because the steps do not get smaller in - // proportion to what is left: measured on a header 3.9 % long, the first round took a third of - // the error and each of the next took a third of the rest. The walk is bounded rather than - // stopped at a size, because there is no size that separates the last step of a real walk from - // the first step of a runaway - and it does not run away: started BELOW the optimum the same fit - // turns back up, so it settles around the answer instead of past it. - constexpr int MAX_GEOMETRY_ROUNDS = 4; - std::optional> adopted_geometry; - int geometry_rounds = 0; - while (!cancelled_ && geometry_rounds < MAX_GEOMETRY_ROUNDS && prepass_detector_geometry_ - && pass2.post_refine && pass2.post_refine->detector_refined - && (pass2.post_refine->large_move || geometry_rounds > 0)) { - adopted_geometry = *prepass_detector_geometry_; - const auto &g = *adopted_geometry; - logger.Info("Two-pass: the post-refinement at the adopted geometry still moves it " - "(distance {:.3f} -> {:.3f} mm) - re-integrating and re-indexing at what it " - "asks for (round {} of at most {})", pass2.post_refine->distance_before_mm, - pass2.post_refine->distance_after_mm, geometry_rounds + 1, - MAX_GEOMETRY_ROUNDS); + // it keeps walking because the steps do not get smaller in proportion to what is left: measured + // on a header 3.9 % long, the first round took a third of the error and each of the next took a + // third of the rest. + // + // But a round is kept only for what it REALISES, not for what the fit predicts: the held-out + // residual of the pass re-integrated and re-indexed at the new geometry has to fall below the + // best round's by more than the cross-validation noise of the two (HeldOutResidualFell). With + // the walk allowed twelve rounds, every round's fit predicted a gain the re-integration did not + // show - the realised residual stayed flat, or passed a minimum and rose again, while the + // distance crept on along the distance/cell-scale degeneracy - so "the fit still commits" never + // ended a walk, and the cap decided where on the drift the run stopped. A round that does not + // pay ends the walk and the run goes back to the best round it reached. So does a fit whose + // committed cell the re-indexing pushes back the same way twice running (ReindexPushesCellBack): + // the fit keeps asking for a cell the indexer at the fit's own geometry does not return, and + // nothing ratifies it. A walk still paying when the rounds run out keeps its best round - the + // last - and says it did not converge; the quality guard below judges it against the header + // pass as it judges every refined pass. + constexpr int MAX_GEOMETRY_ROUNDS = 8; + const auto current_geometry = [&] { + return std::array{experiment_.GetBeamX_pxl(), experiment_.GetBeamY_pxl(), + experiment_.GetDetectorDistance_mm(), + experiment_.GetPoniRot1_rad(), experiment_.GetPoniRot2_rad()}; + }; + const auto set_geometry = [&](const std::array &g) { experiment_.BeamX_pxl(g[0]).BeamY_pxl(g[1]).DetectorDistance_mm(g[2]) .PoniRot1_rad(g[3]).PoniRot2_rad(g[4]); + }; + int geometry_rounds = 0; // passes run at a geometry the walk moved to + int walk_passes = 0; // every pass the walk ran, the return to its best round included + int best_round = 0; + std::array best_geometry = current_geometry(); + PostRefineResult best_fit; // the fit measured in the best round's pass: its realised residual + int last_pushback = 0; + bool geometry_not_converged = false; + std::string walk_stop; // why the walk ended, where anything but the fit settling ended it + std::string walk_rounds; // distance / realised residual per round, for the pass decision + if (pass2.post_refine) { + best_fit = *pass2.post_refine; + walk_rounds = fmt::format("{:.3f} mm / {:.3e}", best_geometry[2], best_fit.held_out_before); + } + while (!cancelled_ && prepass_detector_geometry_ + && pass2.post_refine && pass2.post_refine->detector_refined + && (pass2.post_refine->large_move || geometry_rounds > 0)) { + if (geometry_rounds == MAX_GEOMETRY_ROUNDS) { + geometry_not_converged = true; + walk_stop = fmt::format("the fit still moved the geometry after {} rounds, every one of " + "which lowered the realised residual", MAX_GEOMETRY_ROUNDS); + logger.Warning("Two-pass: {} - the walk did not converge; the run keeps its last round " + "and the quality guard judges it against the header pass", walk_stop); + break; + } + const PostRefineResult fit = *pass2.post_refine; + const std::array g = *prepass_detector_geometry_; + logger.Info("Two-pass: the post-refinement at the adopted geometry still moves it " + "(distance {:.3f} -> {:.3f} mm) - re-integrating and re-indexing at what it " + "asks for (round {} of at most {})", fit.distance_before_mm, + fit.distance_after_mm, geometry_rounds + 1, MAX_GEOMETRY_ROUNDS); + set_geometry(g); pass2 = RunPipeline(observer, /*write_output=*/true, /*geometry_prepass=*/false); ++geometry_rounds; + ++walk_passes; + const double realised = pass2.post_refine ? pass2.post_refine->held_out_before : NAN; + walk_rounds += fmt::format(", {:.3f} mm / {:.3e}", g[2], realised); + if (!pass2.post_refine || !HeldOutResidualFell(best_fit, *pass2.post_refine)) { + walk_stop = fmt::format( + "round {} did not lower the realised held-out residual by more than its noise " + "({:.3e} against {:.3e}, standard errors {:.1e} and {:.1e})", geometry_rounds, + realised, best_fit.held_out_before, + pass2.post_refine ? pass2.post_refine->held_out_before_se : NAN, + best_fit.held_out_before_se); + logger.Info("Two-pass: {}", walk_stop); + break; + } + best_round = geometry_rounds; + best_geometry = g; + best_fit = *pass2.post_refine; + const int pushback = ReindexPushesCellBack(fit, *pass2.post_refine); + if (pushback != 0 && pushback == last_pushback) { + walk_stop = fmt::format( + "re-indexing pushed the cell length {} back against the fit for the second " + "round running, so nothing ratifies the cell the fit asks for", + "abc"[std::abs(pushback) - 1]); + logger.Info("Two-pass: {} - the walk stops at round {}", walk_stop, geometry_rounds); + break; + } + last_pushback = pushback; } postrefine_probe_ = false; + // The canonical files were written by the last round; where that is not the best one, the best + // one is run again to write them. + if (!cancelled_ && geometry_rounds > best_round) { + logger.Info("Two-pass: going back to round {} of the geometry walk (distance {:.3f} mm), the " + "last one that lowered the realised residual", best_round, best_geometry[2]); + set_geometry(best_geometry); + pass2 = RunPipeline(observer, /*write_output=*/true, /*geometry_prepass=*/false); + ++walk_passes; + } // The post-refinement is measured by pass 1 and consumed by pass 2, so carry it onto whichever // result is returned - it is a result of the run, not of the pass that happened to fit it. Where // the walk took rounds they are ONE geometry decision, so what the report names is where the // geometry started and where the run left it, not the first step of the way there. - if (geometry_rounds > 0 && pass1.post_refine && adopted_geometry) { - pass1.post_refine->beam_x_after_px = (*adopted_geometry)[0]; - pass1.post_refine->beam_y_after_px = (*adopted_geometry)[1]; - pass1.post_refine->distance_after_mm = (*adopted_geometry)[2]; + if (best_round > 0 && pass1.post_refine) { + pass1.post_refine->beam_x_after_px = best_geometry[0]; + pass1.post_refine->beam_y_after_px = best_geometry[1]; + pass1.post_refine->distance_after_mm = best_geometry[2]; } pass2.post_refine = pass1.post_refine; - pass2.pass_number = 2 + geometry_rounds; - pass2.pass_count = 2 + geometry_rounds; + pass2.pass_number = 2 + walk_passes; + pass2.pass_count = 2 + walk_passes; if (geometry_rounds > 0) pass2.pass_decision = fmt::format( - "post-refined geometry adopted over {} rounds: the fit was still walking at each of " - "them, and the detector distance ended at {:.3f} mm against the header's {:.3f}", - geometry_rounds + 1, (*adopted_geometry)[2], - pass1.post_refine ? pass1.post_refine->distance_before_mm : 0.0); + "post-refined geometry walked {} round{} and kept round {}, detector distance {:.3f} mm " + "against the header's {:.3f}: {}; distance / realised held-out residual by round: {}", + geometry_rounds, geometry_rounds == 1 ? "" : "s", best_round, best_geometry[2], + pass1.post_refine ? pass1.post_refine->distance_before_mm : 0.0, + walk_stop.empty() ? "the fit at the kept geometry commits no further move" : walk_stop, + walk_rounds); // r2 = r1 + 2 is also the inner edge of the background ring, so widening the signal disk pushes // that ring out into the neighbours; a reflection whose ring is left with five or fewer clean @@ -2157,6 +2231,7 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { pass2.pass_decision = (prepass_detector_geometry_ || prepass_rotation_scale_) ? "post-refined geometry adopted" : "the post-refinement committed no geometry change, so this pass reproduces the first"; + pass2.geometry_not_converged = geometry_not_converged; return pass2; } return RunPipeline(observer, /*write_output=*/true, /*geometry_prepass=*/false); diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index d56907983..91312251a 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -361,6 +361,9 @@ struct ProcessResult { int pass_number = 1; int pass_count = 1; std::string pass_decision; + // The rotation geometry walk used every round it was given and its fit still asked for a move: the + // geometry this result was integrated at is the best round the walk reached, not a settled one. + bool geometry_not_converged = false; // The radiation-damage report (rotation), the same text the log prints. Empty when not measured. std::string radiation_damage_text; diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index 594306c74..12d61a22b 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -793,3 +793,22 @@ TEST_CASE("ResultReport_TranslationalNCS", "[Diagnostics]") { CHECK(lat.find("\nPATHOLOGY_FLAGS= LATTICE_TRANSLATION\n") != std::string::npos); CHECK(lat.find("\nTNCS_SUBLATTICE=") == std::string::npos); } + +TEST_CASE("ResultReport_GeometryNotConverged", "[Diagnostics]") { + DiffractionExperiment x(DetJF(1)); + ProcessResult result; + result.consensus_cell = UnitCell{.a = 79.0f, .b = 79.0f, .c = 38.0f, + .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + result.pass_number = 10; + result.pass_count = 10; + result.pass_decision = "post-refined geometry walked 8 rounds and kept round 8"; + + CHECK(RenderResultReport("p", "in.h5", x, result).find("GEOMETRY_NOT_CONVERGED") == std::string::npos); + + result.geometry_not_converged = true; + const auto text = RenderResultReport("p", "in.h5", x, result); + const auto flags = text.find("\nPATHOLOGY_FLAGS= "); + REQUIRE(flags != std::string::npos); + CHECK(text.substr(flags, text.find('\n', flags + 1) - flags).find("GEOMETRY_NOT_CONVERGED") + != std::string::npos); +} From e6adf8d65013a81b7246b357f19282edc61436af Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 13:02:41 +0200 Subject: [PATCH 13/28] rugnux --model: model structure factors with FFTW instead of pocketfft Every map -> structure-factor transform on the model path - Fcalc from the model density and Fmask from the bulk-solvent mask, in the fit, the rigid-body evaluations (two per evaluation, ~120 evaluations per placement), the frame probe and the stills model reference - went through gemmi's vendored pocketfft, single-threaded. They now go through MapToFPhi (rugnux/ModelFFT.{h,cpp}): an FFTW r2c (fftw3f, already fetched and linked) planned with the guru interface on the same layout gemmi uses (u fastest, w halved), scaled by V/N, into the same FPhiGrid, so gemmi's prepare_asu_data() extracts the reflections exactly as before. gemmi's vendored code is not modified. The F -> map transforms of the output maps are left on gemmi. Plans are FFTW_ESTIMATE only (MEASURE times candidates and could choose differently between runs), made under a mutex - FFTW's planner is not thread-safe, and the null replicates and the parallel Jacobian call this concurrently - cached per grid size for the life of the process, and executed with fftwf_execute_dft_r2c on fftwf_malloc buffers (new-array execution needs the planning arrays' alignment, which a std::vector does not promise). NOT bit-identical with pocketfft: the two libraries order their arithmetic differently, so structure factors move in the last float bits and everything downstream (scale fit, rigid body, R-factors, maps) can move in the last printed digit. Verify by tolerance, separately from the parallelisation commits: MODEL_* keys equal to printed precision and the same MODEL_FIT / hand / indexing decisions on the audit set. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- rugnux/CMakeLists.txt | 4 +- rugnux/ModelFFT.cpp | 73 +++++++++++++++++++++++++++++++++++ rugnux/ModelFFT.h | 18 +++++++++ rugnux/ModelValidation.cpp | 13 ++++--- rugnux/RigidBodyRefine.cpp | 6 +-- tests/ModelValidationTest.cpp | 35 +++++++++++++++++ 6 files changed, 139 insertions(+), 10 deletions(-) create mode 100644 rugnux/ModelFFT.cpp create mode 100644 rugnux/ModelFFT.h diff --git a/rugnux/CMakeLists.txt b/rugnux/CMakeLists.txt index 3484e35e9..37d0b3e0d 100644 --- a/rugnux/CMakeLists.txt +++ b/rugnux/CMakeLists.txt @@ -8,6 +8,8 @@ ADD_LIBRARY(Rugnux STATIC Rugnux.h RugnuxCommandLine.cpp RugnuxCommandLine.h + ModelFFT.cpp + ModelFFT.h ModelScaling.cpp ModelScaling.h ModelValidation.cpp @@ -27,7 +29,7 @@ ADD_LIBRARY(Rugnux STATIC WriteModel.h ) -TARGET_LINK_LIBRARIES(Rugnux JFJochReader JFJochImageAnalysis JFJochWriter gemmi) +TARGET_LINK_LIBRARIES(Rugnux JFJochReader JFJochImageAnalysis JFJochWriter gemmi fftw3f) # rugnux is the single offline analysis CLI; --mode picks what it does: mx (the full pipeline, the # default), azint, scale (re-scale/merge stored reflections) or calibration (powder-ring geometry). diff --git a/rugnux/ModelFFT.cpp b/rugnux/ModelFFT.cpp new file mode 100644 index 000000000..73daab1fc --- /dev/null +++ b/rugnux/ModelFFT.cpp @@ -0,0 +1,73 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "ModelFFT.h" + +#include +#include +#include +#include +#include + +#include + +#include "gemmi/fail.hpp" + +namespace { + +// The r2c plan for an (nu, nv, nw) map stored u fastest, halving w - the layout gemmi's own transform +// reads and writes. FFTW's planner is not thread-safe, so plans are made under a lock; executing one +// on new arrays (fftwf_execute_dft_r2c) is, which is how every caller uses it. +fftwf_plan PlanFor(int nu, int nv, int nw) { + static std::mutex m; + static std::map, fftwf_plan> plans; + std::lock_guard lock(m); + const std::array key{nu, nv, nw}; + if (const auto it = plans.find(key); it != plans.end()) + return it->second; + float *in = fftwf_alloc_real(static_cast(nu) * nv * nw); + fftwf_complex *out = fftwf_alloc_complex(static_cast(nu) * nv * (nw / 2 + 1)); + // The last dimension is the one r2c halves; the strides put u fastest on both sides. + fftwf_iodim dims[3] = {{nu, 1, 1}, {nv, nu, nu}, {nw, nu * nv, nu * nv}}; + fftwf_plan plan = (in != nullptr && out != nullptr) + ? fftwf_plan_guru_dft_r2c(3, dims, 0, nullptr, in, out, FFTW_ESTIMATE) + : nullptr; + fftwf_free(in); + fftwf_free(out); + if (plan == nullptr) + gemmi::fail("MapToFPhi(): no FFTW plan for the grid"); + plans.emplace(key, plan); + return plan; +} + +} // namespace + +gemmi::FPhiGrid MapToFPhi(const gemmi::Grid &map) { + if (map.axis_order == gemmi::AxisOrder::ZYX) + gemmi::fail("MapToFPhi(): ZYX order is not supported"); + gemmi::FPhiGrid hkl; + hkl.unit_cell = map.unit_cell; + hkl.spacegroup = map.spacegroup; + hkl.axis_order = map.axis_order; + hkl.half_l = true; + hkl.set_size_without_checking(map.nu, map.nv, map.nw / 2 + 1); + const float norm = static_cast(map.unit_cell.volume / map.point_count()); + + const fftwf_plan plan = PlanFor(map.nu, map.nv, map.nw); + // Buffers from fftwf_malloc: a plan may only be executed on arrays aligned as the ones it was made + // on were, which the vectors' own allocations do not promise. + float *in = fftwf_alloc_real(map.data.size()); + fftwf_complex *out = fftwf_alloc_complex(hkl.data.size()); + if (in == nullptr || out == nullptr) { + fftwf_free(in); + fftwf_free(out); + gemmi::fail("MapToFPhi(): cannot allocate the FFT buffers"); + } + std::copy(map.data.begin(), map.data.end(), in); + fftwf_execute_dft_r2c(plan, in, out); + for (size_t i = 0; i < hkl.data.size(); ++i) + hkl.data[i] = std::complex(out[i][0], out[i][1]) * norm; + fftwf_free(in); + fftwf_free(out); + return hkl; +} diff --git a/rugnux/ModelFFT.h b/rugnux/ModelFFT.h new file mode 100644 index 000000000..b98778bd2 --- /dev/null +++ b/rugnux/ModelFFT.h @@ -0,0 +1,18 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include "gemmi/grid.hpp" +#include "gemmi/recgrid.hpp" + +// The structure factors of a real-space map on the model path (Fcalc from the model density, Fmask +// from the bulk-solvent mask): what gemmi::transform_map_to_f_phi(map, true) returns - the forward +// transform scaled by V/N, kept for l >= 0 on an (nu, nv, nw/2+1) grid in the map's own XYZ order, +// ready for prepare_asu_data() - computed with FFTW rather than gemmi's single-threaded pocketfft. +// The same transform, not the same bits: the two libraries order their arithmetic differently. +// +// Plans are FFTW_ESTIMATE only - MEASURE would time candidate algorithms and could pick a different +// one from run to run - and made once per grid size and kept, so the rigid body's hundred-odd +// evaluations on one grid plan once. Safe to call from several threads at once. +gemmi::FPhiGrid MapToFPhi(const gemmi::Grid &map); diff --git a/rugnux/ModelValidation.cpp b/rugnux/ModelValidation.cpp index 4deecd583..d85b0430b 100644 --- a/rugnux/ModelValidation.cpp +++ b/rugnux/ModelValidation.cpp @@ -19,7 +19,7 @@ #include // MaybeGzipped #include // IT92 x-ray form factors #include // DensityCalculator -#include // transform_map_to_f_phi, get_f_phi_on_grid, transform_f_phi_grid_to_map +#include // get_f_phi_on_grid, transform_f_phi_grid_to_map (the maps) #include // SolventMasker #include // Scaling (bulk solvent + anisotropic B) #include // Ccp4 map I/O @@ -30,6 +30,7 @@ #include "../common/Logger.h" #include "../common/ParallelFor.h" // ParallelFor #include "../image_analysis/scale_merge/ReindexAmbiguity.h" // ReindexReflections +#include "ModelFFT.h" #include "RigidBodyRefine.h" #include "SigmaA.h" @@ -283,7 +284,7 @@ double frame_probe_r(const gemmi::Structure &st, const gemmi::SpaceGroup *sg, dc.set_refmac_compatible_blur(st.models[0]); dc.put_model_density_on_grid(st.models[0]); gemmi::AsuData> fcalc = - gemmi::transform_map_to_f_phi(dc.grid, true) + MapToFPhi(dc.grid) .prepare_asu_data(probe_d_min, dc.blur, false, false, false); gemmi::GroupOps gops = sg->operations(); @@ -476,7 +477,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector & dc.set_grid_cell_and_spacegroup(ms.st); dc.set_refmac_compatible_blur(ms.st.models[0]); dc.put_model_density_on_grid(ms.st.models[0]); - ms.fcalc = gemmi::transform_map_to_f_phi(dc.grid, true) + ms.fcalc = MapToFPhi(dc.grid) .prepare_asu_data(dc.d_min, dc.blur, false, false, false); // Refmac radii give a slightly lower R than the Cctbx set on our test cases, at no cost. @@ -486,7 +487,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector & mask_grid.spacegroup = dc.grid.spacegroup; mask_grid.set_size_from_spacing(dc.requested_grid_spacing(), gemmi::GridSizeRounding::Up); masker.put_mask_on_grid(mask_grid, ms.st.models[0]); - ms.fmask = gemmi::transform_map_to_f_phi(mask_grid, true).prepare_asu_data(dc.d_min, 0); + ms.fmask = MapToFPhi(mask_grid).prepare_asu_data(dc.d_min, 0); }; compute_model_factors(mdl); @@ -1246,7 +1247,7 @@ std::vector ModelReferenceIntensities(const std::string &model dc.set_refmac_compatible_blur(st.models[0]); dc.put_model_density_on_grid(st.models[0]); gemmi::AsuData> fcalc = - gemmi::transform_map_to_f_phi(dc.grid, true).prepare_asu_data(dc.d_min, dc.blur, false, false, false); + MapToFPhi(dc.grid).prepare_asu_data(dc.d_min, dc.blur, false, false, false); // Flat bulk solvent at the standard constants. Nothing here is fitted - there are no observations // yet - but without it the few lowest-resolution reflections are the largest and the most wrong, @@ -1260,7 +1261,7 @@ std::vector ModelReferenceIntensities(const std::string &model mask_grid.set_size_from_spacing(dc.requested_grid_spacing(), gemmi::GridSizeRounding::Up); masker.put_mask_on_grid(mask_grid, st.models[0]); gemmi::AsuData> fmask = - gemmi::transform_map_to_f_phi(mask_grid, true).prepare_asu_data(dc.d_min, 0); + MapToFPhi(mask_grid).prepare_asu_data(dc.d_min, 0); std::unordered_map> mask_by_hkl; mask_by_hkl.reserve(fmask.v.size()); diff --git a/rugnux/RigidBodyRefine.cpp b/rugnux/RigidBodyRefine.cpp index e9a34551f..91d25843c 100644 --- a/rugnux/RigidBodyRefine.cpp +++ b/rugnux/RigidBodyRefine.cpp @@ -13,11 +13,11 @@ #include #include "gemmi/dencalc.hpp" // DensityCalculator -#include "gemmi/fourier.hpp" // transform_map_to_f_phi #include "gemmi/it92.hpp" // IT92 x-ray form factors #include "gemmi/scaling.hpp" // Scaling (bulk solvent + anisotropic B) #include "gemmi/solmask.hpp" // SolventMasker +#include "ModelFFT.h" // MapToFPhi #include "ModelScaling.h" // FitModelScale #include "../common/JFJochMath.h" // PI #include "../common/Logger.h" @@ -104,7 +104,7 @@ public: dc.set_refmac_compatible_blur(model_); dc.put_model_density_on_grid(model_); gemmi::AsuData> fcalc = - gemmi::transform_map_to_f_phi(dc.grid, true).prepare_asu_data(dc.d_min, dc.blur, false, false, false); + MapToFPhi(dc.grid).prepare_asu_data(dc.d_min, dc.blur, false, false, false); gemmi::SolventMasker masker(gemmi::AtomicRadiiSet::Refmac); gemmi::Grid mask_grid; @@ -113,7 +113,7 @@ public: mask_grid.set_size_from_spacing(dc.requested_grid_spacing(), gemmi::GridSizeRounding::Up); masker.put_mask_on_grid(mask_grid, model_); gemmi::AsuData> fmask = - gemmi::transform_map_to_f_phi(mask_grid, true).prepare_asu_data(dc.d_min, 0); + MapToFPhi(mask_grid).prepare_asu_data(dc.d_min, 0); if (fmask.size() != fcalc.size()) return false; diff --git a/tests/ModelValidationTest.cpp b/tests/ModelValidationTest.cpp index b7e4ec822..943a559cc 100644 --- a/tests/ModelValidationTest.cpp +++ b/tests/ModelValidationTest.cpp @@ -11,8 +11,10 @@ #include #include +#include #include "../common/Logger.h" +#include "../rugnux/ModelFFT.h" #include "../rugnux/ModelValidation.h" #include "../rugnux/RigidBodyRefine.h" #include "../rugnux/SigmaA.h" @@ -587,3 +589,36 @@ TEST_CASE("ModelValidation_CCModelFollowsTheSignalByShell", "[ModelValidation]") std::filesystem::remove(prefix + suffix); std::filesystem::remove(path); } + +// The model path's structure factors come from FFTW; they must be gemmi's own transform to float +// precision, in the same layout, so prepare_asu_data() reads the same reflections from either. +TEST_CASE("ModelValidation_MapToFPhiMatchesGemmi", "[ModelValidation]") { + gemmi::Grid map; + map.unit_cell.set(40.0, 50.0, 60.0, 90.0, 95.0, 90.0); + map.spacegroup = gemmi::find_spacegroup_by_name("P 1"); + map.set_size(20, 24, 30); + std::mt19937 rng(7); + std::uniform_real_distribution dist(-1.0f, 1.0f); + for (auto &x : map.data) + x = dist(rng); + + gemmi::FPhiGrid ref = gemmi::transform_map_to_f_phi(map, true); + gemmi::FPhiGrid ours = MapToFPhi(map); + REQUIRE(ours.nu == ref.nu); + REQUIRE(ours.nv == ref.nv); + REQUIRE(ours.nw == ref.nw); + REQUIRE(ours.half_l == ref.half_l); + REQUIRE(ours.data.size() == ref.data.size()); + double largest = 0, worst = 0; + for (size_t i = 0; i < ref.data.size(); i++) { + largest = std::max(largest, static_cast(std::abs(ref.data[i]))); + worst = std::max(worst, static_cast(std::abs(ours.data[i] - ref.data[i]))); + } + CHECK(worst <= 1e-5 * largest); + + const auto a = ref.prepare_asu_data(4.0, 0, false, false, false); + const auto b = ours.prepare_asu_data(4.0, 0, false, false, false); + REQUIRE(a.v.size() == b.v.size()); + for (size_t i = 0; i < a.v.size(); i++) + CHECK(a.v[i].hkl == b.v[i].hkl); +} From 2f727b1975459628f7590a0d06a4df3e0d576715 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 12:56:13 +0200 Subject: [PATCH 14/28] rugnux --model: fit the bulk-solvent grid points in parallel FitModelScale fits ~113 (k_sol, b_sol) grid points, each a full Levenberg-Marquardt over every working reflection, and it is the largest single cost of --model (every fit_model call: the first fit, the twin-law probe, the fit after the rigid body, each null replicate). It ran on one thread. Each point starts from fit_isotropic_b_approximately(), which sets k_overall and b_star from the data and the point's solvent pair alone, so no point depends on the one before it. The points of each pass (coarse, then the refinement around the coarse winner) now run in contiguous chunks, each on its own copy of the Scaling, and the winner is read off afterwards in grid order with the serial rule (lowest finite R, first on a tie). Each point's arithmetic is unchanged, so the result is the serial loop's bit for bit. Where fit_isotropic_b_approximately() has five or fewer reflections to fit on it returns without setting anything and points would chain, so there the grid is still walked serially on the caller's Scaling. Inside a null replicate (a pool worker) the chunks run inline, as before. To check: MODEL_* keys and md5 of the maps/.mtz/_model.cif identical with and without this commit on the audit set; model-phase time on a fit-dominated set. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- rugnux/ModelScaling.cpp | 72 +++++++++++++++++++++++++++++--------- rugnux/ModelScaling.h | 6 +++- rugnux/ModelValidation.cpp | 2 +- tests/ModelScalingTest.cpp | 23 ++++++++++++ 4 files changed, 84 insertions(+), 19 deletions(-) diff --git a/rugnux/ModelScaling.cpp b/rugnux/ModelScaling.cpp index 9a35b929d..e0c64e626 100644 --- a/rugnux/ModelScaling.cpp +++ b/rugnux/ModelScaling.cpp @@ -5,6 +5,9 @@ #include #include +#include + +#include "../common/ParallelFor.h" namespace { @@ -34,7 +37,15 @@ double RFactor(const gemmi::Scaling &scaling) { // pair is held fixed at each grid point and only the overall scale and the symmetry-constrained // anisotropic B are refined, which is the well-conditioned half of the problem and is left to // gemmi's solver rather than reimplemented. -ModelScaleReport FitModelScale(gemmi::Scaling &scaling, ModelScaleBox box) { +// +// Every grid point is its own fit: fit_isotropic_b_approximately() sets k_overall and b_star from the +// data and the point's solvent pair alone, so a point does not depend on the one fitted before it, and +// the points run in parallel, each chunk on its own copy of `scaling`. The winner is then read off in +// grid order with the serial rule (lowest finite R, the first on a tie), so the answer is the serial +// loop's bit for bit. The one exception is fit_isotropic_b_approximately() finding five or fewer +// reflections to fit on - it then returns without setting anything and a point WOULD start from where +// the previous one ended - so there the grid is walked in order, on `scaling` itself, as it always was. +ModelScaleReport FitModelScale(gemmi::Scaling &scaling, ModelScaleBox box, size_t nthreads) { ModelScaleReport report; report.n_points = static_cast(scaling.points.size()); if (scaling.points.size() < 20) @@ -44,36 +55,63 @@ ModelScaleReport FitModelScale(gemmi::Scaling &scaling, ModelScaleBox box scaling.fix_k_sol = true; // the grid owns the solvent pair; the solver never sees it scaling.fix_b_sol = true; + // The reflections fit_isotropic_b_approximately() fits on (its own filter). + int n_isotropic = 0; + for (const auto &p : scaling.points) + if (!(p.fobs < 1 || p.fobs < p.sigma)) + ++n_isotropic; + const bool independent = n_isotropic > 5; + double best_r = -1, best_k_sol = 0.35, best_b_sol = 46.0, best_k_overall = 1.0; gemmi::SMat33 best_b_star{0, 0, 0, 0, 0, 0}; - auto try_point = [&](double k_sol, double b_sol) { - scaling.k_sol = k_sol; - scaling.b_sol = b_sol; - scaling.fit_isotropic_b_approximately(); // a fresh starting point for this solvent pair - scaling.fit_parameters(); // k_overall + anisotropic B only - ++report.n_grid; - const double r = RFactor(scaling); - // A diverged fit gives r = NaN; latched as best_r it wins every later r < best_r. - if (std::isfinite(r) && (best_r < 0 || r < best_r)) { - best_r = r; - best_k_sol = k_sol; - best_b_sol = b_sol; - best_k_overall = scaling.k_overall; - best_b_star = scaling.b_star; + struct PointFit { double k_sol, b_sol, r = NAN, k_overall = 1.0; gemmi::SMat33 b_star{0, 0, 0, 0, 0, 0}; }; + auto fit_point = [](gemmi::Scaling &s, PointFit &pf) { + s.k_sol = pf.k_sol; + s.b_sol = pf.b_sol; + s.fit_isotropic_b_approximately(); // a fresh starting point for this solvent pair + s.fit_parameters(); // k_overall + anisotropic B only + pf.r = RFactor(s); + pf.k_overall = s.k_overall; + pf.b_star = s.b_star; + }; + auto try_points = [&](std::vector &pts) { + if (independent) + ParallelChunks(static_cast(pts.size()), nthreads, [&](int lo, int hi) { + gemmi::Scaling local = scaling; + for (int i = lo; i < hi; ++i) + fit_point(local, pts[i]); + }); + else + for (auto &pf : pts) + fit_point(scaling, pf); + for (const auto &pf : pts) { + ++report.n_grid; + // A diverged fit gives r = NaN; latched as best_r it wins every later r < best_r. + if (std::isfinite(pf.r) && (best_r < 0 || pf.r < best_r)) { + best_r = pf.r; + best_k_sol = pf.k_sol; + best_b_sol = pf.b_sol; + best_k_overall = pf.k_overall; + best_b_star = pf.b_star; + } } }; // Coarse pass over the whole box, then one refinement pass around the winner. + std::vector coarse; for (double ks = box.k_lo; ks <= box.k_hi + 1e-9; ks += 0.05) for (double bs = box.b_lo; bs <= box.b_hi + 1e-9; bs += 10.0) - try_point(ks, bs); + coarse.push_back(PointFit{ks, bs}); + try_points(coarse); const double k0 = best_k_sol, b0 = best_b_sol; const double k_hi2 = std::min(box.k_hi, k0 + 0.05); const double b_hi2 = std::min(box.b_hi, b0 + 10.0); + std::vector fine; for (double ks = std::max(box.k_lo, k0 - 0.05); ks <= k_hi2 + 1e-9; ks += 0.025) for (double bs = std::max(box.b_lo, b0 - 10.0); bs <= b_hi2 + 1e-9; bs += 5.0) - try_point(ks, bs); + fine.push_back(PointFit{ks, bs}); + try_points(fine); scaling.k_sol = best_k_sol; scaling.b_sol = best_b_sol; diff --git a/rugnux/ModelScaling.h b/rugnux/ModelScaling.h index 0600097d1..9217d6cfc 100644 --- a/rugnux/ModelScaling.h +++ b/rugnux/ModelScaling.h @@ -3,6 +3,8 @@ #pragma once +#include + #include "gemmi/scaling.hpp" // Fit the overall scale, the anisotropic B and the flat bulk solvent of a model to observed @@ -31,4 +33,6 @@ struct ModelScaleReport { double r_work_fit = 1.0; // R on the fitted reflections, at the chosen solution }; -ModelScaleReport FitModelScale(gemmi::Scaling &scaling, ModelScaleBox box = {}); +// nthreads: the grid points are fitted in parallel on up to this many threads; the answer does not +// depend on it. +ModelScaleReport FitModelScale(gemmi::Scaling &scaling, ModelScaleBox box = {}, size_t nthreads = 1); diff --git a/rugnux/ModelValidation.cpp b/rugnux/ModelValidation.cpp index d85b0430b..b3120e309 100644 --- a/rugnux/ModelValidation.cpp +++ b/rugnux/ModelValidation.cpp @@ -545,7 +545,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector & gemmi::Scaling scaling(ucell, sg); scaling.use_solvent = true; scaling.prepare_points(out.fmodel, fobs_work, &ms.fmask); - FitModelScale(scaling); + FitModelScale(scaling, {}, nthreads); scaling.scale_data(out.fmodel, &ms.fmask); // out.fmodel now holds the scaled, solvent-corrected Fmodel out.k_sol = scaling.k_sol; out.b_sol = scaling.b_sol; diff --git a/tests/ModelScalingTest.cpp b/tests/ModelScalingTest.cpp index 155ece16a..28aa20bbc 100644 --- a/tests/ModelScalingTest.cpp +++ b/tests/ModelScalingTest.cpp @@ -141,3 +141,26 @@ TEST_CASE("ModelScaling adds nothing when there is no solvent to model") { CHECK(scaling.b_sol >= 10.0); CHECK(scaling.b_sol <= 80.0); } + +// The grid points are fitted in parallel; the answer must be the serial one, bit for bit. +TEST_CASE("ModelScaling is the same on any number of threads") { + const gemmi::UnitCell cell = Cell(); + const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 21 21 21"); + REQUIRE(sg != nullptr); + gemmi::Scaling serial(cell, sg); + serial.use_solvent = true; + MakePoints(serial, cell, /*k_overall=*/0.5, /*b_iso=*/20.0, + /*k_sol=*/0.35, /*b_sol=*/45.0, /*d_min=*/2.0, /*d_max=*/50.0); + gemmi::Scaling parallel = serial; + + const ModelScaleReport r1 = FitModelScale(serial, {}, 1); + const ModelScaleReport r8 = FitModelScale(parallel, {}, 8); + CHECK(r1.n_grid == r8.n_grid); + CHECK(r1.r_work_fit == r8.r_work_fit); + CHECK(serial.k_sol == parallel.k_sol); + CHECK(serial.b_sol == parallel.b_sol); + CHECK(serial.k_overall == parallel.k_overall); + CHECK(serial.b_star.u11 == parallel.b_star.u11); + CHECK(serial.b_star.u22 == parallel.b_star.u22); + CHECK(serial.b_star.u33 == parallel.b_star.u33); +} From 47bc15c62c42c5ae3b14b45e80063a87fefb84d1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 12:57:04 +0200 Subject: [PATCH 15/28] rugnux --model: drop two fits nothing needed A rejected rigid-body placement put the atoms back and then recomputed the full Fcalc and solvent mask at the restored positions. Nothing reads them: the real model's ModelState is only fitted again after its own compute_model_factors (the re-placement on the merged indexing, the null-failure fallback), and a null replicate's state is discarded. The atoms are still put back; the structure factors are left as they are. Where the lattice has twin laws, the indexing probe fits the identity labelling first, and place_and_fit then made the identical fit again - same ModelState (nothing moves the model between the two), same reflections. The probe now hands that Fit over, for the real model when the identity wins and always for a null replicate (placed against the merged indexing). fit_model is deterministic, so the reused Fit is the one the second call produced. Bit-identical by construction: no value that is read changes. To check: MODEL_* keys and md5 of maps/.mtz/_model.cif identical with and without this commit on the audit set, including sets with twin laws and a null. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- rugnux/ModelValidation.cpp | 42 +++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/rugnux/ModelValidation.cpp b/rugnux/ModelValidation.cpp index b3120e309..06b3e4790 100644 --- a/rugnux/ModelValidation.cpp +++ b/rugnux/ModelValidation.cpp @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -580,14 +581,24 @@ ModelValidationResult ValidateAgainstModel(const std::vector & // The real model and every null replicate below go through this same lambda, so the two are // comparable - a null that was not placed would be the null of a weaker procedure than the one it // is there to judge. It leaves the model where it ends up, and the caller puts it back. + // + // `fitted`: fit_model(ms, obs_in) where the caller already has it - the indexing probe fits the + // identity labelling first, on the same structure factors and the same reflections - so the same + // fit is not made twice in a row. + // + // A rejected placement puts the atoms back but not ms.fcalc/ms.fmask, which are left at the + // rejected placement: nothing reads them afterwards - every later fit_model on this state is + // preceded by its own compute_model_factors - and recomputing them was a full Fcalc and mask + // for nothing. struct Placement { Fit fit; bool rb_applied = false; double rb_angle_deg = 0, rb_shift_A = 0, r_free_before_rb = 0; }; - auto place_and_fit = [&](ModelState &ms, const std::vector &obs_in) -> Placement { + auto place_and_fit = [&](ModelState &ms, const std::vector &obs_in, + std::optional fitted) -> Placement { Placement out; - out.fit = fit_model(ms, obs_in); + out.fit = fitted ? std::move(*fitted) : fit_model(ms, obs_in); // --- rigid-body placement of the model in the data cell --- // Re-fractionalizing the model above puts it in the right box but not in the right place: a @@ -620,7 +631,6 @@ ModelValidationResult ValidateAgainstModel(const std::vector & out.fit = std::move(moved); } else { SetModelPositions(ms.st.models[0], before); - compute_model_factors(ms); } } return out; @@ -647,10 +657,12 @@ ModelValidationResult ValidateAgainstModel(const std::vector & struct IndexingProbe { gemmi::Op op = gemmi::Op::identity(); // the lowest-R-free relabelling double margin = 0; // by how much in R-free it leads the runner-up + std::optional identity_fit; // fit_model(ms, obs), for place_and_fit to reuse }; auto probe_indexing = [&](ModelState &ms) { IndexingProbe out; - std::vector r_free{fit_model(ms, obs).r_free}; // identity first, then the twin laws + out.identity_fit = fit_model(ms, obs); + std::vector r_free{out.identity_fit->r_free}; // identity first, then the twin laws double best_r_free = r_free.front(); for (const auto &op : reindex_ops) { const double cand = fit_model(ms, ReindexReflections(obs, op)).r_free; @@ -679,7 +691,13 @@ ModelValidationResult ValidateAgainstModel(const std::vector & obs_model = &reindexed; } } - Placement real = place_and_fit(mdl, *obs_model); + // The probe's identity fit is this fit exactly when the identity won: same structure factors + // (nothing has moved the model since), same reflections. + std::optional identity_fit; + if (obs_model == &obs) + identity_fit = std::move(indexing.identity_fit); + indexing.identity_fit.reset(); + Placement real = place_and_fit(mdl, *obs_model, std::move(identity_fit)); // --- is there anything for the model to decide? --- // There are only two: the space-group label, where the model asserts the other enantiomorph, and @@ -737,9 +755,13 @@ ModelValidationResult ValidateAgainstModel(const std::vector & SetModelPositions(rep.st.models[0], as_read); reorient_about_centroid(rep.st.models[0], null_rotation[i]); compute_model_factors(rep); - if (!reindex_ops.empty()) - null_margin[i] = probe_indexing(rep).margin; - null_r_work[i] = place_and_fit(rep, obs).fit.r_work; + std::optional identity_fit; + if (!reindex_ops.empty()) { + IndexingProbe probe = probe_indexing(rep); + null_margin[i] = probe.margin; + identity_fit = std::move(probe.identity_fit); // replicates are placed against obs + } + null_r_work[i] = place_and_fit(rep, obs, std::move(identity_fit)).fit.r_work; }); const auto [null_mean, null_sd] = mean_sd(null_r_work); result.fit_tested = true; @@ -794,7 +816,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector & // are now the ones the merge produced, so the fit is remade on them. SetModelPositions(st.models[0], as_read); compute_model_factors(mdl); - real = place_and_fit(mdl, obs); + real = place_and_fit(mdl, obs, std::nullopt); record_fit(real); } } @@ -824,7 +846,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector & result.r_work_sigma = 0.0; SetModelPositions(st.models[0], as_read); compute_model_factors(mdl); - real = place_and_fit(mdl, obs); + real = place_and_fit(mdl, obs, std::nullopt); } Fit &best = real.fit; From 5d931222d700cd198e19081888c48212f9a0491f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 12:58:03 +0200 Subject: [PATCH 16/28] rugnux --model: evaluate the rigid-body Jacobian's columns in parallel The rigid-body target is an Fcalc + solvent mask + scale re-fit per evaluation, and its forward-difference Jacobian made six of them one after the other after the central one - 7 of the ~8 evaluations per LM iteration. The six shifted evaluations now run in parallel, column j on its own Evaluator over its own copy of the model (an evaluation moves every atom, so two cannot share one), set to the same zone and given the central evaluation's bulk-solvent pair - the pair the serial loop's shifted evaluations used, since the zone's solvent is fitted on the zone's first evaluation and every Evaluate starts with the central one. Each column's arithmetic is the serial one and the Jacobian is assembled in column order, so the refinement is bit-identical; the evaluation count is kept as the serial loop kept it (up to and including a failing column). The per-zone solvent fit (FitModelScale) gets the thread count as well. Inside a null replicate (a pool worker) the columns run inline, as before. To check: RIGID_BODY_* and MODEL_* keys and md5 of maps/.mtz/_model.cif identical with and without this commit on the audit set; rigid-body seconds on a large model. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- rugnux/ModelValidation.cpp | 2 +- rugnux/RigidBodyRefine.cpp | 63 ++++++++++++++++++++++++++++------- rugnux/RigidBodyRefine.h | 4 ++- tests/ModelValidationTest.cpp | 40 ++++++++++++++++++++++ 4 files changed, 95 insertions(+), 14 deletions(-) diff --git a/rugnux/ModelValidation.cpp b/rugnux/ModelValidation.cpp index 06b3e4790..4d09805ab 100644 --- a/rugnux/ModelValidation.cpp +++ b/rugnux/ModelValidation.cpp @@ -610,7 +610,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector & // exactly where it was read. const std::vector before = ModelPositions(ms.st.models[0]); const RigidBodyRefineResult rb = - RefineRigidBody(ms.st.models[0], ucell, *sg, out.fit.fobs_work, d_min, logger); + RefineRigidBody(ms.st.models[0], ucell, *sg, out.fit.fobs_work, d_min, logger, nthreads); if (!rb.converged) { // RefineRigidBody leaves the model wherever the solver left it, usable answer or not, so // the restore cannot be conditional on the same flag the re-fit is. diff --git a/rugnux/RigidBodyRefine.cpp b/rugnux/RigidBodyRefine.cpp index 91d25843c..8b36f44d5 100644 --- a/rugnux/RigidBodyRefine.cpp +++ b/rugnux/RigidBodyRefine.cpp @@ -4,6 +4,7 @@ #include "RigidBodyRefine.h" #include +#include #include #include #include @@ -21,6 +22,7 @@ #include "ModelScaling.h" // FitModelScale #include "../common/JFJochMath.h" // PI #include "../common/Logger.h" +#include "../common/ParallelFor.h" namespace { @@ -72,8 +74,8 @@ struct Placement { class Evaluator { public: Evaluator(gemmi::Model &model, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, - const std::vector &base, const Placement &placement) - : model_(model), cell_(cell), sg_(sg), base_(base), placement_(placement) {} + const std::vector &base, const Placement &placement, size_t nthreads) + : model_(model), cell_(cell), sg_(sg), base_(base), placement_(placement), nthreads_(nthreads) {} // The zone's observations, and the scale the residuals are expressed in. void SetZone(const gemmi::AsuData> &fobs, double d_min) { @@ -86,6 +88,14 @@ public: solvent_fitted_ = false; } + // Hold the zone's bulk solvent at a pair another evaluator already fitted, as if this one had + // fitted it itself (the Jacobian's column evaluators take the central evaluation's). + void UseSolvent(double k, double b) { + k_sol = k; + b_sol = b; + solvent_fitted_ = true; + } + size_t NumObservations() const { return fobs_.v.size(); } double JacobianStep() const { return JACOBIAN_STEP_FRACTION * d_min_; } int evaluations = 0; @@ -135,7 +145,7 @@ public: if (scaling.points.empty()) return false; if (!solvent_fitted_) { - FitModelScale(scaling); + FitModelScale(scaling, {}, nthreads_); k_sol = scaling.k_sol; b_sol = scaling.b_sol; solvent_fitted_ = true; @@ -179,6 +189,7 @@ private: double d_min_ = 0; double f_mean_ = 1; bool solvent_fitted_ = false; + size_t nthreads_ = 1; }; // Ceres' own numeric differentiation steps by |x| * relative_step_size, which is zero at the start of @@ -186,9 +197,17 @@ private: // forward differences at a step chosen in the parameters' units. Analytic dF/dp would need // derivatives GEMMI's structure-factor path does not have, and at six parameters it is not worth it: // a Jacobian costs seven evaluations, and the evaluations at 6-3.5 A are cheap. +// +// The six shifted evaluations are independent of each other, so they run in parallel, column j on +// `columns[j]`: an evaluator over its own copy of the model (an evaluation moves every atom to the +// placement it is asked about, so two cannot share one), set to the same zone and handed the solvent +// pair the central evaluation fitted - which is the pair the serial loop's shifted evaluations used, +// since the zone's solvent is fitted by the first evaluation of the zone and every Evaluate starts +// with the central one. Each column's arithmetic is the serial one, so the Jacobian is too. class RigidBodyCost : public ceres::CostFunction { public: - explicit RigidBodyCost(Evaluator &ev) : ev_(ev) { + RigidBodyCost(Evaluator &ev, std::vector &columns, size_t nthreads) + : ev_(ev), columns_(columns), nthreads_(nthreads) { set_num_residuals(static_cast(ev.NumObservations())); mutable_parameter_block_sizes()->push_back(6); } @@ -198,16 +217,24 @@ public: if (!ev_.Residuals(parameters[0], residuals)) return false; if (jacobians != nullptr && jacobians[0] != nullptr) { - std::vector shifted(n); - for (int j = 0; j < 6; j++) { + const double step = ev_.JacobianStep(); + std::vector> shifted(6, std::vector(n)); + std::array ok{}; + ParallelFor(6, nthreads_, [&](int j) { double q[6]; std::copy(parameters[0], parameters[0] + 6, q); - const double step = ev_.JacobianStep(); q[j] += step; - if (!ev_.Residuals(q, shifted.data())) + columns_[j].UseSolvent(ev_.k_sol, ev_.b_sol); + ok[j] = columns_[j].Residuals(q, shifted[j].data()); + }); + // Counted as the serial loop counted them: it stopped at the first column that failed. + for (int j = 0; j < 6; j++) { + ++ev_.evaluations; + ev_.unmatched = columns_[j].unmatched; + if (!ok[j]) return false; for (int i = 0; i < n; i++) - jacobians[0][i * 6 + j] = (shifted[i] - residuals[i]) / step; + jacobians[0][i * 6 + j] = (shifted[j][i] - residuals[i]) / step; } } return true; @@ -215,6 +242,8 @@ public: private: Evaluator &ev_; + std::vector &columns_; + size_t nthreads_; }; // The directions in which this space group's origin is free. Translating the whole cell content along @@ -274,7 +303,8 @@ RigidBodyRefineResult RefineRigidBody(gemmi::Model &model, const gemmi::SpaceGroup &sg, const gemmi::AsuData> &fobs, double d_min, - Logger &logger) { + Logger &logger, + size_t nthreads) { const auto t0 = std::chrono::steady_clock::now(); RigidBodyRefineResult result; @@ -301,7 +331,14 @@ RigidBodyRefineResult RefineRigidBody(gemmi::Model &model, ladder.push_back(d_min); const gemmi::Mat33 gauge = GaugeProjector(sg, cell); - Evaluator ev(model, cell, sg, base, placement); + Evaluator ev(model, cell, sg, base, placement, nthreads); + // The Jacobian's six column evaluators, each over its own copy of the model (see RigidBodyCost). + // The copies only ever hold probe placements; `model` is left at the answer below. + std::vector column_models(6, model); + std::vector columns; + columns.reserve(6); + for (int j = 0; j < 6; j++) + columns.emplace_back(column_models[j], cell, sg, base, placement, 1); double q[6] = {0, 0, 0, 0, 0, 0}; bool any_zone_solved = false; for (double zone : ladder) { @@ -315,9 +352,11 @@ RigidBodyRefineResult RefineRigidBody(gemmi::Model &model, continue; result.zones.push_back(zone); // the ladder WALKED, which a thin zone drops out of ev.SetZone(zone_obs, zone); + for (Evaluator &c : columns) + c.SetZone(zone_obs, zone); ceres::Problem problem; - problem.AddResidualBlock(new RigidBodyCost(ev), nullptr, q); + problem.AddResidualBlock(new RigidBodyCost(ev, columns, nthreads), nullptr, q); ceres::Solver::Options options; options.linear_solver_type = ceres::DENSE_QR; options.max_num_iterations = 15; diff --git a/rugnux/RigidBodyRefine.h b/rugnux/RigidBodyRefine.h index 2f57fdc5d..a17f3489b 100644 --- a/rugnux/RigidBodyRefine.h +++ b/rugnux/RigidBodyRefine.h @@ -3,6 +3,7 @@ #pragma once +#include #include #include "gemmi/asudata.hpp" @@ -41,4 +42,5 @@ RigidBodyRefineResult RefineRigidBody(gemmi::Model &model, const gemmi::SpaceGroup &sg, const gemmi::AsuData> &fobs, double d_min, - Logger &logger); + Logger &logger, + size_t nthreads = 1); diff --git a/tests/ModelValidationTest.cpp b/tests/ModelValidationTest.cpp index 943a559cc..3fcb7bee8 100644 --- a/tests/ModelValidationTest.cpp +++ b/tests/ModelValidationTest.cpp @@ -622,3 +622,43 @@ TEST_CASE("ModelValidation_MapToFPhiMatchesGemmi", "[ModelValidation]") { for (size_t i = 0; i < a.v.size(); i++) CHECK(a.v[i].hkl == b.v[i].hkl); } + +// The Jacobian's six columns are evaluated in parallel; the placement must be the serial one, bit for +// bit. +TEST_CASE("ModelValidation_RigidBodySameOnAnyNumberOfThreads", "[ModelValidation]") { + Logger logger("ModelValidation_RigidBodySameOnAnyNumberOfThreads"); + + const auto path = WriteTemp("rigid_body_threads_test.pdb", ClusterPdb().c_str()); + gemmi::Structure st = gemmi::read_structure_gz(path, gemmi::CoorFormat::Detect); + const gemmi::SpaceGroup *sg = st.find_spacegroup(); + REQUIRE(sg != nullptr); + st.setup_cell_images(); + + const auto ref = ModelReferenceIntensities(path, {}, {}, 3.0, logger); + REQUIRE_FALSE(ref.empty()); + gemmi::AsuData> fobs; + fobs.unit_cell_ = st.cell; + fobs.spacegroup_ = sg; + for (const auto &r : ref) + fobs.v.push_back({{{r.h, r.k, r.l}}, {std::sqrt(r.I), 1.0f}}); + fobs.ensure_sorted(); + + std::vector displaced; + for (const gemmi::Position &p : ModelPositions(st.models[0])) + displaced.emplace_back(p.x + 0.40, p.y - 0.30, p.z + 0.20); + + gemmi::Model serial = st.models[0], parallel = st.models[0]; + SetModelPositions(serial, displaced); + SetModelPositions(parallel, displaced); + const RigidBodyRefineResult r1 = RefineRigidBody(serial, st.cell, *sg, fobs, 3.0, logger, 1); + const RigidBodyRefineResult r6 = RefineRigidBody(parallel, st.cell, *sg, fobs, 3.0, logger, 6); + CHECK(r1.evaluations == r6.evaluations); + CHECK(r1.angle_deg == r6.angle_deg); + CHECK(r1.shift_A == r6.shift_A); + const auto p1 = ModelPositions(serial), p6 = ModelPositions(parallel); + REQUIRE(p1.size() == p6.size()); + for (size_t i = 0; i < p1.size(); i++) + CHECK((p1[i].x == p6[i].x && p1[i].y == p6[i].y && p1[i].z == p6[i].z)); + + std::filesystem::remove(path); +} From 600497cc6703291fbf36f579078f1c9a8dc0739a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 16:04:23 +0200 Subject: [PATCH 17/28] One process-wide lock for FFTW planning FFTW's planner (every fftwf_plan_* and fftwf_destroy_plan) shares global state and is not thread-safe; executing a plan is. FFTIndexerCPU and BeamCenterFFTCPU each guarded their planning with a lock of their own, TranslationalNCS and the viewer's spectrum with none, and ModelFFT with a third - which does not stop two of them planning at once. common/FFTWPlannerLock.h holds the one mutex they all now take. No numerical change. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- common/FFTWPlannerLock.h | 14 +++++++++++++ .../geom_refinement/BeamCenterFFTCPU.cpp | 16 ++++++--------- image_analysis/indexing/FFTIndexerCPU.cpp | 6 +++--- .../scale_merge/TranslationalNCS.cpp | 15 +++++++++++--- rugnux/ModelFFT.cpp | 10 ++++++---- viewer/charts/JFJochDatasetInfoChartView.cpp | 20 +++++++++++++------ 6 files changed, 55 insertions(+), 26 deletions(-) create mode 100644 common/FFTWPlannerLock.h diff --git a/common/FFTWPlannerLock.h b/common/FFTWPlannerLock.h new file mode 100644 index 000000000..a432b7971 --- /dev/null +++ b/common/FFTWPlannerLock.h @@ -0,0 +1,14 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include + +// FFTW's planner - every fftwf_plan_* and fftwf_destroy_plan call - shares global state and is not +// thread-safe; executing a plan is. So every piece of code that makes or destroys a plan takes this +// one lock, not a lock of its own: two private locks do not stop two planners running at once. +inline std::mutex &FFTWPlannerMutex() { + static std::mutex m; + return m; +} diff --git a/image_analysis/geom_refinement/BeamCenterFFTCPU.cpp b/image_analysis/geom_refinement/BeamCenterFFTCPU.cpp index 6c59f77d6..a9be3f21f 100644 --- a/image_analysis/geom_refinement/BeamCenterFFTCPU.cpp +++ b/image_analysis/geom_refinement/BeamCenterFFTCPU.cpp @@ -11,13 +11,9 @@ #include -namespace { +#include "../../common/FFTWPlannerLock.h" -// FFTW planning is not thread-safe (execution is); same idiom as FFTIndexerCPU. -std::mutex &FftwPlanMutex() { - static std::mutex m; - return m; -} +namespace { // One real-to-complex 2D FFT workspace of fixed padded size, reused for every forward transform. class Rfft2 { @@ -32,7 +28,7 @@ public: Rfft2(int64_t ny_, int64_t nx_) : ny(ny_), nx(nx_), nxc(nx_ / 2 + 1) { real.resize(static_cast(ny * nx)); spectrum.resize(static_cast(ny * nxc)); - std::unique_lock lock(FftwPlanMutex()); + std::unique_lock lock(FFTWPlannerMutex()); fwd = fftwf_plan_dft_r2c_2d(static_cast(ny), static_cast(nx), real.data(), reinterpret_cast(spectrum.data()), FFTW_ESTIMATE); @@ -45,7 +41,7 @@ public: Rfft2(const Rfft2 &) = delete; Rfft2 &operator=(const Rfft2 &) = delete; ~Rfft2() { - std::unique_lock lock(FftwPlanMutex()); + std::unique_lock lock(FFTWPlannerMutex()); if (fwd) fftwf_destroy_plan(fwd); if (bwd) @@ -136,7 +132,7 @@ BeamCenterConvSurfaces1D BeamCenterFFTCPU::LineSurfaces(const std::vector fftwf_plan fwd, bwd; { - std::unique_lock lock(FftwPlanMutex()); + std::unique_lock lock(FFTWPlannerMutex()); fwd = fftwf_plan_dft_r2c_1d(static_cast(nfft), in.data(), reinterpret_cast(A.data()), FFTW_ESTIMATE); bwd = fftwf_plan_dft_c2r_1d(static_cast(nfft), @@ -190,7 +186,7 @@ BeamCenterConvSurfaces1D BeamCenterFFTCPU::LineSurfaces(const std::vector out.Q = inverse(accQ); out.D = inverse(accD); { - std::unique_lock lock(FftwPlanMutex()); + std::unique_lock lock(FFTWPlannerMutex()); fftwf_destroy_plan(fwd); fftwf_destroy_plan(bwd); } diff --git a/image_analysis/indexing/FFTIndexerCPU.cpp b/image_analysis/indexing/FFTIndexerCPU.cpp index 8061b62f5..218c95cef 100644 --- a/image_analysis/indexing/FFTIndexerCPU.cpp +++ b/image_analysis/indexing/FFTIndexerCPU.cpp @@ -10,7 +10,7 @@ #include #include -static std::mutex fftw_plan_mutex; +#include "../../common/FFTWPlannerLock.h" static inline double dot_abs(const Coord& a, const Coord& b) { return std::fabs(a.x * b.x + a.y * b.y + a.z * b.z); @@ -35,7 +35,7 @@ FFTIndexerCPU::FFTIndexerCPU(const IndexingSettings& settings) int n[1] = { H }; { - std::unique_lock ul(fftw_plan_mutex); + std::unique_lock ul(FFTWPlannerMutex()); plan = fftwf_plan_many_dft_r2c( 1, n, nDirections, h_input_fft.data(), nullptr, 1, H, @@ -49,7 +49,7 @@ FFTIndexerCPU::FFTIndexerCPU(const IndexingSettings& settings) FFTIndexerCPU::~FFTIndexerCPU() { - std::unique_lock ul(fftw_plan_mutex); + std::unique_lock ul(FFTWPlannerMutex()); fftwf_destroy_plan(plan); } diff --git a/image_analysis/scale_merge/TranslationalNCS.cpp b/image_analysis/scale_merge/TranslationalNCS.cpp index 5fa3b1a96..7b5a91e07 100644 --- a/image_analysis/scale_merge/TranslationalNCS.cpp +++ b/image_analysis/scale_merge/TranslationalNCS.cpp @@ -15,6 +15,8 @@ #include +#include "../../common/FFTWPlannerLock.h" + namespace { // The Patterson is measured in a band, not over the whole dataset: below ~20 A the bulk solvent // and the beam stop dominate, above ~5 A the map answers a question about atoms rather than about @@ -231,8 +233,12 @@ TranslationalNCSResult AnalyzeTranslationalNCS(const std::vector(coeff), map, - FFTW_ESTIMATE); + fftwf_plan plan; + { + std::lock_guard lock(FFTWPlannerMutex()); + plan = fftwf_plan_dft_c2r_3d(nx, ny, nz, reinterpret_cast(coeff), map, + FFTW_ESTIMATE); + } // The near-origin exclusion, as a set of grid points rather than a per-point distance test: only // the points inside a 15 A ball around each origin-equivalent point can be excluded, and there @@ -368,7 +374,10 @@ TranslationalNCSResult AnalyzeTranslationalNCS(const std::vector, fftwf_plan> plans; - std::lock_guard lock(m); + std::lock_guard lock(FFTWPlannerMutex()); const std::array key{nu, nv, nw}; if (const auto it = plans.find(key); it != plans.end()) return it->second; diff --git a/viewer/charts/JFJochDatasetInfoChartView.cpp b/viewer/charts/JFJochDatasetInfoChartView.cpp index 068699d7d..abdfe4305 100644 --- a/viewer/charts/JFJochDatasetInfoChartView.cpp +++ b/viewer/charts/JFJochDatasetInfoChartView.cpp @@ -11,6 +11,7 @@ #include #include "JFJochDatasetInfoChartView.h" +#include "../../common/FFTWPlannerLock.h" namespace { // Qt Charts draws its curves 2 px wide, which is hard to follow across a desk. Take the weight @@ -965,18 +966,25 @@ void JFJochDatasetInfoChartView::buildFFTChart() { std::vector out(static_cast(nComplex)); - fftwf_plan plan = fftwf_plan_dft_r2c_1d( - n, - in.data(), - out.data(), - FFTW_ESTIMATE); + fftwf_plan plan; + { + std::lock_guard lock(FFTWPlannerMutex()); + plan = fftwf_plan_dft_r2c_1d( + n, + in.data(), + out.data(), + FFTW_ESTIMATE); + } if (!plan) { return; } fftwf_execute(plan); - fftwf_destroy_plan(plan); + { + std::lock_guard lock(FFTWPlannerMutex()); + fftwf_destroy_plan(plan); + } // Compute amplitude spectrum and frequencies (0 .. Nyquist) m_fftMagnitudes.resize(static_cast(nComplex)); From a4d1b8eb699fe8609d57261b9594d67982a3c5f0 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 16:08:23 +0200 Subject: [PATCH 18/28] rugnux --model: MapToFPhi returns the conjugate, as gemmi's transform does gemmi::transform_map_to_f_phi conjugates the forward FFT at the end (a structure factor is the sum over exp(+2 pi i h.x), a forward FFT the sum over exp(-2 pi i h.x)); MapToFPhi did not, so every model-path Fcalc and Fmask came out with its phase negated. Amplitudes, and so the R-factors and the scale, were unaffected; the phases of the maps and everything read off them (map coefficients, density at atom centres) were not. Caught by ModelValidation_MapToFPhiMatchesGemmi, which now passes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- rugnux/ModelFFT.cpp | 4 +++- rugnux/ModelFFT.h | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/rugnux/ModelFFT.cpp b/rugnux/ModelFFT.cpp index 304ec673a..03ce9c69c 100644 --- a/rugnux/ModelFFT.cpp +++ b/rugnux/ModelFFT.cpp @@ -67,8 +67,10 @@ gemmi::FPhiGrid MapToFPhi(const gemmi::Grid &map) { } std::copy(map.data.begin(), map.data.end(), in); fftwf_execute_dft_r2c(plan, in, out); + // Conjugated, as gemmi does: FFTW's forward transform carries exp(-2 pi i h.x), and a structure + // factor is the sum over exp(+2 pi i h.x). for (size_t i = 0; i < hkl.data.size(); ++i) - hkl.data[i] = std::complex(out[i][0], out[i][1]) * norm; + hkl.data[i] = std::complex(out[i][0] * norm, -out[i][1] * norm); fftwf_free(in); fftwf_free(out); return hkl; diff --git a/rugnux/ModelFFT.h b/rugnux/ModelFFT.h index b98778bd2..fd807d1de 100644 --- a/rugnux/ModelFFT.h +++ b/rugnux/ModelFFT.h @@ -7,8 +7,8 @@ #include "gemmi/recgrid.hpp" // The structure factors of a real-space map on the model path (Fcalc from the model density, Fmask -// from the bulk-solvent mask): what gemmi::transform_map_to_f_phi(map, true) returns - the forward -// transform scaled by V/N, kept for l >= 0 on an (nu, nv, nw/2+1) grid in the map's own XYZ order, +// from the bulk-solvent mask): what gemmi::transform_map_to_f_phi(map, true) returns - the +// conjugated forward transform (F = sum rho exp(+2 pi i h.x)) scaled by V/N, kept for l >= 0 on an (nu, nv, nw/2+1) grid in the map's own XYZ order, // ready for prepare_asu_data() - computed with FFTW rather than gemmi's single-threaded pocketfft. // The same transform, not the same bits: the two libraries order their arithmetic differently. // From aaca5cc48190b9370a93f9efc3974702988b54d1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 17:56:16 +0200 Subject: [PATCH 19/28] rugnux: with Friedel mates merged separately, a thin hand takes its outlier median from the pair The rot3d merge tests each observation against the weighted median of its reflection, and only from three observations up: at two the median is one of them. With -A each hand of a Friedel pair is its own reflection, so a P1 anomalous merge sits at about two observations a hand and almost nothing is tested - on one P1 sweep the merge rejected 0 observations with -A against tens of thousands without it. A single observation a hundred times its partner's intensity (300 sigma apart, a 3.9 A reflection) then survived into the merge, carried 94% of the weighted variance of its CC1/2 bin, took that bin from 0.80 to 0.18, and the automatic cut with it: 3.46 A and UNUSABLE, where the same data without -A cut at 1.40 A. A hand with fewer than three observations now takes the weighted median of both hands together, where the pair has three. I(+) and I(-) differ by the anomalous signal, a few percent of I and far inside the six-sigma test, so the mate supplies the observations the hand is missing. A hand with three of its own keeps its own median exactly as before, and a Friedel-merged run is unchanged. The medians are formed on the host, so the device merge reads the same ones. Measured with -A: the P1 sweep 3.46 A / UNUSABLE -> 1.49 A (1.63 A XDS reference; 1.40 A without -A, unchanged), 0 -> 3,699 rejections. Two lysozyme and one insulin set: same cut, space group, CC1/2 and ISa. On one of the lysozyme sets 16 more observations are rejected, every shell's CCanom is unchanged, and the overall CCanom goes 0.31 -> -0.02: the whole-range figure was being carried by a handful of wild pairs, while the shells ranged -18% to +12% and XDS's overall anomalous correlation is 1%. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- .../scale_merge/RotationScaleMerge.cpp | 65 +++++++++++++++---- 1 file changed, 53 insertions(+), 12 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index d3597ca03..c3c0112cc 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -3769,27 +3769,65 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // threshold is not the problem - 12 sigma rejects 1.35% of observations and gives the // same answer as 6 sigma - the rule is, and it only has a defined median from three. // Groups below the cut keep a NaN median and the test below skips them. - std::vector start(n_groups + 1, 0); - for (int g = 0; g < n_groups; ++g) - start[g + 1] = start[g] + (cnt[g] >= 3 ? cnt[g] : 0); + // + // With the Friedel mates merged separately (-A) a hand below three borrows its MATE's + // observations: I(+) and I(-) differ by the anomalous signal, a few percent of I and far + // inside a six-sigma test, so together they are a median the hand alone does not have. Without + // it a P1 anomalous merge sits at about two observations a hand and nothing is tested: one + // observation a hundred times its partner's intensity (300 sigma apart) then survived into + // the merge and took a whole CC1/2 bin, and the cut with it, from 0.80 to 0.18. A hand with + // three of its own keeps its own median, as before. Median set g is group g's own; median + // set n_groups + p is Friedel pair p's, filled only where a hand of p needs it. + std::vector pair_of_group; + std::vector pair_needed; + if (!merge_friedel) { + const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, x.GetSpaceGroupOrP1()); + std::unordered_map pair_id; + pair_id.reserve(static_cast(n_groups) + 1); + pair_of_group.resize(n_groups); + for (int g = 0; g < n_groups; ++g) { + const HKLKey gk = anom_keygen(group_h[g], group_k[g], group_l[g]); + const uint64_t key = HKLKey{gk.h, gk.k, gk.l, true}.pack(); + pair_of_group[g] = pair_id.emplace(key, static_cast(pair_id.size())).first->second; + } + std::vector pair_cnt(pair_id.size(), 0); + for (int g = 0; g < n_groups; ++g) pair_cnt[pair_of_group[g]] += cnt[g]; + pair_needed.assign(pair_id.size(), 0); + for (int g = 0; g < n_groups; ++g) + if (cnt[g] > 0 && cnt[g] < 3 && pair_cnt[pair_of_group[g]] >= 3) + pair_needed[pair_of_group[g]] = 1; + } + const int n_sets = n_groups + static_cast(pair_needed.size()); + std::vector set_cnt(n_sets, 0); + for (int g = 0; g < n_groups; ++g) { + if (cnt[g] >= 3) set_cnt[g] = cnt[g]; + if (!pair_needed.empty() && pair_needed[pair_of_group[g]]) + set_cnt[n_groups + pair_of_group[g]] += cnt[g]; + } + std::vector start(n_sets + 1, 0); + for (int t = 0; t < n_sets; ++t) + start[t + 1] = start[t] + set_cnt[t]; // Inverse-variance WEIGHTED median: a frame the crystal barely diffracted on is scaled up by // 1/G together with its sigma, so a plain median lets two such observations outvote one // well-measured one - and the test below then rejects the well-measured one against its // own small sigma. - std::vector> iv(start[n_groups]); // (I*corr, 1/(sigma*corr)^2) + std::vector> iv(start[n_sets]); // (I*corr, 1/(sigma*corr)^2) { std::vector fill(start.begin(), start.end() - 1); for (int i = 0; i < n_full; ++i) { const int g = mf.group[i]; - if (g >= 0 && cnt[g] >= 3) { - const float sc = mf.sigma[i] * mf.corr[i]; - iv[fill[g]++] = {mf.I[i] * mf.corr[i], sc > 0.0f ? 1.0f / (sc * sc) : 0.0f}; - } + if (g < 0) continue; + const float sc = mf.sigma[i] * mf.corr[i]; + const std::pair v{mf.I[i] * mf.corr[i], sc > 0.0f ? 1.0f / (sc * sc) : 0.0f}; + if (cnt[g] >= 3) iv[fill[g]++] = v; + if (!pair_needed.empty() && pair_needed[pair_of_group[g]]) + iv[fill[n_groups + pair_of_group[g]]++] = v; } } - ParallelChunks(n_groups, ThreadsForWork(iv.size(), nthreads), [&](int glo, int ghi) { - for (int g = glo; g < ghi; ++g) { - const int lo = start[g], hi = start[g + 1]; + std::vector set_median(n_sets, NAN); + ParallelChunks(n_sets, ThreadsForWork(iv.size(), nthreads), [&](int tlo, int thi) { + for (int t = tlo; t < thi; ++t) { + const int lo = start[t], hi = start[t + 1]; if (lo == hi) continue; std::sort(iv.begin() + lo, iv.begin() + hi); double total = 0.0; @@ -3800,9 +3838,12 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool run += iv[j].second; if (run >= 0.5 * total) break; } - reject_median[g] = iv[j].first; + set_median[t] = iv[j].first; } }); + for (int g = 0; g < n_groups; ++g) + reject_median[g] = cnt[g] >= 3 ? set_median[g] + : !pair_needed.empty() ? set_median[n_groups + pair_of_group[g]] : NAN; } fit_error_model(samples); } From bd386ba0f23390cb3dbd08975f8aa878f506306a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 17:21:46 +0200 Subject: [PATCH 20/28] tools/battery: a set with a known-wrong reference can be left unscored Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- tools/battery/README.md | 2 ++ tools/battery/score.py | 2 ++ 2 files changed, 4 insertions(+) diff --git a/tools/battery/README.md b/tools/battery/README.md index 982e7d7e5..65d6c2005 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -341,6 +341,8 @@ whole range; the manifest keeps the rule used as `dmin_rule`. control), then run `battery.py refs --arm inhouse --write` to fill in `ref` from the `CORRECT.LP` beside the input. Without `--write`, `refs` only shows what would change. A reference measured by hand goes in `ref_override`, which `refs` leaves alone. + A set whose reference is known to be wrong gets `"unscored": ""`: it still runs, but + scores `unscored` instead of pass or fail. - **Renamed data directories**: `battery.py remap --arm inhouse MAP.json --write`, where `MAP.json` is `{"old": "new"}` (renames the set and its directory) or `{"old id": {"id": "...", "input": "..."}}`. Each old id is recorded under `aliases` in the diff --git a/tools/battery/score.py b/tools/battery/score.py index 81cae0ff1..331722d7d 100644 --- a/tools/battery/score.py +++ b/tools/battery/score.py @@ -159,6 +159,8 @@ def judge(entry, rep, run_note): return verdict("fail", "crash", run_note or "no lattice in the report") if not ref.get("sgno") or not ref.get("cell"): return verdict("unscored", "no_reference", "no reference to score against") + if entry.get("unscored"): # a reference known to be wrong: run, but do not score + return verdict("unscored", "reference_problem", entry["unscored"]) vr, edge = lattice_match(r["cell"], space_group(rep.get("SPACE_GROUP_NAME"), r["sgno"]), ref["cell"], space_group(ref.get("sg"), ref["sgno"])) From e6e506809f77d6bce82b44074d00fc4740a8bda1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 17:48:45 +0200 Subject: [PATCH 21/28] Bragg integration: size each image's reflection list to what is kept Finalize reserved npredicted reflections and kept only the ones whose fit succeeded. On a dense long-axis pattern more than half the predicted reflections lose their background ring, so each retained per-image vector carried more dead capacity than data for the rest of the pass. Count the kept ones first and reserve exactly that. No result changes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- .../bragg_integration/BraggIntegrationEngine.cpp | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp index f44132c1e..ced5d0caf 100644 --- a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp +++ b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp @@ -215,8 +215,14 @@ std::vector BraggIntegrationEngine::Finalize(const std::vector &results, int64_t image_number) const { + // Sized to what is kept, not to what was predicted: this vector is the one every image holds to the + // end of the pass, and a background ring lost to the neighbours drops a reflection here - on a + // dense pattern more than half of them, which a reserve of npredicted would carry as dead capacity. + size_t n_ok = 0; + for (size_t i = 0; i < npredicted; ++i) + n_ok += results[i].ok ? 1 : 0; std::vector out; - out.reserve(npredicted); + out.reserve(n_ok); for (size_t i = 0; i < npredicted; ++i) { const auto &fr = results[i]; if (!fr.ok) From c58510edba883baee008d504d92368f6bcb0e94f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 17:48:45 +0200 Subject: [PATCH 22/28] RotationScaleMerge: ingest without the index-order copy of the sort keys Ingest wrote every observation's 24-byte SortKey into an index-order array and then scattered that array into its h buckets - two key arrays alive together at the top of the ingest. Count the buckets per chunk of frames straight from the source reflections and scatter from them instead: a frame's reflections are consecutive in the key numbering, so the chunks are consecutive index ranges and each bucket lands in the same index order as before. The key array is byte-identical. BuildInRangeObservations also hands back the remap temporaries (old runs, per-run maps, new_idx) before the observation array is built rather than at the end of the function. Measured on a 3600-frame long-axis rotation set (95.7 M partials in the geometry pre-pass): the pre-pass ingest high-water drops 15.45 -> 14.1 GB (sampled RSS, -N 6); merged MTZ/HKL/CIF, the P1 cross-check MTZ and the per-image table are byte-identical. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- .../scale_merge/RotationScaleMerge.cpp | 50 ++++++++++--------- 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index c3c0112cc..8b3e371aa 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -343,7 +343,6 @@ void RotationScaleMerge::Ingest() { // pure waste. Only the survivors are built, from the source reflections, once the runs are known. constexpr int INT_LO = std::numeric_limits::min(); constexpr int INT_HI = std::numeric_limits::max(); - std::vector unsorted(total); std::vector h_lo(n_frames, INT_HI), h_hi(n_frames, INT_LO); ParallelFor(n_frames, nthreads, [&](int o) { if (reference_cell) { @@ -353,13 +352,10 @@ void RotationScaleMerge::Ingest() { double bkg_sum = 0.0; int bkg_n = 0; int lmin = INT_HI, lmax = INT_LO; - int32_t at = src_start[o]; for (const auto &r : partials_out[o].reflections) { if (std::isfinite(r.bkg) && r.bkg > 0.0f) { bkg_sum += r.bkg; ++bkg_n; } - unsorted[at] = SortKey{r.h, r.k, r.l, r.image_number, at, r.d}; lmin = std::min(lmin, r.h); lmax = std::max(lmax, r.h); - ++at; } if (bkg_n > 0) mean_bkg[o] = bkg_sum / bkg_n; h_lo[o] = lmin; h_hi[o] = lmax; @@ -395,30 +391,30 @@ void RotationScaleMerge::Ingest() { if (a.image_number != b.image_number) return a.image_number < b.image_number; return a.idx < b.idx; }; - // Two key arrays for a moment: the sweep above wrote them in index order, and where a key belongs - // in the bucketed order is not known until the whole histogram is. Both together are less than half - // of what the observation array they replace used to occupy on its own. + // One key array, scattered into its h bucket straight from the source reflections. The buckets are + // counted per chunk of FRAMES first; a frame's reflections are consecutive in the key numbering, so + // the chunks are consecutive index ranges and the scatter lands each bucket in index order - the + // same array an index-order copy scattered afterwards gave, without that copy (24 bytes a partial, + // gigabytes on a long axis, at the top of the pass's memory). std::vector keys(total); { // Bucket by h first, then sort each bucket. h is the comparator's leading key, so the sorted // array is exactly the buckets laid end to end - and because the order is total (see above) // the permutation is the one a single std::sort produces, whichever algorithm gets there. // That is what makes this checkable: the output is byte-identical, not merely equivalent. - const int n = static_cast(total); - const int nt = static_cast(std::clamp(nthreads, 1, std::max(1, n))); - const int chunk = (n + nt - 1) / nt; + const int nt = static_cast(std::clamp(nthreads, 1, std::max(1, n_frames))); + const int chunk = (n_frames + nt - 1) / nt; // the split ParallelChunks makes below const int H = h_max - h_min + 1; std::vector> hist(nt, std::vector(H, 0)); - ParallelChunks(nt, nthreads, [&](int tlo, int thi) { - for (int t = tlo; t < thi; ++t) { - const int lo = t * chunk, hi = std::min(n, lo + chunk); - for (int i = lo; i < hi; ++i) hist[t][unsorted[i].h - h_min]++; - } + ParallelChunks(n_frames, nthreads, [&](int lo, int hi) { + std::vector &h_count = hist[lo / chunk]; + for (int o = lo; o < hi; ++o) + for (const auto &r : partials_out[o].reflections) h_count[r.h - h_min]++; }); // Where each chunk writes each bucket: buckets in order, and within a bucket the chunks in - // index order, so the scatter below is stable and lands in original order inside a bucket. + // frame order, so the scatter below is stable and lands in original order inside a bucket. std::vector bstart(H + 1, 0); int32_t acc = 0; for (int b = 0; b < H; ++b) { @@ -431,15 +427,16 @@ void RotationScaleMerge::Ingest() { } bstart[H] = acc; - ParallelChunks(nt, nthreads, [&](int tlo, int thi) { - for (int t = tlo; t < thi; ++t) { - std::vector fill = hist[t]; - const int lo = t * chunk, hi = std::min(n, lo + chunk); - for (int i = lo; i < hi; ++i) keys[fill[unsorted[i].h - h_min]++] = unsorted[i]; + ParallelChunks(n_frames, nthreads, [&](int lo, int hi) { + std::vector fill = hist[lo / chunk]; + for (int o = lo; o < hi; ++o) { + int32_t at = src_start[o]; + for (const auto &r : partials_out[o].reflections) { + keys[fill[r.h - h_min]++] = SortKey{r.h, r.k, r.l, r.image_number, at, r.d}; + ++at; + } } }); - unsorted.clear(); - unsorted.shrink_to_fit(); // Largest bucket first: the tail of this is one bucket, so it should be the big one. std::vector order(H); @@ -687,6 +684,13 @@ void RotationScaleMerge::BuildInRangeObservations(std::vector &keys) { rawrun_start.swap(ns); rawrun_count.swap(nc); rawrun_h.swap(nh); rawrun_k.swap(nk); rawrun_l.swap(nl); rawrun_d.swap(nd); std::vector().swap(keys); // consumed (see the header) + // Everything above that only served the remap - the old runs now in ns..nd, the per-run maps and + // new_idx - is handed back before the observation array below is built, not at the end of the + // function: on a long axis it is a gigabyte beside the largest array of the ingest. + for (auto *v : {&ns, &nc, &nh, &nk, &nl, &dst_run, &dst_start, &new_idx}) + std::vector().swap(*v); + std::vector().swap(nd); + std::vector().swap(keep_run); // The observations themselves, one frame per thread. finite_ok is the per-obs AcceptReflection // finiteness (immutable) - it lets ComputeAsuGroups stamp the ASU-group id per obs from a flat From bddbec7aff2f7c542dac6764fc6aaafbd04446f5 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 18:00:10 +0200 Subject: [PATCH 23/28] rugnux: measure the canonical pass's geometry probe before the scaling engine is built The post-refinement probe on a canonical pass only measures, and it reads nothing the scaling engine produces - the reflections' hkl, I, sigma and positions are final once the images are integrated, and the engine writes back per-frame fields only. It ran after the first merge, beside the engine's arrays, where its gathered observations and rocking events set the pass's memory high-water mark. Run it before the engine is built instead; the pre-pass keeps its original place (it consumes the smoothed mosaicity the merge writes back). Measured on a 3600-frame long-axis rotation set: the canonical pass's high-water drops from 14.8 GB to under 12.8 GB (sampled RSS, -N 6); merged MTZ/HKL/CIF, P1 MTZ, per-image table and the report are byte-identical, the probe's own result included. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- rugnux/Rugnux.cpp | 179 ++++++++++++++++++++++++---------------------- 1 file changed, 95 insertions(+), 84 deletions(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index a6da5bbb2..8e40c0399 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -5006,6 +5006,99 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b const auto &rot_ss = experiment_.GetScalingSettings(); const bool is_rotation = experiment_.IsRotationIndexing(); // rotation indexing -> rotation scaling/merge std::optional rsm; + std::optional prepass_postrefine_obs; + // The rotation geometry post-refinement (see its call sites below for what it is for). + const auto post_refine_geometry = [&] { + const auto &outcomes = indexer->GetIntegrationOutcome(); + if (geometry_prepass) { + prepass_mosaicity_.assign(outcomes.size(), NAN); + for (size_t o = 0; o < outcomes.size(); ++o) + if (outcomes[o].mosaicity_deg) prepass_mosaicity_[o] = *outcomes[o].mosaicity_deg; + } + if (result.consensus_cell.has_value()) { + if (const auto rot = indexer->FinalizeRotationIndexing(); rot && rot->axis) { + if (geometry_prepass) + prepass_result_ = *rot; // full result (lattice+geom+axis), for the supercell fallback + PostRefineSettings prs; + prs.crystal_system = rot->search_result.system; + prs.num_threads = static_cast(config_.nthreads); + prs.refine_geometry = true; + // The centre the pre-scan measured, where the fit was precise enough to be used as + // a centre at all (the same ceiling --estimate-beam-center adopts one on). It does + // not move the geometry - it only tells the bound on step B's beam move where else + // the centre is independently believed to be. + if (measured_beam_center_ + && measured_beam_center_->sigma_pxl <= std::max(MAX_BEAM_CENTER_SIGMA_PXL, + BeamCenterNeed_pxl(experiment_))) + prs.measured_beam_px = std::array{ + static_cast(measured_beam_center_->beam_x_pxl), + static_cast(measured_beam_center_->beam_y_pxl)}; + // The first pass's own geometry, not the header's. The outcomes being post-refined + // were integrated at it, and step B holds the tilt fixed while it fits the beam and + // the distance - so handing it the header tilt fits a beam that has to absorb a tilt + // error the images do not have, and the answer describes neither geometry. + // + // On the pre-pass the observations were gathered right after Ingest (see there) and + // the scaling engine is finished - nothing after this point reads it on that pass + // (the search and the reference re-merge run only on the canonical one) - so it is + // freed BEFORE the fit: the engine holds about twice the partial payload again, + // and with it gone the fit no longer sits on the pass's memory peak. + if (prepass_postrefine_obs) + rsm.reset(); + auto pr_obs = prepass_postrefine_obs + ? std::move(*prepass_postrefine_obs) + : GatherPostRefineObservations(indexer->GetIntegrationOutcome(), + config_.nthreads, + /*release_reflections=*/false, logger); + prepass_postrefine_obs.reset(); + const auto pr = PostRefineRotationGeometry(std::move(pr_obs), *rot->axis, + rot->geom, rot->lattice, prs, logger); + result.post_refine = pr; + if (pr.ok && pr.detector_refined) { + // Store, don't apply: this pass continues to write its own (header-geometry) output; + // Run() applies the refined geometry before the second pass. + prepass_detector_geometry_ = std::array{ + static_cast(pr.beam_x_after_px), + static_cast(pr.beam_y_after_px), + static_cast(pr.distance_after_mm), + rot->geom.GetPoniRot1_rad(), + rot->geom.GetPoniRot2_rad()}; + logger.Info("Two-pass: post-refined detector geometry (distance {:.3f} mm, " + "beam {:.2f},{:.2f} px, tilt {:.4f},{:.4f} deg) - {}", + pr.distance_after_mm, pr.beam_x_after_px, pr.beam_y_after_px, + rot->geom.GetPoniRot1_rad() * 180.0 / PI, + rot->geom.GetPoniRot2_rad() * 180.0 / PI, + geometry_prepass ? "it will drive the second integration pass" + : "what the fit still asks for at the geometry " + "this pass ran at"); + } else { + logger.Info("Two-pass: geometry post-refine committed no detector change{}", + geometry_prepass ? " - the second pass reproduces the first" : ""); + } + // DECISION POINT for the goniometer rotation scale. It does not ride on pr.ok - the + // scale is its own cross-validated fit and the crystals that have a stage fault are + // exactly the ones whose cell and detector steps do NOT pass, because the angle error + // is what their residual is made of. A calibration fault is rare (36 of 37 rotation + // datasets sit at 1.0000) and applying a 1 % angle correction to a healthy dataset + // would silently damage it, so the asymmetry is deliberate: committed only when the + // fit is both cross-validated and outside the tolerance. A manual --rotation-scale is + // already on the goniometer and is left alone. + // Only the pre-pass fits the rotation scale. It is applied from the goniometer + // the file came with, so a scale measured again on angles that have already been + // corrected once is not a correction that can be applied on top of that one. + if (pr.rotation_scale_suspect && !config_.rotation_scale.has_value() && geometry_prepass) + prepass_rotation_scale_ = static_cast(pr.rotation_scale); + } + } + }; + // On a CANONICAL pass the post-refinement only measures (see below), and it reads nothing the + // scaling engine produces: the reflections' hkl, I, sigma and positions are final once the + // images are integrated, and the engine writes back per-frame fields only. So measure it + // before the engine is built rather than beside it - the gathered observations and their + // rocking events are gigabytes on a long axis, and beside the engine's own arrays they set + // the pass's memory high-water mark. + if (postrefine_probe_ && !geometry_prepass && is_rotation) + post_refine_geometry(); if (is_rotation) { if (experiment_.GetRefineRotationWedgeInScaling() || rot_ss.GetRotationWedgeForScaling().has_value()) @@ -5029,7 +5122,6 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // it is read: on a fine-sliced long axis the vectors are gigabytes, held to the end of the // pass for those 32 bytes a partial. A run with a reference MTZ keeps them: resolving the // indexing ambiguity re-reads them (and re-ingests). - std::optional prepass_postrefine_obs; if (geometry_prepass && rsm && config_.reference_data.empty()) prepass_postrefine_obs = GatherPostRefineObservations( indexer->GetIntegrationOutcome(), config_.nthreads, @@ -5211,89 +5303,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // reflections integrated at the geometry after it. The probe measures - it applies nothing and // leaves the smoothed mosaicity and the pre-pass lattice alone, which stay pass 1's for every // pass that consumes them; RunAllPasses decides what to do with the answer. - if ((geometry_prepass || postrefine_probe_) && rsm && experiment_.IsRotationIndexing()) { - const auto &outcomes = indexer->GetIntegrationOutcome(); - if (geometry_prepass) { - prepass_mosaicity_.assign(outcomes.size(), NAN); - for (size_t o = 0; o < outcomes.size(); ++o) - if (outcomes[o].mosaicity_deg) prepass_mosaicity_[o] = *outcomes[o].mosaicity_deg; - } - if (result.consensus_cell.has_value()) { - if (const auto rot = indexer->FinalizeRotationIndexing(); rot && rot->axis) { - if (geometry_prepass) - prepass_result_ = *rot; // full result (lattice+geom+axis), for the supercell fallback - PostRefineSettings prs; - prs.crystal_system = rot->search_result.system; - prs.num_threads = static_cast(config_.nthreads); - prs.refine_geometry = true; - // The centre the pre-scan measured, where the fit was precise enough to be used as - // a centre at all (the same ceiling --estimate-beam-center adopts one on). It does - // not move the geometry - it only tells the bound on step B's beam move where else - // the centre is independently believed to be. - if (measured_beam_center_ - && measured_beam_center_->sigma_pxl <= std::max(MAX_BEAM_CENTER_SIGMA_PXL, - BeamCenterNeed_pxl(experiment_))) - prs.measured_beam_px = std::array{ - static_cast(measured_beam_center_->beam_x_pxl), - static_cast(measured_beam_center_->beam_y_pxl)}; - // The first pass's own geometry, not the header's. The outcomes being post-refined - // were integrated at it, and step B holds the tilt fixed while it fits the beam and - // the distance - so handing it the header tilt fits a beam that has to absorb a tilt - // error the images do not have, and the answer describes neither geometry. - // - // On the pre-pass the observations were gathered right after Ingest (see there) and - // the scaling engine is finished - nothing after this point reads it on that pass - // (the search and the reference re-merge run only on the canonical one) - so it is - // freed BEFORE the fit: the engine holds about twice the partial payload again, - // and with it gone the fit no longer sits on the pass's memory peak. - if (prepass_postrefine_obs) - rsm.reset(); - auto pr_obs = prepass_postrefine_obs - ? std::move(*prepass_postrefine_obs) - : GatherPostRefineObservations(indexer->GetIntegrationOutcome(), - config_.nthreads, - /*release_reflections=*/false, logger); - prepass_postrefine_obs.reset(); - const auto pr = PostRefineRotationGeometry(std::move(pr_obs), *rot->axis, - rot->geom, rot->lattice, prs, logger); - result.post_refine = pr; - if (pr.ok && pr.detector_refined) { - // Store, don't apply: this pass continues to write its own (header-geometry) output; - // Run() applies the refined geometry before the second pass. - prepass_detector_geometry_ = std::array{ - static_cast(pr.beam_x_after_px), - static_cast(pr.beam_y_after_px), - static_cast(pr.distance_after_mm), - rot->geom.GetPoniRot1_rad(), - rot->geom.GetPoniRot2_rad()}; - logger.Info("Two-pass: post-refined detector geometry (distance {:.3f} mm, " - "beam {:.2f},{:.2f} px, tilt {:.4f},{:.4f} deg) - {}", - pr.distance_after_mm, pr.beam_x_after_px, pr.beam_y_after_px, - rot->geom.GetPoniRot1_rad() * 180.0 / PI, - rot->geom.GetPoniRot2_rad() * 180.0 / PI, - geometry_prepass ? "it will drive the second integration pass" - : "what the fit still asks for at the geometry " - "this pass ran at"); - } else { - logger.Info("Two-pass: geometry post-refine committed no detector change{}", - geometry_prepass ? " - the second pass reproduces the first" : ""); - } - // DECISION POINT for the goniometer rotation scale. It does not ride on pr.ok - the - // scale is its own cross-validated fit and the crystals that have a stage fault are - // exactly the ones whose cell and detector steps do NOT pass, because the angle error - // is what their residual is made of. A calibration fault is rare (36 of 37 rotation - // datasets sit at 1.0000) and applying a 1 % angle correction to a healthy dataset - // would silently damage it, so the asymmetry is deliberate: committed only when the - // fit is both cross-validated and outside the tolerance. A manual --rotation-scale is - // already on the goniometer and is left alone. - // Only the pre-pass fits the rotation scale. It is applied from the goniometer - // the file came with, so a scale measured again on angles that have already been - // corrected once is not a correction that can be applied on top of that one. - if (pr.rotation_scale_suspect && !config_.rotation_scale.has_value() && geometry_prepass) - prepass_rotation_scale_ = static_cast(pr.rotation_scale); - } - } - } + if (geometry_prepass && rsm && experiment_.IsRotationIndexing()) + post_refine_geometry(); // For a de-novo search, gate the P1 scaling+merge (the pass that feeds the space-group search) at // >= 1 over thin resolution shells: noise-dominated high-res shells otherwise mislead From dea229bcc0c7af812df8f93b9bbad9627641112e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 18:14:02 +0200 Subject: [PATCH 24/28] RotationScaleMerge: a 32-byte resident ingest record, and smaller staging slices On the resident (GPU) path the ingest's narrow host record sits on top of the source reflections for the whole smoothing and partiality recompute, and on a long axis that moment is the run's memory high-water mark. Store its Miller indices in 16 bits and move the rocking-event flag into their padding: 40 -> 32 bytes a partial. Stage the device upload in slices of two million observations rather than eight, which takes the thirteen staging arrays from ~400 MB to ~100 MB. Neither changes a value. Measured on a 3600-frame long-axis rotation set (-N 6): VmHWM 13.79 -> 12.80 GiB on top of the previous commits (15.0-15.1 GiB before any of them); merged MTZ/HKL/CIF, P1 MTZ, per-image table and the report are byte-identical. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- image_analysis/scale_merge/RotationScaleMerge.cpp | 9 +++++++-- image_analysis/scale_merge/RotationScaleMerge.h | 12 ++++++++---- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 8b3e371aa..c251c7fb7 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -712,7 +712,9 @@ void RotationScaleMerge::BuildInRangeObservations(std::vector &keys) { if (!keep[src]) { ++src; continue; } ++src; ObsIngest &obs = partials_ingest[at]; - obs.h = r.h; obs.k = r.k; obs.l = r.l; + obs.h = static_cast(r.h); + obs.k = static_cast(r.k); + obs.l = static_cast(r.l); obs.frame = o; obs.image_number = r.image_number; obs.d = r.d; @@ -734,7 +736,10 @@ void RotationScaleMerge::BuildInRangeObservations(std::vector &keys) { gpu_->SetPartialsLayout(n_keep, n_frames, frame_start.data(), frame_count.data()); using F = RotationScaleMergeGPU::ObsField; - constexpr int STAGE_SLICE = 8 << 20; + // Two million observations a slice: the thirteen staging arrays are then under a hundred + // megabytes together, where eight million put four hundred on top of the ingest's peak, and a + // slice still spans tens of frames for the threads to share. + constexpr int STAGE_SLICE = 2 << 20; constexpr int NFLOAT = 11; int max_frame_obs = 0; for (int o = 0; o < n_frames; ++o) max_frame_obs = std::max(max_frame_obs, static_cast(frame_count[o])); diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index 533242ad3..81cf8164c 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -144,16 +144,20 @@ private: // observation dump): every immutable per-obs field is uploaded straight from the source // reflections inside BuildInRangeObservations, and the host keeps only what its own remaining // stages touch - the geometry smoothing (delta_phi, then the recomputed partiality), plus the - // rocking-event walk's inputs. Forty bytes against the Obs's eighty, and on that path the fat + // rocking-event walk's inputs. Thirty-two bytes against the Obs's eighty, and on that path the fat // record is never materialised at all: on a fine-sliced long axis it alone is many gigabytes, // held beside the source reflections it was copied from. struct ObsIngest { - int32_t h, k, l, frame; - float image_number, d, zeta, delta_phi, partiality; + // Sixteen bits are plenty for a Miller index - |h| <= a / d_min, in the hundreds even on the + // longest axis at atomic resolution - and on a long axis this array sets the ingest's memory + // high-water mark, so each index costs two bytes here, not four. + int16_t h, k, l; // The rocking-event usability test (RockingEventFrames), evaluated on the ingest-time // values - which is what that walk reads on the resident path, where the host corr is - // never refreshed (see partials_released). + // never refreshed (see partials_released). Beside the indices, where it fills their padding. uint8_t rock_ok; + int32_t frame; + float image_number, d, zeta, delta_phi, partiality; }; // The narrow per-observation record the ingest sort orders: the raw hkl the runs are cut on, the From 2e4196696cc725951097afa929b580bd9ccabe9c Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 18:33:39 +0200 Subject: [PATCH 25/28] rugnux: a rotation run decides with Friedel mates merged, -A or not -A set MergeFriedel(false), and RotationScaleMerge builds one grouping for the whole pass from it, so every decision on an anomalous rotation run was taken on hand-separated groups: per-frame scaling, the correction surfaces, the outlier median, the error model, the CC1/2 half-sets and the resolution cut, the delta-CC1/2 frame ledger and the space-group search. That halves the multiplicity each of those gates was calibrated at, and flipped the bare run's symmetry, resolution and frame-rejection calls on several sets - none of them through the anomalous signal (a 422 refused on an added-operator R, a 23 refused on a chi^2 gate, a screw row read on hand rows, thin hands exempt from the outlier median letting one hot pair own a CC1/2 bin). A rotation run now always merges with Friedel's law: the Rugnux constructor turns MergeFriedel back on when the run is rotation (IsRotationIndexing, which --force-still clears), and --mode scale does the same for its rotation merge. Nothing is lost for anomalous use: the merge always keeps the Bijvoet split, and I(+)/I(-), F(+)/F(-), SigAno and CCanom are written and reported from it whether or not -A was given. -A on a rotation run is therefore a no-op on the processing, which is the point; stills keep -A as their only anomalous route. A bare run is untouched. This is step 1 of the -A design (decide with pairs merged, present with them split). The per-hand statistics table that -A should select is not built yet, so an -A rotation run now reports the Friedel-merged table (FRIEDELS_LAW= TRUE, Laue multiplicity and completeness). The thin-hand pair median of the previous commit is unreachable from rugnux on rotation data but kept for any direct MergeFriedel(false) caller of RotationScaleMerge. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- rugnux/Rugnux.cpp | 17 +++++++++++++++++ rugnux/rugnux_cli.cpp | 3 ++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 8e40c0399..557c79afe 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -497,6 +497,23 @@ Rugnux::Rugnux(JFJochReader &reader, DiffractionExperiment experiment, LogHeaderBeamCenter(experiment_, pixel_mask_); + // Every decision on a rotation run - per-frame scaling, the correction surfaces, the outlier test, + // the error model, the CC1/2 half-sets and the resolution cut, the frame ledger, the space-group + // search - is taken with Friedel mates merged, -A or not. Splitting them halves the multiplicity + // every one of those gates was calibrated at, which is a noisier instrument rather than a more + // careful one: it flipped symmetry, resolution and frame-rejection calls against the bare run on + // several sets, none of them through the anomalous signal. It is also what XDS, AIMLESS, POINTLESS + // and DIALS decide on. The anomalous signal is not lost: the merge always keeps the Bijvoet split, + // and I(+)/I(-), SigAno and CCanom come from it in either mode. Stills keep -A as it was - their + // merge has no separate split. + if (experiment_.IsRotationIndexing() && !experiment_.GetScalingSettings().GetMergeFriedel()) { + ScalingSettings s = experiment_.GetScalingSettings(); + experiment_.ImportScalingSettings(s.MergeFriedel(true)); + Logger("Rugnux").Info("-A on rotation data: every processing decision is made with Friedel mates " + "merged; I(+)/I(-), SigAno and CCanom are reported from the Bijvoet split " + "the merge keeps, exactly as without -A"); + } + // A manually asserted stage calibration applies to everything, before anything reads an angle. if (config_.rotation_scale.has_value()) if (const auto g = experiment_.GetGoniometer()) { diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 6ffbc469d..e4b6fdc9d 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -1577,7 +1577,8 @@ static int RunRugnux(int argc, char **argv) { if (report_resolution_arg) scaling_settings.ReportResolutionRange_A( ParseReportResolution(*report_resolution_arg, scaling_settings, logger)); - scaling_settings.MergeFriedel(!anomalous_mode); + // Rotation merges decide with Friedel mates merged, -A or not (see the Rugnux constructor). + scaling_settings.MergeFriedel(!anomalous_mode || rot); scaling_settings.MinPartiality(min_partiality); scaling_settings.MinCapturedFraction( min_captured_fraction_arg.value_or(scaling_settings.GetMinCapturedFraction())); From 11ca5b866a909815a5772a854af7c7674b6a90dc Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 18:32:15 +0200 Subject: [PATCH 26/28] tools/battery: the XDS references carry the lowest shell's R_meas parse_correct_lp also reads the first row of CORRECT.LP's last resolution table: its high-resolution limit (dmin_low) and R_meas (r_meas_low). inhouse.json regenerated with `refs --arm inhouse --write`; the only change is the two new keys per set (the private manifest was regenerated the same way, outside the repository). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- tools/battery/inhouse.json | 66 +++++++++++++++++++------------------- tools/battery/inputs.py | 8 ++++- 2 files changed, 40 insertions(+), 34 deletions(-) diff --git a/tools/battery/inhouse.json b/tools/battery/inhouse.json index e6d1aefaf..20a7fbd38 100644 --- a/tools/battery/inhouse.json +++ b/tools/battery/inhouse.json @@ -1,37 +1,37 @@ {"arm": "inhouse", "aliases": {"E08_lyso_2_006": "lyso_x10sa_90deg_1", "E08_lyso_2_007": "lyso_x10sa_90deg_2", "lyso_2": "lyso_x06da_atten_wedge", "lyso_6_5keV": "lyso_x06da_5keV", "lysoC_14_f46f0a": "lyso_x06da_ice", "Thau_9": "thau_x10sa_0p1deg", "thau_BL1A_1p9A": "thau_bl1a_6p5keV", "thau_BL1A_2p7A": "thau_bl1a_4p6keV", "thau_BL1A_3p3A": "thau_bl1a_3p8keV", "Ins_H_2_fa9841": "insu_H_x06da_twin", "Ins_H_3_d05684": "insu_H_x06da_notwin", "Ins_I_16": "insu_I_x06da_weak", "Ins_I_2_1a530d": "insu_I_x06da_low_isa", "Ins_I_3_5e90e3": "insu_I_x06da_ref", "Ins_I_6_13keV": "insu_I_x06da_13keV", "Ins_I_6_5keV": "insu_I_x06da_5keV", "Ins_I_6_6keV": "insu_I_x06da_6keV", "Ins_I_7_5keV": "insu_I_x06da_5keV_2", "cytC_10": "cytc_x10sa", "cytC_2_0770b5": "cytc_x06da_1", "cytC_3_874285": "cytc_x06da_2", "MyoB_13": "myob_x10sa", "MyoB_9_446b70": "myob_x06da", "MyoB2-1": "myob_x06da_powder_1", "MyoB2-2": "myob_x06da_powder_2", "MyoB2-4": "myob_x06da_split", "MyoB2-5": "myob_x06da_sparse", "onlyice": "nothing_ice", "nothing_3400ac": "nothing_2", "nothing_9bf604": "nothing_1", "lyso_90deg_1": "lyso_x10sa_90deg_1", "lyso_90deg_2": "lyso_x10sa_90deg_2", "lyso_atten_wedge": "lyso_x06da_atten_wedge", "lyso_5keV": "lyso_x06da_5keV", "lyso_ice": "lyso_x06da_ice", "lyso_ref": "lyso_x06da_ref", "lyso_strong": "lyso_x10sa_strong", "lyso_half_image": "lyso_x06da_half_image", "thau_16keV": "thau_x10sa_16keV", "thau_0p1deg": "thau_x10sa_0p1deg", "thau_1p9A": "thau_bl1a_6p5keV", "thau_2p7A": "thau_bl1a_4p6keV", "thau_3p3A": "thau_bl1a_3p8keV", "thau_injection": "thau_x10sa_injection", "thau_pink": "thau_micromax_pink", "insu_H_twin": "insu_H_x06da_twin", "insu_H_notwin": "insu_H_x06da_notwin", "insu_I_weak": "insu_I_x06da_weak", "insu_I_low_isa": "insu_I_x06da_low_isa", "insu_I_ref": "insu_I_x06da_ref", "insu_I_13keV": "insu_I_x06da_13keV", "insu_I_5keV": "insu_I_x06da_5keV", "insu_I_6keV": "insu_I_x06da_6keV", "insu_I_5keV_2": "insu_I_x06da_5keV_2", "cytc_eiger": "cytc_x10sa", "cytc_pilatus_1": "cytc_x06da_1", "cytc_pilatus_2": "cytc_x06da_2", "myob_eiger": "myob_x10sa", "myob_pilatus": "myob_x06da", "myob_powder_1": "myob_x06da_powder_1", "myob_powder_2": "myob_x06da_powder_2", "myob_split": "myob_x06da_split", "myob_aggregate": "myob_x06da_sparse", "myob_x06da_aggregate": "myob_x06da_sparse"}, "sets": [ - {"id": "lyso_x10sa_90deg_1", "input": "lyso_x10sa_90deg_1/lyso_2_006_master.h5", "ref": {"sgno": 92, "cell": [76.879, 76.879, 36.961, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.75, "completeness": 99.3, "r_meas": 0.182, "cc_half": 0.995, "multiplicity": 3.51, "dmin": 1.966, "dmin_rule": "xds_range", "dmin_xds": 1.966, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, - {"id": "lyso_x10sa_90deg_2", "input": "lyso_x10sa_90deg_2/lyso_2_007_master.h5", "ref": {"sgno": 92, "cell": [76.904, 76.904, 36.976, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.14, "completeness": 99.3, "r_meas": 0.18, "cc_half": 0.994, "multiplicity": 3.51, "dmin": 1.973, "dmin_rule": "xds_range", "dmin_xds": 1.973, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, - {"id": "lyso_x06da_atten_wedge", "input": "lyso_x06da_atten_wedge/lyso_2_002_master.h5", "ref": {"sgno": 92, "cell": [78.235, 78.235, 37.798, 90.0, 90.0, 90.0], "anomalous": true, "isa": 16.56, "completeness": 99.9, "r_meas": 0.239, "cc_half": 0.998, "multiplicity": 10.97, "dmin": 1.264, "dmin_rule": "xds_range", "dmin_xds": 1.264, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, - {"id": "lyso_x06da_5keV", "input": "lyso_x06da_5keV/lyso_6_001_master.h5", "ref": {"sgno": 92, "cell": [78.225, 78.225, 37.897, 90.0, 90.0, 90.0], "anomalous": true, "isa": 19.36, "completeness": 86.4, "r_meas": 0.064, "cc_half": 0.998, "multiplicity": 8.85, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "lysozyme", "long-wavelength"]}, - {"id": "lyso_x06da_ice", "input": "lyso_x06da_ice/lysoC_14_56b5b9_master.h5", "ref": {"sgno": 89, "cell": [78.387, 78.387, 37.427, 90.0, 90.0, 90.0], "anomalous": true, "isa": 23.33, "completeness": 100.0, "r_meas": 0.219, "cc_half": 0.998, "multiplicity": 11.86, "dmin": 1.431, "dmin_rule": "xds_range", "dmin_xds": 1.431, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, - {"id": "lyso_x06da_ref", "input": "lyso_x06da_ref/hewl_35_001_master.h5", "ref": {"sgno": 89, "cell": [78.091, 78.091, 37.704, 90.0, 90.0, 90.0], "anomalous": true, "isa": 28.31, "completeness": 100.0, "r_meas": 0.045, "cc_half": 1.0, "multiplicity": 13.76, "dmin": 1.2, "dmin_rule": "xds_range", "dmin_xds": 1.2, "dmax": 50.0}, "tags": ["h5", "lysozyme"], "tiers": {"smoke": "lysozyme reference set"}}, - {"id": "lyso_x10sa_strong", "input": "lyso_x10sa_strong/lyso_5_f6ac7e_master.h5", "ref": {"sgno": 92, "cell": [77.769, 77.769, 37.442, 90.0, 90.0, 90.0], "anomalous": false, "isa": 8.63, "completeness": 65.8, "r_meas": 0.112, "cc_half": 0.998, "multiplicity": 11.46, "dmin": 1.24, "dmin_rule": "cc_half_0.30", "dmin_xds": 1.18, "dmax": 999.0}, "tags": ["h5", "lysozyme"]}, - {"id": "lyso_x06da_half_image", "input": "lyso_x06da_half_image/lyso3_7_b81d83_master.h5", "ref": {"sgno": 96, "cell": [77.74, 77.74, 37.386, 90.0, 90.0, 90.0], "anomalous": true, "isa": 6.61, "completeness": 99.6, "r_meas": 0.595, "cc_half": 0.958, "multiplicity": 5.31, "dmin": 1.65, "dmin_rule": "xds_range", "dmin_xds": 1.65, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, - {"id": "thau_x10sa_16keV", "input": "thau_x10sa_16keV/lyso_4_d2bb9b_master.h5", "ref": {"sgno": 92, "cell": [57.958, 57.958, 150.547, 90.0, 90.0, 90.0], "anomalous": false, "isa": 44.51, "completeness": 94.3, "r_meas": 0.072, "cc_half": 1.0, "multiplicity": 18.36, "dmin": 1.3, "dmin_rule": "xds_range", "dmin_xds": 1.3, "dmax": 50.0}, "tags": ["h5", "thaumatin"]}, - {"id": "thau_x10sa_0p1deg", "input": "thau_x10sa_0p1deg/Thau_9_002_master.h5", "ref": {"sgno": 92, "cell": [58.488, 58.488, 150.788, 90.0, 90.0, 90.0], "anomalous": true, "isa": 9.15, "completeness": 99.0, "r_meas": 0.23, "cc_half": 0.997, "multiplicity": 11.37, "dmin": 2.197, "dmin_rule": "xds_range", "dmin_xds": 2.197, "dmax": 50.0}, "tags": ["h5", "thaumatin"]}, - {"id": "thau_bl1a_6p5keV", "input": "thau_bl1a_6p5keV/JTH14_1p9_50ds_90T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.031, 58.031, 150.577, 90.0, 90.0, 90.0], "anomalous": true, "isa": 34.46, "completeness": 91.8, "r_meas": 0.072, "cc_half": 0.999, "multiplicity": 7.95, "dmin": 1.78, "dmin_rule": "xds_range", "dmin_xds": 1.78, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]}, - {"id": "thau_bl1a_4p6keV", "input": "thau_bl1a_4p6keV/JTH14_2p7_50ds_19T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.007, 58.007, 150.511, 90.0, 90.0, 90.0], "anomalous": true, "isa": 35.64, "completeness": 92.0, "r_meas": 0.062, "cc_half": 0.998, "multiplicity": 7.94, "dmin": 2.53, "dmin_rule": "xds_range", "dmin_xds": 2.53, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]}, - {"id": "thau_bl1a_3p8keV", "input": "thau_bl1a_3p8keV/JTH14_3p3_50ds_12T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [57.895, 57.895, 150.213, 90.0, 90.0, 90.0], "anomalous": true, "isa": 30.8, "completeness": 92.2, "r_meas": 0.057, "cc_half": 0.997, "multiplicity": 7.92, "dmin": 3.1, "dmin_rule": "xds_range", "dmin_xds": 3.1, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"], "tiers": {"smoke": "long wavelength (3.3 A), KEK BL-1A thaumatin"}}, - {"id": "thau_x10sa_injection", "input": "thau_x10sa_injection/lyso_13_0c7e7a_master.h5", "ref": {"sgno": 92, "cell": [57.922, 57.922, 150.678, 90.0, 90.0, 90.0], "anomalous": true, "isa": 36.23, "completeness": 86.2, "r_meas": 0.037, "cc_half": 1.0, "multiplicity": 9.96, "dmin": 1.28, "dmin_rule": "xds_range", "dmin_xds": 1.28, "dmax": 50.0}, "tags": ["h5", "thaumatin"]}, - {"id": "thau_micromax_pink", "input": "thau_micromax_pink/Thau-Thaumatin_2_master.h5", "ref": {"sgno": 92, "cell": [58.784, 58.784, 152.226, 90.0, 90.0, 90.0], "anomalous": true, "isa": 21.18, "completeness": 98.5, "r_meas": 0.08, "cc_half": 0.999, "multiplicity": 9.38, "dmin": 1.4, "dmin_rule": "xds_range", "dmin_xds": 1.4, "dmax": 50.0}, "tags": ["h5", "thaumatin", "pink-beam"], "tiers": {"smoke": "pink beam"}}, - {"id": "insu_H_x06da_twin", "input": "insu_H_x06da_twin/Ins_H_2_fa9841_master.h5", "ref": {"sgno": 146, "cell": [81.393, 81.393, 33.272, 90.0, 90.0, 120.0], "anomalous": true, "isa": 6.82, "completeness": 95.0, "r_meas": 0.118, "cc_half": 0.988, "multiplicity": 3.61, "dmin": 1.455, "dmin_rule": "xds_range", "dmin_xds": 1.455, "dmax": 50.0}, "tags": ["h5", "insulin"]}, - {"id": "insu_H_x06da_notwin", "input": "insu_H_x06da_notwin/Ins_H_3_d05684_master.h5", "ref": {"sgno": 146, "cell": [81.378, 81.378, 33.269, 90.0, 90.0, 120.0], "anomalous": true, "isa": 17.69, "completeness": 97.4, "r_meas": 0.068, "cc_half": 0.998, "multiplicity": 3.52, "dmin": 1.544, "dmin_rule": "xds_range", "dmin_xds": 1.544, "dmax": 50.0}, "tags": ["h5", "insulin"], "tiers": {"smoke": "rhombohedral insulin (H3)"}}, - {"id": "insu_I_x06da_weak", "input": "insu_I_x06da_weak/Ins_16_776a92_master.h5", "ref": {"sgno": 197, "cell": [77.085, 77.085, 77.085, 90.0, 90.0, 90.0], "anomalous": false, "isa": 18.91, "completeness": 92.4, "r_meas": 0.401, "cc_half": 0.999, "multiplicity": 29.04, "dmin": 1.807, "dmin_rule": "cc_half_0.30", "dmin_xds": 1.08, "dmax": 999.0}, "tags": ["h5", "insulin"]}, - {"id": "insu_I_x06da_low_isa", "input": "insu_I_x06da_low_isa/Ins_I_2_1a530d_master.h5", "ref": {"sgno": 197, "cell": [77.582, 77.582, 77.582, 90.0, 90.0, 90.0], "anomalous": true, "isa": 4.15, "completeness": 90.8, "r_meas": 0.22, "cc_half": 0.998, "multiplicity": 13.33, "dmin": 1.3, "dmin_rule": "xds_range", "dmin_xds": 1.3, "dmax": 50.0}, "tags": ["h5", "insulin"]}, - {"id": "insu_I_x06da_ref", "input": "insu_I_x06da_ref/Ins_I_3_5e90e3_master.h5", "ref": {"sgno": 197, "cell": [77.52, 77.52, 77.52, 90.0, 90.0, 90.0], "anomalous": true, "isa": 25.08, "completeness": 100.0, "r_meas": 0.349, "cc_half": 0.998, "multiplicity": 13.82, "dmin": 1.621, "dmin_rule": "xds_range", "dmin_xds": 1.621, "dmax": 50.0}, "tags": ["h5", "insulin"]}, - {"id": "insu_I_x06da_13keV", "input": "insu_I_x06da_13keV/Ins_I_6_001_master.h5", "ref": {"sgno": 197, "cell": [77.468, 77.468, 77.468, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.83, "completeness": 100.0, "r_meas": 0.279, "cc_half": 0.998, "multiplicity": 17.21, "dmin": 1.635, "dmin_rule": "xds_range", "dmin_xds": 1.635, "dmax": 50.0}, "tags": ["h5", "insulin"]}, - {"id": "insu_I_x06da_5keV", "input": "insu_I_x06da_5keV/Ins_I_6_006_master.h5", "ref": {"sgno": 197, "cell": [77.785, 77.785, 77.785, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.54, "completeness": 91.7, "r_meas": 0.073, "cc_half": 0.999, "multiplicity": 11.17, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]}, - {"id": "insu_I_x06da_6keV", "input": "insu_I_x06da_6keV/Ins_I_6_004_master.h5", "ref": {"sgno": 197, "cell": [77.71, 77.71, 77.71, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.95, "completeness": 92.4, "r_meas": 0.065, "cc_half": 0.999, "multiplicity": 12.42, "dmin": 2.04, "dmin_rule": "xds_range", "dmin_xds": 2.04, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]}, - {"id": "insu_I_x06da_5keV_2", "input": "insu_I_x06da_5keV_2/Ins_I_7_001_master.h5", "ref": {"sgno": 197, "cell": [77.694, 77.694, 77.694, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.01, "completeness": 91.8, "r_meas": 0.076, "cc_half": 0.999, "multiplicity": 12.73, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"], "tiers": {"smoke": "cubic insulin at 5 keV (long wavelength), fast"}}, - {"id": "cytc_x10sa", "input": "cytc_x10sa/cytC_10_002_master.h5", "ref": {"sgno": 152, "cell": [83.659, 83.659, 86.011, 90.0, 90.0, 120.0], "anomalous": true, "isa": 31.75, "completeness": 99.8, "r_meas": 0.232, "cc_half": 0.999, "multiplicity": 10.65, "dmin": 2.267, "dmin_rule": "cc_half_0.30", "dmin_xds": 2.039, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]}, - {"id": "cytc_x06da_1", "input": "cytc_x06da_1/cytC_2_0770b5_master.h5", "ref": {"sgno": 152, "cell": [83.836, 83.836, 86.733, 90.0, 90.0, 120.0], "anomalous": true, "isa": 24.49, "completeness": 99.9, "r_meas": 0.084, "cc_half": 0.999, "multiplicity": 7.67, "dmin": 1.875, "dmin_rule": "xds_range", "dmin_xds": 1.875, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]}, - {"id": "cytc_x06da_2", "input": "cytc_x06da_2/cytC_3_874285_master.h5", "ref": {"sgno": 152, "cell": [83.719, 83.719, 88.617, 90.0, 90.0, 120.0], "anomalous": true, "isa": 26.98, "completeness": 99.9, "r_meas": 0.097, "cc_half": 0.999, "multiplicity": 7.82, "dmin": 1.69, "dmin_rule": "xds_range", "dmin_xds": 1.69, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]}, - {"id": "myob_x10sa", "input": "myob_x10sa/MyoB_13_110ee0_master.h5", "ref": {"sgno": 4, "cell": [35.229, 28.461, 63.076, 90.0, 106.504, 90.0], "anomalous": true, "isa": 5.21, "completeness": 97.7, "r_meas": 0.247, "cc_half": 0.965, "multiplicity": 3.29, "dmin": 1.742, "dmin_rule": "xds_range", "dmin_xds": 1.742, "dmax": 50.0}, "tags": ["h5", "myoglobin"]}, - {"id": "myob_x06da", "input": "myob_x06da/MyoB_9_446b70_master.h5", "ref": {"sgno": 3, "cell": [35.182, 28.544, 62.83, 90.0, 106.138, 90.0], "anomalous": true, "isa": 7.59, "completeness": 99.4, "r_meas": 0.232, "cc_half": 0.987, "multiplicity": 3.47, "dmin": 1.422, "dmin_rule": "xds_range", "dmin_xds": 1.422, "dmax": 50.0}, "tags": ["h5", "myoglobin"], "tiers": {"smoke": "monoclinic myoglobin, screw not arbitrable by XDS"}}, - {"id": "myob_x06da_powder_1", "input": "myob_x06da_powder_1/MyoB2-1_75979a_master.h5", "ref": {"sgno": 1, "cell": [28.705, 34.796, 64.373, 74.314, 89.879, 90.057], "anomalous": false, "isa": 5.54, "completeness": 74.2, "r_meas": 0.493, "cc_half": 0.966, "multiplicity": 2.87, "dmin": 1.495, "dmin_rule": "xds_range", "dmin_xds": 1.495, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"], "tiers": {"smoke": "known hard: triclinic twin"}}, - {"id": "myob_x06da_powder_2", "input": "myob_x06da_powder_2/MyoB2-2_5ddd7d_master.h5", "ref": {"sgno": 3, "cell": [35.576, 28.811, 63.529, 90.0, 105.593, 90.0], "anomalous": false, "isa": 2.19, "completeness": 58.0, "r_meas": 1.106, "cc_half": 0.655, "multiplicity": 4.71, "dmin": 0.99, "dmin_rule": "xds_range", "dmin_xds": 0.99, "dmax": 999.0}, "tags": ["h5", "myoglobin", "twin"]}, - {"id": "myob_x06da_split", "input": "myob_x06da_split/MyoB2-4_079740_master.h5", "ref": {"sgno": 3, "cell": [35.39, 28.783, 63.626, 90.0, 105.544, 90.0], "anomalous": false, "isa": 12.41, "completeness": 77.7, "r_meas": 0.577, "cc_half": 0.984, "multiplicity": 4.38, "dmin": 1.506, "dmin_rule": "xds_range", "dmin_xds": 1.506, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"]}, - {"id": "myob_x06da_sparse", "input": "myob_x06da_sparse/MyoB2-5_7cef9c_master.h5", "ref": {"sgno": 3, "cell": [35.388, 28.752, 64.083, 90.0, 106.352, 90.0], "anomalous": false, "isa": 5.46, "completeness": 73.7, "r_meas": 0.326, "cc_half": 0.972, "multiplicity": 5.21, "dmin": 2.0, "dmin_rule": "xds_range", "dmin_xds": 2.0, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"]}, + {"id": "lyso_x10sa_90deg_1", "input": "lyso_x10sa_90deg_1/lyso_2_006_master.h5", "ref": {"sgno": 92, "cell": [76.879, 76.879, 36.961, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.75, "completeness": 99.3, "r_meas": 0.182, "cc_half": 0.995, "multiplicity": 3.51, "dmin_low": 5.83, "r_meas_low": 0.049, "dmin": 1.966, "dmin_rule": "xds_range", "dmin_xds": 1.966, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, + {"id": "lyso_x10sa_90deg_2", "input": "lyso_x10sa_90deg_2/lyso_2_007_master.h5", "ref": {"sgno": 92, "cell": [76.904, 76.904, 36.976, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.14, "completeness": 99.3, "r_meas": 0.18, "cc_half": 0.994, "multiplicity": 3.51, "dmin_low": 5.86, "r_meas_low": 0.051, "dmin": 1.973, "dmin_rule": "xds_range", "dmin_xds": 1.973, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, + {"id": "lyso_x06da_atten_wedge", "input": "lyso_x06da_atten_wedge/lyso_2_002_master.h5", "ref": {"sgno": 92, "cell": [78.235, 78.235, 37.798, 90.0, 90.0, 90.0], "anomalous": true, "isa": 16.56, "completeness": 99.9, "r_meas": 0.239, "cc_half": 0.998, "multiplicity": 10.97, "dmin_low": 3.76, "r_meas_low": 0.069, "dmin": 1.264, "dmin_rule": "xds_range", "dmin_xds": 1.264, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, + {"id": "lyso_x06da_5keV", "input": "lyso_x06da_5keV/lyso_6_001_master.h5", "ref": {"sgno": 92, "cell": [78.225, 78.225, 37.897, 90.0, 90.0, 90.0], "anomalous": true, "isa": 19.36, "completeness": 86.4, "r_meas": 0.064, "cc_half": 0.998, "multiplicity": 8.85, "dmin_low": 7.22, "r_meas_low": 0.047, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "lysozyme", "long-wavelength"]}, + {"id": "lyso_x06da_ice", "input": "lyso_x06da_ice/lysoC_14_56b5b9_master.h5", "ref": {"sgno": 89, "cell": [78.387, 78.387, 37.427, 90.0, 90.0, 90.0], "anomalous": true, "isa": 23.33, "completeness": 100.0, "r_meas": 0.219, "cc_half": 0.998, "multiplicity": 11.86, "dmin_low": 4.26, "r_meas_low": 0.055, "dmin": 1.431, "dmin_rule": "xds_range", "dmin_xds": 1.431, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, + {"id": "lyso_x06da_ref", "input": "lyso_x06da_ref/hewl_35_001_master.h5", "ref": {"sgno": 89, "cell": [78.091, 78.091, 37.704, 90.0, 90.0, 90.0], "anomalous": true, "isa": 28.31, "completeness": 100.0, "r_meas": 0.045, "cc_half": 1.0, "multiplicity": 13.76, "dmin_low": 3.58, "r_meas_low": 0.029, "dmin": 1.2, "dmin_rule": "xds_range", "dmin_xds": 1.2, "dmax": 50.0}, "tags": ["h5", "lysozyme"], "tiers": {"smoke": "lysozyme reference set"}}, + {"id": "lyso_x10sa_strong", "input": "lyso_x10sa_strong/lyso_5_f6ac7e_master.h5", "ref": {"sgno": 92, "cell": [77.769, 77.769, 37.442, 90.0, 90.0, 90.0], "anomalous": false, "isa": 8.63, "completeness": 65.8, "r_meas": 0.112, "cc_half": 0.998, "multiplicity": 11.46, "dmin_low": 3.52, "r_meas_low": 0.077, "dmin": 1.24, "dmin_rule": "cc_half_0.30", "dmin_xds": 1.18, "dmax": 999.0}, "tags": ["h5", "lysozyme"]}, + {"id": "lyso_x06da_half_image", "input": "lyso_x06da_half_image/lyso3_7_b81d83_master.h5", "ref": {"sgno": 96, "cell": [77.74, 77.74, 37.386, 90.0, 90.0, 90.0], "anomalous": true, "isa": 6.61, "completeness": 99.6, "r_meas": 0.595, "cc_half": 0.958, "multiplicity": 5.31, "dmin_low": 4.91, "r_meas_low": 0.258, "dmin": 1.65, "dmin_rule": "xds_range", "dmin_xds": 1.65, "dmax": 50.0}, "tags": ["h5", "lysozyme"]}, + {"id": "thau_x10sa_16keV", "input": "thau_x10sa_16keV/lyso_4_d2bb9b_master.h5", "ref": {"sgno": 92, "cell": [57.958, 57.958, 150.547, 90.0, 90.0, 90.0], "anomalous": false, "isa": 44.51, "completeness": 94.3, "r_meas": 0.072, "cc_half": 1.0, "multiplicity": 18.36, "dmin_low": 3.89, "r_meas_low": 0.027, "dmin": 1.3, "dmin_rule": "xds_range", "dmin_xds": 1.3, "dmax": 50.0}, "tags": ["h5", "thaumatin"]}, + {"id": "thau_x10sa_0p1deg", "input": "thau_x10sa_0p1deg/Thau_9_002_master.h5", "ref": {"sgno": 92, "cell": [58.488, 58.488, 150.788, 90.0, 90.0, 90.0], "anomalous": true, "isa": 9.15, "completeness": 99.0, "r_meas": 0.23, "cc_half": 0.997, "multiplicity": 11.37, "dmin_low": 6.51, "r_meas_low": 0.104, "dmin": 2.197, "dmin_rule": "xds_range", "dmin_xds": 2.197, "dmax": 50.0}, "tags": ["h5", "thaumatin"]}, + {"id": "thau_bl1a_6p5keV", "input": "thau_bl1a_6p5keV/JTH14_1p9_50ds_90T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.031, 58.031, 150.577, 90.0, 90.0, 90.0], "anomalous": true, "isa": 34.46, "completeness": 91.8, "r_meas": 0.072, "cc_half": 0.999, "multiplicity": 7.95, "dmin_low": 5.33, "r_meas_low": 0.041, "dmin": 1.78, "dmin_rule": "xds_range", "dmin_xds": 1.78, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]}, + {"id": "thau_bl1a_4p6keV", "input": "thau_bl1a_4p6keV/JTH14_2p7_50ds_19T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.007, 58.007, 150.511, 90.0, 90.0, 90.0], "anomalous": true, "isa": 35.64, "completeness": 92.0, "r_meas": 0.062, "cc_half": 0.998, "multiplicity": 7.94, "dmin_low": 7.56, "r_meas_low": 0.039, "dmin": 2.53, "dmin_rule": "xds_range", "dmin_xds": 2.53, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]}, + {"id": "thau_bl1a_3p8keV", "input": "thau_bl1a_3p8keV/JTH14_3p3_50ds_12T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [57.895, 57.895, 150.213, 90.0, 90.0, 90.0], "anomalous": true, "isa": 30.8, "completeness": 92.2, "r_meas": 0.057, "cc_half": 0.997, "multiplicity": 7.92, "dmin_low": 9.23, "r_meas_low": 0.039, "dmin": 3.1, "dmin_rule": "xds_range", "dmin_xds": 3.1, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"], "tiers": {"smoke": "long wavelength (3.3 A), KEK BL-1A thaumatin"}}, + {"id": "thau_x10sa_injection", "input": "thau_x10sa_injection/lyso_13_0c7e7a_master.h5", "ref": {"sgno": 92, "cell": [57.922, 57.922, 150.678, 90.0, 90.0, 90.0], "anomalous": true, "isa": 36.23, "completeness": 86.2, "r_meas": 0.037, "cc_half": 1.0, "multiplicity": 9.96, "dmin_low": 3.83, "r_meas_low": 0.022, "dmin": 1.28, "dmin_rule": "xds_range", "dmin_xds": 1.28, "dmax": 50.0}, "tags": ["h5", "thaumatin"]}, + {"id": "thau_micromax_pink", "input": "thau_micromax_pink/Thau-Thaumatin_2_master.h5", "ref": {"sgno": 92, "cell": [58.784, 58.784, 152.226, 90.0, 90.0, 90.0], "anomalous": true, "isa": 21.18, "completeness": 98.5, "r_meas": 0.08, "cc_half": 0.999, "multiplicity": 9.38, "dmin_low": 4.17, "r_meas_low": 0.04, "dmin": 1.4, "dmin_rule": "xds_range", "dmin_xds": 1.4, "dmax": 50.0}, "tags": ["h5", "thaumatin", "pink-beam"], "tiers": {"smoke": "pink beam"}}, + {"id": "insu_H_x06da_twin", "input": "insu_H_x06da_twin/Ins_H_2_fa9841_master.h5", "ref": {"sgno": 146, "cell": [81.393, 81.393, 33.272, 90.0, 90.0, 120.0], "anomalous": true, "isa": 6.82, "completeness": 95.0, "r_meas": 0.118, "cc_half": 0.988, "multiplicity": 3.61, "dmin_low": 4.34, "r_meas_low": 0.106, "dmin": 1.455, "dmin_rule": "xds_range", "dmin_xds": 1.455, "dmax": 50.0}, "tags": ["h5", "insulin"]}, + {"id": "insu_H_x06da_notwin", "input": "insu_H_x06da_notwin/Ins_H_3_d05684_master.h5", "ref": {"sgno": 146, "cell": [81.378, 81.378, 33.269, 90.0, 90.0, 120.0], "anomalous": true, "isa": 17.69, "completeness": 97.4, "r_meas": 0.068, "cc_half": 0.998, "multiplicity": 3.52, "dmin_low": 4.61, "r_meas_low": 0.045, "dmin": 1.544, "dmin_rule": "xds_range", "dmin_xds": 1.544, "dmax": 50.0}, "tags": ["h5", "insulin"], "tiers": {"smoke": "rhombohedral insulin (H3)"}}, + {"id": "insu_I_x06da_weak", "input": "insu_I_x06da_weak/Ins_16_776a92_master.h5", "ref": {"sgno": 197, "cell": [77.085, 77.085, 77.085, 90.0, 90.0, 90.0], "anomalous": false, "isa": 18.91, "completeness": 92.4, "r_meas": 0.401, "cc_half": 0.999, "multiplicity": 29.04, "dmin_low": 3.22, "r_meas_low": 0.054, "dmin": 1.807, "dmin_rule": "cc_half_0.30", "dmin_xds": 1.08, "dmax": 999.0}, "tags": ["h5", "insulin"]}, + {"id": "insu_I_x06da_low_isa", "input": "insu_I_x06da_low_isa/Ins_I_2_1a530d_master.h5", "ref": {"sgno": 197, "cell": [77.582, 77.582, 77.582, 90.0, 90.0, 90.0], "anomalous": true, "isa": 4.15, "completeness": 90.8, "r_meas": 0.22, "cc_half": 0.998, "multiplicity": 13.33, "dmin_low": 3.92, "r_meas_low": 0.167, "dmin": 1.3, "dmin_rule": "xds_range", "dmin_xds": 1.3, "dmax": 50.0}, "tags": ["h5", "insulin"]}, + {"id": "insu_I_x06da_ref", "input": "insu_I_x06da_ref/Ins_I_3_5e90e3_master.h5", "ref": {"sgno": 197, "cell": [77.52, 77.52, 77.52, 90.0, 90.0, 90.0], "anomalous": true, "isa": 25.08, "completeness": 100.0, "r_meas": 0.349, "cc_half": 0.998, "multiplicity": 13.82, "dmin_low": 4.82, "r_meas_low": 0.143, "dmin": 1.621, "dmin_rule": "xds_range", "dmin_xds": 1.621, "dmax": 50.0}, "tags": ["h5", "insulin"]}, + {"id": "insu_I_x06da_13keV", "input": "insu_I_x06da_13keV/Ins_I_6_001_master.h5", "ref": {"sgno": 197, "cell": [77.468, 77.468, 77.468, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.83, "completeness": 100.0, "r_meas": 0.279, "cc_half": 0.998, "multiplicity": 17.21, "dmin_low": 4.85, "r_meas_low": 0.093, "dmin": 1.635, "dmin_rule": "xds_range", "dmin_xds": 1.635, "dmax": 50.0}, "tags": ["h5", "insulin"]}, + {"id": "insu_I_x06da_5keV", "input": "insu_I_x06da_5keV/Ins_I_6_006_master.h5", "ref": {"sgno": 197, "cell": [77.785, 77.785, 77.785, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.54, "completeness": 91.7, "r_meas": 0.073, "cc_half": 0.999, "multiplicity": 11.17, "dmin_low": 7.21, "r_meas_low": 0.048, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]}, + {"id": "insu_I_x06da_6keV", "input": "insu_I_x06da_6keV/Ins_I_6_004_master.h5", "ref": {"sgno": 197, "cell": [77.71, 77.71, 77.71, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.95, "completeness": 92.4, "r_meas": 0.065, "cc_half": 0.999, "multiplicity": 12.42, "dmin_low": 6.03, "r_meas_low": 0.047, "dmin": 2.04, "dmin_rule": "xds_range", "dmin_xds": 2.04, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]}, + {"id": "insu_I_x06da_5keV_2", "input": "insu_I_x06da_5keV_2/Ins_I_7_001_master.h5", "ref": {"sgno": 197, "cell": [77.694, 77.694, 77.694, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.01, "completeness": 91.8, "r_meas": 0.076, "cc_half": 0.999, "multiplicity": 12.73, "dmin_low": 7.21, "r_meas_low": 0.048, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"], "tiers": {"smoke": "cubic insulin at 5 keV (long wavelength), fast"}}, + {"id": "cytc_x10sa", "input": "cytc_x10sa/cytC_10_002_master.h5", "ref": {"sgno": 152, "cell": [83.659, 83.659, 86.011, 90.0, 90.0, 120.0], "anomalous": true, "isa": 31.75, "completeness": 99.8, "r_meas": 0.232, "cc_half": 0.999, "multiplicity": 10.65, "dmin_low": 6.05, "r_meas_low": 0.036, "dmin": 2.267, "dmin_rule": "cc_half_0.30", "dmin_xds": 2.039, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]}, + {"id": "cytc_x06da_1", "input": "cytc_x06da_1/cytC_2_0770b5_master.h5", "ref": {"sgno": 152, "cell": [83.836, 83.836, 86.733, 90.0, 90.0, 120.0], "anomalous": true, "isa": 24.49, "completeness": 99.9, "r_meas": 0.084, "cc_half": 0.999, "multiplicity": 7.67, "dmin_low": 5.58, "r_meas_low": 0.033, "dmin": 1.875, "dmin_rule": "xds_range", "dmin_xds": 1.875, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]}, + {"id": "cytc_x06da_2", "input": "cytc_x06da_2/cytC_3_874285_master.h5", "ref": {"sgno": 152, "cell": [83.719, 83.719, 88.617, 90.0, 90.0, 120.0], "anomalous": true, "isa": 26.98, "completeness": 99.9, "r_meas": 0.097, "cc_half": 0.999, "multiplicity": 7.82, "dmin_low": 5.03, "r_meas_low": 0.032, "dmin": 1.69, "dmin_rule": "xds_range", "dmin_xds": 1.69, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]}, + {"id": "myob_x10sa", "input": "myob_x10sa/MyoB_13_110ee0_master.h5", "ref": {"sgno": 4, "cell": [35.229, 28.461, 63.076, 90.0, 106.504, 90.0], "anomalous": true, "isa": 5.21, "completeness": 97.7, "r_meas": 0.247, "cc_half": 0.965, "multiplicity": 3.29, "dmin_low": 5.16, "r_meas_low": 0.186, "dmin": 1.742, "dmin_rule": "xds_range", "dmin_xds": 1.742, "dmax": 50.0}, "tags": ["h5", "myoglobin"]}, + {"id": "myob_x06da", "input": "myob_x06da/MyoB_9_446b70_master.h5", "ref": {"sgno": 3, "cell": [35.182, 28.544, 62.83, 90.0, 106.138, 90.0], "anomalous": true, "isa": 7.59, "completeness": 99.4, "r_meas": 0.232, "cc_half": 0.987, "multiplicity": 3.47, "dmin_low": 4.23, "r_meas_low": 0.105, "dmin": 1.422, "dmin_rule": "xds_range", "dmin_xds": 1.422, "dmax": 50.0}, "tags": ["h5", "myoglobin"], "tiers": {"smoke": "monoclinic myoglobin, screw not arbitrable by XDS"}}, + {"id": "myob_x06da_powder_1", "input": "myob_x06da_powder_1/MyoB2-1_75979a_master.h5", "ref": {"sgno": 1, "cell": [28.705, 34.796, 64.373, 74.314, 89.879, 90.057], "anomalous": false, "isa": 5.54, "completeness": 74.2, "r_meas": 0.493, "cc_half": 0.966, "multiplicity": 2.87, "dmin_low": 4.44, "r_meas_low": 0.125, "dmin": 1.495, "dmin_rule": "xds_range", "dmin_xds": 1.495, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"], "tiers": {"smoke": "known hard: triclinic twin"}}, + {"id": "myob_x06da_powder_2", "input": "myob_x06da_powder_2/MyoB2-2_5ddd7d_master.h5", "ref": {"sgno": 3, "cell": [35.576, 28.811, 63.529, 90.0, 105.593, 90.0], "anomalous": false, "isa": 2.19, "completeness": 58.0, "r_meas": 1.106, "cc_half": 0.655, "multiplicity": 4.71, "dmin_low": 2.98, "r_meas_low": 0.706, "dmin": 0.99, "dmin_rule": "xds_range", "dmin_xds": 0.99, "dmax": 999.0}, "tags": ["h5", "myoglobin", "twin"]}, + {"id": "myob_x06da_split", "input": "myob_x06da_split/MyoB2-4_079740_master.h5", "ref": {"sgno": 3, "cell": [35.39, 28.783, 63.626, 90.0, 105.544, 90.0], "anomalous": false, "isa": 12.41, "completeness": 77.7, "r_meas": 0.577, "cc_half": 0.984, "multiplicity": 4.38, "dmin_low": 4.48, "r_meas_low": 0.091, "dmin": 1.506, "dmin_rule": "xds_range", "dmin_xds": 1.506, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"]}, + {"id": "myob_x06da_sparse", "input": "myob_x06da_sparse/MyoB2-5_7cef9c_master.h5", "ref": {"sgno": 3, "cell": [35.388, 28.752, 64.083, 90.0, 106.352, 90.0], "anomalous": false, "isa": 5.46, "completeness": 73.7, "r_meas": 0.326, "cc_half": 0.972, "multiplicity": 5.21, "dmin_low": 5.92, "r_meas_low": 0.146, "dmin": 2.0, "dmin_rule": "xds_range", "dmin_xds": 2.0, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"]}, {"id": "nothing_2", "input": "nothing_2/test-28_3400ac_master.h5", "expect": "no_lattice", "tags": ["h5", "control"]}, {"id": "nothing_1", "input": "nothing_1/test-28_9bf604_master.h5", "expect": "no_lattice", "tags": ["h5", "control"]} ]} diff --git a/tools/battery/inputs.py b/tools/battery/inputs.py index 9d37cef7a..71012ccf5 100644 --- a/tools/battery/inputs.py +++ b/tools/battery/inputs.py @@ -107,7 +107,8 @@ def input_format(path): def parse_correct_lp(path): """Reference values from an XDS CORRECT.LP: the space group and cell CORRECT refined, the resolution range it merged (dmin_xds, dmax), the reference d_min read off its statistics - (dmin, dmin_rule - see reference_dmin) and its overall merging statistics. + (dmin, dmin_rule - see reference_dmin), its overall merging statistics, and the R_meas of its + lowest-resolution shell (r_meas_low, over dmax to dmin_low). CORRECT.LP states the group and cell twice - first as the INPUT to INTEGRATE (routinely P1), then as CORRECT's own answer. The last SPACE_GROUP_NUMBER and the refined UNIT CELL PARAMETERS @@ -143,6 +144,11 @@ def parse_correct_lp(path): ref["r_meas"] = round(num(total[9]) / 100, 4) ref["cc_half"] = round(num(total[10]) / 100, 4) ref["multiplicity"] = round(int(total[1]) / int(total[2]), 2) if int(total[2]) else None + if shells: + # the lowest-resolution shell, dmax to dmin_low: strong reflections only, so its R_meas + # is the merge's accuracy rather than where the signal fades + ref["dmin_low"] = float(shells[0][0]) + ref["r_meas_low"] = round(num(shells[0][9]) / 100, 4) # The range XDS merged: the explicit INCLUDE_RESOLUTION_RANGE, or - when XDS.INP left the high # limit at 0.0, "the whole detector" - the finest shell actually tabulated. The pooled statistics From 342ef5f66ae6e546b3bd6d0a057bebf66ba767be Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 18:32:15 +0200 Subject: [PATCH 27/28] tools/battery: merge quality is CC1/2 against XDS only The owner's scoring decision: no R_meas criterion, and merge quality judged only by CC1/2 relative to XDS. - The global rule (R_meas > 60% or CC1/2 < 0.5 fails) is gone. - XDS arms: a set fails `merge` when (1/CC1/2 - 1) of the reference-range table is more than twice XDS's, XDS's CC1/2 taken at the bottom of its 0.1% rounding. CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S at any CC1/2, and E goes as 1/observations: 2x is XDS's merge with half its observations. Stored as cc_half_noise_ratio. No REFRES table or no XDS CC1/2: no criterion. - Open arm: no merge criterion; CC1/2 and R_meas stay reported numbers. - Low-resolution R_meas is reported, not scored: the lowest shell of the own table, the reference-range table and XDS's (lowres_*), with lowres_r_meas_ratio in the like-for-like table and a ratio plot. - Every schema-3 run is re-scored when it is read (report, compare, baseline delta) with today's scorer and today's manifest rows, so both sides of a comparison are scored alike. results.json keeps the verdicts as scored at run time; rows without a lattice keep them. Re-scoring the rc172 reference run: inhouse 29 -> 27 pass (three new merge fails, one R_meas fail lifted), open 137 -> 138 (one R_meas fail lifted). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- tools/battery/README.md | 50 +++++++++++++++++++++++------ tools/battery/battery.py | 2 ++ tools/battery/report.py | 65 +++++++++++++++++++++++++++++++++---- tools/battery/score.py | 69 ++++++++++++++++++++++++++++++++-------- 4 files changed, 156 insertions(+), 30 deletions(-) diff --git a/tools/battery/README.md b/tools/battery/README.md index 65d6c2005..80d33c4b0 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -15,14 +15,35 @@ with each other. | **inhouse** | standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), plus no-crystal controls | XDS, from the `CORRECT.LP` beside each dataset | `inhouse.json` (committed) | | **private** | user data | XDS, like inhouse | outside the repository; the local site config gives its path | -Scoring checks four things in order, and the first one that fails decides the verdict: did it -run, is the lattice right, is the symmetry right, is the merge usable (R_meas at most 60%, CC1/2 at -least 0.5). The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio +Scoring checks these things in order, and the first one that fails decides the verdict: did it +run, is the lattice right, is the symmetry right, and on the XDS arms, is the merge as good as +XDS's. The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is scored with `sgequiv`. On the XDS arms only the point group is scored, because XDS never tests a screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported. `score.py` has the details. +**Merge quality is judged only against XDS, and only by CC1/2.** No R_meas is a pass/fail +criterion, and no absolute CC1/2 either. On the XDS arms the set fails `merge` when + + (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) > 2 + +with rugnux's CC1/2 from the reference-range table (`REFRES_CC_HALF`, XDS's own range) and XDS's +from the `CORRECT.LP` total. CC1/2 = S / (S + E), with S the signal variance and E the half-set +error variance, so 1/CC1/2 - 1 = E/S at any CC1/2, and E goes as 1/(observations): a ratio above 2 +is a merge noisier than XDS's would be with half of its observations. XDS prints CC1/2 to 0.1%, so +its value is taken at the bottom of that rounding (-0.0005), and a printed 100.0 does not demand a +perfect CC1/2. The ratio is stored as `cc_half_noise_ratio`. A set without the reference-range +table or without an XDS CC1/2 (the controls, a hand-measured reference) has no merge criterion. +**The open arm has no merge criterion at all**: it is scored on lattice and symmetry, and its +resolution, CC1/2 and R_meas are reported, not judged. + +**Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own +table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's +`CORRECT.LP` (`lowres_r_meas_ref`, stored in the manifest as `r_meas_low`), with the shells' +high-resolution limits. The reference-range table uses XDS's shell limits, so its lowest shell and +XDS's cover the same reflections; `lowres_r_meas_ratio` is the one over the other, plotted per set. + ## One command per arm Every set runs **once**, with one command per arm: @@ -52,9 +73,9 @@ the reference: - **`--report-resolution`** (XDS arms): a second statistics table from the same merge, binned over XDS's own range (`dmin_xds` and XDS's d_max as given, 999 included, so it covers exactly what XDS's CORRECT.LP totals cover), and `-A` where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the - manifest), so the two tables count the same way. Its REFRES_* numbers are scored against XDS's - CORRECT.LP totals: completeness, multiplicity, R_meas, CC1/2 and ISa (I/sigma is shown; the - manifest holds no XDS value for it). Where rugnux's own cut is coarser than the reference + manifest), so the two tables count the same way. Its REFRES_* numbers are shown beside XDS's + CORRECT.LP totals: completeness, multiplicity, R_meas, the lowest shell's R_meas, CC1/2 and ISa + (I/sigma is shown; the manifest holds no XDS value for it). Only CC1/2 is scored (above). Where rugnux's own cut is coarser than the reference (`REFRES_SHELLS_PAST_LIMIT` > 0), the shells past it are not merged, so the completeness is read as **coverage** of the reference range, not as a quality regression, and the other numbers are over the shells rugnux reached. @@ -211,6 +232,13 @@ This is a flat list with one object per set, and every arm uses the same keys. M `null`. `manifest.json` records the schema version as `results_schema`, and each arm's command as `command_doc`. +**Verdicts are scored again whenever a run is read** (`report`, `compare`, the baseline delta): +every row whose `p_report.txt` has a lattice (or that is a control) goes through today's +`score.judge` with today's manifest row (reference, `unscored`, `expect`), so the two sides of a +comparison are always scored by the same rules against the same references. A row whose report +has no lattice keeps its stored verdict, because why it failed came from the run. `results.json` +keeps the verdicts as they were scored when the run finished. + Schema 3 is one row per set again: `variant` and `first_read` are gone, and `refres_*`, `isa_ratio`, `r_meas_ratio`, `multiplicity_ref`, `cc_model`, `sg_label`, `model` and `model_note` are new. `sgno`/`sg` are now the data's own space group where `--model` relabelled the hand. @@ -228,7 +256,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. |---|---| | `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran | | `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) | -| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` | +| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge` (XDS arms: CC1/2 noise ratio above 2), `false_lattice`; or `no_reference` / `no_input` / `reference_problem` | | `reason` | one line for a human | | `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours (the data's own determination) / the reference's | | `sg_label` | the space group rugnux reported where `--model` put the model's enantiomorph on it, else null | @@ -240,6 +268,8 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. | `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref`, `multiplicity_ref` | XDS's overall values, over XDS's own range | | `refres_range`, `refres_shells_past_limit`, `refres_unique_reflections`, `refres_completeness`, `refres_multiplicity`, `refres_i_over_sigma`, `refres_r_meas`, `refres_cc_half`, `refres_isa` | XDS arms: the REFRES_* keys, the same merge over the reference range | | `isa_ratio`, `r_meas_ratio` | refres_isa / isa_ref and refres_r_meas / r_meas_ref | +| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), the merge criterion | +| `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref | | `model`, `model_note` | open arm: the deposited coordinates given to `--model`, or why there were none | | `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited | | `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm) | @@ -256,8 +286,8 @@ SVG, no external files): - a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of the main metrics; - plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference) - for each arm; on the XDS arms the reference-range ISa and R_meas over XDS's; on the open arm - R_free / published R_free; + for each arm; on the XDS arms the reference-range ISa, R_meas, lowest-shell R_meas and CC1/2 + noise over XDS's; on the open arm R_free / published R_free; - on the XDS arms the **like-for-like table**: the reference-range numbers beside XDS's, with the rows where rugnux's cut is coarser marked as coverage; - the **failures**, and **one row per set** with all the numbers; @@ -286,7 +316,7 @@ $B compare RUN_A RUN_B --rerun-changed # rerun the changed sets wit `compare` pairs rows by arm and set id, following the manifests' `aliases` across renames. A run from before schema 3 is compared through its `bare` rows (see the schema above). It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell, R-free, -reference-range R_meas or ISa (where both runs have them), or time moved by more than the noise +reference-range R_meas, lowest-shell R_meas or ISa (where both runs have them), or time moved by more than the noise thresholds in `report.py` (`NOISE`). Those thresholds are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with A's saved binary and options. A set that moves again under the same binary is noise. A set that diff --git a/tools/battery/battery.py b/tools/battery/battery.py index 27ae2c150..72ed37663 100644 --- a/tools/battery/battery.py +++ b/tools/battery/battery.py @@ -662,6 +662,8 @@ def main(): a = ap.parse_args() site = load_site(a.site) report.ALIASES = load_aliases(site) + report.MANIFESTS = {arm: {e["id"]: e for e in load_sets(site, arm)} + for arm, cfg in site["arms"].items() if os.path.exists(cfg["manifest"])} {"run": cmd_run, "list": cmd_list, "compare": cmd_compare, "report": cmd_report, "abort": cmd_abort, "discover": cmd_discover, "refs": cmd_refs, "remap": cmd_remap}[a.cmd](a, site) diff --git a/tools/battery/report.py b/tools/battery/report.py index df86c2e13..23ea0089a 100644 --- a/tools/battery/report.py +++ b/tools/battery/report.py @@ -5,6 +5,7 @@ import socket import statistics import sys +import score from render import Doc, ratio_dots, verdict_bars # A change smaller than these is treated as run-to-run noise. They are a starting point, to be @@ -19,6 +20,7 @@ NOISE = { "rfree": ("abs", 0.005), "refres_r_meas": ("rel", 0.05), "refres_isa": ("rel", 0.05), + "refres_lowres_r_meas": ("rel", 0.05), "wall_s": ("rel", 0.25), # and at least 10 s - see beyond_noise() } @@ -53,9 +55,27 @@ def load_run(run_dir, allow_incomplete=False): schema1(man, res) if schema < 3: res[:] = schema2(res) + else: + rescore(run_dir, man, res) return man, res +def rescore(run_dir, man, res): + """Score every row again from its saved report, with today's scorer and today's manifest rows + (references, `unscored`, `expect`), so a report or a compare never mixes two scorings or two + references. results.json keeps the verdicts as they were scored when the run finished. + + A row whose report has no lattice keeps its verdict: why it failed (crash, reader, timeout) + came from the run, not from the report.""" + own = {(e["arm"], e["id"]): e for e in man["sets"]} + for r in res: + e = MANIFESTS.get(r["arm"], {}).get(key(r)[1]) or own.get((r["arm"], r["set"])) + rep = score.read_report(os.path.join(run_dir, "work", r["arm"], r["set"], "p_report.txt")) + lattice = rep.get("SPACE_GROUP_NUMBER") and rep.get("UNIT_CELL_CONSTANTS") + if e and (lattice or e.get("expect") == "no_lattice"): + r.update(score.judge(dict(e, arm=r["arm"]), rep, "")) + + def schema1(man, res): """Read a run from before the variants (results schema 1) as one: it ran each set once, plain on the open arm and with XDS's settings on the XDS arms - unless --unforced, which left the @@ -114,6 +134,9 @@ RESULTS_SCHEMA = 3 # dataset was renamed still pairs with one made after ALIASES = {} +# {arm: {set id: manifest row}} of today's manifests (battery.py sets it); rescore() scores with them +MANIFESTS = {} + def key(r): return (r["arm"], ALIASES.get(r["arm"], {}).get(r["set"], r["set"])) @@ -166,7 +189,7 @@ def changes(ra, rb): if beyond_noise(name, ra.get(name), rb.get(name)): out.append(name) # the like-for-like table only where both runs have one (a pre-schema-3 run has none) - for name in ("refres_r_meas", "refres_isa"): + for name in ("refres_r_meas", "refres_isa", "refres_lowres_r_meas"): if ra.get(name) is not None and rb.get(name) is not None and beyond_noise(name, ra[name], rb[name]): out.append(name) if beyond_noise("wall_s", ra.get("wall_s"), rb.get("wall_s")): @@ -192,7 +215,7 @@ def arrow(x, y, fmt=lambda v: str(v)): def compare_table(doc, rows, only_changed=True): head = ["set", "arm", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2", - "cell dev %", "R_free", "ref-range R_meas", "time s", "beyond noise"] + "cell dev %", "R_free", "ref-range R_meas", "ref-range low-res R_meas", "time s", "beyond noise"] out = [] for (arm, s), ra, rb, ch in rows: if only_changed and not ch: @@ -207,6 +230,8 @@ def compare_table(doc, rows, only_changed=True): arrow(ra.get("cell_dev_pct"), rb.get("cell_dev_pct"), lambda v: f"{v:.2f}"), arrow(ra.get("rfree"), rb.get("rfree"), lambda v: f"{v:.3f}"), arrow(ra.get("refres_r_meas"), rb.get("refres_r_meas"), lambda v: f"{100 * v:.1f}%"), + arrow(ra.get("refres_lowres_r_meas"), rb.get("refres_lowres_r_meas"), + lambda v: f"{100 * v:.1f}%"), arrow(ra.get("wall_s"), rb.get("wall_s"), lambda v: f"{v:.0f}"), ", ".join(ch)]) doc.table(head, out, row_class=lambda r: "fail" if "verdict" in r[-1] else "") @@ -301,6 +326,9 @@ def build(run_dir, baseline=None, allow_incomplete=False): ("cc_half", "CC1/2", "{:.3f}"), ("res_gain_pct", "res. gain %", "{:+.1f}"), ("isa_ratio", "ref-range ISa / XDS", "{:.3f}"), ("r_meas_ratio", "ref-range R_meas / XDS", "{:.3f}"), + ("lowres_r_meas", "low-res shell R_meas", None), + ("lowres_r_meas_ratio", "ref-range low-res R_meas / XDS", "{:.3f}"), + ("cc_half_noise_ratio", "ref-range CC1/2 noise / XDS", "{:.2f}"), ("rfree", "R_free (placement)", "{:.3f}"), ("rfree_ratio", "R_free ratio", "{:.3f}"), ("wall_s", "time s", "{:.0f}")): q = quartiles(r.get(name) for r in rs) @@ -322,7 +350,13 @@ def build(run_dir, baseline=None, allow_incomplete=False): ("isa_ratio", "ISa", "ISa of the reference-range table (the error model refitted on it) / " "XDS's ISa; above 1 = rugnux's error model is the better one"), ("r_meas_ratio", "R_meas", "R_meas over the reference range / XDS's R_meas; below 1 = " - "rugnux's merge is the more consistent one")): + "rugnux's merge is the more consistent one"), + ("lowres_r_meas_ratio", "low-res R_meas", "R_meas of the lowest-resolution shell of the " + "reference-range table / of XDS's table; below 1 = " + "rugnux's strong reflections agree better"), + ("cc_half_noise_ratio", "CC1/2 noise", "half-set noise over the reference range read off CC1/2 " + "(1 / CC1/2 - 1) / XDS's; above 2 fails (the merge is as " + "noisy as XDS's with half its observations)")): for arm in arms: pts = [(r["set"], r[name], r["verdict"], f"{what} ratio, {refres_note(r)}") for r in res if r["arm"] == arm and r.get(name)] @@ -363,11 +397,21 @@ def build(run_dir, baseline=None, allow_incomplete=False): f(r.get("refres_multiplicity"), "{:.1f}") + " / " + f(r.get("multiplicity_ref"), "{:.1f}"), f(r.get("refres_i_over_sigma"), "{:.1f}"), pct(r.get("refres_r_meas")) + " / " + pct(r.get("r_meas_ref")), - f(r.get("refres_cc_half"), "{:.3f}") + " / " + f(r.get("cc_half_ref"), "{:.3f}"), + lowres_cell(r), + f(r.get("refres_cc_half"), "{:.4f}") + " / " + f(r.get("cc_half_ref"), "{:.3f}"), + f(r.get("cc_half_noise_ratio"), "{:.2f}"), f(r.get("refres_isa"), "{:.1f}") + " / " + f(r.get("isa_ref"), "{:.1f}"), refres_note(r)]) doc.table(["set", "arm", "range A", "own d_min", "compl % (rugnux / XDS)", "mult", "I/sigma", - "R_meas", "CC1/2", "ISa", "reading"], rows, num=(3, 6)) + "R_meas", "low-res R_meas (to d A)", "CC1/2", "CC1/2 noise ratio", "ISa", "reading"], + rows, num=(3, 6, 10)) + doc.p("Merge quality is scored here and only here, by CC1/2 against XDS's over the same range. " + "CC1/2 = S / (S + E) (signal and half-set error variances), so 1 / CC1/2 - 1 = E / S is the " + "half-set noise; the noise ratio is rugnux's over XDS's, with XDS's CC1/2 taken at the " + "bottom of its printed rounding (-0.0005). A set fails `merge` when the ratio is above " + f"{score.CC_HALF_NOISE_LIMIT:g}: noisier than XDS's merge would be with half its " + "observations. The low-resolution R_meas is the lowest shell of each table (the shells " + "are XDS's, so both cover the same reflections); it is reported, not scored.") derived = {r["set"]: r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"} if derived: @@ -394,7 +438,7 @@ def build(run_dir, baseline=None, allow_incomplete=False): model += [("REFMAC R_free", "refmac_rfree_depflags"), ("REFMAC dep data", "refmac_rfree_depdata"), ("REFMAC ratio", "refmac_rfree_ratio")] if any(r.get("refmac_status") for r in res) else [] head = ["set", "arm", "verdict", "space group", "ref", "cell dev %", "V ratio", "d_min", "ref d_min", - "gain", "compl %", "mult", "R_meas", "CC1/2", "ISa", "ref ISa", "idx"] + \ + "gain", "compl %", "mult", "R_meas", "low-res R_meas", "CC1/2", "ISa", "ref ISa", "idx"] + \ [h for h, _ in model] + (["model fit"] if model else []) + ["time s", "note"] rows = [] for r in sorted(res, key=lambda r: (order.get(r["arm"], 9), r["set"])): @@ -404,7 +448,8 @@ def build(run_dir, baseline=None, allow_incomplete=False): f(r.get("cell_dev_pct")), f(r.get("volume_ratio"), "{:.3f}"), f(r.get("d_min")), f(r.get("d_min_ref")), f(r.get("res_gain_pct"), "{:+.1f}%"), f(r.get("completeness"), "{:.1f}"), f(r.get("multiplicity"), "{:.1f}"), - pct(r.get("r_meas")), f(r.get("cc_half"), "{:.3f}"), f(r.get("isa"), "{:.1f}"), + pct(r.get("r_meas")), pct(r.get("lowres_r_meas")), f(r.get("cc_half"), "{:.3f}"), + f(r.get("isa"), "{:.1f}"), f(r.get("isa_ref"), "{:.1f}"), pct(r.get("indexing_rate"), 0)] + [f(r.get(k), "{:.3f}") for _, k in model] + ([r.get("model_fit") or "-"] if model else []) @@ -430,6 +475,12 @@ def build(run_dir, baseline=None, allow_incomplete=False): return doc +def lowres_cell(r): + """rugnux / XDS R_meas of the lowest-resolution shell, with the shells' high-resolution limits.""" + return (pct(r.get("refres_lowres_r_meas")) + " / " + pct(r.get("lowres_r_meas_ref")) + + f" ({f(r.get('refres_lowres_d'))} / {f(r.get('lowres_d_ref'))})") + + def refres_note(r): """How to read a row of the reference-range table.""" if r.get("refres_shells_past_limit"): diff --git a/tools/battery/score.py b/tools/battery/score.py index 331722d7d..865daf681 100644 --- a/tools/battery/score.py +++ b/tools/battery/score.py @@ -1,9 +1,9 @@ """Scoring one rugnux run against its reference. One dataset, one verdict, one cause, decided in this order: did it run -> is the lattice right -> -is the symmetry right -> is the merge usable. The order matters: a halved axis also makes the -screw along it unobservable, and counting that row twice would hide the defect that cost the -reflections. +is the symmetry right -> (XDS arms only) is the merge as good as XDS's by CC1/2. The order matters: +a halved axis also makes the screw along it unobservable, and counting that row twice would hide +the defect that cost the reflections. """ import itertools import os @@ -17,13 +17,28 @@ REPORT_LINE = re.compile(r"^([A-Z_0-9]+)=\s*(.*)$") def read_report(path): - """KEY= value lines of a rugnux _report.txt.""" + """KEY= value lines of a rugnux _report.txt, plus the lowest-resolution shell of its two shell + tables: LOWRES_D / LOWRES_R_MEAS from the table over its own range, REFRES_LOWRES_D / + REFRES_LOWRES_R_MEAS from the one over the reference range (it follows REFRES_RANGE=).""" out = {} - if os.path.exists(path): - for line in open(path, errors="replace"): - m = REPORT_LINE.match(line.rstrip("\n")) - if m: - out[m.group(1)] = m.group(2).strip() + if not os.path.exists(path): + return out + header = None + for line in open(path, errors="replace"): + line = line.rstrip("\n") + m = REPORT_LINE.match(line) + if m: + out[m.group(1)] = m.group(2).strip() + continue + f = line.lstrip("#").split() + if f[:2] == ["d_min", "N_obs"]: + header = f + elif header and f and re.match(r"^\d+\.\d+$", f[0]): + prefix = "REFRES_" if "REFRES_RANGE" in out else "" + out[prefix + "LOWRES_D"] = f[0] + r_meas = f[header.index("R_meas")].rstrip("%") # "3.7%", or "-" for none + out[prefix + "LOWRES_R_MEAS"] = "" if r_meas == "-" else str(round(float(r_meas) / 100, 4)) + header = None # only the first row of each table return out @@ -74,6 +89,24 @@ def cell_dev_pct(cell, ref): for p in itertools.permutations(cell[:3])) +# XDS prints CC1/2 in % with one decimal; its value is taken at the bottom of that rounding, so a +# printed 100.0 does not demand a perfect CC1/2 of rugnux +XDS_CC_HALF_ROUNDING = 0.0005 +CC_HALF_NOISE_LIMIT = 2.0 + + +def cc_half_noise_ratio(cc, cc_xds): + """rugnux's half-set noise over XDS's, both read off CC1/2 over XDS's range; None without both. + + CC1/2 = S / (S + E), with S the variance of the signal and E that of the half-set error, so + E / S = 1 / CC1/2 - 1 at any CC1/2. E goes as 1 / (observations): a ratio of 2 + (CC_HALF_NOISE_LIMIT) is a merge as noisy as XDS's would be with half of its observations. + A CC1/2 at or below 0 is no signal at all, counted as 0.001.""" + if cc is None or cc_xds is None: + return None + return round((1 / max(cc, 0.001) - 1) / (1 / (cc_xds - XDS_CC_HALF_ROUNDING) - 1), 3) + + def point_group(sgno): return gemmi.find_spacegroup_by_number(int(sgno)).point_group_hm() @@ -115,6 +148,15 @@ def judge(entry, rep, run_note): r["isa_ratio"] = round(r["refres_isa"] / r["isa_ref"], 4) if r["refres_isa"] and r["isa_ref"] else None r["r_meas_ratio"] = (round(r["refres_r_meas"] / r["r_meas_ref"], 4) if r["refres_r_meas"] and r["r_meas_ref"] else None) + # R_meas of the lowest-resolution shell: rugnux's own table, the reference-range table and XDS's + # (reported, not scored) + r.update(lowres_d=fnum(rep.get("LOWRES_D")), lowres_r_meas=fnum(rep.get("LOWRES_R_MEAS")), + refres_lowres_d=fnum(rep.get("REFRES_LOWRES_D")), + refres_lowres_r_meas=fnum(rep.get("REFRES_LOWRES_R_MEAS")), + lowres_d_ref=ref.get("dmin_low"), lowres_r_meas_ref=ref.get("r_meas_low")) + r["lowres_r_meas_ratio"] = (round(r["refres_lowres_r_meas"] / r["lowres_r_meas_ref"], 4) + if r["refres_lowres_r_meas"] and r["lowres_r_meas_ref"] else None) + r["cc_half_noise_ratio"] = cc_half_noise_ratio(r["refres_cc_half"], r["cc_half_ref"]) if rep.get("SPACE_GROUP_NUMBER"): r["sgno"] = int(fnum(rep["SPACE_GROUP_NUMBER"])) r["sg"] = rep.get("SPACE_GROUP_NAME") or sg_name(r["sgno"]) @@ -191,10 +233,11 @@ def judge(entry, rep, run_note): "sym_over" if order > order_ref else "sym_other") return verdict("fail", cause, f"{r['sg']} vs reference {r['sg_ref']}") - if r["r_meas"] is not None and r["r_meas"] > 0.6: - return verdict("fail", "merge", f"R_meas {r['r_meas']:.0%} overall") - if r["cc_half"] is not None and r["cc_half"] < 0.5: - return verdict("fail", "merge", f"CC1/2 {r['cc_half']:.2f} overall") + # Merge quality is judged only against XDS, by CC1/2 over XDS's own range (cc_half_noise_ratio). + # The open arm has no merge criterion. + if r["cc_half_noise_ratio"] is not None and r["cc_half_noise_ratio"] > CC_HALF_NOISE_LIMIT: + return verdict("fail", "merge", f"CC1/2 {r['refres_cc_half']:.4f} vs XDS {r['cc_half_ref']:.3f} " + f"over XDS's range: {r['cc_half_noise_ratio']:.1f}x XDS's half-set noise") note = "" if r["sgno"] != ref["sgno"]: note = f"{r['sg']} vs reference {r['sg_ref']} ({r.get('sg_relation') or 'screws not judged against XDS'})" From 80e39541aa6ad0fd022819a2fa0a4cdb738c43aa Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 18:41:43 +0200 Subject: [PATCH 28/28] tools/battery: verdicts judge lattice and symmetry only Data quality - resolution, CC1/2, R_meas, completeness, ISa - is no longer a pass/fail criterion anywhere; it is reported beside the reference in the tables and plots for a human to judge. The CC1/2 noise ratio against XDS stays as a reported guide. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- tools/battery/README.md | 24 ++++++++---------------- tools/battery/report.py | 13 ++++++------- tools/battery/score.py | 13 ++++--------- 3 files changed, 18 insertions(+), 32 deletions(-) diff --git a/tools/battery/README.md b/tools/battery/README.md index 80d33c4b0..71869800e 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -23,20 +23,12 @@ scored with `sgequiv`. On the XDS arms only the point group is scored, because X screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported. `score.py` has the details. -**Merge quality is judged only against XDS, and only by CC1/2.** No R_meas is a pass/fail -criterion, and no absolute CC1/2 either. On the XDS arms the set fails `merge` when - - (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) > 2 - -with rugnux's CC1/2 from the reference-range table (`REFRES_CC_HALF`, XDS's own range) and XDS's -from the `CORRECT.LP` total. CC1/2 = S / (S + E), with S the signal variance and E the half-set -error variance, so 1/CC1/2 - 1 = E/S at any CC1/2, and E goes as 1/(observations): a ratio above 2 -is a merge noisier than XDS's would be with half of its observations. XDS prints CC1/2 to 0.1%, so -its value is taken at the bottom of that rounding (-0.0005), and a printed 100.0 does not demand a -perfect CC1/2. The ratio is stored as `cc_half_noise_ratio`. A set without the reference-range -table or without an XDS CC1/2 (the controls, a hand-measured reference) has no merge criterion. -**The open arm has no merge criterion at all**: it is scored on lattice and symmetry, and its -resolution, CC1/2 and R_meas are reported, not judged. +**Data quality is not scored.** A verdict is about the lattice and the symmetry only. Resolution, +CC1/2, R_meas (overall and lowest shell), completeness, multiplicity and ISa are reported beside +XDS's or the deposition's values, in the per-set tables and plots, for a human to judge. On the XDS +arms `cc_half_noise_ratio` = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) over XDS's own +range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as +XDS's would be with half of its observations. **Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's @@ -256,7 +248,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. |---|---| | `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran | | `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) | -| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge` (XDS arms: CC1/2 noise ratio above 2), `false_lattice`; or `no_reference` / `no_input` / `reference_problem` | +| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `false_lattice`; or `no_reference` / `no_input` / `reference_problem` | | `reason` | one line for a human | | `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours (the data's own determination) / the reference's | | `sg_label` | the space group rugnux reported where `--model` put the model's enantiomorph on it, else null | @@ -268,7 +260,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. | `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref`, `multiplicity_ref` | XDS's overall values, over XDS's own range | | `refres_range`, `refres_shells_past_limit`, `refres_unique_reflections`, `refres_completeness`, `refres_multiplicity`, `refres_i_over_sigma`, `refres_r_meas`, `refres_cc_half`, `refres_isa` | XDS arms: the REFRES_* keys, the same merge over the reference range | | `isa_ratio`, `r_meas_ratio` | refres_isa / isa_ref and refres_r_meas / r_meas_ref | -| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), the merge criterion | +| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), reported, not scored | | `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref | | `model`, `model_note` | open arm: the deposited coordinates given to `--model`, or why there were none | | `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited | diff --git a/tools/battery/report.py b/tools/battery/report.py index 23ea0089a..724d0e6a2 100644 --- a/tools/battery/report.py +++ b/tools/battery/report.py @@ -355,8 +355,8 @@ def build(run_dir, baseline=None, allow_incomplete=False): "reference-range table / of XDS's table; below 1 = " "rugnux's strong reflections agree better"), ("cc_half_noise_ratio", "CC1/2 noise", "half-set noise over the reference range read off CC1/2 " - "(1 / CC1/2 - 1) / XDS's; above 2 fails (the merge is as " - "noisy as XDS's with half its observations)")): + "(1 / CC1/2 - 1) / XDS's; 2 = as noisy as XDS's merge " + "would be with half its observations (not scored)")): for arm in arms: pts = [(r["set"], r[name], r["verdict"], f"{what} ratio, {refres_note(r)}") for r in res if r["arm"] == arm and r.get(name)] @@ -405,13 +405,12 @@ def build(run_dir, baseline=None, allow_incomplete=False): doc.table(["set", "arm", "range A", "own d_min", "compl % (rugnux / XDS)", "mult", "I/sigma", "R_meas", "low-res R_meas (to d A)", "CC1/2", "CC1/2 noise ratio", "ISa", "reading"], rows, num=(3, 6, 10)) - doc.p("Merge quality is scored here and only here, by CC1/2 against XDS's over the same range. " + doc.p("Data quality is not scored; this table is for a human to judge. " "CC1/2 = S / (S + E) (signal and half-set error variances), so 1 / CC1/2 - 1 = E / S is the " "half-set noise; the noise ratio is rugnux's over XDS's, with XDS's CC1/2 taken at the " - "bottom of its printed rounding (-0.0005). A set fails `merge` when the ratio is above " - f"{score.CC_HALF_NOISE_LIMIT:g}: noisier than XDS's merge would be with half its " - "observations. The low-resolution R_meas is the lowest shell of each table (the shells " - "are XDS's, so both cover the same reflections); it is reported, not scored.") + "bottom of its printed rounding (-0.0005); 2 would mean noisier than XDS's merge with half " + "its observations. The low-resolution R_meas is the lowest shell of each table (the shells " + "are XDS's, so both cover the same reflections); reported like everything here.") derived = {r["set"]: r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"} if derived: diff --git a/tools/battery/score.py b/tools/battery/score.py index 865daf681..af44f3d57 100644 --- a/tools/battery/score.py +++ b/tools/battery/score.py @@ -1,7 +1,7 @@ """Scoring one rugnux run against its reference. One dataset, one verdict, one cause, decided in this order: did it run -> is the lattice right -> -is the symmetry right -> (XDS arms only) is the merge as good as XDS's by CC1/2. The order matters: +is the symmetry right. Data quality is not scored: it is reported for a human to judge. The order matters: a halved axis also makes the screw along it unobservable, and counting that row twice would hide the defect that cost the reflections. """ @@ -92,15 +92,14 @@ def cell_dev_pct(cell, ref): # XDS prints CC1/2 in % with one decimal; its value is taken at the bottom of that rounding, so a # printed 100.0 does not demand a perfect CC1/2 of rugnux XDS_CC_HALF_ROUNDING = 0.0005 -CC_HALF_NOISE_LIMIT = 2.0 def cc_half_noise_ratio(cc, cc_xds): """rugnux's half-set noise over XDS's, both read off CC1/2 over XDS's range; None without both. CC1/2 = S / (S + E), with S the variance of the signal and E that of the half-set error, so - E / S = 1 / CC1/2 - 1 at any CC1/2. E goes as 1 / (observations): a ratio of 2 - (CC_HALF_NOISE_LIMIT) is a merge as noisy as XDS's would be with half of its observations. + E / S = 1 / CC1/2 - 1 at any CC1/2. E goes as 1 / (observations): a ratio of 2 is a merge as + noisy as XDS's would be with half of its observations. Reported, not scored. A CC1/2 at or below 0 is no signal at all, counted as 0.001.""" if cc is None or cc_xds is None: return None @@ -233,11 +232,7 @@ def judge(entry, rep, run_note): "sym_over" if order > order_ref else "sym_other") return verdict("fail", cause, f"{r['sg']} vs reference {r['sg_ref']}") - # Merge quality is judged only against XDS, by CC1/2 over XDS's own range (cc_half_noise_ratio). - # The open arm has no merge criterion. - if r["cc_half_noise_ratio"] is not None and r["cc_half_noise_ratio"] > CC_HALF_NOISE_LIMIT: - return verdict("fail", "merge", f"CC1/2 {r['refres_cc_half']:.4f} vs XDS {r['cc_half_ref']:.3f} " - f"over XDS's range: {r['cc_half_noise_ratio']:.1f}x XDS's half-set noise") + # Data quality (CC1/2, R_meas, resolution, ISa) is reported, never scored: that is for a human. note = "" if r["sgno"] != ref["sgno"]: note = f"{r['sg']} vs reference {r['sg_ref']} ({r.get('sg_relation') or 'screws not judged against XDS'})"