diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 658d8be7b..eb4dbc5ab 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -475,7 +475,7 @@ A reference dataset (`--reference`, MTZ or SF-mmCIF) supplies known intensities **Serial stills without a reference or model.** The data cannot say which indexing is the *true* one, but they can say which crystals are in the *same* one, and that is all a merge needs. Where the space group is fixed (`-S`) and its lattice admits alternative indexings, the crystals are made consistent with each other before anything is scaled, following Brehm & Diederichs (2014). Each crystal's partiality- and Lorentz-corrected intensities are divided by the run's mean intensity in their resolution shell (20 shells of equal reciprocal volume), standardised within the crystal so that its unknown scale and offset drop out, and weighted by $1/(1+(\sigma/s)^2)$, $s$ being the crystal's spread, so that a weak or barely recorded reflection counts for little. The agreement of two crystals is the sum of the products of these values over the reflections they share, and the objective is that agreement summed over every pair of crystals, each in the indexing assigned to it. That sum is $(|M|^2 - \mathrm{const})/2$ with $M$ the per-reflection sum over all crystals, so a crystal's agreement with all the others is one dot product with $M$ less its own contribution: each crystal in turn takes the indexing in which that dot product is largest, $M$ is updated, and the passes repeat until no more than one crystal in a thousand moves. A pass costs one read of the reflections, not one per pair of crystals, which is what keeps tens of thousands of crystals to seconds. The objective only grows, so it converges; it is started four times (once from the indexing as found, three times from a seeded random assignment) and the start that ends highest is kept, and of the equally consistent answers related by an overall relabelling the one that moves the fewest crystals is taken. The order of every floating-point sum is fixed (crystals in image order, sequential passes), so the result does not depend on the thread count. The run reports how clearly the crystals separated: for each crystal the correlation with the mean of all the others in its chosen indexing and in the best other one, compared by Fisher's $z$ in units of its standard error ($\sqrt{2/(n-3)}$ for $n$ shared reflections) — the median of that margin and the share of crystals above $3\sigma$. -A twin law is only an ambiguity for crystals whose own lattice cannot tell the indexings apart. On a pseudo-merohedral cell whose axes differ by a few per cent, each crystal's refined cell already says which axis is which, and the indexer has used that; letting the intensities overrule it there moves crystals that were right. So for each law the share of crystals whose refined cell fits the run's cell better as indexed than after the law is measured (a tie counting half — on an exactly merohedral lattice the share is about one half). Where it is at least 0.9 the law is decided by the cells: a crystal whose cell prefers its indexing stays there, as an anchor, and only the crystals whose cell does not are decided by the intensities, against the anchors. With a reference or a model the per-image choice above stands, and the same resolution is run only to measure it: the report gives the number of crystals on which the data alone would have chosen otherwise. +A twin law is only an ambiguity for crystals whose own lattice cannot tell the indexings apart. On a pseudo-merohedral cell whose axes differ by a few per cent, each crystal's refined cell already says which axis is which, and the indexer has used that; letting the intensities overrule it there moves crystals that were right. So for each law the share of crystals whose refined cell fits the run's cell better as indexed than after the law is measured (a tie counting half — on an exactly merohedral lattice the share is about one half). Where it is at least 0.9 the law is decided by the cells: a crystal whose cell prefers its indexing stays there, as an anchor, and only the crystals whose cell does not are decided by the intensities, against the anchors. With a reference or a model the per-image choice above stands except where the cells decide: once all images are integrated, a crystal its own cell anchors goes back to that indexing, and the reference chooses only among the indexings left open to the others; the same data-only resolution is then run only to measure it: the report gives the number of crystals on which the data alone would have chosen otherwise. ### 10.10 Ice rings at the scale and merge stages diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index 9f2c89f1c..261f34de4 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -194,6 +194,9 @@ This is the only point at which a model affects which observations are merged to not gated by the null — there is no merge yet to test the model on. It chooses only between indexings that are equally valid for each crystal; a model that does not belong to the crystal carries no information to choose with, so the crystals stay mixed, as they would without a model. +Where the lattice is only pseudo-merohedral and the crystals' own cells already say which indexing is +which, the cells win: a crystal its cell anchors keeps that indexing, and the model decides +only the crystals whose cell cannot tell. After the merge, the model is validated as in the steps above. Without `-C` and `-S`, or on rotation data, the model is not used before the merge at all. diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index 8d899bec1..2846487e5 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -948,7 +948,7 @@ void IndexAndRefine::ScaleImage(DataMessage &msg, IntegrationOutcome& outcome) { return; auto scaling_start_time = std::chrono::steady_clock::now(); - reindex_resolver->Resolve(outcome.reflections, outcome.latt); + outcome.reference_cc = reindex_resolver->Resolve(outcome.reflections, outcome.latt); auto scaling_end_time = std::chrono::steady_clock::now(); msg.image_scale_time_s = std::chrono::duration(scaling_end_time - scaling_start_time).count(); } diff --git a/image_analysis/IntegrationOutcome.h b/image_analysis/IntegrationOutcome.h index a16f41957..2d1b37cc0 100644 --- a/image_analysis/IntegrationOutcome.h +++ b/image_analysis/IntegrationOutcome.h @@ -4,6 +4,7 @@ #pragma once #include +#include #include "../common/Reflection.h" #include "../common/CrystalLattice.h" @@ -23,6 +24,9 @@ struct IntegrationOutcome { std::optional image_scale_cc_n; std::optional image_scale_g; std::optional image_scale_wedge_deg; + // Stills with a reference: the image's CC with it in each indexing it was offered, as indexed first + // (ReindexAmbiguityResolver::Resolve); empty otherwise. + std::vector reference_cc; }; // How far apart, in frames, two partials of one raw hkl may sit and still belong to the same rocking diff --git a/image_analysis/scale_merge/ReferenceFreeReindex.cpp b/image_analysis/scale_merge/ReferenceFreeReindex.cpp index e13e541cc..f23cc3689 100644 --- a/image_analysis/scale_merge/ReferenceFreeReindex.cpp +++ b/image_analysis/scale_merge/ReferenceFreeReindex.cpp @@ -40,6 +40,85 @@ namespace { } } +MetricAnchors AnchorOnCells(const std::vector &outcomes, const UnitCell &cell, + const std::vector &ops) { + std::vector indexings{gemmi::Op::identity()}; + indexings.insert(indexings.end(), ops.begin(), ops.end()); + const size_t K = indexings.size(); + + // How much better an image's own refined cell fits the run's cell as indexed than after indexing g. + const auto cell_preference = [&](const IntegrationOutcome &o, size_t g) { + return CellDistance(ReindexLattice(o.latt, indexings[g]).GetUnitCell(), cell) + - CellDistance(o.latt.GetUnitCell(), cell); + }; + // Which indexings the cells decide: those the indexed images' cells prefer the identity over (a tie + // counting half) for at least METRIC_DECIDED of them. + MetricAnchors out; + std::vector decided(K, false); + for (size_t g = 1; g < K; ++g) { + double prefer = 0, n = 0; + for (const auto &o : outcomes) { + if (o.reflections.empty()) + continue; + const double p = cell_preference(o, g); + prefer += std::fabs(p) < METRIC_TIE ? 0.5 : p > 0 ? 1.0 : 0.0; + n += 1; + } + out.metric_preference.push_back(n > 0 ? prefer / n : 0.5); + decided[g] = out.metric_preference.back() >= METRIC_DECIDED; + } + // Where the cells decide an indexing, an image whose own cell prefers the identity stays as indexed. + out.allowed.assign(outcomes.size() * K, 1); + for (size_t o = 0; o < outcomes.size(); ++o) + for (size_t g = 1; g < K; ++g) + if (decided[g] && !outcomes[o].reflections.empty() && cell_preference(outcomes[o], g) >= METRIC_TIE) + out.allowed[o * K + g] = 0; + return out; +} + +ReferenceAnchorResult AnchorReferenceReindex(std::vector &outcomes, const UnitCell &cell, + const std::vector &ops) { + std::vector indexings{gemmi::Op::identity()}; + indexings.insert(indexings.end(), ops.begin(), ops.end()); + const size_t K = indexings.size(); + // The indexing best correlated with the reference among those open: the one as indexed unless another + // beats it; a score that is not finite never wins (the rule of the per-image resolver). + const auto best = [&](const std::vector &cc, const uint8_t *open) { + size_t b = 0; + for (size_t g = 1; g < K; ++g) + if (open[g] && std::isfinite(cc[g]) && (!std::isfinite(cc[b]) || cc[g] > cc[b])) + b = g; + return b; + }; + const std::vector all_open(K, 1); + + // Each image back in the indexing it was indexed in. + std::vector by_reference(outcomes.size(), 0); + for (size_t o = 0; o < outcomes.size(); ++o) { + auto &out = outcomes[o]; + if (out.reference_cc.size() != K) + continue; + by_reference[o] = best(out.reference_cc, all_open.data()); + if (by_reference[o] != 0) + ReindexImage(out.reflections, out.latt, indexings[by_reference[o]].inverse()); + } + + const MetricAnchors anchors = AnchorOnCells(outcomes, cell, ops); + ReferenceAnchorResult result; + result.metric_preference = anchors.metric_preference; + for (size_t o = 0; o < outcomes.size(); ++o) { + auto &out = outcomes[o]; + if (out.reference_cc.size() != K) + continue; + const size_t g = best(out.reference_cc, &anchors.allowed[o * K]); + result.overruled += g != by_reference[o]; + if (g != 0) + ReindexImage(out.reflections, out.latt, indexings[g]); + out.reference_cc.clear(); // the scores were for the indexing as indexed, which is no longer known + } + return result; +} + ReferenceFreeReindexResult ResolveReindexReferenceFree(std::vector &outcomes, const DiffractionExperiment &x, const UnitCell &cell, @@ -56,26 +135,8 @@ ReferenceFreeReindexResult ResolveReindexReferenceFree(std::vector(indexings.size()); - // How much better an image's own refined cell fits the run's cell as indexed than after indexing g. - const auto cell_preference = [&](const IntegrationOutcome &o, int g) { - return CellDistance(ReindexLattice(o.latt, indexings[g]).GetUnitCell(), cell) - - CellDistance(o.latt.GetUnitCell(), cell); - }; - // Which indexings the cells decide: those the indexed images' cells prefer the identity over (a tie - // counting half) for at least METRIC_DECIDED of them. - std::vector decided(K, false); - for (int g = 1; g < K; ++g) { - double prefer = 0, n = 0; - for (const auto &o : outcomes) { - if (o.reflections.empty()) - continue; - const double p = cell_preference(o, g); - prefer += std::fabs(p) < METRIC_TIE ? 0.5 : p > 0 ? 1.0 : 0.0; - n += 1; - } - out.metric_preference.push_back(n > 0 ? prefer / n : 0.5); - decided[g] = out.metric_preference.back() >= METRIC_DECIDED; - } + const MetricAnchors anchors = AnchorOnCells(outcomes, cell, all_ops); + out.metric_preference = anchors.metric_preference; const auto accept = [&](const Reflection &r) { return !r.on_ice_ring && AcceptReflection(r, s.GetHighResolutionLimit_A(), s.GetLowResolutionLimit_A()) @@ -124,16 +185,14 @@ ReferenceFreeReindexResult ResolveReindexReferenceFree(std::vector allowed(static_cast(N) * K, 1); + // The indexings each crystal may take (AnchorOnCells): a crystal its own cell anchors is one of those + // the others are decided against. + std::vector allowed(static_cast(N) * K); std::vector free(N, false); for (int c = 0; c < N; ++c) - for (int g = 1; g < K; ++g) { - if (decided[g] && cell_preference(outcomes[crystal_outcome[c]], g) >= METRIC_TIE) - allowed[c * K + g] = 0; - free[c] = free[c] || allowed[c * K + g]; + for (int g = 0; g < K; ++g) { + allowed[c * K + g] = anchors.allowed[crystal_outcome[c] * K + g]; + free[c] = free[c] || (g > 0 && allowed[c * K + g]); } out.crystals = std::count(free.begin(), free.end(), true); if (out.crystals == 0) @@ -362,7 +421,8 @@ ReferenceFreeReindexResult ResolveReindexReferenceFree(std::vector= METRIC_DECIDED; }); for (int h = 0; h < (anchored ? 1 : K); ++h) { size_t kept = 0; for (int c = 0; c < N; ++c) diff --git a/image_analysis/scale_merge/ReferenceFreeReindex.h b/image_analysis/scale_merge/ReferenceFreeReindex.h index 0220d82e3..2527b1d54 100644 --- a/image_analysis/scale_merge/ReferenceFreeReindex.h +++ b/image_analysis/scale_merge/ReferenceFreeReindex.h @@ -4,6 +4,7 @@ #pragma once #include +#include #include #include "gemmi/symmetry.hpp" @@ -27,6 +28,30 @@ // share is a half. constexpr double METRIC_DECIDED = 0.9; +// Which indexings each image's own refined cell leaves open. Indexing g is the image as indexed for g = 0 +// and after ops[g - 1] otherwise; `cell` is the run's cell. An operator the cells decide is closed to +// every image whose cell fits `cell` better as indexed; everything else is open. +struct MetricAnchors { + std::vector metric_preference; // per operator: share of the images whose refined cell is + // closer to `cell` as indexed than after the operator + std::vector allowed; // allowed[o * (ops.size() + 1) + g]: indexing g open to image o +}; +MetricAnchors AnchorOnCells(const std::vector &outcomes, const UnitCell &cell, + const std::vector &ops); + +// With a reference - a reference MTZ or intensities from the model - each image has been put, image by +// image, into the indexing that correlates best with it (ReindexAmbiguityResolver, which leaves the scores in +// IntegrationOutcome::reference_cc, over the same `ops`). The reference knows nothing of the cells, so +// here the cells have their say first, as above: an image its own cell anchors keeps that indexing, +// and the reference decides among the indexings left open to the others. +struct ReferenceAnchorResult { + std::vector metric_preference; // as in MetricAnchors + size_t overruled = 0; // images the reference had put into an indexing their cell + // excludes, now in the best open one instead +}; +ReferenceAnchorResult AnchorReferenceReindex(std::vector &outcomes, const UnitCell &cell, + const std::vector &ops); + struct ReferenceFreeReindexResult { size_t crystals = 0; // crystals that took part (enough reflections to say anything, and an // indexing their own cell leaves open) diff --git a/image_analysis/scale_merge/ReindexAmbiguity.cpp b/image_analysis/scale_merge/ReindexAmbiguity.cpp index 74be6e69e..42486ce0f 100644 --- a/image_analysis/scale_merge/ReindexAmbiguity.cpp +++ b/image_analysis/scale_merge/ReindexAmbiguity.cpp @@ -266,17 +266,18 @@ double ReindexAmbiguityResolver::ReferenceCC(std::span reflect // Serial stills index each crystal in one of the merohedrally-equivalent hands at random; pick, for this // image alone, the reindexing whose intensities correlate best with the external reference and apply it. -void ReindexAmbiguityResolver::Resolve(std::span reflections, CrystalLattice &latt) const { +std::vector ReindexAmbiguityResolver::Resolve(std::span reflections, CrystalLattice &latt) const { if (ops.empty()) - return; + return {}; - const BestReindex best = PickBestReindex( - ReferenceCC(reflections, gemmi::Op::identity()), ops, - [&](const gemmi::Op &op) { return ReferenceCC(reflections, op); }); - if (best.is_identity) - return; - - ReindexImage(reflections, latt, best.op); + std::vector cc{ReferenceCC(reflections, gemmi::Op::identity())}; + const BestReindex best = PickBestReindex(cc[0], ops, [&](const gemmi::Op &op) { + cc.push_back(ReferenceCC(reflections, op)); + return cc.back(); + }); + if (!best.is_identity) + ReindexImage(reflections, latt, best.op); + return cc; } void ReindexImage(std::span reflections, CrystalLattice &latt, const gemmi::Op &op) { diff --git a/image_analysis/scale_merge/ReindexAmbiguity.h b/image_analysis/scale_merge/ReindexAmbiguity.h index 429abd7c4..57346d18d 100644 --- a/image_analysis/scale_merge/ReindexAmbiguity.h +++ b/image_analysis/scale_merge/ReindexAmbiguity.h @@ -138,9 +138,11 @@ public: // The reference must be an external, correctly-handed dataset (not the data's own running merge). ReindexAmbiguityResolver(const DiffractionExperiment &x, const std::vector &reference); - // Reindex this image's reflections in place into the hand best-correlated with the reference (done - // once per image, for good). No-op when there is no ambiguity. + // Reindex this image's reflections in place into the hand best-correlated with the reference. No-op + // when there is no ambiguity. // Relabels the image's reflections and moves its lattice onto the same axes, so that a spot's new - // index still predicts it where it was measured. - void Resolve(std::span reflections, CrystalLattice &latt) const; + // index still predicts it where it was measured. Returns the CC in each indexing, as indexed first and + // then the operators of ReindexAmbiguityOperators(cell, space group) - empty when there are none - + // for AnchorReferenceReindex to revisit the choice once the whole run's cells are known. + std::vector Resolve(std::span reflections, CrystalLattice &latt) const; }; diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index efca00db0..6b26e0bfa 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -1510,6 +1510,25 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { const bool guard_beside = merge_first_part && guard_ahead.valid(); if (guard_beside) rsm->SetWriteBackPerFrameScale(false); + // Stills with a reference or a model: each image was put into the indexing that agrees best with it, + // image by image. Where the crystals' own cells decide an indexing, those cells come first, and the + // reference decides only the crystals they leave open. Before the relabel below, which changes axes. + if (!rsm && ambiguity_reference && result.consensus_cell && experiment_.GetUnitCell() + && experiment_.GetGemmiSpaceGroup()) { + const auto ops = ReindexAmbiguityOperators(*experiment_.GetUnitCell(), experiment_.GetSpaceGroupOrP1()); + if (!ops.empty()) { + const auto anchored = AnchorReferenceReindex(indexer->GetIntegrationOutcome(), *result.consensus_cell, + ops); + for (size_t i = 0; i < ops.size(); ++i) + logger.Info("Indexing ambiguity {}: {:.1f}% of the crystals' own cells fit better as indexed{}", + IndexTriplet(ops[i]), 100.0 * anchored.metric_preference[i], + anchored.metric_preference[i] >= METRIC_DECIDED + ? " - the cells decide it; the reference decides only crystals whose cell does not" + : ""); + logger.Info("Indexing ambiguity: {} crystal(s) the reference had put against their own cell kept " + "in an indexing the cell allows", anchored.overruled); + } + } // Stills: a crystal indexed in a lattice that contains the run's - an axis multiple, or the run's in // another setting - is put on the run's cell before anything is scaled, rather than left out. if (!rsm && result.consensus_cell) { diff --git a/tests/ReindexAmbiguityTest.cpp b/tests/ReindexAmbiguityTest.cpp index b3ed40918..143b4e9aa 100644 --- a/tests/ReindexAmbiguityTest.cpp +++ b/tests/ReindexAmbiguityTest.cpp @@ -465,3 +465,93 @@ TEST_CASE("Reindex: where the cells decide, the crystals whose cell disagrees ar for (int c = 0; c < n_crystals; ++c) CHECK((result.choice[c] != 0) == swapped[c]); } + +TEST_CASE("Reindex: with a reference, a crystal its own cell anchors keeps that indexing", "[reindex]") { + // a and b 3% apart: the crystals' cells decide the twin law. The reference is written in the other + // indexing, so image by image it moves every crystal; afterwards, the crystals whose cell says which + // axis is which go back, and only those whose cell cannot tell (a = b) stay where the reference put them. + const UnitCell cell{78.0f, 80.4f, 37.0f, 90.0f, 90.0f, 90.0f}; + const UnitCell undecided{79.2f, 79.2f, 37.0f, 90.0f, 90.0f, 90.0f}; + DiffractionExperiment x; + x.SetUnitCell(cell).SpaceGroupNumber(16); // P222 + const auto laws = ReindexAmbiguityOperators(cell, x.GetSpaceGroupOrP1()); + REQUIRE(laws.size() == 1); + + const HKLKeyGenerator key(true, x.GetSpaceGroupOrP1()); + std::mt19937 rng(5); + std::exponential_distribution wilson(1.0); + std::normal_distribution noise(0.0, 0.2); + std::unordered_map structure; + std::vector> hkls; + for (int h = -12; h <= 12; ++h) + for (int k = -12; k <= 12; ++k) + for (int l = -6; l <= 6; ++l) + if (h != 0 || k != 0 || l != 0) { + hkls.push_back({h, k, l}); + structure.try_emplace(key(h, k, l).pack(), wilson(rng)); + } + const auto d_of = [&](const std::array &t) { + return 1.0f / std::sqrt(t[0] * t[0] / (cell.a * cell.a) + t[1] * t[1] / (cell.b * cell.b) + + t[2] * t[2] / (cell.c * cell.c)); + }; + std::vector reference; + std::unordered_set seen; + for (const auto &t : hkls) { + if (!seen.insert(key(t[0], t[1], t[2]).pack()).second) + continue; + const auto m = laws[0].apply_to_hkl({{t[0], t[1], t[2]}}); + MergedReflection r; + r.h = t[0]; r.k = t[1]; r.l = t[2]; + r.I = static_cast(structure.at(key(m[0], m[1], m[2]).pack())); + r.sigma = 1.0f; + r.d = d_of(t); + reference.push_back(r); + } + + const int n_crystals = 200; + std::vector outcomes(n_crystals); + std::vector open(n_crystals); + for (int c = 0; c < n_crystals; ++c) { + open[c] = c % 10 == 3; + outcomes[c].latt = CrystalLattice(open[c] ? undecided : cell); + for (int i = 0; i < 150; ++i) { + const auto &t = hkls[rng() % hkls.size()]; + Reflection r{}; + r.h = t[0]; r.k = t[1]; r.l = t[2]; + r.d = d_of(t); + r.I = static_cast(structure.at(key(t[0], t[1], t[2]).pack()) * (1.0 + noise(rng))); + r.sigma = 0.1f; + r.partiality = 1.0f; + r.prescaling_corr = 1.0f; + outcomes[c].reflections.push_back(r); + } + } + const auto as_indexed = outcomes; + const auto as_indexed_now = [&](int c) { + for (size_t i = 0; i < outcomes[c].reflections.size(); ++i) { + const auto &r = outcomes[c].reflections[i], &r0 = as_indexed[c].reflections[i]; + if (r.h != r0.h || r.k != r0.k || r.l != r0.l) + return false; + } + return true; + }; + + const ReindexAmbiguityResolver resolver(x, reference); + for (auto &o : outcomes) { + o.reference_cc = resolver.Resolve(o.reflections, o.latt); + REQUIRE(o.reference_cc.size() == 2); + } + for (int c = 0; c < n_crystals; ++c) + REQUIRE_FALSE(as_indexed_now(c)); // the reference moved them all + + const auto result = AnchorReferenceReindex(outcomes, cell, laws); + REQUIRE(result.metric_preference.size() == 1); + CHECK(result.metric_preference[0] >= METRIC_DECIDED); + CHECK(result.overruled == static_cast(std::count(open.begin(), open.end(), false))); + for (int c = 0; c < n_crystals; ++c) { + CHECK(as_indexed_now(c) == !open[c]); + CHECK(outcomes[c].reference_cc.empty()); + if (!open[c]) + CHECK(outcomes[c].latt.GetUnitCell().b == Catch::Approx(cell.b).epsilon(1e-4)); + } +}