Space-group search: ask twice - all observations, and only the well-measured ones

--search-min-zeta rescues a point group that the full merge cannot confirm, but used on its own it is a trade: on the crystal it was built for it recovers the correct 422, and on four others it costs the space group outright, because discarding 40-80% of the observations starves operator correlations that were perfectly healthy. Both ways of applying it - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade the SAME crystals, so the cut itself is the problem, not where it is applied. Filip's observation makes it one-way: every disagreement between the two is a LOST operator, never an invented one. Discarding observations can starve a correlation; it cannot manufacture symmetry that is not there. So run the search on both merges and keep whichever found MORE symmetry, and the failure mode disappears - each arm rescues the other exactly where it fails. crystal all observations Lorentz-filtered adopted thaumatin (weak) 222 422 422 tetragonal lysozyme 422 222 422 cubic insulin x3 23 2 / 222 23 The filtered merge is used ONLY to rescue the point group. The screw and centering determination always comes from the merge with all the observations, because systematic absences are decided by the WEAK reflections and the filter throws most of them away. Preferring the filtered arm on a tie is not a conservative choice, it is a wrong one: it cost four crystals their screw axes (P2(1) read as P2, P4(1)2(1)2 as P42(1)2) with the point group and every intensity statistic identical - a regression invisible to CC1/2, R_meas and ISa. Where the two find the same ORDER but different symmetry, nothing can prefer one, so the run says so: it names both space groups, states that the data do not decide, reports which one processing continued in, and gives the flag to force the other. Two candidates of the same order imply different molecular replacement searches, and trying both is cheap next to reprocessing - much cheaper than a confident wrong answer. Rotation battery, 33 crystals, both spot finders: fixed-threshold finder 30/33 - ZERO crystals differ from the single search adaptive finder 30/33 - the same three mismatches, gap CLOSED The adaptive finder now matches the fixed-threshold one exactly, which it has not done before: its last remaining loss was the thaumatin set whose 4-fold sits 88.9 deg from the spindle, and it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken). A merohedral twin stays refused in BOTH arms at all three frame ranges where it over-promotes, and at one of them the second opinion is strictly better than shipping behaviour - the full merge collapses to P1 where the filtered one finds the correct H3. Cost is the extra scale-combine-merge on already-ingested partials, with no re-integration: 47.2 s against 47.8 s on the same crystal back to back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 21:11:05 +02:00
parent f2b92e3f4d
commit 25458265d3
2 changed files with 71 additions and 0 deletions
@@ -72,6 +72,11 @@ public:
    // <I/sigma> >= 1 without cutting the final in-symmetry merge. Reset to the manual limit afterwards.
    void SetDMinLimit(std::optional<double> d_min_A) { d_min_limit = d_min_A; }

+    // Toggle the search-pass Lorentz filter (see search_min_zeta) between Run() calls, so the caller can
+    // produce both a filtered and an unfiltered search merge from the same ingested partials.
+    void SetSearchMinZeta(double zeta) { search_min_zeta = zeta; }
+    [[nodiscard]] double GetSearchMinZeta() const { return search_min_zeta; }
+
 private:
    // One integrated observation - a per-frame partial during scaling/combine, or a combined full during
    // scale-fulls/merge. Flat (not nested per image); a POD so the arrays translate straight to CUDA.
@@ -1204,6 +1204,72 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
                sg_opts.lattice_system = end_msg.rotation_lattice_type->crystal_system;
            auto sg_search = SearchSpaceGroup(sm.merged, sg_opts);

+            // Second opinion from a merge that keeps only well-measured observations (see
+            // RotationScaleMerge::search_min_zeta), and take whichever search found MORE symmetry.
+            // The two disagree in one direction only: every crystal on which the Lorentz filter changes
+            // the answer LOSES an operator, because discarding 40-80% of the observations starves the
+            // operator correlations - it never invents one. So the higher of the two is the safe pick,
+            // and it costs nothing on crystals where the filter is irrelevant. Both searches keep their
+            // vetoes, so a merohedral twin stays refused in each (verified on a real twin at the three
+            // frame ranges where it over-promotes: refused in both arms every time).
+            if (rsm && rsm->GetSearchMinZeta() > 0.0 && !sg_search.point_group_hm.empty()) {
+                const double zeta = rsm->GetSearchMinZeta();
+                rsm->SetSearchMinZeta(0.0);
+                auto sm_all = scale_and_merge("P1, all observations", true);
+                rsm->SetSearchMinZeta(zeta);
+                const auto alt = SearchSpaceGroup(sm_all.merged, sg_opts);
+                // Point-group order = the number of rotations, which is what sym_ops holds (centering
+                // lives in cen_ops). Taken from the space group the search chose - a point-group symbol
+                // like "422" is not a space-group name and cannot be looked up.
+                const auto order_of = [](const SearchSpaceGroupResult &r) -> size_t {
+                    return r.best_space_group.has_value()
+                           ? r.best_space_group->operations().sym_ops.size() : 0;
+                };
+                logger.Info("Space-group search: all-observation merge -> {} (order {}), "
+                            "Lorentz-filtered -> {} (order {})",
+                            alt.point_group_hm.empty() ? "?" : alt.point_group_hm, order_of(alt),
+                            sg_search.point_group_hm.empty() ? "?" : sg_search.point_group_hm,
+                            order_of(sg_search));
+                // The filtered merge is used ONLY to rescue a point group the full merge failed to
+                // confirm. Everything else - and the screw/centering determination in particular - comes
+                // from the merge with all the observations, because systematic absences are decided by
+                // the WEAK reflections and the filter throws most of them away. Preferring the filtered
+                // arm on a tie cost four crystals their screw axes (P2(1) read as P2) for exactly that
+                // reason, with the point group and every intensity statistic identical.
+                if (order_of(sg_search) <= order_of(alt)) {
+                    if (order_of(alt) > order_of(sg_search))
+                        logger.Info("Space-group search: all-observation merge supports {} where the "
+                                    "Lorentz-filtered one supports {} - taking the higher symmetry",
+                                    alt.point_group_hm, sg_search.point_group_hm);
+                    sg_search = alt;
+                    sm = std::move(sm_all);
+                } else {
+                    logger.Info("Space-group search: Lorentz-filtered merge supports {} where the "
+                                "all-observation one supports {} - taking the higher symmetry",
+                                sg_search.point_group_hm, alt.point_group_hm);
+                }
+                if (order_of(alt) > 0 && order_of(alt) == order_of(sg_search)
+                    && alt.point_group_hm != sg_search.point_group_hm) {
+                    // Same order, different symmetry: the two merges disagree about WHICH operators are
+                    // real, and neither is higher, so there is nothing to prefer. Say so instead of
+                    // picking silently - the two imply different molecular-replacement searches, and
+                    // trying both is cheap next to processing the data again.
+                    const std::string a = alt.best_space_group.has_value()
+                                          ? alt.best_space_group->short_name() : alt.point_group_hm;
+                    const std::string b = sg_search.best_space_group.has_value()
+                                          ? sg_search.best_space_group->short_name() : sg_search.point_group_hm;
+                    const std::string msg = fmt::format(
+                        "Space group is AMBIGUOUS between {} and {} (point groups {} and {}, the same "
+                        "order): the merge of all observations and the merge of only the well-measured "
+                        "ones each support one of them, and neither is the higher symmetry, so the data "
+                        "do not decide. Processing continues in {} - try BOTH in molecular replacement, "
+                        "or re-run with --search-min-zeta 0 to force the all-observation choice.",
+                        a, b, alt.point_group_hm, sg_search.point_group_hm, a);
+                    logger.Warning("{}", msg);
+                    stats_text << "  !! " << msg << "\n\n";
+                }
+            }
+
            // Twinning evidence measured BEFORE any promotion, on the P1/subgroup merge the search was
            // given. Reported alongside the post-adoption analysis: once a higher Laue class has been
            // adopted, its own twinning test can only say "no twin law exists within this class", which