Rotation indexing: do not keep a metric symmetry that indexes almost nothing

The Bravais class is decided from the UNREFINED FFT candidate against a fixed
3 degree angular tolerance (LatticeSearch). A lattice that is pseudo-symmetric to
a few tenths of a degree is therefore promoted a class too far, and the constraint
then snaps a real angle to the ideal one - which throws nearly every reflection of
every frame out of tolerance. Measured on a monoclinic crystal that is
pseudo-C-orthorhombic to 0.42 degrees: the promoted cell indexes 2 of 60
validation frames and the run dies, where its own primitive cell indexes 39. It is
the same lattice in a different setting, b_oC = -(a + 2c), volume exactly 2.00x.

The perverse part is that BETTER SPOTS MAKE IT WORSE. LatticeSearch applied to the
true cell returns the promoted class deterministically; runs that succeed escape
only because the raw FFT candidate is inaccurate enough to miss the promotion
window. So it is bistable and non-monotone in every knob - 190 spots per image
gives 44/60, 195 gives 12/60, 200 gives 2/60 - and it will bite harder as spot
finding improves.

The indexer already refines a free triclinic cell alongside each constrained
candidate, but decides between them on the fraction of the accumulated first-pass
cloud that indexes, where the two differ by less than a factor 2 (measured 0.243
vs 0.135, missing both of that guard's bars). The caller has a far sharper
statistic: it already counts how many of 60 validation frames a candidate indexes,
and there the same pair differs by more than 20x. So keep the triclinic cell
instead of dropping it, and let the first pass settle it.

The bar is a clear majority, not a margin, and that is the part that took a
battery to get right: the unconstrained refinement holds NO cell parameter fixed,
so it can only index at least as many frames as the constrained one, and on
genuine symmetry it does index a few more. A 10 % margin - the bar a later scheme
needs to displace an earlier one - demoted a real I-centred orthorhombic crystal
to P1 (47 -> 54 frames) and perturbed an F-cubic one (49 -> 58). Only a
constrained cell that fails outright while its unconstrained cell works is
evidence of a false promotion, so demand exactly that. It is the same "fails to
index half the frames" test the long-axis rescue below already uses.

Battery over 37 rotation crystals: 33/37 space groups matching XDS with one hard
failure becomes 34/37 with none, and the other 36 crystals are identical in every
printed statistic (checked against a repeat run of the previous binary, which
itself differs on one crystal by one observation). The extra validation pass runs
only where the constrained cell already failed - 71 ms in a 15 s run - and not at
all on the 34 crystals whose constrained cell indexes a majority.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

image_analysis/rotation_indexer/RotationIndexer.cpp

@@ -274,6 +274,7 @@ void RotationIndexer::RunIndexing() {
         size_t best_ci = 0;
         XtalOptimizerData best_data;
         LatticeSearchResult best_sr;
+        std::shared_ptr<RotationIndexerResult> best_alt;
         for (size_t ci = 0; ci < n_try; ci++) {
             if (!work[ci].viable)
                 continue;
@@ -290,16 +291,33 @@ void RotationIndexer::RunIndexing() {
             // symmetry (incl. R-centred) indexes comparably (ratio ~0.7). Preferring the constrained
             // cell on a near-tie keeps the real symmetry/centering; the intensities settle the final
             // space group.
+            // When the guard leaves the constrained cell in place, hand the refined triclinic cell to
+            // the caller instead of dropping it: the accumulated-spot fraction separates a false
+            // promotion from genuine symmetry by less than 2x on a lattice that is pseudo-symmetric
+            // to a few tenths of a degree, while the caller's per-frame validation separates the same
+            // pair by more than 20x.
+            std::shared_ptr<RotationIndexerResult> tri_alt;
             if (work[ci].has_tri) {
                 Solved t = tri_f[ci].get();
+                auto as_triclinic = [](LatticeSearchResult s) {
+                    s.system = gemmi::CrystalSystem::Triclinic;
+                    s.centering = 'P';
+                    s.conventional = s.primitive_reduced;
+                    s.reindex = gemmi::Mat33(1, 0, 0, 0, 1, 0, 0, 0, 1);
+                    return s;
+                };
                 if (t.ok && t.frac > 0.3f && frac < 0.5f * t.frac) {
+                    sr = as_triclinic(sr);
                     data = std::move(t.data);
                     ok = true;
                     frac = t.frac;
-                    sr.system = gemmi::CrystalSystem::Triclinic;
-                    sr.centering = 'P';
-                    sr.conventional = sr.primitive_reduced;
-                    sr.reindex = gemmi::Mat33(1, 0, 0, 0, 1, 0, 0, 0, 1);
+                } else if (t.ok) {
+                    tri_alt = std::make_shared<RotationIndexerResult>(RotationIndexerResult{
+                        .lattice = t.data.latt,
+                        .search_result = as_triclinic(sr),
+                        .geom = t.data.geom,
+                        .axis = t.data.axis,
+                    });
                 }
             }
 
@@ -364,6 +382,7 @@ void RotationIndexer::RunIndexing() {
                 best_data = std::move(data);
                 best_sr = sr;
                 best_ci = ci;
+                best_alt = std::move(tri_alt);
             }
         }
 
@@ -372,6 +391,7 @@ void RotationIndexer::RunIndexing() {
             indexed_lattice = best_data.latt;
             updated_geom_ = best_data.geom;
             axis_ = best_data.axis;
+            unconstrained_ = std::move(best_alt);
         }
 
         // Extra (twin) lattices: MultiLatticeSearch derives each rotation by relating the FFT's primary
@@ -469,6 +489,7 @@ std::optional<RotationIndexerResult> RotationIndexer::GetLattice() const {
         .search_result = search_result_,
         .geom = updated_geom_,
         .axis = axis_,
+        .unconstrained = unconstrained_,
     };
 }
 
@@ -479,6 +500,7 @@ void RotationIndexer::ForceResult(const RotationIndexerResult &result) {
     search_result_ = result.search_result;
     updated_geom_ = result.geom;
     axis_ = result.axis;
+    unconstrained_ = result.unconstrained;
 }
 
 bool RotationIndexer::AccumulationFull() const {

image_analysis/rotation_indexer/RotationIndexer.h

@@ -3,6 +3,7 @@
 
 #pragma once
 
+#include <memory>
 #include <vector>
 #include <mutex>
 
@@ -17,6 +18,12 @@ struct RotationIndexerResult {
     LatticeSearchResult search_result;
     DiffractionGeometry geom;
     std::optional<GoniometerAxis> axis;
+    // The same lattice refined WITHOUT the metric symmetry (triclinic P), when there is one to
+    // compare against. The Bravais class is decided on the UNREFINED FFT candidate against a fixed
+    // angular tolerance, so a pseudo-symmetric cell can be promoted a class too far and the
+    // constraint then snaps a real angle to the ideal one; here the caller can settle that on a
+    // statistic the accumulated-spot fraction is too blunt for. Null on the alternative itself.
+    std::shared_ptr<RotationIndexerResult> unconstrained;
 };
 
 class RotationIndexer {
@@ -38,6 +45,7 @@ class RotationIndexer {
     DiffractionGeometry updated_geom_;
     LatticeSearchResult search_result_;
     std::vector<CrystalLattice> extra_lattices_;
+    std::shared_ptr<RotationIndexerResult> unconstrained_;
 
     IndexerThreadPool &indexer_;