v1.0.0-rc.160 (#70)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/IndexAndRefine.cpp

@@ -10,6 +10,7 @@
 #include "indexing/FFTIndexer.h"
 #include "indexing/MultiLatticeSearch.h"
 #include "lattice_search/LatticeSearch.h"
+#include "scale_merge/ReindexAmbiguity.h"
 #include "scale_merge/ScaleOnTheFly.h"
 
 IndexAndRefine::IndexAndRefine(const DiffractionExperiment &x, IndexerThreadPool *indexer, bool retain_outcomes)
@@ -176,6 +177,23 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data
     return outcome;
 }
 
+namespace {
+// Count spots whose fractional Miller index falls within the indexing tolerance of an integer for a
+// given lattice + geometry - the "how well does this model explain the spots" score used by -r multi.
+int CountIndexedSpots(const DiffractionGeometry &geom, const CrystalLattice &latt,
+                      const std::vector<SpotToSave> &spots, float tol_sq) {
+    const Coord a = latt.Vec0(), b = latt.Vec1(), c = latt.Vec2();
+    int n = 0;
+    for (const auto &s : spots) {
+        const Coord recip = s.ReciprocalCoord(geom);
+        const float hf = recip * a, kf = recip * b, lf = recip * c;
+        const float dh = hf - std::round(hf), dk = kf - std::round(kf), dl = lf - std::round(lf);
+        if (dh * dh + dk * dk + dl * dl < tol_sq) ++n;
+    }
+    return n;
+}
+}   // namespace
+
 void IndexAndRefine::RefineGeometryIfNeeded(DataMessage &msg, IndexAndRefine::IndexingOutcome &outcome) {
     if (!outcome.lattice_candidate)
         return;
@@ -187,6 +205,11 @@ void IndexAndRefine::RefineGeometryIfNeeded(DataMessage &msg, IndexAndRefine::In
         .latt = *outcome.lattice_candidate,
         .crystal_system = outcome.symmetry.crystal_system,
         .min_spots = experiment.GetIndexingSettings().GetViableCellMinSpots(),
+        // Match the [30,150] deg bound the indexers already use (FFBIDXIndexer, FFT settings):
+        // the struct default [60,120] clamps a monoclinic beta outside that window (e.g. a C2
+        // beta near 132 deg) to the boundary, corrupting the per-frame cell refinement.
+        .min_angle_deg = 30.0f,
+        .max_angle_deg = 150.0f,
         .refine_beam_center = true,
         .refine_distance_mm = false,
         .refine_detector_angles = false,
@@ -213,6 +236,39 @@ void IndexAndRefine::RefineGeometryIfNeeded(DataMessage &msg, IndexAndRefine::In
                 outcome.beam_center_updated = true;
             }
             break;
+        case GeomRefinementAlgorithmEnum::Flex: {
+            // Try all three refinements per image and keep whichever indexes the most spots. Beam+cell
+            // refinement helps some stills but diverges on sparse spot lists (few spots, long axes),
+            // where it pushes a good lattice out of tolerance; scoring by indexed-spot count lets each
+            // image fall back to orientation-only or no refinement when refinement would hurt. Ties
+            // prefer less refinement (strict >, not >=) to avoid overfitting.
+            const float tol = experiment.GetIndexingSettings().GetTolerance();
+            const float tol_sq = tol * tol;
+
+            XtalOptimizerData d_none = data;
+            XtalOptimizerData d_orient = data;
+            XtalOptimizerRotationOnly(d_orient, msg.spots, 0.2);
+            XtalOptimizerRotationOnly(d_orient, msg.spots, 0.1);
+            XtalOptimizerRotationOnly(d_orient, msg.spots, 0.05);
+            XtalOptimizerData d_beam = data;
+            const bool beam_ok = XtalOptimizer(d_beam, {msg.spots});
+
+            const int s_none = CountIndexedSpots(d_none.geom, d_none.latt, msg.spots, tol_sq);
+            const int s_orient = CountIndexedSpots(d_orient.geom, d_orient.latt, msg.spots, tol_sq);
+            const int s_beam = beam_ok ? CountIndexedSpots(d_beam.geom, d_beam.latt, msg.spots, tol_sq) : -1;
+
+            if (s_beam > s_none && s_beam > s_orient) {
+                data = d_beam;
+                outcome.experiment.BeamX_pxl(data.geom.GetBeamX_pxl())
+                        .BeamY_pxl(data.geom.GetBeamY_pxl());
+                outcome.beam_center_updated = true;
+            } else if (s_orient > s_none) {
+                data = d_orient;
+            } else {
+                data = d_none;
+            }
+            break;
+        }
     }
 
     outcome.lattice_candidate = data.latt;
@@ -302,6 +358,16 @@ void IndexAndRefine::QuickPredictAndIntegrate(DataMessage &msg,
             mos_deg = msg.mosaicity_deg.value();
             mosaicity[msg.number] = mos_deg;
         }
+        // Second pass of the rotation two-pass: widen the prediction to the frame-order-smoothed mosaicity
+        // that RotationScaleMerge fitted in the first pass. Take the MAX with this frame's own estimate so
+        // prediction is never NARROWER than the first pass - a too-narrow smoothed value would otherwise drop
+        // reflections and collapse the multiplicity. (A wider value only helps prediction cover the spot.)
+        if (msg.number >= 0 && msg.number < static_cast<int64_t>(prediction_mosaicity_override_.size())
+            && std::isfinite(prediction_mosaicity_override_[msg.number])
+            && prediction_mosaicity_override_[msg.number] > 0.0f) {
+            mos_deg = std::max(mos_deg, prediction_mosaicity_override_[msg.number]);
+            mosaicity[msg.number] = mos_deg;
+        }
     }
 
     IntegrationOutcome i_outcome{
@@ -324,7 +390,11 @@ void IndexAndRefine::QuickPredictAndIntegrate(DataMessage &msg,
         .wedge_deg = std::fabs(wedge_deg),
         .mosaicity_deg = std::fabs(mos_deg),
         // FWHM -> sigma; 0 when monochromatic, leaving the prediction unchanged.
-        .bandwidth_sigma = experiment.GetBandwidthFWHM().value_or(0.0f) / 2.3548f
+        .bandwidth_sigma = experiment.GetBandwidthFWHM().value_or(0.0f) / 2.3548f,
+        // Experimental stills partiality (off by default): sigma = ewald_dist_cutoff/2 = the per-image
+        // profile radius, so a reflection at the acceptance edge (dist_ewald ~ 2*sigma) keeps p ~ exp(-2).
+        .still_partiality = experiment.GetBraggIntegrationSettings().GetStillPartiality(),
+        .profile_radius_recipA = ewald_dist_cutoff * 0.5f
     };
 
     // Predict, then integrate with the selected integrator (box-sum or profile-fit).
@@ -431,23 +501,29 @@ std::optional<RotationIndexerResult> IndexAndRefine::FinalizeRotationIndexing()
 }
 
 IndexAndRefine &IndexAndRefine::ReferenceIntensities(std::vector<MergedReflection> &reference) {
-    scaling_engine = std::make_unique<ScaleOnTheFly>(experiment, reference);
+    // An external reference is trusted to be in the correct hand, so use it to break the merohedral
+    // indexing ambiguity per image (serial stills index each crystal independently).
+    reindex_resolver = std::make_unique<ReindexAmbiguityResolver>(experiment, reference);
     return *this;
 }
 
 void IndexAndRefine::ScaleImage(DataMessage &msg, IntegrationOutcome& outcome) {
-    if (!scaling_engine)
+    if (!reindex_resolver)
+        return;
+
+    // The external reference fixes the cell/space group, breaks the indexing ambiguity and reports CCref,
+    // but is NEVER a scale anchor: scaling an image against a foreign dataset injects cross-dataset
+    // systematics and is a worse reference than the data's own merge, so scaling self-references at the
+    // post-measurement merge for both workflows. Rotation resolves the ambiguity globally and self-scales
+    // in RotationScaleMerge (ChooseReindex / ReferenceIntensityCC), so there is nothing to do per image.
+    // Stills resolve the merohedral ambiguity per image here (each crystal indexes in a random hand; pick
+    // the hand best-correlated with the reference, once and for good).
+    if (experiment.IsRotationIndexing())
         return;
 
     auto scaling_start_time = std::chrono::steady_clock::now();
-    scaling_engine->Scale(outcome);
+    reindex_resolver->Resolve(outcome.reflections);
     auto scaling_end_time = std::chrono::steady_clock::now();
-
-    scale_cc[msg.number] = outcome.image_scale_cc.value_or(NAN);
-    msg.image_scale_b_factor = outcome.image_scale_b_factor_Ang2;
-    msg.image_scale_factor = outcome.image_scale_g;
-    msg.image_scale_mosaicity = outcome.mosaicity_deg;
-    msg.image_scale_cc = outcome.image_scale_cc;
     msg.image_scale_time_s = std::chrono::duration<float>(scaling_end_time - scaling_start_time).count();
 }
 

image_analysis/IndexAndRefine.h

@@ -17,6 +17,7 @@
 #include "lattice_search/LatticeSearch.h"
 #include "rotation_indexer/RotationIndexer.h"
 #include "rotation_indexer/RotationIndexerCounter.h"
+#include "scale_merge/ReindexAmbiguity.h"
 #include "scale_merge/ScaleOnTheFly.h"
 #include "scale_merge/ScalingResult.h"
 #include "IntegrationOutcome.h"
@@ -63,6 +64,11 @@ class IndexAndRefine {
     mutable std::mutex reflections_mutex;
     std::vector<IntegrationOutcome> integration_outcome;
     std::vector<float> mosaicity;
+    // Optional per-frame mosaicity used for Bragg prediction, indexed by image number. When set (the
+    // second pass of the rotation two-pass), it overrides the per-image spot-shape estimate so prediction
+    // uses the frame-order-SMOOTHED mosaicity that RotationScaleMerge already fitted in the first pass,
+    // rather than re-deriving it from scratch.
+    std::vector<float> prediction_mosaicity_override_;
     std::vector<float> scale_cc;
     std::vector<std::optional<UnitCell> > unit_cells;
 
@@ -81,7 +87,7 @@ class IndexAndRefine {
                                  const BraggIntegrateFn &integrate,
                                  const IndexingOutcome &outcome);
 
-    std::unique_ptr<ScaleOnTheFly> scaling_engine;
+    std::unique_ptr<ReindexAmbiguityResolver> reindex_resolver;
     void ScaleImage(DataMessage &msg, IntegrationOutcome& outcome);
 
     std::optional<float> RotationAngle(int64_t image) const;   // mid-exposure angle for the indexer
@@ -91,6 +97,10 @@ public:
     void AddImageToRotationIndexer(DataMessage &msg);
     void ForceRotationIndexerLattice(const CrystalLattice& lattice);
     void ForceRotationIndexerResult(const RotationIndexerResult& result);
+    // Supply a per-frame (by image number) mosaicity for prediction, overriding the per-image estimate.
+    void SetPredictionMosaicityOverride(std::vector<float> mosaicity_per_frame) {
+        prediction_mosaicity_override_ = std::move(mosaicity_per_frame);
+    }
 
     void ProcessImage(DataMessage &msg, const SpotFindingSettings &settings,
                       BraggPrediction &prediction, const BraggIntegrateFn &integrate);

image_analysis/LoadFCalcFromMtz.cpp

@@ -36,6 +36,18 @@ const gemmi::Mtz::Column* SelectDefaultColumn(const gemmi::Mtz& mtz, bool& squar
             square = false;
             return &c;
         }
+    // Fall back to a plain structure-factor amplitude (type F): a deposited/observed FP or FOBS, or a
+    // lone FC, squared to an intensity like F-model. Lets a reference MTZ that carries only amplitudes
+    // (no F-model or IMEAN/I - e.g. a deposition that has only FP) still seed CCref
+    // without an explicit --reference-column. Only these named observed/calculated labels are matched:
+    // a bare "first type-F column" catch-all would silently pick up map coefficients (FWT, DELFWT,
+    // FC_ALL) and seed CCref from the wrong quantity - require --reference-column in that case instead.
+    for (const char* label : {"FP", "FOBS", "F", "FC", "Fobs", "F-obs"}) {
+        if (const auto* col = mtz.column_with_label(label, nullptr, 'F')) {
+            square = true;
+            return col;
+        }
+    }
     return nullptr;
 }
 
@@ -63,7 +75,7 @@ ReferenceMtzData LoadReferenceMtz(const std::string& path,
     } else {
         col = SelectDefaultColumn(mtz, out.squared);
         if (col == nullptr)
-            throw std::runtime_error("MTZ has no F-model or intensity (J) column to use as reference");
+            throw std::runtime_error("MTZ has no F-model, intensity (J) or amplitude (F) column to use as reference");
         out.default_column = true;
     }
     out.used_column = col->label;

image_analysis/WriteReflections.cpp

@@ -3,9 +3,13 @@
 
 #include "WriteReflections.h"
 #include "scale_merge/Merge.h"
+#include "scale_merge/HKLKey.h"
 #include "scale_merge/TwinningAnalysis.h"
 
+#include <algorithm>
 #include <cmath>
+#include <map>
+#include <tuple>
 #include <fstream>
 #include <iomanip>
 #include <sstream>
@@ -55,6 +59,84 @@ std::string CifStr(const std::string& s) {
     return s;
 }
 
+// One output row per reflection in the standard CCP4 anomalous layout: the merged mean (IMEAN / F)
+// plus the two Bijvoet mates (I(+)/I(-), F(+)/F(-)). The merge keeps the two mates as separate rows
+// (I+ under the ASU representative hkl, I- under -hkl); this collapses them into one row so the MTZ
+// and SHELX writers share one row list. has_anom is set false when NO reflection carries an anomalous
+// split (e.g. the stills path) - callers then omit the +/- columns.
+struct MergedOutRow {
+    int h = 0, k = 0, l = 0;
+    float Imean = NAN, sImean = NAN, Ip = NAN, sIp = NAN, Im = NAN, sIm = NAN;
+    float Fmean = NAN, sFmean = NAN, Fp = NAN, sFp = NAN, Fm = NAN, sFm = NAN;
+    int rfree = 0;
+};
+
+std::vector<MergedOutRow> BuildMergedRows(const std::vector<MergedReflection> &reflections,
+                                          const DiffractionExperiment &experiment,
+                                          bool &has_anom) {
+    std::vector<MergedOutRow> out_rows;
+    has_anom = true;
+
+    if (experiment.GetScalingSettings().GetMergeFriedel()) {
+        // Friedel-merged: IMEAN is the already-merged intensity (r.I). I(+)/I(-) are carried verbatim
+        // from the Bijvoet split the merge kept (rotation always does; scaled non-anomalously), so a weak
+        // anomalous signal is preserved without reprocessing. A reflection with only one mate, or a
+        // centric, gets a missing value (NaN) for the absent hand. When NO reflection has an anomalous
+        // split (e.g. the stills path, which does not compute one) the anomalous columns are omitted.
+        has_anom = std::any_of(reflections.begin(), reflections.end(),
+            [](const MergedReflection& r){ return std::isfinite(r.I_plus) || std::isfinite(r.I_minus); });
+        out_rows.reserve(reflections.size());
+        for (const auto& r : reflections)
+            out_rows.push_back({r.h, r.k, r.l, r.I, r.sigma, r.I_plus, r.sigma_plus, r.I_minus,
+                                r.sigma_minus, r.F, r.sigmaF, r.F_plus, r.sigmaF_plus, r.F_minus,
+                                r.sigmaF_minus, r.rfree_flag ? 1 : 0});
+    } else {
+        // Anomalous: group the two mates by their (shared) Friedel-merged ASU representative, then form
+        // IMEAN / F as their inverse-variance Friedel mean. A single generator gives both the group key
+        // (its hkl, identical for +hkl and -hkl) and which mate this row is (.plus).
+        const HKLKeyGenerator key_gen(false, experiment.GetSpaceGroupNumber().value_or(1));
+        struct AnomRow {
+            int h = 0, k = 0, l = 0;
+            float Ip = NAN, sIp = NAN, Im = NAN, sIm = NAN;
+            float Fp = NAN, sFp = NAN, Fm = NAN, sFm = NAN;
+            int rfree = 0;
+        };
+        std::map<std::tuple<int, int, int>, AnomRow> rows;
+        for (const auto& r : reflections) {
+            const HKLKey key = key_gen(r);
+            AnomRow& row = rows[{key.h, key.k, key.l}];
+            row.h = key.h; row.k = key.k; row.l = key.l;
+            row.rfree = r.rfree_flag ? 1 : 0;
+            if (key.plus) { row.Ip = r.I; row.sIp = r.sigma; row.Fp = r.F; row.sFp = r.sigmaF; }
+            else          { row.Im = r.I; row.sIm = r.sigma; row.Fm = r.F; row.sFm = r.sigmaF; }
+        }
+
+        // Friedel-mean of the two mates by inverse variance (the single mate, if only one was measured).
+        const auto combine = [](float a, float sa, float b, float sb, float& val, float& sig) {
+            const bool ok_a = std::isfinite(a) && sa > 0.0f;
+            const bool ok_b = std::isfinite(b) && sb > 0.0f;
+            if (ok_a && ok_b) {
+                const double wa = 1.0 / (static_cast<double>(sa) * sa);
+                const double wb = 1.0 / (static_cast<double>(sb) * sb);
+                val = static_cast<float>((wa * a + wb * b) / (wa + wb));
+                sig = static_cast<float>(1.0 / std::sqrt(wa + wb));
+            } else if (ok_a) { val = a; sig = sa; }
+            else if (ok_b)   { val = b; sig = sb; }
+            else             { val = NAN; sig = NAN; }
+        };
+
+        out_rows.reserve(rows.size());
+        for (const auto& [hkl, row] : rows) {
+            float i_mean, sig_i_mean, f_mean, sig_f_mean;
+            combine(row.Ip, row.sIp, row.Im, row.sIm, i_mean, sig_i_mean);
+            combine(row.Fp, row.sFp, row.Fm, row.sFm, f_mean, sig_f_mean);
+            out_rows.push_back({row.h, row.k, row.l, i_mean, sig_i_mean, row.Ip, row.sIp, row.Im,
+                                row.sIm, f_mean, sig_f_mean, row.Fp, row.sFp, row.Fm, row.sFm, row.rfree});
+        }
+    }
+    return out_rows;
+}
+
 } // namespace
 
 
@@ -83,10 +165,9 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
     out << "#\n";
 
     // ---------- _software ----------
-    out << "_software.name         'Jungfraujoch'\n";
-
+    out << "_software.name         'Rugnux'\n";
     out << "_software.version      " << CifStr(jfjoch_version()) << "\n";
-    out << "_software.classification reduction\n";
+    out << "_software.classification 'data reduction'\n";
     out << "#\n";
 
     // ---------- _cell ----------
@@ -115,6 +196,12 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
 
     out << "_diffrn_radiation_wavelength.wavelength  " << Fmt(experiment.GetWavelength_A(), 5) << "\n";
     out << "_diffrn_detector.detector  " << CifStr(experiment.GetDetectorDescription()) << "\n";
+    // Detector geometry actually used for integration (refined, when geometry refinement ran - the rotation
+    // two-pass or the stills global refinement update it on experiment_ before the written pass). jfjoch_
+    // local-data-name items: feedback of the distance / beam centre the data was reduced with.
+    out << "_diffrn_detector.jfjoch_distance_mm       " << Fmt(experiment.GetDetectorDistance_mm(), 4) << "\n";
+    out << "_diffrn_detector.jfjoch_beam_center_x_pxl " << Fmt(experiment.GetBeamX_pxl(), 2) << "\n";
+    out << "_diffrn_detector.jfjoch_beam_center_y_pxl " << Fmt(experiment.GetBeamY_pxl(), 2) << "\n";
     out << "#\n";
 
     // ---------- merging statistics (_reflns overall + _reflns_shell loop) ----------
@@ -131,6 +218,9 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
                ? 100.0 * static_cast<double>(s.unique_reflections) / s.possible_unique_reflections : 0.0; };
     if (!statistics.shells.empty()) {
         const auto &ov = statistics.overall;
+        // Anomalous signal-to-noise (SigAno) is written only when an anomalous split was made, so a
+        // non-anomalous merge keeps its previous stats block / shell-loop columns unchanged.
+        const bool has_anom = std::isfinite(ov.abs_diff_over_sigma_anomalous);
         out << "_reflns.d_resolution_high         " << Fmt(ov.d_min, 2) << "\n";
         out << "_reflns.d_resolution_low          " << Fmt(ov.d_max, 2) << "\n";
         out << "_reflns.number_obs                " << ov.unique_reflections << "\n";
@@ -140,7 +230,14 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
         out << "_reflns.pdbx_netI_over_sigmaI     " << Fmt(ov.mean_i_over_sigma, 2) << "\n";
         out << "_reflns.pdbx_Rrim_I_all           " << Fmt(ov.r_meas, 4) << "\n";
         out << "_reflns.pdbx_CC_half              " << Fmt(ov.cc_half, 4) << "\n";
+        if (has_anom)
+            out << "_reflns.pdbx_absDiff_over_sigma_anomalous  " << Fmt(ov.abs_diff_over_sigma_anomalous, 3)
+                << "   # SigAno = <|dano|>/<sigma(dano)>\n";
         out << "_reflns.jfjoch_diffrn_ISa             " << CifStr(isa) << "   # asymptotic I/sigma (Diederichs)\n";
+        // Dataset-wide isotropic Wilson B-factor estimate (standard PDBx item), analogous to XDS's
+        // "WILSON LINE ... B=". Emitted only when the log-linear fit succeeded.
+        if (std::isfinite(statistics.wilson_b) && statistics.wilson_b > 0.0)
+            out << "_reflns.B_iso_Wilson_estimate     " << Fmt(statistics.wilson_b, 2) << "\n";
         // Twinning indicators (no standard mmCIF item; same jfjoch local prefix as ISa above).
         if (twinning.l_test_pairs > 0) {
             out << "_reflns.jfjoch_L_test_mean_abs_L      " << Fmt(twinning.mean_abs_l, 3)
@@ -151,8 +248,28 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
         if (twinning.moment_reflections > 0)
             out << "_reflns.jfjoch_second_moment_I        " << Fmt(twinning.second_moment, 3)
                 << "   # <I^2>/<I>^2 (untwinned 2.00, perfect twin 1.50)\n";
+        // Radiation-damage monitor (rotation): the relative Debye-Waller B change from the first to the last
+        // frame (A^2). A large magnitude flags a dose-dependent resolution-scale change = radiation damage;
+        // positive is the typical direction (high-resolution intensity fades with dose). No standard mmCIF item.
+        if (std::isfinite(statistics.radiation_damage_delta_b))
+            out << "_reflns.jfjoch_radiation_damage_relative_B  " << Fmt(statistics.radiation_damage_delta_b, 2)
+                << "   # relative-B first->last over the run (A^2); + = high-res fades with dose\n";
         out << "#\n";
 
+        // Per-batch relative-B curve (the radiation-damage monitor, rotation): one relative Debye-Waller B
+        // per rotation-range batch, measured before any correction. rotation_start_deg = id * batch_deg.
+        if (!statistics.radiation_damage_b_batch.empty()) {
+            out << "loop_\n";
+            out << "_jfjoch_radiation_damage_batch.id\n";
+            out << "_jfjoch_radiation_damage_batch.rotation_start_deg\n";
+            out << "_jfjoch_radiation_damage_batch.relative_B\n";
+            for (size_t i = 0; i < statistics.radiation_damage_b_batch.size(); ++i)
+                out << "  " << (i + 1) << "  "
+                    << Fmt(static_cast<double>(i) * statistics.radiation_damage_batch_deg, 1) << "  "
+                    << Fmt(statistics.radiation_damage_b_batch[i], 2) << "\n";
+            out << "#\n";
+        }
+
         out << "loop_\n";
         out << "_reflns_shell.d_res_high\n";
         out << "_reflns_shell.d_res_low\n";
@@ -163,13 +280,18 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
         out << "_reflns_shell.meanI_over_sigI_obs\n";
         out << "_reflns_shell.pdbx_Rrim_I_all\n";
         out << "_reflns_shell.pdbx_CC_half\n";
+        if (has_anom)
+            out << "_reflns_shell.pdbx_absDiff_over_sigma_anomalous\n";
         for (const auto &s : statistics.shells) {
             if (s.unique_reflections == 0)
                 continue;
             out << Fmt(s.d_min, 2) << " " << Fmt(s.d_max, 2) << " "
                 << s.total_observations << " " << s.unique_reflections << " "
                 << Fmt(mult(s), 2) << " " << Fmt(compl_pct(s), 1) << " "
-                << Fmt(s.mean_i_over_sigma, 2) << " " << Fmt(s.r_meas, 4) << " " << Fmt(s.cc_half, 4) << "\n";
+                << Fmt(s.mean_i_over_sigma, 2) << " " << Fmt(s.r_meas, 4) << " " << Fmt(s.cc_half, 4);
+            if (has_anom)
+                out << " " << Fmt(s.abs_diff_over_sigma_anomalous, 3);
+            out << "\n";
         }
         out << "#\n";
     }
@@ -181,8 +303,16 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
     out << "_refln.index_l\n";
     out << "_refln.intensity_meas\n";
     out << "_refln.intensity_sigma\n";
+    out << "_refln.pdbx_I_plus\n";
+    out << "_refln.pdbx_I_plus_sigma\n";
+    out << "_refln.pdbx_I_minus\n";
+    out << "_refln.pdbx_I_minus_sigma\n";
     out << "_refln.F_meas_au\n";
     out << "_refln.F_meas_sigma_au\n";
+    out << "_refln.pdbx_F_plus\n";
+    out << "_refln.pdbx_F_plus_sigma\n";
+    out << "_refln.pdbx_F_minus\n";
+    out << "_refln.pdbx_F_minus_sigma\n";
     out << "_refln.status_free\n";
     out << "_refln.status\n";
 
@@ -192,8 +322,16 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
             << std::setw(5) << r.l << " "
             << std::setw(14) << Fmt(r.I, 4) << " "
             << std::setw(14) << Fmt(r.sigma, 4) << " "
+            << std::setw(14) << Fmt(r.I_plus, 4) << " "
+            << std::setw(14) << Fmt(r.sigma_plus, 4) << " "
+            << std::setw(14) << Fmt(r.I_minus, 4) << " "
+            << std::setw(14) << Fmt(r.sigma_minus, 4) << " "
             << std::setw(14) << Fmt(r.F, 4) << " "
             << std::setw(14) << Fmt(r.sigmaF, 4) << " "
+            << std::setw(14) << Fmt(r.F_plus, 4) << " "
+            << std::setw(14) << Fmt(r.sigmaF_plus, 4) << " "
+            << std::setw(14) << Fmt(r.F_minus, 4) << " "
+            << std::setw(14) << Fmt(r.sigmaF_minus, 4) << " "
             << (r.rfree_flag ? 1 : 0) << " "
             << "o"  // 'o' = observed
             << "\n";
@@ -215,57 +353,111 @@ void WriteMtzReflections(const std::vector<MergedReflection> &reflections,
         experiment.GetSpaceGroupNumber().value_or(1));
     mtz.set_cell_for_all(unitCell);
 
+    // Producing-software provenance in the MTZ header (title + HISTORY, the CCP4 convention).
+    mtz.title = "Rugnux merged reflections";
+    mtz.history.push_back("From Rugnux " + jfjoch_version() + ", data reduction");
+
     // Add dataset
     gemmi::Mtz::Dataset& ds = mtz.add_dataset("native");
     ds.crystal_name = experiment.GetSampleName();
     ds.wavelength = experiment.GetWavelength_A();
 
-    int dataset_id = ds.id;
+    const int dataset_id = ds.id;
+
+    // One row per reflection in the CCP4 anomalous layout (IMEAN + I(+)/I(-), and the same split for
+    // the French-Wilson amplitude), which aimless / ctruncate / mtz2sca / ANODE read directly.
+    bool has_anom = true;
+    const std::vector<MergedOutRow> out_rows = BuildMergedRows(reflections, experiment, has_anom);
 
-    // Add columns
     mtz.add_column("H", 'H', dataset_id, -1, false);
     mtz.add_column("K", 'H', dataset_id, -1, false);
     mtz.add_column("L", 'H', dataset_id, -1, false);
     mtz.add_column("IMEAN", 'J', dataset_id, -1, false);
     mtz.add_column("SIGIMEAN", 'Q', dataset_id, -1, false);
-    mtz.add_column("F", 'F', dataset_id, -1, false);      // French-Wilson amplitude
+    if (has_anom) {
+        mtz.add_column("I(+)", 'K', dataset_id, -1, false);
+        mtz.add_column("SIGI(+)", 'M', dataset_id, -1, false);
+        mtz.add_column("I(-)", 'K', dataset_id, -1, false);
+        mtz.add_column("SIGI(-)", 'M', dataset_id, -1, false);
+    }
+    mtz.add_column("F", 'F', dataset_id, -1, false);       // French-Wilson amplitude
     mtz.add_column("SIGF", 'Q', dataset_id, -1, false);
+    if (has_anom) {
+        mtz.add_column("F(+)", 'G', dataset_id, -1, false);
+        mtz.add_column("SIGF(+)", 'L', dataset_id, -1, false);
+        mtz.add_column("F(-)", 'G', dataset_id, -1, false);
+        mtz.add_column("SIGF(-)", 'L', dataset_id, -1, false);
+    }
     mtz.add_column("FreeR_flag", 'I', dataset_id, -1, false);
 
-    // Number of rows
-    mtz.nreflections = reflections.size();
-
-    // Allocate data table
-    mtz.data.reserve(reflections.size() * 8);
-
-    for (const auto& r : reflections) {
-        mtz.data.push_back(static_cast<float>(r.h));
-        mtz.data.push_back(static_cast<float>(r.k));
-        mtz.data.push_back(static_cast<float>(r.l));
-        mtz.data.push_back(r.I);
-        mtz.data.push_back(r.sigma);
-        mtz.data.push_back(r.F);
-        mtz.data.push_back(r.sigmaF);
-        mtz.data.push_back(r.rfree_flag ? 1 : 0);
+    mtz.nreflections = static_cast<int>(out_rows.size());
+    mtz.data.reserve(out_rows.size() * (has_anom ? 16 : 8));
+    for (const auto& row : out_rows) {
+        mtz.data.push_back(static_cast<float>(row.h));
+        mtz.data.push_back(static_cast<float>(row.k));
+        mtz.data.push_back(static_cast<float>(row.l));
+        mtz.data.push_back(row.Imean);
+        mtz.data.push_back(row.sImean);
+        if (has_anom) {
+            mtz.data.push_back(row.Ip);
+            mtz.data.push_back(row.sIp);
+            mtz.data.push_back(row.Im);
+            mtz.data.push_back(row.sIm);
+        }
+        mtz.data.push_back(row.Fmean);
+        mtz.data.push_back(row.sFmean);
+        if (has_anom) {
+            mtz.data.push_back(row.Fp);
+            mtz.data.push_back(row.sFp);
+            mtz.data.push_back(row.Fm);
+            mtz.data.push_back(row.sFm);
+        }
+        mtz.data.push_back(static_cast<float>(row.rfree));
     }
-
-    // Write MTZ
     mtz.write_to_file(filename);
 }
 
-void WriteHKLReflections(const std::vector<MergedReflection> &reflections,
-                         const std::string &filename) {
-    std::ofstream hkl_file(filename);
-    if (!hkl_file)
-        throw JFJochException(JFJochExceptionCategory::FileWriteError, "Cannot open file " + filename + " for writing");
+void WriteShelxHklReflections(const std::vector<MergedReflection> &reflections,
+                              const DiffractionExperiment &experiment,
+                              const std::string &filename) {
+    bool has_anom = true;
+    const std::vector<MergedOutRow> rows = BuildMergedRows(reflections, experiment, has_anom);
 
-    for (const auto &r: reflections)
-        hkl_file << r.h << " " << r.k << " " << r.l << " "
-                << r.I << " " << r.sigma << " "
-                << (r.rfree_flag ? 1 : 0) << " "
-                << r.F << " " << r.sigmaF << std::endl;
+    // SHELX HKLF 4 (SHELXC / ANODE input): fixed FORMAT(3I4,2F8.2), one record per reflection as
+    // h k l I sigma(I). The Bijvoet mates are written separately - I(+) at +hkl, I(-) at -hkl - so the
+    // anomalous differences survive; a reflection with no anomalous split is written once as its mean.
+    // Intensities are put on a common scale so the largest value fits the F8.2 field (the absolute scale
+    // is irrelevant to SHELXC / ANODE, which use only ratios); I and sigma share the scale, so the
+    // anomalous signal is untouched. The file ends with a 0 0 0 terminator record.
+    const auto usable = [](float v, float s) { return std::isfinite(v) && std::isfinite(s) && s > 0.0f; };
 
-    hkl_file.close();
+    double max_abs = 0.0;
+    for (const auto& r : rows) {
+        if (usable(r.Ip, r.sIp)) max_abs = std::max({max_abs, std::fabs(double(r.Ip)), double(r.sIp)});
+        if (usable(r.Im, r.sIm)) max_abs = std::max({max_abs, std::fabs(double(r.Im)), double(r.sIm)});
+        if (!usable(r.Ip, r.sIp) && !usable(r.Im, r.sIm) && usable(r.Imean, r.sImean))
+            max_abs = std::max({max_abs, std::fabs(double(r.Imean)), double(r.sImean)});
+    }
+    const double scale = (std::isfinite(max_abs) && max_abs > 0.0) ? 9999.0 / max_abs : 1.0;
+
+    std::ofstream out(filename);
+    if (!out)
+        throw std::runtime_error("WriteShelxHklReflections: cannot open " + filename);
+    out << std::fixed << std::setprecision(2);
+    const auto emit = [&out, scale](int h, int k, int l, float I, float sigma) {
+        out << std::setw(4) << h << std::setw(4) << k << std::setw(4) << l
+            << std::setw(8) << scale * I << std::setw(8) << scale * sigma << "\n";
+    };
+    for (const auto& r : rows) {
+        const bool plus = usable(r.Ip, r.sIp);
+        const bool minus = usable(r.Im, r.sIm);
+        if (plus)  emit(r.h, r.k, r.l, r.Ip, r.sIp);
+        if (minus) emit(-r.h, -r.k, -r.l, r.Im, r.sIm);
+        if (!plus && !minus && usable(r.Imean, r.sImean))
+            emit(r.h, r.k, r.l, r.Imean, r.sImean);
+    }
+    emit(0, 0, 0, 0.0f, 0.0f);   // HKLF-4 end-of-data marker
+    out.close();
 }
 
 void WriteReflections(const std::vector<MergedReflection> &reflections,
@@ -275,15 +467,10 @@ void WriteReflections(const std::vector<MergedReflection> &reflections,
                       const std::string &isa,
                       const TwinningAnalysisResult &twinning,
                       const std::string &filename) {
-    switch (experiment.GetScalingSettings().GetFileFormat()) {
-        case IntensityFormat::Text:
-            WriteHKLReflections(reflections, filename + ".hkl");
-            break;
-        case IntensityFormat::mmCIF:
-            WriteMmcifReflections(reflections, unitCell, experiment, statistics, isa, twinning, filename + ".cif");
-            break;
-        case IntensityFormat::MTZ:
-            WriteMtzReflections(reflections, unitCell, experiment, filename + ".mtz");
-            break;
-    }
+    // Write an MTZ, an mmCIF and a SHELX HKLF-4 .hkl - each has its uses downstream (MTZ for the CCP4 /
+    // phenix reflection tools, mmCIF for deposition and as the self-describing native format, HKLF-4 as
+    // the SHELXC / ANODE substructure-solution input).
+    WriteMtzReflections(reflections, unitCell, experiment, filename + ".mtz");
+    WriteMmcifReflections(reflections, unitCell, experiment, statistics, isa, twinning, filename + ".cif");
+    WriteShelxHklReflections(reflections, experiment, filename + ".hkl");
 }

image_analysis/WriteReflections.h

@@ -26,8 +26,10 @@ void WriteMtzReflections(const std::vector<MergedReflection> &reflections,
                          const DiffractionExperiment &experiment,
                          const std::string &filename);
 
-void WriteHKLReflections(const std::vector<MergedReflection> &reflections,
-                          const std::string &filename);
+// SHELX HKLF-4 text file (h k l I sigma(I), Bijvoet mates separate) for SHELXC / ANODE.
+void WriteShelxHklReflections(const std::vector<MergedReflection> &reflections,
+                              const DiffractionExperiment &experiment,
+                              const std::string &filename);
 
 void WriteReflections(const std::vector<MergedReflection> &reflections,
                          const UnitCell &unitCell,

image_analysis/beam_stop/SHADOW_FINDER.md

@@ -0,0 +1,138 @@
+# Beam-stop shadow finder
+
+`ShadowFinder` detects the beam-stop shadow — the central **disk** plus its **holder arm** —
+from a small number of images, and returns a per-pixel mask of the shadowed region. It exists
+so the pipeline can mask the beam stop **automatically** instead of relying on a hand-drawn user
+mask.
+
+It is written in the accumulate-then-finalize shape of `DarkMaskAnalysis`: feed frames with
+`AddImage()`, then read the mask once with `GetMask()`. **It is not yet wired into any workflow**
+— see *Wiring plan* below. The class is self-contained (`common/` only) and unit-testable.
+
+## API
+
+```cpp
+ShadowFinder finder(experiment);                 // geometry (beam centre, converted size)
+std::vector<uint8_t> scratch;
+for (int i : first_N_frames)                      // ~20–50 spread frames is plenty (see below)
+    finder.AddImage(read_frame(i), scratch);      // DataMessage with .image, like DarkMaskAnalysis
+std::vector<uint32_t> shadow = finder.GetMask();  // 1 = shadow, 0 = keep; converted geometry
+```
+
+- **Input** (`AddImage`): a full **converted-geometry** image inside a `DataMessage`, dispatched by
+  pixel type exactly like `DarkMaskAnalysis::AnalyzeImage`. A pixel equal to its type's sentinel
+  extreme (`INT*_MIN` for signed, `UINT*_MAX` for unsigned) is treated as *no data* (module gap /
+  masked) and skipped. For signed types the opposite extreme (`INT*_MAX`, a genuine saturation
+  clamp) is kept, so saturated reflections still register as bright. Thread-safe.
+- **Output** (`GetMask`): a `0/1` vector of size `GetPixelsNumConv()`, ready to feed
+  `PixelMask::LoadMask(..., bit)`. Recomputed from the accumulators on each call — call it once at
+  the end.
+- **Settings** (`ShadowFinderSettings`): plain struct with defaults (below). When wired, this can
+  move onto `DiffractionExperiment` the way `DarkMaskSettings` does.
+
+## Algorithm
+
+The shadow is a **stationary, azimuthal anomaly**: air scatter makes the background fall off
+smoothly with radius but stay roughly constant around each ring, so the beam stop is a localized
+dip at a fixed place. Detection therefore compares each pixel to the typical background **at its
+own radius**, not to a global threshold.
+
+1. **Projection.** Per pixel accumulate the **max** and the **mean** (sum + count) over the added
+   frames, skipping the no-data sentinel. The stop is stationary, so it stays dark while spots and
+   the air-scatter halo build up around it.
+2. **Robust radial baseline.** Take the **median of the mean-projection in each 1-pixel ring**
+   about the beam centre. The median ignores the shadow (a minority of each ring) and the spots.
+   Iterate 3×, excluding the shadow it finds, so the baseline is not pulled down by the shadow.
+3. **Ratio.** `ratio = mean / baseline(r)`. Shadowed pixels sit far below 1; the smooth halo and
+   symmetric ice rings cancel to ≈1 and are **not** flagged.
+4. **Core + connectivity.** `core = ratio < shadow_ratio`, then keep only the part **connected to
+   the beam centre** (flood from a small central seed through a `bridge_px`-dilated copy, so the
+   arm crosses module gaps). The centre anchor is what makes it specific to the beam stop rather
+   than any random low patch.
+5. **Reflection guard.** A pixel whose **max**-projection reached `min_reflection` recorded a real
+   reflection — a beam stop cannot block a reflection that was measured — so it is **never masked**
+   (a small-cluster test drops single-pixel zingers).
+6. **Central disk (low-res guard).** From the azimuthal **blocked fraction** about the beam centre,
+   mask a full disk out to the radius where nearly every azimuth is blocked (a disk blocks all
+   directions; a thin arm/gap does not). Returns nothing when the centre is not behind a disk, and
+   is **capped just inside the innermost reflection** so a real edge spot is never swallowed. This
+   guards the low-resolution zone the stop makes un-measurable, including a disk the raw data has
+   already voided.
+7. **Soft boundary.** Grow the core+disk into partially-shadowed pixels (`ratio < penumbra_ratio`)
+   within `penumbra_max_px`, then a morphological close rounds the contour and hole-filling closes
+   the disk interior.
+8. **Carve reflections last.** Remove any lit pixel from the final mask (no fill afterwards) so a
+   spot the geometry still covered is given back rather than re-enclosed.
+
+Three findings from the prototype are baked in: the **mean** ratio (not the max) does the
+detecting — the max projection is a great *picture* but a noisier detector; the **centre anchor**
+is what distinguishes the stop from stray low patches; and **a lit pixel is never masked**.
+Detection converges in **~2–3 frames** (the shadow cuts the halo present in every frame); 20–50 is
+just safe headroom.
+
+## Parameters (`ShadowFinderSettings`)
+
+| field | default | meaning |
+|---|---|---|
+| `shadow_ratio` | 0.35 | core threshold: mean below this fraction of the radial baseline |
+| `penumbra_ratio` | 0.72 | soft boundary grows out to this ratio … |
+| `penumbra_max_px` | 14 | … but no further than this from the core |
+| `bridge_px` | 6 | bridge module gaps / breaks the arm crosses |
+| `min_reflection` | 25 | max-projection value at/above which a pixel is a protected reflection |
+
+## Complexity & memory
+
+Per `AddImage`: `O(pixels)`. `GetMask`: a handful of `O(pixels)` sweeps (three ring-median passes
+and a few BFS-based morphology ops), i.e. a couple of seconds on a full ~18 MP detector — a
+one-time finalize. Accumulators are `int32 + int64 + uint32` per pixel (~0.3 GB at 18 MP); the
+finalize adds a few transient `float`/`char` buffers. Fine for the offline (Rugnux/viewer)
+context it targets; not intended for the online per-frame hot path.
+
+## Wiring plan (deferred — implement later)
+
+The detector is deliberately reader-agnostic and bit-agnostic: one algorithm, fed frames by each
+caller, with the caller deciding where the resulting mask lands. Two intended consumers:
+
+**1. Build.** Add `beam_stop/ShadowFinder.cpp` + `.h` to the `JFJochImageAnalysis` source list in
+`image_analysis/CMakeLists.txt` (mirroring `dark_mask_analysis/DarkMaskAnalysis.*` at lines 27–28).
+
+**2. Mask bit + loader** (`common/PixelMask.{h,cpp}`): add `BeamStopShadowBit = 9` (bit 9 is free;
+`PixelMask.h:31-36`), a `LoadBeamStopMask` mirroring `LoadDarkBadPixelMask` (`.cpp:298-315`), a
+`beam_stop_pixel` counter in `PixelMaskStatistics` / `GetStatistics` (`.cpp:186-205`), and
+optionally `GetBeamStopMask()` (mirroring `GetUserMask`). Leave `GetUserMask` untouched (keeps the
+user-mask channel clean). Application is then automatic: the CPU/GPU preprocessor masks any pixel
+with a non-zero bit (`ImagePreprocessorCPU.cpp:10`, `ImagePreprocessorGPU.cu:104`), and HDF5
+write/read + CBOR already carry the full bitfield, so bit 9 persists to `_process.h5` and reloads
+with no extra code.
+
+**3. Rugnux offline processing → bit 9** (dataset-specific, per run). Add a `detect_beam_stop`
+flag to `ProcessConfig` (`rugnux/Rugnux.h`) and a `--detect-beam-stop` CLI flag mirroring
+`--detect-ice-rings` in `rugnux/rugnux_cli.cpp`. In `Rugnux::Run()`, after geometry/goniometer
+setup and **before** the azimuthal mapping (`Rugnux.cpp:281`) and the `start_message.pixel_mask`
+persist (`:296`), run a `ShadowFinder` pre-pass over a spread sample (`select_equally_spaced_image_ordinals`
++ `reader_.GetRawImage`, as `RefineStillsGeometry` does at `:96-157`) and
+`pixel_mask_.LoadBeamStopMask(...)`. Everything downstream then applies and persists it for free.
+
+**4. Viewer standalone → user mask (bit 8)** and a viewer preview. A "Find beam stop" action reads
+the first N frames via the worker's `file_reader`, runs `ShadowFinder`, and holds the result as
+bit 9 for a distinct-coloured, toggleable overlay; a "Transfer to user mask" action ORs it into the
+user mask through the existing chokepoint `JFJochImageReadingWorker::UpdateUserMask_i`
+(`.cpp:783`, same idiom as `LoadUserMaskTIFF` add-mode `:851-858`). The distinct colour needs a
+`ColorScaleSpecial::BeamStop` + sentinel branch in `JFJochReaderImage.cpp:117-124`; the on/off
+toggle mirrors "Show saturated pixels".
+
+**5. Docs.** Add a "Bit 9 – beam-stop shadow" row to the bit table in `docs/PIXEL_MASK.md:9-21`.
+
+Not needed: the online broker / receiver / FPGA / REST-API / React-frontend paths — the auto
+beam-stop mask is an **offline** (Rugnux + viewer) feature and never goes to the detector.
+
+## Known limitations / future refinements
+
+- **Boundary smoothing** uses a morphological close; the prototype used a Gaussian level-set for a
+  slightly smoother edge. Upgrade if the edge looks blocky.
+- **Off-centre disk**: the low-res disk guard keys on the *stored* beam centre. If that is off the
+  physical stop (seen on one stills detector), a small already-dead disk gets no extra circular
+  guard — a data-driven disk centre would close it.
+- **Unsigned saturation**: for unsigned images a detector-saturating reflection collides with the
+  no-data sentinel and is skipped; harmless in practice (such reflections do not sit under the
+  stop), and absent entirely on the signed/preprocessed path.

image_analysis/beam_stop/ShadowFinder.cpp

@@ -0,0 +1,338 @@
+// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#include "ShadowFinder.h"
+
+#include <algorithm>
+#include <cmath>
+#include <limits>
+#include <queue>
+#include <type_traits>
+
+#include "../../common/JFJochException.h"
+
+// ---------------------------------------------------------------------------------
+// Small binary-image helpers on a width*height frame stored row-major as char (0/1).
+// All run once, at GetMask() time. The BFS forms keep them O(pixels) rather than
+// O(pixels * radius), so a radius-14 dilation is still a single sweep.
+// ---------------------------------------------------------------------------------
+namespace {
+
+// 8-connected dilation by `r` pixels (Chebyshev), via a multi-source BFS.
+std::vector<char> Dilate(const std::vector<char> &in, int W, int H, int r) {
+    if (r <= 0)
+        return in;
+    std::vector<int> dist(in.size(), -1);
+    std::queue<int> q;
+    for (size_t i = 0; i < in.size(); i++)
+        if (in[i]) { dist[i] = 0; q.push(static_cast<int>(i)); }
+    while (!q.empty()) {
+        const int i = q.front(); q.pop();
+        if (dist[i] >= r)
+            continue;
+        const int y = i / W, x = i % W;
+        for (int dy = -1; dy <= 1; dy++)
+            for (int dx = -1; dx <= 1; dx++) {
+                const int yy = y + dy, xx = x + dx;
+                if (yy < 0 || yy >= H || xx < 0 || xx >= W)
+                    continue;
+                const int j = yy * W + xx;
+                if (dist[j] < 0) { dist[j] = dist[i] + 1; q.push(j); }
+            }
+    }
+    std::vector<char> out(in.size());
+    for (size_t i = 0; i < out.size(); i++)
+        out[i] = (dist[i] >= 0) ? 1 : 0;
+    return out;
+}
+
+// Erosion by `r` = dilation of the complement (image border counts as outside).
+std::vector<char> Erode(const std::vector<char> &in, int W, int H, int r) {
+    std::vector<char> comp(in.size());
+    for (size_t i = 0; i < in.size(); i++)
+        comp[i] = !in[i];
+    const auto grown = Dilate(comp, W, H, r);
+    std::vector<char> out(in.size());
+    for (size_t i = 0; i < out.size(); i++)
+        out[i] = !grown[i];
+    return out;
+}
+
+// Pixels of `passable` reachable from any of `seeds` (8-connected flood).
+std::vector<char> Flood(const std::vector<char> &passable, int W, int H, const std::vector<int> &seeds) {
+    std::vector<char> visited(passable.size(), 0);
+    std::queue<int> q;
+    for (const int s : seeds)
+        if (s >= 0 && s < static_cast<int>(passable.size()) && passable[s] && !visited[s]) {
+            visited[s] = 1; q.push(s);
+        }
+    while (!q.empty()) {
+        const int i = q.front(); q.pop();
+        const int y = i / W, x = i % W;
+        for (int dy = -1; dy <= 1; dy++)
+            for (int dx = -1; dx <= 1; dx++) {
+                const int yy = y + dy, xx = x + dx;
+                if (yy < 0 || yy >= H || xx < 0 || xx >= W)
+                    continue;
+                const int j = yy * W + xx;
+                if (passable[j] && !visited[j]) { visited[j] = 1; q.push(j); }
+            }
+    }
+    return visited;
+}
+
+// Fill holes: background not reachable from the image border becomes region.
+std::vector<char> FillHoles(const std::vector<char> &region, int W, int H) {
+    std::vector<char> bg_visited(region.size(), 0);
+    std::queue<int> q;
+    auto push = [&](int i) { if (!region[i] && !bg_visited[i]) { bg_visited[i] = 1; q.push(i); } };
+    for (int x = 0; x < W; x++) { push(x); push((H - 1) * W + x); }
+    for (int y = 0; y < H; y++) { push(y * W); push(y * W + W - 1); }
+    while (!q.empty()) {
+        const int i = q.front(); q.pop();
+        const int y = i / W, x = i % W;
+        for (int dy = -1; dy <= 1; dy++)
+            for (int dx = -1; dx <= 1; dx++) {
+                const int yy = y + dy, xx = x + dx;
+                if (yy < 0 || yy >= H || xx < 0 || xx >= W)
+                    continue;
+                const int j = yy * W + xx;
+                if (!region[j] && !bg_visited[j]) { bg_visited[j] = 1; q.push(j); }
+            }
+    }
+    std::vector<char> out = region;
+    for (size_t i = 0; i < out.size(); i++)
+        if (!region[i] && !bg_visited[i])
+            out[i] = 1;
+    return out;
+}
+
+// Median of `values` per integer radius, over the pixels flagged in `use`.
+std::vector<float> RingMedian(const std::vector<float> &values, const std::vector<char> &use,
+                              const std::vector<int> &radius, int max_radius) {
+    std::vector<std::vector<float>> bins(max_radius + 1);
+    for (size_t i = 0; i < values.size(); i++)
+        if (use[i])
+            bins[radius[i]].push_back(values[i]);
+    std::vector<float> median(max_radius + 1, 0.0f);
+    for (int r = 0; r <= max_radius; r++) {
+        auto &b = bins[r];
+        if (!b.empty()) {
+            const size_t k = b.size() / 2;
+            std::nth_element(b.begin(), b.begin() + k, b.end());
+            median[r] = b[k];
+        }
+    }
+    return median;
+}
+
+// Fraction of each integer-radius ring that is flagged in `blocked`.
+std::vector<float> RingFraction(const std::vector<char> &blocked, const std::vector<int> &radius, int max_radius) {
+    std::vector<int64_t> num(max_radius + 1, 0), den(max_radius + 1, 0);
+    for (size_t i = 0; i < blocked.size(); i++) {
+        den[radius[i]]++;
+        if (blocked[i]) num[radius[i]]++;
+    }
+    std::vector<float> frac(max_radius + 1, 0.0f);
+    for (int r = 0; r <= max_radius; r++)
+        frac[r] = den[r] ? static_cast<float>(num[r]) / static_cast<float>(den[r]) : 0.0f;
+    return frac;
+}
+
+}  // namespace
+
+// ---------------------------------------------------------------------------------
+
+ShadowFinder::ShadowFinder(const DiffractionExperiment &experiment, ShadowFinderSettings in_settings)
+    : width(static_cast<int>(experiment.GetXPixelsNumConv())),
+      height(static_cast<int>(experiment.GetYPixelsNumConv())),
+      beam_x(experiment.GetBeamX_pxl()),
+      beam_y(experiment.GetBeamY_pxl()),
+      settings(in_settings),
+      max_value(static_cast<size_t>(width) * height, 0),
+      sum_value(static_cast<size_t>(width) * height, 0),
+      valid_count(static_cast<size_t>(width) * height, 0) {}
+
+template<class T>
+void ShadowFinder::Add(const T *ptr) {
+    // The pixel type's sentinel extreme marks "no data" (module gap / masked): the
+    // preprocessor/writer stores INT*_MIN for signed and UINT*_MAX for unsigned. For
+    // signed types the opposite extreme (INT*_MAX) is a genuine saturated value and is
+    // kept, so a saturated reflection still registers as bright.
+    T masked;
+    if constexpr (std::is_signed_v<T>)
+        masked = std::numeric_limits<T>::min();
+    else
+        masked = std::numeric_limits<T>::max();
+
+    std::unique_lock ul(m);
+    for (size_t i = 0; i < max_value.size(); i++) {
+        const T v = ptr[i];
+        if (v == masked)
+            continue;
+        const int32_t vi = static_cast<int32_t>(v);
+        if (valid_count[i] == 0 || vi > max_value[i])
+            max_value[i] = vi;
+        sum_value[i] += vi;
+        valid_count[i]++;
+    }
+    frames++;
+}
+
+void ShadowFinder::AddImage(const DataMessage &data, std::vector<uint8_t> buffer) {
+    if (static_cast<size_t>(data.image.GetWidth()) * data.image.GetHeight() != max_value.size())
+        throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
+                              "ShadowFinder: image size does not match the detector");
+
+    const auto ptr = data.image.GetUncompressedPtr(buffer);
+    switch (data.image.GetMode()) {
+        case CompressedImageMode::Int8:   Add(reinterpret_cast<const int8_t   *>(ptr)); break;
+        case CompressedImageMode::Uint8:  Add(reinterpret_cast<const uint8_t  *>(ptr)); break;
+        case CompressedImageMode::Int16:  Add(reinterpret_cast<const int16_t  *>(ptr)); break;
+        case CompressedImageMode::Uint16: Add(reinterpret_cast<const uint16_t *>(ptr)); break;
+        case CompressedImageMode::Int32:  Add(reinterpret_cast<const int32_t  *>(ptr)); break;
+        case CompressedImageMode::Uint32: Add(reinterpret_cast<const uint32_t *>(ptr)); break;
+        default:
+            throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
+                                  "ShadowFinder: unsupported image mode");
+    }
+}
+
+uint32_t ShadowFinder::GetFrameCount() const {
+    std::unique_lock ul(m);
+    return frames;
+}
+
+std::vector<uint32_t> ShadowFinder::GetMask() const {
+    std::unique_lock ul(m);
+
+    const int W = width, H = height;
+    const int N = W * H;
+    const ShadowFinderSettings &S = settings;
+
+    std::vector<uint32_t> mask(N, 0);
+    if (frames == 0)
+        return mask;
+
+    // --- mean projection, per-pixel validity and radius from the beam centre ---
+    std::vector<float> mean(N, 0.0f);
+    std::vector<char>  valid(N, 0);
+    std::vector<int>   radius(N, 0);
+    int max_radius = 0;
+    for (int y = 0; y < H; y++)
+        for (int x = 0; x < W; x++) {
+            const int i = y * W + x;
+            if (valid_count[i] > 0) {
+                mean[i] = static_cast<float>(static_cast<double>(sum_value[i]) / valid_count[i]);
+                valid[i] = 1;
+            }
+            const double dx = x - beam_x, dy = y - beam_y;
+            const int r = static_cast<int>(std::lround(std::sqrt(dx * dx + dy * dy)));
+            radius[i] = r;
+            if (r > max_radius) max_radius = r;
+        }
+
+    // --- robust radial baseline; iterate to keep the shadow out of its own baseline ---
+    std::vector<float> ratio(N, 1.0f);
+    std::vector<char>  excluded(N, 0);
+    for (int iter = 0; iter < 3; iter++) {
+        std::vector<char> use(N);
+        for (int i = 0; i < N; i++)
+            use[i] = valid[i] && !excluded[i];
+        const auto baseline = RingMedian(mean, use, radius, max_radius);
+        for (int i = 0; i < N; i++)
+            if (valid[i])
+                ratio[i] = mean[i] / std::max(baseline[radius[i]], 1e-6f);
+        for (int i = 0; i < N; i++)
+            excluded[i] = valid[i] && ratio[i] < S.shadow_ratio;
+    }
+
+    // --- shadow core: low-ratio pixels connected to the beam centre (bridging gaps) ---
+    std::vector<char> low(N);
+    for (int i = 0; i < N; i++)
+        low[i] = valid[i] && ratio[i] < S.shadow_ratio;
+
+    const std::vector<char> grown = Dilate(low, W, H, S.bridge_px);
+    std::vector<int> seeds;   // a small disk at the beam centre
+    for (int y = 0; y < H; y++)
+        for (int x = 0; x < W; x++) {
+            const double dx = x - beam_x, dy = y - beam_y;
+            if (dx * dx + dy * dy < 4.0 * 4.0)
+                seeds.push_back(y * W + x);
+        }
+    const std::vector<char> connected = Flood(grown, W, H, seeds);
+    std::vector<char> core(N);
+    for (int i = 0; i < N; i++)
+        core[i] = low[i] && connected[i];
+
+    // --- real reflections: any pixel that recorded signal is never masked. Require a
+    //     small cluster so a single-frame zinger does not count as a reflection. ---
+    std::vector<char> lit(N, 0);
+    for (int i = 0; i < N; i++)
+        lit[i] = (valid_count[i] > 0) && (max_value[i] >= static_cast<int32_t>(S.min_reflection));
+    std::vector<char> reflection(N, 0);
+    for (int y = 0; y < H; y++)
+        for (int x = 0; x < W; x++) {
+            const int i = y * W + x;
+            if (!lit[i]) continue;
+            int neighbours = 0;
+            for (int dy = -1; dy <= 1; dy++)
+                for (int dx = -1; dx <= 1; dx++) {
+                    const int yy = y + dy, xx = x + dx;
+                    if ((dx || dy) && yy >= 0 && yy < H && xx >= 0 && xx < W && lit[yy * W + xx])
+                        neighbours++;
+                }
+            reflection[i] = (neighbours >= 2);
+        }
+
+    // --- central low-res disk: the fully-blocked region about the beam centre. Sized by
+    //     the azimuthal blocked fraction (a disk blocks ~every azimuth; a thin arm or
+    //     gap does not), and capped just inside the innermost reflection. ---
+    std::vector<char> blocked(N);
+    for (int i = 0; i < N; i++)
+        blocked[i] = (valid_count[i] == 0) || low[i];
+    const auto blocked_frac = RingFraction(blocked, radius, max_radius);
+
+    int disk_radius = 0;
+    {
+        float head = 0.0f; int head_n = 0;
+        for (int r = 0; r <= std::min(5, max_radius); r++) { head += blocked_frac[r]; head_n++; }
+        if (head_n > 0 && head / head_n >= 0.65f) {   // the beam centre is behind a disk
+            disk_radius = max_radius;
+            for (int r = 1; r <= max_radius; r++)
+                if (blocked_frac[r] < 0.55f) { disk_radius = r; break; }
+        }
+    }
+    int reflection_radius = max_radius + 1;   // innermost reflection (ignore the very centre)
+    for (int i = 0; i < N; i++)
+        if (reflection[i] && radius[i] > 12 && radius[i] < reflection_radius)
+            reflection_radius = radius[i];
+    if (disk_radius > reflection_radius - 4)
+        disk_radius = reflection_radius - 4;
+    if (disk_radius < 0)
+        disk_radius = 0;
+
+    // --- assemble: core + disk, grow the soft penumbra, round, fill the disk interior ---
+    std::vector<char> region(N);
+    for (int i = 0; i < N; i++)
+        region[i] = core[i] || (disk_radius > 0 && radius[i] < disk_radius);
+
+    const std::vector<char> near = Dilate(region, W, H, S.penumbra_max_px);
+    for (int i = 0; i < N; i++)
+        if (near[i] && valid[i] && ratio[i] < S.penumbra_ratio)
+            region[i] = 1;
+
+    region = Erode(Dilate(region, W, H, 2), W, H, 2);   // close: round the boundary
+    region = FillHoles(region, W, H);
+
+    // Expose recorded reflections - done last, with no fill afterwards, so a spot the
+    // geometry still covered is given back rather than re-enclosed.
+    const std::vector<char> reflection_grown = Dilate(reflection, W, H, 1);
+    for (int i = 0; i < N; i++)
+        if (reflection_grown[i])
+            region[i] = 0;
+
+    for (int i = 0; i < N; i++)
+        mask[i] = region[i] ? 1 : 0;
+    return mask;
+}

image_analysis/beam_stop/ShadowFinder.h

@@ -0,0 +1,76 @@
+// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#pragma once
+
+#include <cstdint>
+#include <mutex>
+#include <vector>
+
+#include "../../common/CompressedImage.h"
+#include "../../common/DiffractionExperiment.h"
+#include "../../common/JFJochMessages.h"   // DataMessage
+
+// Tunable parameters for ShadowFinder. Plain struct with sensible defaults; when the
+// finder is wired into the workflow these can move onto DiffractionExperiment the way
+// DarkMaskSettings does. See SHADOW_FINDER.md for what each one does.
+struct ShadowFinderSettings {
+    // A pixel is "shadow core" when its mean is below this fraction of the typical
+    // (azimuthal-median) background at the same radius.
+    float shadow_ratio = 0.35f;
+
+    // The soft boundary grows outward into partially-shadowed pixels down to this
+    // fraction of the background, but no further than penumbra_max_px from the core.
+    float penumbra_ratio = 0.72f;
+    int   penumbra_max_px = 14;
+
+    // Bridge module gaps / small breaks that the holder arm crosses (pixels).
+    int   bridge_px = 6;
+
+    // A pixel whose max-projection reaches this value recorded a real reflection and is
+    // never masked - a beam stop cannot block a reflection that was measured. This also
+    // caps the central disk just inside the innermost such reflection.
+    float min_reflection = 25.0f;
+};
+
+// Detects the beam-stop shadow (central disk + holder arm) from a small number of
+// images, mirroring the accumulate-then-finalize shape of DarkMaskAnalysis: feed frames
+// with AddImage(), then read the mask once with GetMask(). The returned mask is in
+// converted geometry and is 1 where the beam stop shadows the detector.
+//
+// The shadow is treated as an azimuthal anomaly: a per-radius background baseline is
+// robust to the shadow, so a localized dip connected to the beam centre is the beam
+// stop. See SHADOW_FINDER.md for the full algorithm and the (deferred) wiring plan.
+// Thread-safe: AddImage may be called from several worker threads.
+class ShadowFinder {
+    mutable std::mutex m;
+
+    const int    width;
+    const int    height;
+    const double beam_x;
+    const double beam_y;
+    const ShadowFinderSettings settings;
+
+    // Per-pixel projection over the frames added so far (converted geometry).
+    std::vector<int32_t>  max_value;    // maximum over frames
+    std::vector<int64_t>  sum_value;    // sum of valid values
+    std::vector<uint32_t> valid_count;  // number of frames the pixel carried data
+    uint32_t frames = 0;
+
+    template<class T> void Add(const T *ptr);
+
+public:
+    ShadowFinder(const DiffractionExperiment &experiment, ShadowFinderSettings settings = {});
+
+    // Accumulate one full converted-geometry image into the projection. Gap / masked
+    // pixels (the pixel type's sentinel extreme) are skipped. `buffer` is scratch space
+    // for decompression (mirrors DarkMaskAnalysis::AnalyzeImage).
+    void AddImage(const DataMessage &data, std::vector<uint8_t> buffer);
+
+    // Compute the beam-stop shadow mask (1 = shadow, 0 = keep). Size is the converted
+    // pixel count. Recomputed from the accumulators on each call - meant to be called
+    // once at the end; not cheap (see SHADOW_FINDER.md).
+    [[nodiscard]] std::vector<uint32_t> GetMask() const;
+
+    [[nodiscard]] uint32_t GetFrameCount() const;
+};

image_analysis/bragg_integration/BraggIntegrationEngine.cpp

@@ -68,6 +68,11 @@ BraggIntegrationEngine::BraggIntegrationEngine(const DiffractionExperiment &expe
     beam_y = geom.GetBeamY_pxl();
     use_ellipse = !empirical && (bw_sigma > 0.0 || c_radial > 0.0);
 
+    // Trimmed-mean background applies to monochromatic data - both rotation AND stills (the discriminator
+    // is the beam, not the acquisition mode). Broadband (non-zero bandwidth: pink-beam / DMM) data keep the
+    // tuned high-side sigma-clip instead, so the trim is forced off there. The fraction comes from settings.
+    bkg_trim = broadband ? 0.0f : settings.GetBackgroundTrimFraction();
+
     polarization = experiment.GetPolarizationFactor();
 }
 

image_analysis/bragg_integration/BraggIntegrationEngine.h

@@ -97,6 +97,10 @@ protected:
     double c_radial;             // radial variance coefficient of tan^2(2theta): parallax + capture
     double F_px;                 // detector distance expressed in pixels
     float  beam_x, beam_y;
+    // Effective symmetric trimmed-mean background fraction (BraggIntegrationSettings): the configured
+    // fraction for monochromatic (rotation) data, forced to 0 for broadband (stills, which keep their
+    // high-side sigma-clip). 0 = plain ring mean. Read by both the CPU and GPU engines.
+    float  bkg_trim = 0.0f;
 
     DiffractionGeometry geom;    // kept for the per-reflection polarization correction
     std::optional<float> polarization;

image_analysis/bragg_integration/BraggIntegrationEngineCPU.cpp

@@ -7,6 +7,7 @@
 #include <cmath>
 #include <cstdint>
 #include <limits>
+#include <vector>
 
 #include "../../common/CompressedImage.h"
 #include "../../common/JFJochException.h"
@@ -56,6 +57,12 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
     const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
     const bool do_clip = apply_bkg_clip && mode != IntegratorMode::BoxSum;
 
+    // Symmetric trimmed-mean background fraction (BraggIntegrationSettings, rugnux --background-trim):
+    // replaces the r2..r3 ring MEAN with an f-trimmed mean, robust to the high-side contamination
+    // (neighbour wings, tails) that biases the plain mean up and makes it over-subtract weak high-angle
+    // reflections. Already forced to 0 for broadband/stills by the base ctor (they keep the clip below).
+    const double bkg_trim_frac = bkg_trim;
+
     auto grid_idx = [this](int dx, int dy) { return (dy + R) * G + (dx + R); };
 
     // --- Reflection mask: mark the r2 signal disk of every predicted reflection so a neighbour's
@@ -82,6 +89,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
     };
     std::vector<Rough> rough(npredicted);
     double inv_d2_min = std::numeric_limits<double>::max(), inv_d2_max = 0.0;
+    std::vector<int32_t> bkg_vals;   // reused per reflection for the trimmed-mean background (idea 1)
 
     for (size_t i = 0; i < npredicted; ++i) {
         const auto &r = predicted[i];
@@ -94,6 +102,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
         int64_t I_sum = 0, I_sum_x = 0, I_sum_y = 0, n_inner = 0, n_inner_valid = 0;
         double bkg_sum = 0.0;
         int n_bkg = 0;
+        bkg_vals.clear();
         for (int y = y0; y <= y1; ++y)
             for (int x = x0; x <= x1; ++x) {
                 const double d2 = (x - r.predicted_x) * (x - r.predicted_x) + (y - r.predicted_y) * (y - r.predicted_y);
@@ -109,15 +118,28 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
                     if (refl_mask[y * W + x]) continue;
                     if (!valid(px)) continue;
                     bkg_sum += static_cast<double>(px);
+                    if (bkg_trim_frac > 0.0) bkg_vals.push_back(px);
                     ++n_bkg;
                 }
             }
 
         if (n_inner_valid == n_inner && n_bkg > 5) {
             out.bkg = bkg_sum / n_bkg;
-            // One high-outlier sigma-clip pass on the background ring (stills-only): reject pixels
-            // above mean + 3*sqrt(mean) to strip a bandwidth-streaked neighbour that biases the mean.
-            if (do_clip) {
+            if (bkg_trim_frac > 0.0 && bkg_vals.size() > 5) {
+                // Symmetric trimmed mean over the background ring (idea 1): drop the lowest and highest
+                // bkg_trim_frac of the pixels, average the rest. Robust to the high-side contamination
+                // that biases the plain ring mean and makes it over-subtract at high resolution.
+                std::sort(bkg_vals.begin(), bkg_vals.end());
+                const size_t lo = static_cast<size_t>(bkg_vals.size() * bkg_trim_frac);
+                const size_t hi = bkg_vals.size() - lo;
+                if (hi > lo) {
+                    double s = 0.0;
+                    for (size_t t = lo; t < hi; ++t) s += bkg_vals[t];
+                    out.bkg = s / static_cast<double>(hi - lo);
+                }
+            } else if (do_clip) {
+                // One high-outlier sigma-clip pass on the background ring (stills-only): reject pixels
+                // above mean + 3*sqrt(mean) to strip a bandwidth-streaked neighbour that biases the mean.
                 const double thr = out.bkg + 3.0 * std::sqrt(std::max(out.bkg, 1.0));
                 double s = 0.0;
                 int n = 0;
@@ -323,8 +345,11 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
             I = rh.I;
             sigma = rh.sigma;
         }
+        // Carry the Pass-A box-sum intensity-weighted centroid (observed spot position) through the
+        // profile path too - post-refinement uses it as the observed position (beam-centre / distance).
         results[i] = {static_cast<float>(I), static_cast<float>(sigma),
-                      static_cast<float>(rh.bkg), 0.0f, 0.0f, true, false};
+                      static_cast<float>(rh.bkg),
+                      static_cast<float>(rh.obs_x), static_cast<float>(rh.obs_y), true, rh.has_obs};
     }
 
     return Finalize(predicted, npredicted, results, image_number);

image_analysis/bragg_integration/BraggIntegrationEngineGPU.cu

@@ -26,10 +26,14 @@ struct BraggGpuParams {
     float c_radial;
     float F_px;
     float beam_x, beam_y;
+    float bkg_trim;       // idea 1: symmetric trimmed-mean background fraction (0 = plain ring mean)
 };
 
 __device__ inline bool valid(int32_t v) { return v != INT32_MIN && v != INT32_MAX; }
 
+// idea 1: max annulus pixels held in shared memory for the per-block trimmed-mean sort (else flat mean).
+constexpr int BKG_TRIM_MAX = 512;
+
 // --- Mark the r2 signal disk of every predicted reflection (race-free: all writes are 1). ---
 __global__ void mark_mask(const float *px_x, const float *px_y, uint8_t *mask, BraggGpuParams p, int n) {
     const int i = blockIdx.x;
@@ -111,8 +115,57 @@ __global__ void boxsum(const float *px_x, const float *px_y, const float *dd,
     }
     __syncthreads();
 
+    // Trimmed-mean background (idea 1): collect the annulus into shared memory, bitonic-sort the block, and
+    // average the middle (1 - 2*bkg_trim) fraction - robust to the high-side contamination that biases the
+    // plain ring mean. Falls back to the flat mean when the ring exceeds the shared buffer.
+    __shared__ int s_bvals[BKG_TRIM_MAX];
+    __shared__ int s_bn;
+    if (threadIdx.x == 0) s_bn = 0;
+    __syncthreads();
+    const bool do_trim = s_accept && p.bkg_trim > 0.0f && s_nbkg > 5 && s_nbkg <= BKG_TRIM_MAX;
+    if (do_trim) {
+        for (int t = threadIdx.x; t < area; t += blockDim.x) {
+            const int x = x0 + t % bw, y = y0 + t / bw;
+            const float ddx = (float) x - cx, ddy = (float) y - cy;
+            const float d2 = ddx * ddx + ddy * ddy;
+            if (!(d2 >= p.r2_sq && d2 < p.r3_sq)) continue;
+            if (mask[y * p.W + x]) continue;
+            const int32_t px = img[y * p.W + x];
+            if (!valid(px)) continue;
+            const int slot = atomicAdd(&s_bn, 1);
+            if (slot < BKG_TRIM_MAX) s_bvals[slot] = px;
+        }
+        __syncthreads();
+        const int nb = min(s_bn, BKG_TRIM_MAX);
+        int n2 = 1;
+        while (n2 < nb) n2 <<= 1;
+        for (int t = threadIdx.x + nb; t < n2; t += blockDim.x) s_bvals[t] = INT32_MAX;   // pad to power of 2
+        __syncthreads();
+        for (int k = 2; k <= n2; k <<= 1)                       // ascending bitonic sort of s_bvals[0..n2)
+            for (int j = k >> 1; j > 0; j >>= 1) {
+                for (int idx = threadIdx.x; idx < n2; idx += blockDim.x) {
+                    const int ixj = idx ^ j;
+                    if (ixj > idx) {
+                        const bool up = ((idx & k) == 0);
+                        const int a = s_bvals[idx], b = s_bvals[ixj];
+                        if ((up && a > b) || (!up && a < b)) { s_bvals[idx] = b; s_bvals[ixj] = a; }
+                    }
+                }
+                __syncthreads();
+            }
+        if (threadIdx.x == 0) {
+            const int lo = (int) (nb * p.bkg_trim), hi = nb - lo;
+            if (hi > lo) {
+                double s = 0.0;
+                for (int t = lo; t < hi; ++t) s += (double) s_bvals[t];
+                s_bkg = s / (double) (hi - lo);
+            }
+        }
+        __syncthreads();
+    }
+
     // Second ring pass for the stills sigma-clip (re-reads the annulus; avoids storing bkg values).
-    if (s_accept && p.do_clip) {
+    if (s_accept && p.do_clip && !do_trim) {
         double c_l = 0.0; int cn_l = 0;
         for (int t = threadIdx.x; t < area; t += blockDim.x) {
             const int x = x0 + t % bw, y = y0 + t / bw;
@@ -437,6 +490,7 @@ std::vector<Reflection> BraggIntegrationEngineGPU::Run(const ImagePreprocessorBu
         .bw_sigma = static_cast<float>(bw_sigma), .c_radial = static_cast<float>(c_radial),
         .F_px = static_cast<float>(F_px),
         .beam_x = beam_x, .beam_y = beam_y,
+        .bkg_trim = bkg_trim,   // effective trim fraction (0 for stills), set by the base ctor from settings
     };
 
     // Pass A: reset accumulators, mask, then box-sum.
@@ -464,12 +518,11 @@ std::vector<Reflection> BraggIntegrationEngineGPU::Run(const ImagePreprocessorBu
     cuda_err(cudaMemcpyAsync(h_sigma.data(), d_sigma, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
     cuda_err(cudaMemcpyAsync(h_bkg.data(), d_bkg, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
     cuda_err(cudaMemcpyAsync(h_ok.data(), d_ok, sizeof(uint8_t) * npredicted, cudaMemcpyDeviceToHost, *stream));
-    const bool boxsum_mode = mode == IntegratorMode::BoxSum;
-    if (boxsum_mode) {
-        cuda_err(cudaMemcpyAsync(h_obs_x.data(), d_obs_x, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
-        cuda_err(cudaMemcpyAsync(h_obs_y.data(), d_obs_y, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
-        cuda_err(cudaMemcpyAsync(h_has_obs.data(), d_has_obs, sizeof(uint8_t) * npredicted, cudaMemcpyDeviceToHost, *stream));
-    }
+    // Pass A always fills the box-sum centroid (observed spot position), so copy it back in all modes -
+    // post-refinement uses it as the observed position (beam-centre / distance).
+    cuda_err(cudaMemcpyAsync(h_obs_x.data(), d_obs_x, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
+    cuda_err(cudaMemcpyAsync(h_obs_y.data(), d_obs_y, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
+    cuda_err(cudaMemcpyAsync(h_has_obs.data(), d_has_obs, sizeof(uint8_t) * npredicted, cudaMemcpyDeviceToHost, *stream));
     cuda_err(cudaStreamSynchronize(*stream));
 
     for (size_t i = 0; i < npredicted; ++i) {
@@ -478,7 +531,7 @@ std::vector<Reflection> BraggIntegrationEngineGPU::Run(const ImagePreprocessorBu
         results[i].sigma = h_sigma[i];
         results[i].bkg = h_bkg[i];
         results[i].ok = true;
-        if (boxsum_mode && h_has_obs[i]) {
+        if (h_has_obs[i]) {
             results[i].observed_x = h_obs_x[i];
             results[i].observed_y = h_obs_y[i];
             results[i].has_observed = true;

image_analysis/bragg_integration/CalcISigma.cpp

@@ -1,10 +1,21 @@
 // SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
 // SPDX-License-Identifier: GPL-3.0-only
 
+#include <algorithm>
+#include <cmath>
+#include <limits>
+
 #include "CalcISigma.h"
 #include "Regression.h"
 #include "../../common/ResolutionShells.h"
 
+// Upper bound on a physically plausible macromolecular isotropic B-factor (A^2). A fit that lands
+// outside (0, WILSON_B_MAX) comes from a bad frame / bad dataset (too few or mis-indexed reflections,
+// a degenerate resolution range) rather than real Debye-Waller falloff, so it is rejected as
+// indeterminate rather than reported. Radiation-damaged / low-resolution data rarely exceeds ~120 A^2;
+// 200 leaves generous head-room while still excluding the hundreds-of-A^2 garbage.
+static constexpr float WILSON_B_MAX = 200.0f;
+
 void CalcISigma(DataMessage &msg) {
     CalcISigma(msg, msg.reflections);
 }
@@ -81,11 +92,80 @@ void CalcWilsonBFactor(DataMessage &msg,
 
     if (replace_b && valid_shells > 2) {
         auto reg_result = regression(shells_mean_one_over_d_square, log_I_mean, valid_shells);
+        const float b_est = -2.0f * reg_result.slope;
 
-        if (reg_result.r_square > 0.3)
-             msg.b_factor = -2.0f * reg_result.slope;
+        // Accept only a well-correlated, physically plausible fit. Occasional bad frames (an indexing
+        // glitch, too few reflections) produce a wildly steep Wilson line and a B of several hundred
+        // A^2 that pollutes the per-image plot; leaving b_factor unset (rendered as NaN) is better than
+        // emitting garbage.
+        if (reg_result.r_square > 0.3 && std::isfinite(b_est) && b_est > 0.0f && b_est < WILSON_B_MAX)
+             msg.b_factor = b_est;
     }
 
     msg.integration_B_logI = log_I_mean;
     msg.integration_B_one_over_d_square = shells_mean_one_over_d_square;
 }
+
+GlobalWilsonB CalcGlobalWilsonB(const std::vector<MergedReflection> &merged) {
+    GlobalWilsonB out;
+    // A dataset-wide estimate needs enough reflections to average the shell means; below this the
+    // per-image estimate is the only thing on offer and a global number would be meaningless.
+    if (merged.size() < 100)
+        return out;
+
+    float d_min = std::numeric_limits<float>::infinity(), d_max = 0.0f;
+    for (const auto &r : merged) {
+        if (std::isfinite(r.I) && r.d > 0.0f) {
+            d_min = std::min(d_min, r.d);
+            d_max = std::max(d_max, r.d);
+        }
+    }
+    if (!(d_min < d_max))
+        return out;
+
+    // Below ~4 A the Wilson plot is non-linear (bonding/solvent structure), so when the data extend to
+    // lower resolution than that, restrict the fit to d <= 4 A - the standard Wilson-B convention. If
+    // the whole dataset is coarser than 4 A, fall back to using all of it.
+    constexpr double WILSON_LOW_RES_LIMIT_A = 4.0;
+    const float d_low = (d_max > WILSON_LOW_RES_LIMIT_A && d_min < WILSON_LOW_RES_LIMIT_A)
+                        ? static_cast<float>(WILSON_LOW_RES_LIMIT_A) : d_max;
+
+    const int nshells = 20;
+    ResolutionShells shells(d_min, d_low, nshells);
+    std::vector<double> I_sum(nshells, 0.0), sig_sum(nshells, 0.0), count(nshells, 0.0);
+    for (const auto &r : merged) {
+        if (!std::isfinite(r.I) || r.d <= 0.0f)
+            continue;
+        auto s = shells.GetShell(r.d);          // reflections coarser than d_low fall outside -> skipped
+        if (s) {
+            I_sum[*s] += r.I;
+            if (std::isfinite(r.sigma) && r.sigma > 0.0f)
+                sig_sum[*s] += r.sigma;
+            ++count[*s];
+        }
+    }
+
+    const auto s2 = shells.GetShellMeanOneOverResSq();
+    std::vector<float> x, y;
+    for (int i = 0; i < nshells; ++i) {
+        // Skip empty / net-negative shells, and shells past the signal limit (mean I/sigma < 1). The
+        // latter keeps the fit out of the noise floor: without a resolution cut the weakest high-angle
+        // shells are background-residual-dominated and flatten the Wilson line, deflating B. This mirrors
+        // XDS/ctruncate fitting only over the meaningful range and makes the estimate insensitive to how
+        // far the merged data were carried.
+        if (count[i] > 0 && I_sum[i] > 0.0 && I_sum[i] > sig_sum[i]) {
+            x.push_back(s2[i]);
+            y.push_back(std::log(static_cast<float>(I_sum[i] / count[i])));
+        }
+    }
+    if (x.size() < 3)
+        return out;
+
+    const auto reg = regression(x, y, x.size());
+    const double b = -2.0 * reg.slope;          // <I> ~ exp(-2 B s^2), s^2 = 1/(4 d^2), x = 1/d^2
+    out.n_shells = static_cast<int>(x.size());
+    out.correlation = std::sqrt(std::clamp(static_cast<double>(reg.r_square), 0.0, 1.0));
+    if (std::isfinite(b) && b > 0.0 && b < WILSON_B_MAX)
+        out.b = b;
+    return out;
+}

image_analysis/bragg_integration/CalcISigma.h

@@ -3,6 +3,9 @@
 
 #pragma once
 
+#include <limits>
+#include <vector>
+
 #include "../../common/JFJochMessages.h"
 #include "../../common/Reflection.h"
 
@@ -12,3 +15,14 @@ void CalcWilsonBFactor(DataMessage &msg, bool replace_b = true);
 void CalcISigma(DataMessage &msg, const std::vector<Reflection> &reflections);
 void CalcWilsonBFactor(DataMessage &msg, const std::vector<Reflection> &reflections, bool replace_b = true);
 
+// Dataset-wide isotropic Wilson B-factor estimate from a log-linear fit of the shell-averaged merged
+// intensity against 1/d^2 (B = -2*slope). Robust because it averages over the whole dataset - the
+// per-image CalcWilsonBFactor above is a per-frame estimate and much noisier. Diagnostic only (like
+// XDS's "WILSON LINE ... B="), not used in scaling. Returns b = NaN when it cannot be determined.
+struct GlobalWilsonB {
+    double b = std::numeric_limits<double>::quiet_NaN();   // Wilson B-factor estimate (A^2)
+    double correlation = std::numeric_limits<double>::quiet_NaN();  // |correlation| of the log-linear fit
+    int n_shells = 0;                                      // resolution shells actually used
+};
+GlobalWilsonB CalcGlobalWilsonB(const std::vector<MergedReflection> &merged);
+

image_analysis/bragg_prediction/BraggPrediction.cpp

@@ -156,6 +156,13 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal
                         continue;
 
                     float d = 1.0f / sqrtf(recip_sq);
+                    float partiality = 1.0f;
+                    if (settings.still_partiality && settings.profile_radius_recipA > 0.0f) {
+                        const float sig_bw = settings.bandwidth_sigma * std::fabs(recip_z);
+                        const float sigma2 = settings.profile_radius_recipA * settings.profile_radius_recipA
+                                             + sig_bw * sig_bw;
+                        partiality = std::exp(-0.5f * dist_ewald_sphere * dist_ewald_sphere / sigma2);
+                    }
                     reflections[i] = Reflection{
                         .h = h,
                         .k = k,
@@ -168,7 +175,7 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal
                         .d = d,
                         .dist_ewald = dist_ewald_sphere,
                         .rlp = 1.0,
-                        .partiality = 1.0,
+                        .partiality = partiality,
                         .zeta = 1.0,
                         .image_scale_corr = 1.0
                     };

image_analysis/bragg_prediction/BraggPrediction.h

@@ -21,9 +21,15 @@ struct BraggPredictionSettings {
     // Relative X-ray bandwidth Δλ/λ expressed as a Gaussian sigma (0 = monochromatic).
     // When > 0 the Ewald-shell acceptance is thickened radially per reflection by
     // σ_bw = |recip_z|·bandwidth_sigma (= bλ/2d²), so the 1/d² pink-beam smear no
-    // longer clips high-resolution reflections. Must stay the last member so existing
-    // designated initializers remain valid.
+    // longer clips high-resolution reflections.
     float bandwidth_sigma = 0.0f;
+    // Experimental stills partiality (rugnux --still-partiality). When still_partiality is set and
+    // profile_radius_recipA > 0, each reflection gets a Gaussian excitation-error partiality
+    // p = exp(-dist_ewald^2 / (2*sigma^2)), sigma^2 = profile_radius_recipA^2 + (bandwidth_sigma*|recip_z|)^2,
+    // instead of the fixed 1.0 (full). Off by default. Keep these two as the trailing members so the
+    // existing designated initializers (which stop at bandwidth_sigma) remain valid.
+    bool still_partiality = false;
+    float profile_radius_recipA = 0.0f;
 };
 
 class BraggPrediction {

image_analysis/bragg_prediction/BraggPredictionGPU.cu

@@ -127,7 +127,13 @@ namespace {
         out.d = 1.0f / sqrtf(recip_sq);
         out.dist_ewald = dist_ewald;
         out.rlp = 1.0f;
-        out.partiality = 1.0f;
+        float partiality = 1.0f;
+        if (C.still_partiality && C.profile_radius_recipA > 0.0f) {
+            const float sig_bw = C.bandwidth_sigma * fabsf(recip_z);
+            const float sigma2 = C.profile_radius_recipA * C.profile_radius_recipA + sig_bw * sig_bw;
+            partiality = expf(-0.5f * dist_ewald * dist_ewald / sigma2);
+        }
+        out.partiality = partiality;
         out.zeta = 1.0f;
         out.image_scale_corr = 1.0f;
         return true;
@@ -158,7 +164,9 @@ namespace {
                                           float high_res_A,
                                           float ewald_dist_cutoff,
                                           char centering,
-                                          float bandwidth_sigma) {
+                                          float bandwidth_sigma,
+                                          bool still_partiality,
+                                          float profile_radius_recipA) {
         KernelConsts kc{};
         auto geom = experiment.GetDiffractionGeometry();
         kc.det_width_pxl = static_cast<float>(experiment.GetXPixelsNum());
@@ -171,6 +179,8 @@ namespace {
         kc.one_over_wavelength = 1.0f / geom.GetWavelength_A();
         kc.ewald_cutoff = ewald_dist_cutoff;
         kc.bandwidth_sigma = bandwidth_sigma;
+        kc.still_partiality = still_partiality;
+        kc.profile_radius_recipA = profile_radius_recipA;
         kc.Astar = lattice.Astar();
         kc.Bstar = lattice.Bstar();
         kc.Cstar = lattice.Cstar();
@@ -193,7 +203,8 @@ int BraggPredictionGPU::Calc(const DiffractionExperiment &experiment,
                              const BraggPredictionSettings &settings) {
     // Build constants on host
     KernelConsts hK = BuildKernelConsts(experiment, lattice, settings.high_res_A, settings.ewald_dist_cutoff,
-                                        settings.centering, settings.bandwidth_sigma);
+                                        settings.centering, settings.bandwidth_sigma,
+                                        settings.still_partiality, settings.profile_radius_recipA);
     cudaMemcpyAsync(dK, &hK, sizeof(KernelConsts), cudaMemcpyHostToDevice, stream);
     cudaMemsetAsync(d_count, 0, sizeof(int), stream);
 

image_analysis/bragg_prediction/BraggPredictionGPU.h

@@ -19,6 +19,8 @@ struct KernelConsts {
     float one_over_dmax_sq;
     float ewald_cutoff;
     float bandwidth_sigma;   // relative Δλ/λ (sigma); 0 = monochromatic
+    bool still_partiality;        // experimental stills excitation-error partiality (off => p = 1, fulls)
+    float profile_radius_recipA;  // Gaussian sigma [1/A] for the stills partiality
     Coord Astar, Bstar, Cstar, S0;
     float rot[9];
     char centering;

image_analysis/geom_refinement/CMakeLists.txt

@@ -6,6 +6,11 @@ ADD_LIBRARY(JFJochGeomRefinement STATIC
         AssignSpotsToRings.h
         XtalOptimizer.cpp
         XtalOptimizer.h
+        XtalResidual.h
+        PostRefine.cpp
+        PostRefine.h
+        GeometryRefiner.cpp
+        GeometryRefiner.h
         LatticeReduction.cpp
         LatticeReduction.h
 )

image_analysis/geom_refinement/GeometryRefiner.cpp

@@ -0,0 +1,311 @@
+// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#include "GeometryRefiner.h"
+
+#include <algorithm>
+#include <array>
+#include <cmath>
+#include <vector>
+
+#include "../../common/JFJochMath.h"   // PI
+#include "LatticeReduction.h"
+#include "XtalResidual.h"
+#include "ceres/ceres.h"
+
+namespace {
+
+// Trigonal is described with the hexagonal cell + B, same as the per-image XtalOptimizer.
+gemmi::CrystalSystem MapSystem(gemmi::CrystalSystem s) {
+    return (s == gemmi::CrystalSystem::Trigonal) ? gemmi::CrystalSystem::Hexagonal : s;
+}
+
+// The cell angles (radians) that XtalResidual bakes into its B matrix for a given system. Only
+// monoclinic (beta) and triclinic (all three) read p2; the rest use fixed symmetry angles. We keep the
+// angles constant in v1, so these are set once from the input cell.
+void InitCellAngles(gemmi::CrystalSystem system, const UnitCell &cell, double ang[3]) {
+    const double d2r = PI / 180.0;
+    switch (system) {
+        case gemmi::CrystalSystem::Hexagonal:
+            ang[0] = PI / 2.0; ang[1] = PI / 2.0; ang[2] = 2.0 * PI / 3.0;
+            break;
+        case gemmi::CrystalSystem::Monoclinic:
+            // XtalResidual reads p2[0] as beta.
+            ang[0] = cell.beta * d2r; ang[1] = PI / 2.0; ang[2] = PI / 2.0;
+            break;
+        case gemmi::CrystalSystem::Triclinic:
+            ang[0] = cell.alpha * d2r; ang[1] = cell.beta * d2r; ang[2] = cell.gamma * d2r;
+            break;
+        default:  // orthorhombic / tetragonal / cubic
+            ang[0] = PI / 2.0; ang[1] = PI / 2.0; ang[2] = PI / 2.0;
+            break;
+    }
+}
+
+// Effective (lengths, alpha,beta,gamma) that reproduce the lattice XtalResidual builds from (len, ang),
+// so a frame's lattice can be reconstructed with AngleAxisAndCellToLattice for HKL re-assignment.
+void EffectiveCell(gemmi::CrystalSystem system, const double len[3], const double ang[3],
+                   double lengths[3], double &alpha, double &beta, double &gamma) {
+    switch (system) {
+        case gemmi::CrystalSystem::Hexagonal:
+            lengths[0] = len[0]; lengths[1] = len[0]; lengths[2] = len[2];
+            alpha = PI / 2.0; beta = PI / 2.0; gamma = 2.0 * PI / 3.0;
+            break;
+        case gemmi::CrystalSystem::Tetragonal:
+            lengths[0] = len[0]; lengths[1] = len[0]; lengths[2] = len[2];
+            alpha = PI / 2.0; beta = PI / 2.0; gamma = PI / 2.0;
+            break;
+        case gemmi::CrystalSystem::Cubic:
+            lengths[0] = len[0]; lengths[1] = len[0]; lengths[2] = len[0];
+            alpha = PI / 2.0; beta = PI / 2.0; gamma = PI / 2.0;
+            break;
+        case gemmi::CrystalSystem::Monoclinic:
+            lengths[0] = len[0]; lengths[1] = len[1]; lengths[2] = len[2];
+            alpha = PI / 2.0; beta = ang[0]; gamma = PI / 2.0;
+            break;
+        case gemmi::CrystalSystem::Triclinic:
+            lengths[0] = len[0]; lengths[1] = len[1]; lengths[2] = len[2];
+            alpha = ang[0]; beta = ang[1]; gamma = ang[2];
+            break;
+        default:  // orthorhombic
+            lengths[0] = len[0]; lengths[1] = len[1]; lengths[2] = len[2];
+            alpha = PI / 2.0; beta = PI / 2.0; gamma = PI / 2.0;
+            break;
+    }
+}
+
+// Orientation seed (angle-axis) for a frame's lattice, using the per-system decomposition the per-image
+// optimizer uses so it matches XtalResidual's B convention.
+void OrientationSeed(gemmi::CrystalSystem system, const CrystalLattice &latt, double rod[3]) {
+    double scratch[3];
+    if (system == gemmi::CrystalSystem::Hexagonal) {
+        LatticeToRodriguesAndLengths_Hex(latt, rod, scratch);
+    } else if (system == gemmi::CrystalSystem::Monoclinic) {
+        double beta;
+        LatticeToRodriguesLengthsBeta_Mono(latt, rod, scratch, beta);
+    } else {
+        LatticeToRodriguesAndLengths_GS(latt, rod, scratch);
+    }
+}
+
+UnitCell BuildUnitCell(gemmi::CrystalSystem system, const double len[3], const double ang[3]) {
+    const double r2d = 180.0 / PI;
+    UnitCell uc{};
+    switch (system) {
+        case gemmi::CrystalSystem::Hexagonal:
+            uc = {(float)len[0], (float)len[0], (float)len[2], 90.0f, 90.0f, 120.0f};
+            break;
+        case gemmi::CrystalSystem::Tetragonal:
+            uc = {(float)len[0], (float)len[0], (float)len[2], 90.0f, 90.0f, 90.0f};
+            break;
+        case gemmi::CrystalSystem::Cubic:
+            uc = {(float)len[0], (float)len[0], (float)len[0], 90.0f, 90.0f, 90.0f};
+            break;
+        case gemmi::CrystalSystem::Monoclinic:
+            uc = {(float)len[0], (float)len[1], (float)len[2], 90.0f, (float)(ang[0] * r2d), 90.0f};
+            break;
+        case gemmi::CrystalSystem::Triclinic:
+            uc = {(float)len[0], (float)len[1], (float)len[2],
+                  (float)(ang[0] * r2d), (float)(ang[1] * r2d), (float)(ang[2] * r2d)};
+            break;
+        default:  // orthorhombic
+            uc = {(float)len[0], (float)len[1], (float)len[2], 90.0f, 90.0f, 90.0f};
+            break;
+    }
+    return uc;
+}
+
+// Anchors the cell lengths to the input cell. Penalises each length's deviation from its input value
+// with a large weight, breaking the low-resolution distance<->cell-scale degeneracy (position ~
+// distance / cell_scale) so distance is determined by the data with the cell pinned.
+struct CellLengthRegularizer {
+    double w0, w1, w2;
+    double l0, l1, l2;
+    template <typename T>
+    bool operator()(const T *const len, T *r) const {
+        r[0] = T(w0) * (len[0] - T(l0));
+        r[1] = T(w1) * (len[1] - T(l1));
+        r[2] = T(w2) * (len[2] - T(l2));
+        return true;
+    }
+};
+
+}   // namespace
+
+GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_geom,
+                                           const UnitCell &input_cell,
+                                           const std::vector<GeomRefineFrame> &frames,
+                                           const GeometryRefinerSettings &settings) {
+    GeometryRefinerResult result;
+    try {
+        const gemmi::CrystalSystem system = MapSystem(settings.crystal_system);
+
+        // Shared geometry parameter blocks (refined). Detector tilt and the rotation axis are constant.
+        double beam[2] = {nominal_geom.GetBeamX_pxl(), nominal_geom.GetBeamY_pxl()};
+        double distance_mm = nominal_geom.GetDetectorDistance_mm();
+        double detector_rot[2] = {nominal_geom.GetPoniRot1_rad(), nominal_geom.GetPoniRot2_rad()};
+        double rot_vec[3] = {1.0, 0.0, 0.0};
+        const double rot3 = nominal_geom.GetPoniRot3_rad();
+        const double lambda = nominal_geom.GetWavelength_A();
+        const double pixel = nominal_geom.GetPixelSize_mm();
+
+        // Shared cell blocks. Lengths refined (regularized); angles constant.
+        double cell_len[3] = {input_cell.a, input_cell.b, input_cell.c};
+        const double cell_len0[3] = {input_cell.a, input_cell.b, input_cell.c};
+        double cell_ang[3];
+        InitCellAngles(system, input_cell, cell_ang);
+
+        double eff_len[3], eff_alpha, eff_beta, eff_gamma;   // recomputed each round from cell_len/ang
+
+        // Per-frame orientation seeds.
+        std::vector<std::array<double, 3>> orient(frames.size());
+        for (size_t i = 0; i < frames.size(); ++i)
+            OrientationSeed(system, frames[i].lattice, orient[i].data());
+
+        // Reciprocal length of one detector pixel at the nominal geometry, used to scale the robust
+        // loss (~3 px) into the reciprocal-space residual units.
+        const double recip_per_px = pixel / (distance_mm * lambda);
+        const double loss_scale = 3.0 * recip_per_px;
+
+        // Tolerance schedule (fractional-hkl norm), tightening each round like the per-image optimizer.
+        auto round_tol = [&](int r) {
+            if (settings.rounds <= 1) return 0.2;
+            const double t = static_cast<double>(r) / (settings.rounds - 1);
+            return 0.3 - (0.3 - 0.1) * t;
+        };
+
+        for (int round = 0; round < settings.rounds; ++round) {
+            EffectiveCell(system, cell_len, cell_ang, eff_len, eff_alpha, eff_beta, eff_gamma);
+
+            DiffractionGeometry cur = nominal_geom;
+            cur.BeamX_pxl(beam[0]).BeamY_pxl(beam[1]).DetectorDistance_mm(distance_mm);
+
+            const double tol = round_tol(round);
+            const double tol_sq = tol * tol;
+
+            ceres::Problem problem;
+            auto ordering = std::make_shared<ceres::ParameterBlockOrdering>();
+
+            int spots_used = 0, frames_used = 0;
+            for (size_t i = 0; i < frames.size(); ++i) {
+                const CrystalLattice latt = AngleAxisAndCellToLattice(
+                    orient[i].data(), eff_len, eff_alpha, eff_beta, eff_gamma);
+                const Coord v0 = latt.Vec0(), v1 = latt.Vec1(), v2 = latt.Vec2();
+
+                int used_in_frame = 0;
+                for (const auto &s : frames[i].spots) {
+                    const Coord recip = cur.DetectorToRecip(s.x, s.y);
+                    const double hf = recip * v0, kf = recip * v1, lf = recip * v2;
+                    const double h = std::round(hf), k = std::round(kf), l = std::round(lf);
+                    const double dn = (hf - h) * (hf - h) + (kf - k) * (kf - k) + (lf - l) * (lf - l);
+                    if (dn > tol_sq)
+                        continue;
+
+                    problem.AddResidualBlock(
+                        new ceres::AutoDiffCostFunction<XtalResidual, 3, 2, 1, 2, 3, 3, 3, 3>(
+                            new XtalResidual(s.x, s.y, lambda, pixel, rot3, 0.0, h, k, l, system)),
+                        new ceres::CauchyLoss(loss_scale),
+                        beam, &distance_mm, detector_rot, rot_vec, orient[i].data(), cell_len, cell_ang);
+                    ++used_in_frame;
+                    ++spots_used;
+                }
+                if (used_in_frame > 0) {
+                    ordering->AddElementToGroup(orient[i].data(), 0);   // eliminate orientations first
+                    ++frames_used;
+                }
+            }
+
+            if (spots_used < settings.min_spots_total) {
+                if (round == 0)
+                    return result;   // ok stays false
+                break;               // keep the previous round's solution
+            }
+
+            // Cell-length regularizer (weight ~ coeff * sqrt(N_obs) / L0^2 -> effectively pins the cell).
+            const double sqrtN = std::sqrt(static_cast<double>(spots_used));
+            CellLengthRegularizer *reg = new CellLengthRegularizer{
+                settings.cell_reg_coeff * sqrtN / (cell_len0[0] * cell_len0[0]),
+                settings.cell_reg_coeff * sqrtN / (cell_len0[1] * cell_len0[1]),
+                settings.cell_reg_coeff * sqrtN / (cell_len0[2] * cell_len0[2]),
+                cell_len0[0], cell_len0[1], cell_len0[2]};
+            problem.AddResidualBlock(
+                new ceres::AutoDiffCostFunction<CellLengthRegularizer, 3, 3>(reg), nullptr, cell_len);
+
+            // Shared blocks go in the second elimination group; constant ones are stripped by Ceres.
+            for (double *b : {beam, &distance_mm, detector_rot, rot_vec, cell_len, cell_ang})
+                ordering->AddElementToGroup(b, 1);
+
+            // Beam + distance bounds around the nominal value; cell lengths loosely bounded (the
+            // regularizer does the anchoring).
+            problem.SetParameterLowerBound(beam, 0, nominal_geom.GetBeamX_pxl() - settings.beam_range_px);
+            problem.SetParameterUpperBound(beam, 0, nominal_geom.GetBeamX_pxl() + settings.beam_range_px);
+            problem.SetParameterLowerBound(beam, 1, nominal_geom.GetBeamY_pxl() - settings.beam_range_px);
+            problem.SetParameterUpperBound(beam, 1, nominal_geom.GetBeamY_pxl() + settings.beam_range_px);
+            problem.SetParameterLowerBound(&distance_mm, 0,
+                std::max(1.0, nominal_geom.GetDetectorDistance_mm() - settings.distance_range_mm));
+            problem.SetParameterUpperBound(&distance_mm, 0,
+                nominal_geom.GetDetectorDistance_mm() + settings.distance_range_mm);
+            for (int j = 0; j < 3; ++j) {
+                problem.SetParameterLowerBound(cell_len, j, 0.7 * cell_len0[j]);
+                problem.SetParameterUpperBound(cell_len, j, 1.3 * cell_len0[j]);
+            }
+
+            problem.SetParameterBlockConstant(detector_rot);   // tilt not refined
+            problem.SetParameterBlockConstant(rot_vec);
+            problem.SetParameterBlockConstant(cell_ang);       // angles not refined in v1
+
+            ceres::Solver::Options options;
+            options.linear_solver_type = ceres::DENSE_SCHUR;
+            options.linear_solver_ordering = ordering;
+            options.num_threads = std::max(1, settings.num_threads);
+            options.max_num_iterations = 100;
+            options.max_solver_time_in_seconds = 60.0;
+            options.logging_type = ceres::LoggingType::SILENT;
+            options.minimizer_progress_to_stdout = false;
+
+            ceres::Solver::Summary summary;
+            ceres::Solve(options, &problem, &summary);
+
+            result.frames_used = frames_used;
+            result.spots_used = spots_used;
+        }
+
+        // Final fit quality: median detector residual of the retained spots at the refined geometry.
+        EffectiveCell(system, cell_len, cell_ang, eff_len, eff_alpha, eff_beta, eff_gamma);
+        DiffractionGeometry cur = nominal_geom;
+        cur.BeamX_pxl(beam[0]).BeamY_pxl(beam[1]).DetectorDistance_mm(distance_mm);
+        std::vector<double> residuals;
+        for (size_t i = 0; i < frames.size(); ++i) {
+            const CrystalLattice latt = AngleAxisAndCellToLattice(
+                orient[i].data(), eff_len, eff_alpha, eff_beta, eff_gamma);
+            const Coord a = latt.Astar(), b = latt.Bstar(), c = latt.Cstar();
+            const Coord v0 = latt.Vec0(), v1 = latt.Vec1(), v2 = latt.Vec2();
+            for (const auto &s : frames[i].spots) {
+                const Coord recip = cur.DetectorToRecip(s.x, s.y);
+                const double hf = recip * v0, kf = recip * v1, lf = recip * v2;
+                const double h = std::round(hf), k = std::round(kf), l = std::round(lf);
+                const double dn = (hf - h) * (hf - h) + (kf - k) * (kf - k) + (lf - l) * (lf - l);
+                if (dn > 0.1 * 0.1)
+                    continue;
+                const Coord recip_pred = a * (float)h + b * (float)k + c * (float)l;
+                const auto [px, py] = cur.RecipToDetector(recip_pred);
+                if (std::isfinite(px) && std::isfinite(py))
+                    residuals.push_back(std::hypot(px - s.x, py - s.y));
+            }
+        }
+        if (!residuals.empty()) {
+            std::nth_element(residuals.begin(), residuals.begin() + residuals.size() / 2, residuals.end());
+            result.median_residual_px = residuals[residuals.size() / 2];
+        }
+
+        result.ok = result.spots_used >= settings.min_spots_total && result.frames_used > 0;
+        result.beam_x_px = beam[0];
+        result.beam_y_px = beam[1];
+        result.distance_mm = distance_mm;
+        result.cell = BuildUnitCell(system, cell_len, cell_ang);
+        return result;
+    } catch (...) {
+        result.ok = false;
+        return result;
+    }
+}

image_analysis/geom_refinement/GeometryRefiner.h

@@ -0,0 +1,59 @@
+// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#pragma once
+
+#include <cstdint>
+#include <vector>
+
+#include "../../common/CrystalLattice.h"
+#include "../../common/DiffractionGeometry.h"
+#include "../../common/UnitCell.h"
+#include "gemmi/symmetry.hpp"
+
+// Offline global (multi-frame) geometry refinement for serial stills. Pass 1 indexes many frames
+// independently; this joint bundle adjustment then determines the ONE shared detector geometry (beam
+// centre, distance, cell scale) that per-image refinement cannot reach from a single sparse still, so
+// re-indexing with it converts more frames. Mirrors the validated Python prototype: one Ceres problem
+// with shared beam/distance/cell blocks and a per-frame orientation block, robust loss, and a cell
+// regularizer that anchors the (known) cell to break the distance<->cell-scale degeneracy at low
+// resolution. Detector tilt (rot1/rot2) is NOT refined (gauge-coupled with the beam; zero indexing gain).
+
+struct GeomRefineSpot {
+    float x = 0, y = 0;         // observed (raw) spot position, pixels
+    int32_t h = 0, k = 0, l = 0; // integer Miller index assigned in pass 1
+};
+
+struct GeomRefineFrame {
+    CrystalLattice lattice;             // per-frame indexed lattice (real space), for the orientation seed
+    std::vector<GeomRefineSpot> spots;  // indexed spots on this frame
+};
+
+struct GeometryRefinerSettings {
+    gemmi::CrystalSystem crystal_system = gemmi::CrystalSystem::Triclinic;
+    // Cell-scale regularizer strength (anchors the cell lengths to the input cell so the otherwise
+    // degenerate distance is determined). Large => cell effectively fixed (the validated safe regime).
+    double cell_reg_coeff = 30.0;
+    int rounds = 3;                 // hkl-reassignment / tolerance-tightening rounds
+    double beam_range_px = 8.0;     // beam bound around the nominal value
+    double distance_range_mm = 8.0; // distance bound around the nominal value
+    int min_spots_total = 100;      // refuse to refine below this many indexed spots
+    int num_threads = 1;
+};
+
+struct GeometryRefinerResult {
+    bool ok = false;
+    double beam_x_px = 0, beam_y_px = 0;   // refined beam centre (== direct beam, tilt not refined)
+    double distance_mm = 0;
+    UnitCell cell{};                       // refined cell
+    int frames_used = 0;
+    int spots_used = 0;
+    double median_residual_px = 0;         // final per-spot detector residual (fit quality)
+};
+
+// nominal_geom supplies the starting beam/distance/tilt/wavelength/pixel size; input_cell is anchored by
+// the regularizer. Returns ok=false (and leaves the caller's geometry untouched) on any failure.
+GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_geom,
+                                           const UnitCell &input_cell,
+                                           const std::vector<GeomRefineFrame> &frames,
+                                           const GeometryRefinerSettings &settings);

image_analysis/geom_refinement/PostRefine.cpp

@@ -0,0 +1,366 @@
+// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#include "PostRefine.h"
+
+#include <algorithm>
+#include <cmath>
+
+#include "../../common/JFJochMath.h"   // PI
+#include "XtalResidual.h"              // XtalResidual (the positional detector<->reciprocal residual, step B)
+#include "LatticeReduction.h"
+#include "ceres/ceres.h"
+#include "ceres/rotation.h"
+
+namespace {
+
+// One integrated partial, flattened across all images.
+struct Partial {
+    int h, k, l;
+    float img;
+    double I, sigma, zeta;
+    double angle_rad;   // frame mid-exposure goniometer angle
+    double obs_x, obs_y; // observed spot centroid (pixels); NAN if the box sum found no centroid
+};
+
+// A rocking event and its precomputed reference reciprocal vector (phi=0 frame, from the indexed lattice).
+struct Event {
+    double phi_obs;     // rad, intensity-weighted rocking centroid
+    double weight;      // sqrt(sum I / sum sigma)
+    double e_ref[3];    // h*a* + k*b* + l*c* at the reference (unrefined) cell/orientation
+    int h, k, l;
+};
+
+// Distance-INDEPENDENT Ewald excitation residual for a uniform cell-scale parameter s and a refined
+// goniometer axis (3-vector). The header-distance miscalibration leaves a uniform cell scale; the axis is
+// the other phi_obs lever. Both are phi_obs-constrained (distance-independent). e_ref is the reference
+// reciprocal (h*a* + k*b* + l*c* at the indexed cell). On the Ewald sphere <=> |p|^2 + 2 p_z/lambda == 0.
+struct ScaleAxisExcitationResidual {
+    ScaleAxisExcitationResidual(double lambda, double angle_rad, double weight, const double e_ref[3])
+        : inv_lambda(1.0 / lambda), angle_rad(angle_rad), weight(weight),
+          ex(e_ref[0]), ey(e_ref[1]), ez(e_ref[2]) {}
+    template<typename T>
+    bool operator()(const T *const s, const T *const axis, T *residual) const {
+        const T inv_s = T(1) / s[0];
+        const T p_ref[3] = {T(ex) * inv_s, T(ey) * inv_s, T(ez) * inv_s};
+        const T aa[3] = {T(-angle_rad) * axis[0], T(-angle_rad) * axis[1], T(-angle_rad) * axis[2]};
+        T p_lab[3];
+        ceres::AngleAxisRotatePoint(aa, p_ref, p_lab);
+        const T zeta = p_lab[0] * p_lab[0] + p_lab[1] * p_lab[1] + p_lab[2] * p_lab[2]
+                       + T(2.0) * p_lab[2] * T(inv_lambda);
+        residual[0] = T(weight) * zeta * T(0.5) / T(inv_lambda);
+        return true;
+    }
+    const double inv_lambda, angle_rad, weight, ex, ey, ez;
+};
+
+}  // namespace
+
+PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome> &outcomes,
+                                            const GoniometerAxis &axis,
+                                            const DiffractionGeometry &nominal_geom,
+                                            const CrystalLattice &reference_latt,
+                                            const PostRefineSettings &settings,
+                                            Logger &logger) {
+    PostRefineResult result;
+    result.geom = nominal_geom;
+    result.cell = reference_latt.GetUnitCell();
+    result.distance_before_mm = nominal_geom.GetDetectorDistance_mm();
+    result.distance_after_mm = nominal_geom.GetDetectorDistance_mm();
+    try {
+        const double wedge_half = axis.GetWedge_deg() / 2.0;
+        const double lambda = nominal_geom.GetWavelength_A();
+        const Coord ax = axis.GetAxis();
+        const Coord Astar = reference_latt.Astar(), Bstar = reference_latt.Bstar(), Cstar = reference_latt.Cstar();
+
+        std::vector<Partial> pts;
+        for (const auto &o : outcomes)
+            for (const auto &r : o.reflections) {
+                if (!std::isfinite(r.I) || !std::isfinite(r.sigma) || r.sigma <= 0.0f) continue;
+                const double mid_deg = axis.GetAngle_deg(r.image_number) + wedge_half;
+                const double ox = std::isfinite(r.observed_x) ? r.observed_x : NAN;
+                const double oy = std::isfinite(r.observed_y) ? r.observed_y : NAN;
+                pts.push_back(Partial{r.h, r.k, r.l, r.image_number, r.I, r.sigma,
+                                      std::isfinite(r.zeta) ? r.zeta : 0.0, mid_deg * PI / 180.0, ox, oy});
+            }
+        logger.Info("Post-refine: {} partials gathered", pts.size());
+        if (pts.size() < static_cast<size_t>(settings.min_events)) return result;
+
+        std::sort(pts.begin(), pts.end(), [](const Partial &a, const Partial &b) {
+            if (a.h != b.h) return a.h < b.h;
+            if (a.k != b.k) return a.k < b.k;
+            if (a.l != b.l) return a.l < b.l;
+            return a.img < b.img;
+        });
+
+        // Split into rocking events (same raw hkl, adjacent frames). Only >=2-frame events carry an
+        // unbiased phi_obs (a single-frame centroid is just the frame centre); precompute e_ref per event.
+        constexpr float MAX_FRAME_GAP = 2.0f;
+        std::vector<Event> events;
+        std::vector<double> rock_std;   // per-event observed rocking width (rad); task-3 mosaicity signal
+        std::vector<double> zeta_ev;    // per-event intensity-weighted zeta (aligned with rock_std)
+        size_t i = 0;
+        while (i < pts.size()) {
+            size_t j = i + 1;
+            while (j < pts.size() && pts[j].h == pts[i].h && pts[j].k == pts[i].k && pts[j].l == pts[i].l
+                   && pts[j].img - pts[j - 1].img <= MAX_FRAME_GAP)
+                ++j;
+            if (j - i >= 2) {
+                double sumI = 0, sumIphi = 0, sumIphi2 = 0, sumSig = 0, sumIzeta = 0;
+                for (size_t m = i; m < j; ++m) {
+                    const double Ipos = std::max(0.0, pts[m].I);
+                    sumI += Ipos; sumIphi += Ipos * pts[m].angle_rad;
+                    sumIphi2 += Ipos * pts[m].angle_rad * pts[m].angle_rad; sumSig += pts[m].sigma;
+                    sumIzeta += Ipos * pts[m].zeta;
+                }
+                if (sumI > 0.0 && sumSig > 0.0) {
+                    const double phi = sumIphi / sumI;
+                    // Intensity-weighted rocking width (rad) - the observed spread of the reflection over the
+                    // frames it spans (mosaicity + beam-divergence + oscillation, sampled by phi_obs).
+                    const double rvar = std::max(0.0, sumIphi2 / sumI - phi * phi);
+                    rock_std.push_back(std::sqrt(rvar));
+                    zeta_ev.push_back(sumIzeta / sumI);
+                    const Coord e = Astar * static_cast<float>(pts[i].h) + Bstar * static_cast<float>(pts[i].k)
+                                    + Cstar * static_cast<float>(pts[i].l);
+                    events.push_back(Event{phi, std::sqrt(sumI / sumSig), {e.x, e.y, e.z},
+                                           pts[i].h, pts[i].k, pts[i].l});
+                }
+            }
+            i = j;
+        }
+        logger.Info("Post-refine: {} multi-frame rocking events", events.size());
+        if (static_cast<int>(events.size()) < settings.min_events) return result;
+
+        // Task 3 signal: the observed rocking width (median over events) vs the frame oscillation. If the
+        // rocking width materially exceeds the oscillation, mosaicity/beam-divergence is resolved and could
+        // be refined from it; if it is ~ the oscillation, the reflections are barely rocking (under-sampled).
+        if (!rock_std.empty()) {
+            // Deconvolve the oscillation from the observed rocking width and convert the residual angular
+            // spread to a mosaicity: sigma_mos_angular = sqrt(rock_std^2 - (osc/sqrt12)^2), mosaicity = it*zeta.
+            const double osc_rad = axis.GetWedge_deg() * PI / 180.0;
+            const double osc_var = (osc_rad / std::sqrt(12.0)) * (osc_rad / std::sqrt(12.0));
+            std::vector<double> mos;
+            mos.reserve(rock_std.size());
+            for (size_t e = 0; e < rock_std.size(); ++e) {
+                const double sig_ang = std::sqrt(std::max(0.0, rock_std[e] * rock_std[e] - osc_var));
+                mos.push_back(sig_ang * zeta_ev[e] * 180.0 / PI);
+            }
+            std::nth_element(rock_std.begin(), rock_std.begin() + rock_std.size() / 2, rock_std.end());
+            const double med_rock_deg = rock_std[rock_std.size() / 2] * 180.0 / PI;
+            std::nth_element(mos.begin(), mos.begin() + mos.size() / 2, mos.end());
+            result.est_mosaicity_deg = mos[mos.size() / 2];
+            logger.Info("Post-refine mosaicity signal: median rocking width {:.4f} deg (osc {:.4f}) -> "
+                        "estimated mosaicity {:.4f} deg", med_rock_deg, axis.GetWedge_deg(), result.est_mosaicity_deg);
+        }
+
+        constexpr size_t MAX_EVENTS = 20000;
+        if (events.size() > MAX_EVENTS) {
+            std::nth_element(events.begin(), events.begin() + MAX_EVENTS, events.end(),
+                             [](const Event &a, const Event &b) { return a.weight > b.weight; });
+            events.resize(MAX_EVENTS);
+        }
+
+        // ---- GEOMETRY REFINEMENT: the XtalOptimizer-equivalent, done as TWO SEPARATE
+        // cross-validated steps rather than one joint fit (the same lesson as integration: refining the
+        // profile width and the scale jointly fails, refining them separately works). Each step is committed
+        // only if it lowers a HELD-OUT (deterministic split-half) residual - otherwise that part of the
+        // geometry is left at nominal ("quit when things go wrong"):
+        //   Step A: cell scale + rotation axis from phi_obs (distance-independent excitation residual).
+        //   Step B: detector distance + beam centre from the observed spot positions, with the cell FIXED at
+        //           step A (so the positional residual is no longer degenerate with the cell scale).
+        // Detector tilt is held fixed (gauge-coupled to orientation on a single crystal). ----
+        if (settings.refine_geometry) {
+            const gemmi::CrystalSystem sys =
+                (settings.crystal_system == gemmi::CrystalSystem::Trigonal) ? gemmi::CrystalSystem::Hexagonal
+                                                                            : settings.crystal_system;
+            const double ax0[3] = {ax.x, ax.y, ax.z};
+            const double lambda_l = lambda;
+            const double rot3 = nominal_geom.GetPoniRot3_rad();
+            const double pixel_mm = nominal_geom.GetPixelSize_mm();
+            const double det_rot[2] = {nominal_geom.GetPoniRot1_rad(), nominal_geom.GetPoniRot2_rad()};
+            const UnitCell r0 = reference_latt.GetUnitCell();
+
+            // Deterministic split of the reflections into a fit half and a held-out half. Avalanche-mix the
+            // hkl hash so the split bit is decorrelated from the LSB - a plain h+k+l parity collides with the
+            // lattice centering condition (e.g. an I-centred lattice has h+k+l even for EVERY present
+            // reflection, so a parity split would leave the validation half empty).
+            auto is_val = [](int h, int k, int l) {
+                unsigned u = static_cast<unsigned>(h) * 2654435761u + static_cast<unsigned>(k) * 2246822519u
+                             + static_cast<unsigned>(l) * 3266489917u;
+                u ^= u >> 15; u *= 2246822519u; u ^= u >> 13;
+                return (u & 1u) != 0u;
+            };
+            enum Subset { FIT, VAL, ALL };
+            auto in = [&](int h, int k, int l, Subset s) {
+                return s == ALL || (is_val(h, k, l) == (s == VAL)); };
+
+            // ===== Step A: cell scale s + rotation axis from phi_obs =====
+            auto excit_cost = [&](Subset s, double sc, const double axv[3]) {
+                double c = 0.0; int n = 0;
+                for (const auto &ev : events) {
+                    if (!in(ev.h, ev.k, ev.l, s)) continue;
+                    ScaleAxisExcitationResidual r(lambda_l, ev.phi_obs, 1.0, ev.e_ref);
+                    double sd = sc, av[3] = {axv[0], axv[1], axv[2]}, resid = 0.0;
+                    r(&sd, av, &resid); c += resid * resid; ++n;
+                }
+                return n ? c / n : 0.0;
+            };
+            auto solve_scale_axis = [&](Subset s, double &s_out, double ax_out[3]) {
+                double sc = 1.0, axv[3] = {ax0[0], ax0[1], ax0[2]};
+                ceres::Problem p;
+                for (const auto &ev : events) {
+                    if (!in(ev.h, ev.k, ev.l, s)) continue;
+                    p.AddResidualBlock(new ceres::AutoDiffCostFunction<ScaleAxisExcitationResidual, 1, 1, 3>(
+                        new ScaleAxisExcitationResidual(lambda_l, ev.phi_obs, settings.excitation_weight, ev.e_ref)),
+                        new ceres::CauchyLoss(0.02), &sc, axv);
+                }
+                p.SetParameterLowerBound(&sc, 0, 0.9); p.SetParameterUpperBound(&sc, 0, 1.1);
+                for (int j = 0; j < 3; ++j) { p.SetParameterLowerBound(axv, j, ax0[j] - 0.05);
+                                              p.SetParameterUpperBound(axv, j, ax0[j] + 0.05); }
+                ceres::Solver::Options o; o.linear_solver_type = ceres::DENSE_QR; o.max_num_iterations = 50;
+                o.num_threads = std::max(1, settings.num_threads); o.logging_type = ceres::LoggingType::SILENT;
+                ceres::Solver::Summary sum; ceres::Solve(o, &p, &sum);
+                s_out = sc; ax_out[0] = axv[0]; ax_out[1] = axv[1]; ax_out[2] = axv[2];
+                return sum.IsSolutionUsable();
+            };
+            double s_fit = 1.0, ax_fit[3];
+            const bool convA = solve_scale_axis(FIT, s_fit, ax_fit);
+            const double cvA_nom = excit_cost(VAL, 1.0, ax0);
+            const double cvA_ref = excit_cost(VAL, s_fit, ax_fit);
+            // Commit the cell scale only for a small, credible move: a well-calibrated header needs < ~0.6 %,
+            // so a > 1 % scale is a red flag (on multi-lattice / noisy data the excitation fit is biased the
+            // same way in every cross-validation fold, so the relative-improvement gate cannot catch it).
+            result.cell_refined = convA && cvA_ref < 0.98 * cvA_nom && std::fabs(s_fit - 1.0) < 0.01;
+            double s = 1.0, axv[3] = {ax0[0], ax0[1], ax0[2]};
+            if (result.cell_refined) solve_scale_axis(ALL, s, axv);   // commit: re-fit on all data
+            const double axlen = std::sqrt(axv[0]*axv[0] + axv[1]*axv[1] + axv[2]*axv[2]);
+            const double axdev = std::acos(std::clamp((axv[0]*ax0[0]+axv[1]*ax0[1]+axv[2]*ax0[2])
+                                                      / std::max(1e-9, axlen), -1.0, 1.0)) * 180.0 / PI;
+            logger.Info("Post-refine GEOM step A (cell/axis): s = {:.5f}, rot-axis {:.3f} deg, held-out excit "
+                        "{:.3e} -> {:.3e} => {}", s, axdev, cvA_nom, cvA_ref,
+                        result.cell_refined ? "COMMIT" : "reject (kept nominal cell)");
+
+            // Cell (scale s, shape fixed) as the XtalResidual parameter blocks p0/p1/p2, held CONSTANT in step B.
+            double p0[3] = {0, 0, 0}, p1[3] = {0, 0, 0}, p2[3] = {0, 0, 0};
+            double beta = r0.beta;
+            switch (sys) {
+                case gemmi::CrystalSystem::Tetragonal:
+                    LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1);
+                    p1[0] = (p1[0] + p1[1]) / 2.0; break;
+                case gemmi::CrystalSystem::Cubic:
+                    LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1);
+                    p1[0] = (p1[0] + p1[1] + p1[2]) / 3.0; break;
+                case gemmi::CrystalSystem::Hexagonal:
+                    LatticeToRodriguesAndLengths_Hex(reference_latt, p0, p1); break;
+                case gemmi::CrystalSystem::Monoclinic:
+                    LatticeToRodriguesLengthsBeta_Mono(reference_latt, p0, p1, beta);
+                    p2[0] = beta; break;
+                case gemmi::CrystalSystem::Orthorhombic:
+                    LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1); break;
+                default:
+                    LatticeToRodriguesAndLengths_GS(reference_latt, p0, p1);
+                    p2[0] = r0.alpha * PI / 180.0; p2[1] = r0.beta * PI / 180.0; p2[2] = r0.gamma * PI / 180.0; break;
+            }
+            for (int j = 0; j < 3; ++j) p1[j] *= s;                     // apply the committed cell scale
+            double rot_vec[3] = {axv[0], axv[1], axv[2]};              // committed (or nominal) axis
+
+            // ===== Step B: detector distance + beam from the observed positions, cell fixed =====
+            std::vector<const Partial *> obs;
+            for (const auto &pp : pts)
+                if (std::isfinite(pp.obs_x) && std::isfinite(pp.obs_y)) obs.push_back(&pp);
+            constexpr size_t MAX_OBS = 20000;
+            if (obs.size() > MAX_OBS) {
+                std::nth_element(obs.begin(), obs.begin() + MAX_OBS, obs.end(),
+                                 [](const Partial *a, const Partial *b) {
+                                     return a->I / std::max(1e-9, a->sigma) > b->I / std::max(1e-9, b->sigma); });
+                obs.resize(MAX_OBS);
+            }
+            result.obs_used = static_cast<int>(obs.size());
+            const double beam_x0 = nominal_geom.GetBeamX_pxl(), beam_y0 = nominal_geom.GetBeamY_pxl();
+            const double dist0 = nominal_geom.GetDetectorDistance_mm();
+            auto pos_cost = [&](Subset s, const double beam[2], const double dist[1]) {
+                double c = 0.0; int n = 0;
+                for (const Partial *pp : obs) {
+                    if (!in(pp->h, pp->k, pp->l, s)) continue;
+                    XtalResidual r(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, rot3, pp->angle_rad,
+                                   pp->h, pp->k, pp->l, sys);
+                    double resid[3] = {0, 0, 0};
+                    r(beam, dist, det_rot, rot_vec, p0, p1, p2, resid);
+                    c += resid[0]*resid[0] + resid[1]*resid[1] + resid[2]*resid[2]; ++n;
+                }
+                return n ? c / n : 0.0;
+            };
+            auto solve_detector = [&](Subset s, double beam_out[2], double &dist_out) {
+                double beam[2] = {beam_x0, beam_y0}, dist[1] = {dist0};
+                ceres::Problem p;
+                for (const Partial *pp : obs) {
+                    if (!in(pp->h, pp->k, pp->l, s)) continue;
+                    p.AddResidualBlock(new ceres::AutoDiffCostFunction<XtalResidual, 3, 2, 1, 2, 3, 3, 3, 3>(
+                        new XtalResidual(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, rot3, pp->angle_rad,
+                                         pp->h, pp->k, pp->l, sys)),
+                        new ceres::CauchyLoss(0.02), beam, dist,
+                        const_cast<double *>(det_rot), rot_vec, p0, p1, p2);
+                    p.SetParameterBlockConstant(const_cast<double *>(det_rot));
+                    p.SetParameterBlockConstant(rot_vec);
+                    p.SetParameterBlockConstant(p0); p.SetParameterBlockConstant(p1); p.SetParameterBlockConstant(p2);
+                }
+                if (p.NumResidualBlocks() == 0) { beam_out[0] = beam_x0; beam_out[1] = beam_y0; dist_out = dist0; return false; }
+                p.SetParameterLowerBound(dist, 0, dist0 * 0.95); p.SetParameterUpperBound(dist, 0, dist0 * 1.05);
+                for (int j = 0; j < 2; ++j) { p.SetParameterLowerBound(beam, j, beam[j] - 15.0);
+                                              p.SetParameterUpperBound(beam, j, beam[j] + 15.0); }
+                ceres::Solver::Options o; o.linear_solver_type = ceres::DENSE_QR; o.max_num_iterations = 60;
+                o.num_threads = std::max(1, settings.num_threads); o.logging_type = ceres::LoggingType::SILENT;
+                ceres::Solver::Summary sum; ceres::Solve(o, &p, &sum);
+                beam_out[0] = beam[0]; beam_out[1] = beam[1]; dist_out = dist[0];
+                return sum.IsSolutionUsable();
+            };
+            double beam[2] = {beam_x0, beam_y0}, dist = dist0;
+            if (obs.size() >= static_cast<size_t>(settings.min_events)) {
+                double beam_fit[2], dist_fit;
+                const bool convB = solve_detector(FIT, beam_fit, dist_fit);
+                const double b_nom[2] = {beam_x0, beam_y0}, d_nom[1] = {dist0};
+                const double b_ref[2] = {beam_fit[0], beam_fit[1]}, d_ref[1] = {dist_fit};
+                const double cvB_nom = pos_cost(VAL, b_nom, d_nom);
+                const double cvB_ref = pos_cost(VAL, b_ref, d_ref);
+                // Commit the detector geometry only for a small, credible move: distance < 1 % (a calibrated
+                // header needs < ~0.6 %). A larger move is the red flag for an unreliable fit - typically a
+                // second lattice whose spots bias every cross-validation fold identically, so the relative
+                // "it improved" gate is blind to it and pulls a spurious distance<->cell pair (the radial
+                // degeneracy) far off. The absolute size of the move discriminates a genuine header correction
+                // from that failure far better than the absolute residual, which real marginal (noisy / iced)
+                // data shares with the multi-lattice case.
+                const bool in_bounds = std::fabs(dist_fit - dist0) < 0.01 * dist0
+                                       && std::hypot(beam_fit[0] - beam_x0, beam_fit[1] - beam_y0) < 15.0;
+                result.detector_refined = convB && cvB_ref < 0.98 * cvB_nom && in_bounds;
+                if (result.detector_refined) { double bo[2]; solve_detector(ALL, bo, dist); beam[0] = bo[0]; beam[1] = bo[1]; }
+                logger.Info("Post-refine GEOM step B (distance/beam): dist {:.3f} -> {:.3f} mm, beam "
+                            "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), held-out pos {:.3e} -> {:.3e} => {}",
+                            dist0, result.detector_refined ? dist : dist0, beam_x0, beam_y0,
+                            result.detector_refined ? beam[0] : beam_x0, result.detector_refined ? beam[1] : beam_y0,
+                            cvB_nom, cvB_ref, result.detector_refined ? "COMMIT" : "reject (kept nominal detector)");
+            } else {
+                logger.Info("Post-refine GEOM step B: only {} positional observations - skipped", obs.size());
+            }
+
+            // Assemble the committed geometry.
+            UnitCell cellA = r0;
+            if (result.cell_refined) { cellA.a = static_cast<float>(r0.a * s); cellA.b = static_cast<float>(r0.b * s);
+                                       cellA.c = static_cast<float>(r0.c * s); }
+            result.cell = cellA;
+            result.distance_after_mm = dist;
+            result.beam_x_before_px = beam_x0; result.beam_x_after_px = beam[0];
+            result.beam_y_before_px = beam_y0; result.beam_y_after_px = beam[1];
+            result.events_used = static_cast<int>(events.size());
+            result.ok = result.cell_refined || result.detector_refined;
+            if (!result.ok)
+                logger.Info("Post-refine GEOM: neither step passed cross-validation - geometry left at nominal");
+            return result;
+        }
+
+        return result;   // refine_geometry is the only supported mode; nothing refined otherwise
+    } catch (...) {
+        result.ok = false;
+        return result;
+    }
+}
+

image_analysis/geom_refinement/PostRefine.h

@@ -0,0 +1,59 @@
+// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#pragma once
+
+#include <vector>
+
+#include "../../common/DiffractionGeometry.h"
+#include "../../common/CrystalLattice.h"
+#include "../../common/GoniometerAxis.h"
+#include "../../common/UnitCell.h"
+#include "../../common/Logger.h"
+#include "../IntegrationOutcome.h"
+#include "gemmi/symmetry.hpp"
+
+// Post-integration geometry refinement for rotation data. Unlike the at-indexing XtalOptimizer, this runs
+// AFTER integration/merge, where each reflection has an OBSERVED rocking centroid phi_obs (the intensity-
+// weighted mean goniometer angle over the frames it spans) and an observed spot position. It refines one
+// shared crystal orientation + cell (+ optionally the detector distance) against two residuals:
+//   * an Ewald excitation residual evaluated at phi_obs (distance-independent) -> pins the absolute cell
+//     scale that the positional residual leaves degenerate with the distance. Because phi_obs is the real
+//     rocking angle (not a frame centre) it is unbiased.
+//   * the positional detector<->reciprocal residual at each partial's observed spot -> pins the distance.
+// Reflections are weighted by their merged I/sigma (strong, well-measured reflections dominate).
+
+struct PostRefineResult {
+    bool ok = false;
+    DiffractionGeometry geom;                 // refined (distance / beam left as configured)
+    UnitCell cell{};                          // refined unit cell
+    int events_used = 0;
+    int obs_used = 0;
+    double phi_rms_deg = 0.0;                 // RMS(phi_obs - phi_pred) at the refined model (fit quality)
+    double distance_before_mm = 0.0, distance_after_mm = 0.0;
+    double beam_x_before_px = 0.0, beam_x_after_px = 0.0;   // refined beam centre (GEOM mode)
+    double beam_y_before_px = 0.0, beam_y_after_px = 0.0;
+    bool cell_refined = false;                // GEOM step A (cell scale + axis) passed cross-validation
+    bool detector_refined = false;            // GEOM step B (distance + beam) passed cross-validation
+    double est_mosaicity_deg = 0.0;           // mosaicity estimated from the observed rocking widths (0 = n/a)
+};
+
+struct PostRefineSettings {
+    gemmi::CrystalSystem crystal_system = gemmi::CrystalSystem::Triclinic;
+    bool refine_geometry = false;     // XtalOptimizer-equivalent: cell scale + axis (from phi_obs) and detector
+                                      // distance + beam centre (from the observed spot positions X,Y), as two
+                                      // separate cross-validated steps. The only supported refinement mode.
+    double excitation_weight = 1.0;   // weight of the phi/excitation residual vs the positional one
+    int min_events = 50;
+    int num_threads = 1;
+};
+
+// nominal_geom / reference_latt: the current detector geometry and the phi=0 reference lattice (orientation
+// + cell) from rotation indexing. outcomes: the per-image integrated reflections (observed_x/y, I, sigma,
+// image_number). axis: the goniometer. Returns ok=false (geometry untouched) on failure.
+PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome> &outcomes,
+                                            const GoniometerAxis &axis,
+                                            const DiffractionGeometry &nominal_geom,
+                                            const CrystalLattice &reference_latt,
+                                            const PostRefineSettings &settings,
+                                            Logger &logger);

image_analysis/geom_refinement/XtalOptimizer.cpp

@@ -5,201 +5,24 @@
 #include <Eigen/Dense>
 
 #include "XtalOptimizer.h"
+#include "XtalResidual.h"
 #include "ceres/ceres.h"
 #include "ceres/rotation.h"
 #include "LatticeReduction.h"
 
-struct XtalResidual {
-    XtalResidual(double x, double y,
-                 double lambda,
-                 double pixel_size,
-                 double rot3,
-                 double angle_rad,
-                 double exp_h, double exp_k,
-                 double exp_l,
-                 gemmi::CrystalSystem symmetry)
-        : obs_x(x), obs_y(y),
-          inv_lambda(1.0/lambda),
-          pixel_size(pixel_size),
-          rot3(rot3),
-          exp_h(exp_h),
-          exp_k(exp_k),
-          exp_l(exp_l),
-          angle_rad(angle_rad),
-          symmetry(symmetry) {
-        if (std::fabs(lambda) < 1e-6)
-            throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
-                "Lambda cannot be close to zero");
-    }
-
+// Soft header prior on ONE beam-centre component (the spindle-parallel, gauge-weak one). Residual = w*(b - b0);
+// the caller sets w so the prior behaves like a sigma-pixel restraint that competes with the (unit-weight)
+// positional residuals - strong enough to pin the gauge direction, negligible in the well-constrained one.
+struct BeamComponentPrior {
+    BeamComponentPrior(int component, double b0, double weight)
+        : component(component), b0(b0), weight(weight) {}
     template<typename T>
-    bool operator()(const T *const beam,
-                    const T *const distance_mm,
-                    const T *const detector_rot,
-                    const T *const rotation_axis,
-                    const T *const p0,
-                    const T *const p1,
-                    const T *const p2,
-                    T *residual) const {
-        // PyFAI convention: poni_rot = Rz(-rot3) * Rx(-rot2) * Ry(+rot1).
-        // detector_rot[0] = rot1, detector_rot[1] = rot2 are refined; rot3 is fixed
-        // (e.g. from a PONI import) and baked in here as a constant so that a non-zero
-        // rot3 is not silently dropped during refinement.
-        const T rot1 = detector_rot[0];
-        const T rot2 = detector_rot[1];
-
-        // Ry(+rot1): rotation around Y-axis
-        const T c1 = ceres::cos(rot1);
-        const T s1 = ceres::sin(rot1);
-
-        // Rx(-rot2): rotation around X-axis with inverted sign (PyFAI left-handed)
-        const T c2 = ceres::cos(rot2);
-        const T s2 = ceres::sin(rot2);
-
-        // Rz(-rot3): rotation around Z (beam); constant, identity when rot3 == 0
-        const T c3 = T(cos(rot3));
-        const T s3 = T(sin(rot3));
-
-        // Detector coordinates in mm
-        const T det_x = (T(obs_x) - beam[0]) * T(pixel_size);
-        const T det_y = (T(obs_y) - beam[1]) * T(pixel_size);
-        const T det_z = T(distance_mm[0]);
-
-        // Apply Ry(rot1) first: rotate around Y
-        const T t1_x = c1 * det_x + s1 * det_z;
-        const T t1_y = det_y;
-        const T t1_z = -s1 * det_x + c1 * det_z;
-
-        // Then apply Rx(-rot2): rotate around X
-        const T t2_x = t1_x;
-        const T t2_y = c2 * t1_y + s2 * t1_z;
-        const T t2_z = -s2 * t1_y + c2 * t1_z;
-
-        // Then apply Rz(-rot3): rotate around Z (beam)
-        const T x = c3 * t2_x + s3 * t2_y;
-        const T y = -s3 * t2_x + c3 * t2_y;
-        const T z = t2_z;
-
-        // convert to recip space
-        const T lab_norm = ceres::sqrt(x * x + y * y + z * z);
-        const T inv_norm = T(1) / lab_norm;
-
-        T recip_raw[3];
-        recip_raw[0] = x * inv_norm * T(inv_lambda);
-        recip_raw[1] = y * inv_norm * T(inv_lambda);
-        recip_raw[2] = (z * inv_norm - T(1.0)) * T(inv_lambda);
-
-        // Apply goniometer "back-to-start" rotation:
-        // brings observed reciprocal from image orientation into reference crystal frame
-        const T aa_back[3] = {
-            T(angle_rad) * rotation_axis[0],
-            T(angle_rad) * rotation_axis[1],
-            T(angle_rad) * rotation_axis[2]
-        };
-
-        T recip_obs[3];
-        ceres::AngleAxisRotatePoint(aa_back, recip_raw, recip_obs);
-
-        const Eigen::Matrix<T, 3, 1> e_obs_recip(recip_obs[0], recip_obs[1], recip_obs[2]);
-
-        // Build unit cell lengths and B (convention: columns are a, b, c prior to global rotation)
-        Eigen::Matrix<T, 3, 1> e_uc_len = Eigen::Matrix<T, 3, 1>::Zero();
-        Eigen::Matrix<T, 3, 3> B = Eigen::Matrix<T, 3, 3>::Identity();
-
-        if (symmetry == gemmi::CrystalSystem::Hexagonal) {
-            e_uc_len << p1[0], p1[0], p1[2];
-            B(0, 1) = T(-0.5); // cos(120)
-            B(1, 1) = T(sqrt(3.0) / 2.0); // sin(120)
-        } else if (symmetry == gemmi::CrystalSystem::Orthorhombic) {
-            e_uc_len << p1[0], p1[1], p1[2];
-        } else if (symmetry == gemmi::CrystalSystem::Tetragonal) {
-            e_uc_len << p1[0], p1[0], p1[2];
-        } else if (symmetry == gemmi::CrystalSystem::Cubic) {
-            e_uc_len << p1[0], p1[0], p1[0];
-        } else if (symmetry == gemmi::CrystalSystem::Monoclinic) {
-            // Unique axis b: alpha = gamma = 90°, beta free (angle between a and c)
-            e_uc_len << p1[0], p1[1], p1[2];
-            B(0, 2) = ceres::cos(p2[0]);
-            B(2, 2) = ceres::sin(p2[0]);
-        } else {
-            // Triclinic: p1 = (a,b,c), p2 = (alpha, beta, gamma) in radians
-            const T ca = ceres::cos(p2[0]);
-            const T cb = ceres::cos(p2[1]);
-            const T cg = ceres::cos(p2[2]);
-            const T sg = ceres::sin(p2[2]);
-
-            e_uc_len << p1[0], p1[1], p1[2];
-
-            B(0, 0) = T(1);
-            B(1, 0) = T(0);
-            B(2, 0) = T(0);
-            B(0, 1) = cg;
-            B(1, 1) = sg;
-            B(2, 1) = T(0);
-
-            // c vector components:
-            const T cx = cb;
-            const T cy = (ca - cb * cg) / sg;
-            const T v = T(1) - cx * cx - cy * cy;
-            const T cz = (v >= T(0)) ? ceres::sqrt(v) : T(0);
-
-            B(0, 2) = cx;
-            B(1, 2) = cy;
-            B(2, 2) = cz;
-        }
-
-        // Build unrotated direct lattice columns: (B * D), then rotate them by p0.
-        // This avoids AngleAxisToRotationMatrix + matrix multiplications.
-        const T L0 = e_uc_len[0];
-        const T L1 = e_uc_len[1];
-        const T L2 = e_uc_len[2];
-
-        T col0_unrot[3] = {B(0, 0) * L0, B(1, 0) * L0, B(2, 0) * L0};
-        T col1_unrot[3] = {B(0, 1) * L1, B(1, 1) * L1, B(2, 1) * L1};
-        T col2_unrot[3] = {B(0, 2) * L2, B(1, 2) * L2, B(2, 2) * L2};
-
-        T col0_rot[3], col1_rot[3], col2_rot[3];
-        ceres::AngleAxisRotatePoint(p0, col0_unrot, col0_rot);
-        ceres::AngleAxisRotatePoint(p0, col1_unrot, col1_rot);
-        ceres::AngleAxisRotatePoint(p0, col2_unrot, col2_rot);
-
-        const Eigen::Matrix<T, 3, 1> A(col0_rot[0], col0_rot[1], col0_rot[2]);
-        const Eigen::Matrix<T, 3, 1> Bv(col1_rot[0], col1_rot[1], col1_rot[2]);
-        const Eigen::Matrix<T, 3, 1> C(col2_rot[0], col2_rot[1], col2_rot[2]);
-
-        const Eigen::Matrix<T, 3, 1> BxC = Bv.cross(C);
-        const Eigen::Matrix<T, 3, 1> CxA = C.cross(A);
-        const Eigen::Matrix<T, 3, 1> AxB = A.cross(Bv);
-
-        const T V = A.dot(BxC);
-        const T invV = T(1) / V;
-
-        const Eigen::Matrix<T, 3, 1> Astar = BxC * invV;
-        const Eigen::Matrix<T, 3, 1> Bstar = CxA * invV;
-        const Eigen::Matrix<T, 3, 1> Cstar = AxB * invV;
-
-        const T h = T(exp_h);
-        const T k = T(exp_k);
-        const T l = T(exp_l);
-
-        const Eigen::Matrix<T, 3, 1> e_pred_recip = Astar * h + Bstar * k + Cstar * l;
-
-        residual[0] = e_obs_recip[0] - e_pred_recip[0];
-        residual[1] = e_obs_recip[1] - e_pred_recip[1];
-        residual[2] = e_obs_recip[2] - e_pred_recip[2];
-
+    bool operator()(const T *const beam, T *residual) const {
+        residual[0] = T(weight) * (beam[component] - T(b0));
         return true;
     }
-
-    const double obs_x, obs_y;
-    const double inv_lambda;
-    const double pixel_size;
-    const double rot3;
-    const double exp_h;
-    const double exp_k;
-    const double exp_l;
-    const double angle_rad;
-    gemmi::CrystalSystem symmetry;
+    int component;
+    double b0, weight;
 };
 
 struct XtalResidualRotationOnlyPrecomp {
@@ -391,6 +214,29 @@ bool XtalOptimizerInternal(XtalOptimizerData &data,
 
         if (!data.refine_beam_center)
             problem.SetParameterBlockConstant(beam);
+        else if (data.axis) {
+            // Gauge handling (single-axis rotation): rotating the whole experiment about the spindle leaves every
+            // spot position unchanged, so the beam-centre component PARALLEL to the spindle is a null/gauge-weak
+            // direction. Refining it freely lets it wander (~+3 px) and absorb centroid systematics into a wrong
+            // beam that the co-refined orientation keeps position-consistent. Rather than freeze it (the beam
+            // does drift - it is only LaB6-monitored to ~a few px), RESTRAIN it toward the header with a soft
+            // prior: the gauge direction has ~zero data sensitivity so the prior pins it near the header, while a
+            // real, well-supported drift can still overcome it. The spindle is along a detector axis in standard
+            // geometry, so restrain the dominant of X / Y.
+            const Coord spindle = data.axis->GetAxis();
+            const int parallel = (std::fabs(spindle.x) >= std::fabs(spindle.y)) ? 0 : 1;
+            // Weight so the prior is a sigma_px-pixel restraint that competes with the positional residuals.
+            // k = d|recip|/d(beam_px) ~ pixel/(distance*lambda) [A^-1/px]; scaling by sqrt(#residuals) makes the
+            // prior's curvature ~ (1/9) of the well-constrained-data curvature at sigma_px=3, i.e. data wins the
+            // perpendicular direction, the prior wins the gauge one.
+            constexpr double sigma_px = 3.0;
+            const double k = data.geom.GetPixelSize_mm() / (distance_mm * data.geom.GetWavelength_A());
+            const double w = k * std::sqrt(static_cast<double>(problem.NumResidualBlocks())) / sigma_px;
+            problem.AddResidualBlock(
+                new ceres::AutoDiffCostFunction<BeamComponentPrior, 1, 2>(
+                    new BeamComponentPrior(parallel, beam[parallel], w)),
+                nullptr, beam);
+        }
 
         if (!data.refine_detector_angles) {
             problem.SetParameterBlockConstant(detector_rot);

image_analysis/geom_refinement/XtalResidual.h

@@ -0,0 +1,214 @@
+// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#pragma once
+
+#include <cmath>
+
+#include <Eigen/Dense>
+
+#include "ceres/ceres.h"
+#include "ceres/rotation.h"
+#include "gemmi/symmetry.hpp"
+
+#include "../../common/JFJochException.h"
+
+// Detector -> reciprocal geometry residual, shared by the per-image XtalOptimizer (one lattice, one
+// frame) and the offline GeometryRefiner (shared beam/distance/cell blocks, one orientation block per
+// frame). Parameter blocks: beam(2), distance_mm(1), detector_rot(2 = rot1,rot2), rotation_axis(3),
+// p0 = lattice orientation angle-axis(3), p1 = unit-cell lengths(3), p2 = unit-cell angles(3, rad).
+// The residual is the difference between the observed reciprocal vector (from the detector position via
+// the current beam/distance/tilt) and the predicted one (h*a* + k*b* + l*c* for the current cell +
+// orientation), in Angstrom^-1. rot3 and the goniometer angle_rad are baked in as constants.
+struct XtalResidual {
+    XtalResidual(double x, double y,
+                 double lambda,
+                 double pixel_size,
+                 double rot3,
+                 double angle_rad,
+                 double exp_h, double exp_k,
+                 double exp_l,
+                 gemmi::CrystalSystem symmetry)
+        : obs_x(x), obs_y(y),
+          inv_lambda(1.0/lambda),
+          pixel_size(pixel_size),
+          rot3(rot3),
+          exp_h(exp_h),
+          exp_k(exp_k),
+          exp_l(exp_l),
+          angle_rad(angle_rad),
+          symmetry(symmetry) {
+        if (std::fabs(lambda) < 1e-6)
+            throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
+                "Lambda cannot be close to zero");
+    }
+
+    template<typename T>
+    bool operator()(const T *const beam,
+                    const T *const distance_mm,
+                    const T *const detector_rot,
+                    const T *const rotation_axis,
+                    const T *const p0,
+                    const T *const p1,
+                    const T *const p2,
+                    T *residual) const {
+        // PyFAI convention: poni_rot = Rz(-rot3) * Rx(-rot2) * Ry(+rot1).
+        // detector_rot[0] = rot1, detector_rot[1] = rot2 are refined; rot3 is fixed
+        // (e.g. from a PONI import) and baked in here as a constant so that a non-zero
+        // rot3 is not silently dropped during refinement.
+        const T rot1 = detector_rot[0];
+        const T rot2 = detector_rot[1];
+
+        // Ry(+rot1): rotation around Y-axis
+        const T c1 = ceres::cos(rot1);
+        const T s1 = ceres::sin(rot1);
+
+        // Rx(-rot2): rotation around X-axis with inverted sign (PyFAI left-handed)
+        const T c2 = ceres::cos(rot2);
+        const T s2 = ceres::sin(rot2);
+
+        // Rz(-rot3): rotation around Z (beam); constant, identity when rot3 == 0
+        const T c3 = T(cos(rot3));
+        const T s3 = T(sin(rot3));
+
+        // Detector coordinates in mm
+        const T det_x = (T(obs_x) - beam[0]) * T(pixel_size);
+        const T det_y = (T(obs_y) - beam[1]) * T(pixel_size);
+        const T det_z = T(distance_mm[0]);
+
+        // Apply Ry(rot1) first: rotate around Y
+        const T t1_x = c1 * det_x + s1 * det_z;
+        const T t1_y = det_y;
+        const T t1_z = -s1 * det_x + c1 * det_z;
+
+        // Then apply Rx(-rot2): rotate around X
+        const T t2_x = t1_x;
+        const T t2_y = c2 * t1_y + s2 * t1_z;
+        const T t2_z = -s2 * t1_y + c2 * t1_z;
+
+        // Then apply Rz(-rot3): rotate around Z (beam)
+        const T x = c3 * t2_x + s3 * t2_y;
+        const T y = -s3 * t2_x + c3 * t2_y;
+        const T z = t2_z;
+
+        // convert to recip space
+        const T lab_norm = ceres::sqrt(x * x + y * y + z * z);
+        const T inv_norm = T(1) / lab_norm;
+
+        T recip_raw[3];
+        recip_raw[0] = x * inv_norm * T(inv_lambda);
+        recip_raw[1] = y * inv_norm * T(inv_lambda);
+        recip_raw[2] = (z * inv_norm - T(1.0)) * T(inv_lambda);
+
+        // Apply goniometer "back-to-start" rotation:
+        // brings observed reciprocal from image orientation into reference crystal frame
+        const T aa_back[3] = {
+            T(angle_rad) * rotation_axis[0],
+            T(angle_rad) * rotation_axis[1],
+            T(angle_rad) * rotation_axis[2]
+        };
+
+        T recip_obs[3];
+        ceres::AngleAxisRotatePoint(aa_back, recip_raw, recip_obs);
+
+        const Eigen::Matrix<T, 3, 1> e_obs_recip(recip_obs[0], recip_obs[1], recip_obs[2]);
+
+        // Build unit cell lengths and B (convention: columns are a, b, c prior to global rotation)
+        Eigen::Matrix<T, 3, 1> e_uc_len = Eigen::Matrix<T, 3, 1>::Zero();
+        Eigen::Matrix<T, 3, 3> B = Eigen::Matrix<T, 3, 3>::Identity();
+
+        if (symmetry == gemmi::CrystalSystem::Hexagonal) {
+            e_uc_len << p1[0], p1[0], p1[2];
+            B(0, 1) = T(-0.5); // cos(120)
+            B(1, 1) = T(sqrt(3.0) / 2.0); // sin(120)
+        } else if (symmetry == gemmi::CrystalSystem::Orthorhombic) {
+            e_uc_len << p1[0], p1[1], p1[2];
+        } else if (symmetry == gemmi::CrystalSystem::Tetragonal) {
+            e_uc_len << p1[0], p1[0], p1[2];
+        } else if (symmetry == gemmi::CrystalSystem::Cubic) {
+            e_uc_len << p1[0], p1[0], p1[0];
+        } else if (symmetry == gemmi::CrystalSystem::Monoclinic) {
+            // Unique axis b: alpha = gamma = 90°, beta free (angle between a and c)
+            e_uc_len << p1[0], p1[1], p1[2];
+            B(0, 2) = ceres::cos(p2[0]);
+            B(2, 2) = ceres::sin(p2[0]);
+        } else {
+            // Triclinic: p1 = (a,b,c), p2 = (alpha, beta, gamma) in radians
+            const T ca = ceres::cos(p2[0]);
+            const T cb = ceres::cos(p2[1]);
+            const T cg = ceres::cos(p2[2]);
+            const T sg = ceres::sin(p2[2]);
+
+            e_uc_len << p1[0], p1[1], p1[2];
+
+            B(0, 0) = T(1);
+            B(1, 0) = T(0);
+            B(2, 0) = T(0);
+            B(0, 1) = cg;
+            B(1, 1) = sg;
+            B(2, 1) = T(0);
+
+            // c vector components:
+            const T cx = cb;
+            const T cy = (ca - cb * cg) / sg;
+            const T v = T(1) - cx * cx - cy * cy;
+            const T cz = (v >= T(0)) ? ceres::sqrt(v) : T(0);
+
+            B(0, 2) = cx;
+            B(1, 2) = cy;
+            B(2, 2) = cz;
+        }
+
+        // Build unrotated direct lattice columns: (B * D), then rotate them by p0.
+        // This avoids AngleAxisToRotationMatrix + matrix multiplications.
+        const T L0 = e_uc_len[0];
+        const T L1 = e_uc_len[1];
+        const T L2 = e_uc_len[2];
+
+        T col0_unrot[3] = {B(0, 0) * L0, B(1, 0) * L0, B(2, 0) * L0};
+        T col1_unrot[3] = {B(0, 1) * L1, B(1, 1) * L1, B(2, 1) * L1};
+        T col2_unrot[3] = {B(0, 2) * L2, B(1, 2) * L2, B(2, 2) * L2};
+
+        T col0_rot[3], col1_rot[3], col2_rot[3];
+        ceres::AngleAxisRotatePoint(p0, col0_unrot, col0_rot);
+        ceres::AngleAxisRotatePoint(p0, col1_unrot, col1_rot);
+        ceres::AngleAxisRotatePoint(p0, col2_unrot, col2_rot);
+
+        const Eigen::Matrix<T, 3, 1> A(col0_rot[0], col0_rot[1], col0_rot[2]);
+        const Eigen::Matrix<T, 3, 1> Bv(col1_rot[0], col1_rot[1], col1_rot[2]);
+        const Eigen::Matrix<T, 3, 1> C(col2_rot[0], col2_rot[1], col2_rot[2]);
+
+        const Eigen::Matrix<T, 3, 1> BxC = Bv.cross(C);
+        const Eigen::Matrix<T, 3, 1> CxA = C.cross(A);
+        const Eigen::Matrix<T, 3, 1> AxB = A.cross(Bv);
+
+        const T V = A.dot(BxC);
+        const T invV = T(1) / V;
+
+        const Eigen::Matrix<T, 3, 1> Astar = BxC * invV;
+        const Eigen::Matrix<T, 3, 1> Bstar = CxA * invV;
+        const Eigen::Matrix<T, 3, 1> Cstar = AxB * invV;
+
+        const T h = T(exp_h);
+        const T k = T(exp_k);
+        const T l = T(exp_l);
+
+        const Eigen::Matrix<T, 3, 1> e_pred_recip = Astar * h + Bstar * k + Cstar * l;
+
+        residual[0] = e_obs_recip[0] - e_pred_recip[0];
+        residual[1] = e_obs_recip[1] - e_pred_recip[1];
+        residual[2] = e_obs_recip[2] - e_pred_recip[2];
+
+        return true;
+    }
+
+    const double obs_x, obs_y;
+    const double inv_lambda;
+    const double pixel_size;
+    const double rot3;
+    const double exp_h;
+    const double exp_k;
+    const double exp_l;
+    const double angle_rad;
+    gemmi::CrystalSystem symmetry;
+};

image_analysis/indexing/FFTIndexer.cpp

@@ -23,7 +23,7 @@ FFTIndexer::FFTIndexer(const IndexingSettings &settings)
     // recovered lengths are independent of it. The padding factor is 2*pi: a historical value
     // from when the extent was mistakenly written as 2*pi/d (the Q convention); it is kept
     // because the exact amount sets which marginal frames index - rounding it to a nearby
-    // integer shifts the indexing rate ~0.5-1% (validated on the lyso datasets).
+    // integer shifts the indexing rate ~0.5-1% (measured on the test datasets).
     const float oversampling = 2.0f * static_cast<float>(PI);
     const float one_over_d_max = 1.0f / settings.GetFFT_HighResolution_A();
 

image_analysis/indexing/FFTIndexerCPU.cpp

@@ -88,7 +88,7 @@ void FFTIndexerCPU::ExecuteFFT(const std::vector<Coord> &coord, size_t nspots) {
     const double len_coeff = 2.0 * static_cast<double>(max_length_A) / static_cast<double>(H);
     // Background half-window ~15 A (in length, so it is independent of the histogram sizing);
     // wide enough to span the envelope yet narrow enough not to smooth real peaks. Validated
-    // optimum on the lyso jet (de-novo indexing 0% -> 24%, vs FFBIDX 27%); crystal 2 unchanged.
+    // optimum on serial-still jet data (de-novo indexing improved markedly vs FFBIDX); a second dataset unchanged.
     constexpr double BG_HALF_WIDTH_A = 15.0;
     const int bg_half = std::max(1, static_cast<int>(std::lround(BG_HALF_WIDTH_A / len_coeff)));
 

image_analysis/indexing/IndexerThreadPool.cpp

@@ -222,6 +222,42 @@ IndexerResult IndexerThreadPool::Run(const DiffractionExperiment &experiment, co
                               "Internal error: indexing algorithm must be resolved (not Auto) "
                               "before reaching the indexer pool");
 
+    // The workers built their indexers from the raw requested algorithm, but the algorithm actually
+    // dispatched is the RESOLVED one (rotation, for instance, always resolves to the GPU FFT indexer
+    // when a GPU is present, ignoring the request). If the resolution lands on an algorithm this host
+    // did not build an indexer for, fail here with an explanation instead of the opaque "no indexer
+    // available for the resolved algorithm" from deep inside a worker.
+    const auto requested = experiment.GetIndexingSettings().GetAlgorithm();
+    const bool have_gpu = get_gpu_count() > 0;
+#ifdef JFJOCH_USE_FFTW
+    constexpr bool fftw_built = true;
+#else
+    constexpr bool fftw_built = false;
+#endif
+    const bool servable =
+        (algorithm == IndexingAlgorithmEnum::FFT    && have_gpu &&
+             (requested == IndexingAlgorithmEnum::Auto || requested == IndexingAlgorithmEnum::FFT)) ||
+        (algorithm == IndexingAlgorithmEnum::FFBIDX && have_gpu &&
+             (requested == IndexingAlgorithmEnum::Auto || requested == IndexingAlgorithmEnum::FFBIDX)) ||
+        (algorithm == IndexingAlgorithmEnum::FFTW   && fftw_built &&
+             ((requested == IndexingAlgorithmEnum::Auto && !have_gpu) || requested == IndexingAlgorithmEnum::FFTW));
+    if (!servable) {
+        std::string msg;
+        if (requested == IndexingAlgorithmEnum::FFTW && have_gpu)
+            msg = "FFTW is the CPU indexer and is not available on a node with a GPU. Rotation indexing "
+                  "always uses the GPU FFT indexer here; select FFT or Auto, or run FFTW on a CPU-only node.";
+        else if (algorithm == IndexingAlgorithmEnum::FFT && !have_gpu)
+            msg = "FFT is the GPU indexer but no GPU is available. Select FFTW or Auto for CPU indexing.";
+        else if (algorithm == IndexingAlgorithmEnum::FFBIDX && !have_gpu)
+            msg = "FFBIDX is a GPU indexer but no GPU is available. Select FFTW or Auto for CPU indexing.";
+        else if (algorithm == IndexingAlgorithmEnum::FFTW)
+            msg = "FFTW (CPU) indexing was requested but this build has no FFTW indexer.";
+        else
+            msg = "the requested indexing algorithm resolved to one with no indexer available on this host.";
+        throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
+                              "Cannot index: " + msg);
+    }
+
     // Check if there is available worker
    const int task = GetFreeWorker();
 

image_analysis/indexing/PostIndexingRefinement.cpp

@@ -170,6 +170,34 @@ std::vector<CrystalLattice> Refine(const std::vector<Coord> &in_spots,
             }
             if (!lengths_ok)
                 continue;
+
+            // Also require the angles to match the reference. Fold each to its acute complement
+            // (min(x,180-x)) so the obtuse/acute setting choice is irrelevant, then compare the
+            // sorted triples. Guards against a right-edges/wrong-angle cell (a pseudo-symmetric
+            // near-metric, e.g. a monoclinic beta refined to the wrong value) passing on lengths.
+            auto fold = [](float deg) { return std::min(deg, 180.0f - deg); };
+            auto row_angle = [&](int i, int j) {
+                return std::acos(std::clamp(cell_rows.row(i).normalized().dot(cell_rows.row(j).normalized()),
+                                            -1.0f, 1.0f)) * 180.0f / PI;
+            };
+            std::array<float, 3> obs_ang = {fold(row_angle(1, 2)), fold(row_angle(0, 2)), fold(row_angle(0, 1))};
+            std::array<float, 3> ref_ang = {
+                fold(static_cast<float>(p.reference_unit_cell->alpha)),
+                fold(static_cast<float>(p.reference_unit_cell->beta)),
+                fold(static_cast<float>(p.reference_unit_cell->gamma))
+            };
+            std::sort(obs_ang.begin(), obs_ang.end());
+            std::sort(ref_ang.begin(), ref_ang.end());
+
+            bool angles_ok = true;
+            for (int k = 0; k < 3; ++k) {
+                if (std::abs(obs_ang[k] - ref_ang[k]) > REFINE_ANGLE_TOLERANCE_VS_REFERENCE_DEG) {
+                    angles_ok = false;
+                    break;
+                }
+            }
+            if (!angles_ok)
+                continue;
         } else {
             if (row_norms.minCoeff() < p.min_length_A || row_norms.maxCoeff() > p.max_length_A)
                 continue;

image_analysis/indexing/PostIndexingRefinement.h

@@ -17,6 +17,9 @@ constexpr float REFINE_CANDIDATE_VOLUME_RATIO_THRESHOLD = 1.05f;
 constexpr float REFINE_CANDIDATE_OVERLAP_RATIO_THRESHOLD = 0.4f;
 constexpr float REFINE_MIN_VOLUME_EPSILON = 1e-12f;
 constexpr float REFINE_MIN_REFERENCE_LENGTH_EPSILON = 1e-6f;
+// Max deviation (deg) of a candidate cell angle from the reference cell angle, compared after
+// folding both to the acute complement (min(x,180-x)) so the obtuse/acute setting is irrelevant.
+constexpr float REFINE_ANGLE_TOLERANCE_VS_REFERENCE_DEG = 10.0f;
 
 struct RefineParameters {
     int64_t viable_cell_min_spots;

image_analysis/rotation_indexer/RotationIndexer.cpp

@@ -12,6 +12,14 @@
 #include <future>
 
 namespace {
+    // Sub-cell override thresholds used in candidate selection to undo a spurious axis doubling:
+    // a later candidate replaces the chosen cell when it is smaller by more than this volume ratio
+    // (a doubling is 2x, well past 1.5) and indexes within this fraction slack of it. The slack is
+    // far below the indexed-fraction gap a real superstructure opens between its true cell and its
+    // sub-cell, so genuine large cells are kept.
+    constexpr float ROT_SUBCELL_VOLUME_RATIO = 1.5f;
+    constexpr float ROT_SUBCELL_FRAC_SLACK = 0.02f;
+
     // Re-express a primitive hexagonal/trigonal lattice in the conventional hexagonal setting
     // (a = b, gamma = 120). The Niggli-reduced primitive cell carries the two equal-length axes
     // at gamma = 60; replacing b with b - a opens that angle to 120 without changing the lattice.
@@ -158,6 +166,10 @@ void RotationIndexer::RunIndexing() {
                 .crystal_system = sys,
                 .min_spots = experiment.GetIndexingSettings().GetViableCellMinSpots(),
                 .max_length_A = length_bound_A,
+                // Match the indexers' [30,150] deg bound so a monoclinic beta outside [60,120]
+                // (e.g. beta>120) is refined, not clamped to the boundary.
+                .min_angle_deg = 30.0f,
+                .max_angle_deg = 150.0f,
                 .refine_beam_center = true,
                 .refine_distance_mm =  false,
                 .refine_detector_angles = true,
@@ -224,6 +236,7 @@ void RotationIndexer::RunIndexing() {
 
         // Assemble and select serially, in candidate order - identical to refining them one by one.
         float best_frac = -1.0f;
+        float best_vol = 0.0f;
         bool have_best = false;
         size_t best_ci = 0;
         XtalOptimizerData best_data;
@@ -264,8 +277,21 @@ void RotationIndexer::RunIndexing() {
             // marginally-higher alternative from displacing the primary when both index poorly (e.g. a
             // twin, where the accumulated-spot fraction is a noisy proxy) - only a decisively better
             // cell (a superstructure's true cell vs its sublattice) takes over.
-            if (!have_best || (frac > best_frac + 0.05f && frac > 0.15f)) {
+            // Displace the current best when the candidate indexes clearly more, OR when it is a
+            // genuine sub-cell: a meaningfully smaller cell that still indexes at least as many spots.
+            // The sub-cell branch unmasks a spurious supercell (axis doubling): the primitive cell
+            // always indexes >= its integer multiple, so a doubled cell that wins ci-order by the
+            // hysteresis margin is overridden by its own primitive. A real superstructure's true
+            // (larger) cell indexes MORE than its sub-cell and is kept by the clearly-more branch;
+            // twin lattices share the cell volume, so this never disturbs twin selection.
+            const float cand_vol = std::abs(data.latt.CalcVolume());
+            const bool clearly_more = frac > best_frac + 0.05f && frac > 0.15f;
+            const bool smaller_subcell = have_best && frac > 0.15f
+                                         && frac >= best_frac - ROT_SUBCELL_FRAC_SLACK
+                                         && cand_vol < best_vol / ROT_SUBCELL_VOLUME_RATIO;
+            if (!have_best || clearly_more || smaller_subcell) {
                 best_frac = frac;
+                best_vol = cand_vol;
                 have_best = true;
                 best_data = std::move(data);
                 best_sr = sr;

image_analysis/scale_merge/FrenchWilson.cpp

@@ -59,16 +59,24 @@ Posterior integrate_posterior(double I, double sigma, double sigma_wilson, bool
 
 void ApplyFrenchWilson(std::vector<MergedReflection> &merged, int32_t space_group_number,
                        const FrenchWilsonOptions &opts) {
-    auto naive = [](MergedReflection &r) {
-        const double ip = std::max(r.I, 0.0f);
-        r.F = static_cast<float>(std::sqrt(ip));
-        r.sigmaF = (ip > 0.0 && std::isfinite(r.sigma)) ? static_cast<float>(r.sigma / (2.0 * std::sqrt(ip)))
-                                                        : NAN;
+    // Naive amplitude sqrt(max(I,0)) for a missing / strong / untrusted intensity; NaN in -> NaN out
+    // (a missing Bijvoet hand stays missing). Fills one (F, sigmaF) pair.
+    auto naive_one = [](float I, float sigma, float &F, float &sigF) {
+        if (!std::isfinite(I)) { F = NAN; sigF = NAN; return; }
+        const double ip = std::max(I, 0.0f);
+        F = static_cast<float>(std::sqrt(ip));
+        sigF = (ip > 0.0 && std::isfinite(sigma)) ? static_cast<float>(sigma / (2.0 * std::sqrt(ip))) : NAN;
+    };
+    // The mean intensity and each measured hand share the reflection's Wilson prior, so fill all three.
+    auto naive_all = [&](MergedReflection &r) {
+        naive_one(r.I, r.sigma, r.F, r.sigmaF);
+        naive_one(r.I_plus, r.sigma_plus, r.F_plus, r.sigmaF_plus);
+        naive_one(r.I_minus, r.sigma_minus, r.F_minus, r.sigmaF_minus);
     };
 
     const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_number(space_group_number);
     if (sg == nullptr || merged.empty()) {
-        for (auto &r : merged) naive(r);
+        for (auto &r : merged) naive_all(r);
         return;
     }
     const gemmi::GroupOps gops = sg->operations();
@@ -80,7 +88,7 @@ void ApplyFrenchWilson(std::vector<MergedReflection> &merged, int32_t space_grou
             d_max = std::max(d_max, r.d);
         }
     if (!(d_min < d_max && d_min > 0.0f)) {
-        for (auto &r : merged) naive(r);
+        for (auto &r : merged) naive_all(r);
         return;
     }
 
@@ -110,22 +118,22 @@ void ApplyFrenchWilson(std::vector<MergedReflection> &merged, int32_t space_grou
         if (shell_count[s] >= opts.min_reflections_per_shell)
             shell_mean[s] = std::max(shell_sum[s] / shell_count[s], 1e-10);
 
-    for (auto &r : merged) {
-        if (!std::isfinite(r.I) || !std::isfinite(r.sigma) || r.sigma <= 0.0f) {
-            naive(r);
-            continue;
-        }
+    // French-Wilson |F| for one intensity of reflection r (its mean, or one Bijvoet hand); the shell
+    // Wilson prior, epsilon and centric flag are the reflection's, shared by all three.
+    auto fw_one = [&](const MergedReflection &r, float I, float sigma, float &F, float &sigF) {
+        if (!std::isfinite(I) || !std::isfinite(sigma) || sigma <= 0.0f) { naive_one(I, sigma, F, sigF); return; }
         // Strong reflections: the FW correction is negligible, <|F|> = sqrt(I).
-        if (r.I > opts.strong_cutoff * r.sigma) {
-            naive(r);
-            continue;
-        }
+        if (I > opts.strong_cutoff * sigma) { naive_one(I, sigma, F, sigF); return; }
         const auto s = shells.GetShell(r.d);
         const double sigma_wilson = epsilon(r) * (s ? shell_mean[*s] : global_mean);
         const bool centric = gops.is_reflection_centric({{r.h, r.k, r.l}});
-        const Posterior post = integrate_posterior(r.I, r.sigma, sigma_wilson, centric,
-                                                   opts.integration_points);
-        r.F = static_cast<float>(post.mean_F);
-        r.sigmaF = static_cast<float>(std::sqrt(std::max(0.0, post.mean_I - post.mean_F * post.mean_F)));
+        const Posterior post = integrate_posterior(I, sigma, sigma_wilson, centric, opts.integration_points);
+        F = static_cast<float>(post.mean_F);
+        sigF = static_cast<float>(std::sqrt(std::max(0.0, post.mean_I - post.mean_F * post.mean_F)));
+    };
+    for (auto &r : merged) {
+        fw_one(r, r.I, r.sigma, r.F, r.sigmaF);
+        fw_one(r, r.I_plus, r.sigma_plus, r.F_plus, r.sigmaF_plus);
+        fw_one(r, r.I_minus, r.sigma_minus, r.F_minus, r.sigmaF_minus);
     }
 }

image_analysis/scale_merge/Merge.cpp

@@ -93,7 +93,7 @@ void MergeOnTheFly::AddImage(const IntegrationOutcome &outcome, int64_t image_id
             continue;
         auto hkl = generator(r);
         auto hkl_key = hkl.pack();
-        sigma_corr = CorrectedSigma(I_corr, sigma_corr, hkl_key);
+        sigma_corr = CorrectedSigma(I_corr, sigma_corr, hkl_key, r.partiality);
 
         // Robust outlier rejection: drop this observation if it sits more than
         // reject_nsigma error-model sigmas from the reflection's median. Needs the active
@@ -131,7 +131,138 @@ void MergeOnTheFly::AddImage(const IntegrationOutcome &outcome, int64_t image_id
     }
 }
 
-float MergeOnTheFly::CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl_key) const {
+double MergeOnTheFly::RefineModulation(std::vector<IntegrationOutcome> &outcomes) {
+    // A minimum held-out generalizing gain (fraction of the held-out scatter) before the surface is
+    // applied - a margin, so a noise-level "improvement" never engages the correction. Matches the
+    // rotation ApplyCellSurface gate.
+    constexpr double CV_MIN_RELATIVE_GAIN = 0.02;
+    constexpr int NB = 16;
+    const int ncell = NB * NB;
+
+    // One scaled observation reduced to what the surface fit needs. `parity` (image index & 1) drives the
+    // even/odd cross-validation split - the stills analogue of the rotation frame parity.
+    struct MObs { double I, sigma, corr; float px, py; int32_t group; int parity; int cell; };
+    std::vector<MObs> obs;
+
+    // Accept exactly what AddImage merges (systematic absence, scale/resolution/ice/partiality filters),
+    // and additionally require a finite detector position. Returns the dense ASU-group id or -1.
+    std::unordered_map<uint64_t, int> group_of;
+    auto accept = [&](const Reflection &r, MObs &out) -> bool {
+        if (generator.IsSystematicallyAbsent(r)) return false;
+        if (r.image_scale_corr <= 0.0f || !std::isfinite(r.image_scale_corr)) return false;
+        if (!AcceptReflection(r, high_resolution_limit)) return false;
+        if (exclude_ice_rings && r.on_ice_ring) return false;
+        if (IsMaskedRing(r)) return false;
+        if (r.partiality < min_partiality) return false;
+        if (!std::isfinite(r.predicted_x) || !std::isfinite(r.predicted_y)) return false;
+        const float I_corr = r.I * r.image_scale_corr, sigma_corr = r.sigma * r.image_scale_corr;
+        if (!std::isfinite(I_corr) || !std::isfinite(sigma_corr) || sigma_corr <= 0.0f) return false;
+        const uint64_t key = generator(r).pack();
+        auto [it, inserted] = group_of.try_emplace(key, static_cast<int>(group_of.size()));
+        out.I = r.I; out.sigma = r.sigma; out.corr = r.image_scale_corr;
+        out.px = r.predicted_x; out.py = r.predicted_y; out.group = it->second;
+        return true;
+    };
+
+    float pxmin = std::numeric_limits<float>::infinity(), pxmax = -pxmin, pymin = pxmin, pymax = -pxmin;
+    for (size_t i = 0; i < outcomes.size(); ++i) {
+        const int parity = static_cast<int>(i & 1);
+        for (const auto &r : outcomes[i].reflections) {
+            MObs m{};
+            if (!accept(r, m)) continue;
+            m.parity = parity;
+            pxmin = std::min(pxmin, m.px); pxmax = std::max(pxmax, m.px);
+            pymin = std::min(pymin, m.py); pymax = std::max(pymax, m.py);
+            obs.push_back(m);
+        }
+    }
+    const int n_groups = static_cast<int>(group_of.size());
+    if (obs.size() < static_cast<size_t>(8 * ncell) || !(pxmax > pxmin) || !(pymax > pymin))
+        return 0.0;   // too sparse to over-determine a 16x16 surface, or degenerate detector footprint
+
+    const float sx = NB / (pxmax - pxmin), sy = NB / (pymax - pymin);
+    for (auto &m : obs) {
+        const int ix = std::clamp(static_cast<int>((m.px - pxmin) * sx), 0, NB - 1);
+        const int iy = std::clamp(static_cast<int>((m.py - pymin) * sy), 0, NB - 1);
+        m.cell = ix * NB + iy;
+    }
+
+    // Fit the per-cell factor over {parity subset} (parity < 0 = all obs), n_iter alternating rounds against
+    // that subset's own inverse-variance reference: Tikhonov pull to 1, gauge-fixed to a den-weighted
+    // geometric mean of 1 so it never drifts the overall scale. Mirrors RotationScaleMerge::ApplyCellSurface.
+    constexpr int N_ITER = 3;
+    auto fit_surface = [&](int parity) -> std::vector<double> {
+        std::vector<double> A(ncell, 1.0);
+        for (int it = 0; it < N_ITER; ++it) {
+            std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
+            for (const auto &o : obs) {
+                if (parity >= 0 && o.parity != parity) continue;
+                const double a = A[o.cell], sc = o.sigma * o.corr * a, w = 1.0 / (sc * sc);
+                sw[o.group] += w; swI[o.group] += w * o.I * o.corr * a;
+            }
+            std::vector<double> num(ncell, 0.0), den(ncell, 0.0);
+            for (const auto &o : obs) {
+                if ((parity >= 0 && o.parity != parity) || sw[o.group] <= 0.0) continue;
+                const double Iref = swI[o.group] / sw[o.group], a = A[o.cell];
+                const double Is = o.I * o.corr * a, sc = o.sigma * o.corr * a;
+                if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0) || !(sc > 0.0)) continue;
+                const double w = 1.0 / (sc * sc);
+                num[o.cell] += w * Is * Iref; den[o.cell] += w * Is * Is;
+            }
+            std::vector<double> dsorted = den;
+            std::nth_element(dsorted.begin(), dsorted.begin() + dsorted.size() / 2, dsorted.end());
+            const double lambda = 0.1 * std::max(1e-30, dsorted[dsorted.size() / 2]);
+            double logsum = 0.0, wsum = 0.0;
+            std::vector<double> upd(ncell, 1.0);
+            for (int c = 0; c < ncell; ++c) upd[c] = (num[c] + lambda) / (den[c] + lambda);
+            for (int c = 0; c < ncell; ++c) if (den[c] > 0.0) { logsum += den[c] * std::log(upd[c]); wsum += den[c]; }
+            const double gm = wsum > 0.0 ? std::exp(logsum / wsum) : 1.0;
+            for (int c = 0; c < ncell; ++c) A[c] = std::clamp(A[c] * upd[c] / gm, 0.25, 4.0);
+        }
+        return A;
+    };
+    // Sigma-independent (R-meas-like) agreement of the held-out equivalents: sum|Is - Iref| / sum|Iref|.
+    // A fractional metric cannot be gamed by a surface that merely reshapes sigma via corr.
+    auto score = [&](int parity, const std::vector<double> &A) -> double {
+        std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
+        for (const auto &o : obs) {
+            if (o.parity != parity) continue;
+            const double a = A[o.cell], Is = o.I * o.corr * a, sc = o.sigma * o.corr * a, w = 1.0 / (sc * sc);
+            sw[o.group] += w; swI[o.group] += w * Is;
+        }
+        double num = 0.0, den = 0.0;
+        for (const auto &o : obs) {
+            if (o.parity != parity || sw[o.group] <= 0.0) continue;
+            const double a = A[o.cell], Is = o.I * o.corr * a, Iref = swI[o.group] / sw[o.group];
+            if (!std::isfinite(Iref) || Iref <= 0.0) continue;
+            num += std::abs(Is - Iref); den += Iref;
+        }
+        return den > 0.0 ? num / den : 0.0;
+    };
+
+    // Cross-validate: fit on even images, score the held-out odd equivalents (and vice versa). Apply the
+    // full-data surface only if the held-out agreement improves by a clear margin.
+    const std::vector<double> ident(ncell, 1.0);
+    const std::vector<double> A_even = fit_surface(0), A_odd = fit_surface(1);
+    const double base = score(1, ident) + score(0, ident);
+    const double gain = base - (score(1, A_even) + score(0, A_odd));
+    if (!(gain > CV_MIN_RELATIVE_GAIN * base))
+        return 0.0;   // not cross-validated: the correction stays a no-op (caller logs)
+    const std::vector<double> A = fit_surface(-1);
+
+    // Fold the surface into each accepted reflection's image_scale_corr (recompute its cell deterministically).
+    for (auto &outcome : outcomes)
+        for (auto &r : outcome.reflections) {
+            MObs m{};
+            if (!accept(r, m)) continue;
+            const int ix = std::clamp(static_cast<int>((m.px - pxmin) * sx), 0, NB - 1);
+            const int iy = std::clamp(static_cast<int>((m.py - pymin) * sy), 0, NB - 1);
+            r.image_scale_corr = static_cast<float>(r.image_scale_corr * A[ix * NB + iy]);
+        }
+    return gain / std::max(base, 1e-30);   // held-out gain fraction (caller logs)
+}
+
+float MergeOnTheFly::CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl_key, float partiality) const {
     if (!error_model_active)
         return sigma_corr;
 
@@ -140,8 +271,21 @@ float MergeOnTheFly::CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl
     const auto it = error_model_mean_I.find(hkl_key);
     const double I_for_b = (it != error_model_mean_I.end()) ? it->second : I_corr;
 
-    const double v = error_model_a * static_cast<double>(sigma_corr) * sigma_corr
-                     + (error_model_b * I_for_b) * (error_model_b * I_for_b);
+    double v = error_model_a * static_cast<double>(sigma_corr) * sigma_corr
+               + (error_model_b * I_for_b) * (error_model_b * I_for_b);
+
+    // Partiality-model uncertainty: a reflection recorded at fraction p carries a systematic intensity
+    // error ~ (dp/p) that is proportional to <I> and grows as p falls - plain counting sigma misses it,
+    // so strong low-p partials would otherwise be over-trusted. This is the stills-partiality analog of
+    // the rotation --capture-uncertainty term ((1-captured_fraction)*I in RotationScaleMerge). Inert when
+    // partiality == 1 (no stills partiality model). Gated on a real systematic (error_model_b > 1, i.e.
+    // ISa < 1): on weak counting-limited data (small b) it would only over-concentrate the merge and hurt.
+    const double c = scaling_settings.GetPartialityUncertaintyCoeff();
+    if (c > 0.0 && error_model_b > 1.0) {
+        const double one_minus_p = std::max(0.0, std::min(1.0, 1.0 - static_cast<double>(partiality)));
+        const double t = c * I_for_b * one_minus_p;
+        v += t * t;
+    }
     return (v > 0.0) ? static_cast<float>(std::sqrt(v)) : sigma_corr;
 }
 
@@ -330,9 +474,6 @@ std::vector<char> MergeOnTheFly::DeltaCChalfReject(const std::vector<Integration
     std::unordered_map<uint64_t, Acc> acc;
     std::vector<int> img_half(outcomes.size(), 0);
 
-    std::mt19937 lrng{2026061600u};
-    std::bernoulli_distribution hd{0.5};
-
     // ---- pass 1: accumulate half-set sums, record each image's half ----
     auto contribution = [&](const Reflection &r, uint64_t &key, double &wI, double &w) -> bool {
         if (generator.IsSystematicallyAbsent(r)) return false;
@@ -349,7 +490,9 @@ std::vector<char> MergeOnTheFly::DeltaCChalfReject(const std::vector<Integration
     };
 
     for (size_t i = 0; i < outcomes.size(); ++i) {
-        const int half = hd(lrng) ? 1 : 0;
+        // Same deterministic half-set as the merge (HalfForImage), so deltaCChalf is measured on the
+        // exact CC1/2 split the statistics report - not an independent, order-dependent RNG draw.
+        const int half = HalfForImage(static_cast<int64_t>(i));
         img_half[i] = half;
         for (const auto &r : outcomes[i].reflections) {
             uint64_t key; double wI, w;
@@ -743,7 +886,7 @@ std::ostream &operator<<(std::ostream &output, const MergeStatisticsShell &in) {
     double multiplicity = in.unique_reflections > 0
         ? static_cast<double>(in.total_observations) / in.unique_reflections : 0.0;
 
-    output << fmt::format("{:8d} {:8d} {:8d} {:7.1f}% {:7.1f} {:8.1f} {:7.1f}% {:7.1f}% {:7.1f}%",
+    output << fmt::format("{:8d} {:8d} {:8d} {:7.1f}% {:7.1f} {:8.1f} {:7.1f}% {:7.1f}% {:7.1f}% {:8.2f}",
                 in.total_observations,
                 in.unique_reflections,
                 in.possible_unique_reflections,
@@ -752,17 +895,18 @@ std::ostream &operator<<(std::ostream &output, const MergeStatisticsShell &in) {
                 in.mean_i_over_sigma,
                 in.r_meas*100.0,
                 in.cc_half*100.0,
-                in.cc_ref*100.0);
+                in.cc_ref*100.0,
+                in.abs_diff_over_sigma_anomalous);
     return output;
 }
 
 std::ostream &operator<<(std::ostream &output, const MergeStatistics &in) {
     output << std::endl;
-    output << fmt::format("  {:>8s} {:>8s} {:>8s} {:>8s} {:>8s} {:>7s} {:>8s} {:>8s} {:>8s} {:>8s}",
-        "d_min", "N_obs", "N_uniq", "N_possib", "Compl", "Mult", "<I/sig>", "R_meas", "CC1/2", "CCref")
+    output << fmt::format("  {:>8s} {:>8s} {:>8s} {:>8s} {:>8s} {:>7s} {:>8s} {:>8s} {:>8s} {:>8s} {:>8s}",
+        "d_min", "N_obs", "N_uniq", "N_possib", "Compl", "Mult", "<I/sig>", "R_meas", "CC1/2", "CCref", "SigAno")
     << std::endl;
-    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s}",
-        "", "", "", "", "", "", "", "", "", "") << std::endl;
+    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s} {:->8s}",
+        "", "", "", "", "", "", "", "", "", "", "") << std::endl;
     for (const auto &sh: in.shells) {
         if (sh.unique_reflections == 0)
             continue;
@@ -770,12 +914,19 @@ std::ostream &operator<<(std::ostream &output, const MergeStatistics &in) {
         output << sh;
         output << std::endl;
     }
-    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s}",
-        "", "", "", "", "", "", "", "", "", "") << std::endl;
+    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s} {:->8s}",
+        "", "", "", "", "", "", "", "", "", "", "") << std::endl;
 
     output << fmt::format("  {:>8s} ", "Overall");
     output << in.overall;
     output << std::endl;
+    if (std::isfinite(in.wilson_b) && in.wilson_b > 0.0)
+        output << fmt::format("  Wilson B-factor estimate: {:.2f} A^2 (correlation {:.3f})",
+                              in.wilson_b, in.wilson_b_correlation) << std::endl;
+    if (std::isfinite(in.radiation_damage_delta_b))
+        output << fmt::format("  Radiation damage: relative B-factor change over run = {:+.2f} A^2 "
+                              "(first->last, {} batches)",
+                              in.radiation_damage_delta_b, in.radiation_damage_b_batch.size()) << std::endl;
     output << std::endl;
     return output;
 }

image_analysis/scale_merge/Merge.h

@@ -46,11 +46,31 @@ struct MergeStatisticsShell {
     // Redundancy-independent merging R-factor (Diederichs & Karplus 1997), computed over the
     // observations that enter the merge: R_meas = sum_hkl sqrt(n/(n-1)) sum_i|I_i-<I>| / sum I_i.
     double r_meas = NAN;
+
+    // Anomalous signal-to-noise (XDS "SigAno" / mmCIF pdbx_absDiff_over_sigma_anomalous):
+    // <|I(+)-I(-)|> / <sigma(I(+)-I(-))> over acentric reflections measured in both hands. NaN when
+    // there is no anomalous split (e.g. Friedel-merged with no mates, or the stills path).
+    double abs_diff_over_sigma_anomalous = NAN;
 };
 
 struct MergeStatistics {
     std::vector<MergeStatisticsShell> shells;
     MergeStatisticsShell overall;
+
+    // Dataset-wide isotropic Wilson B-factor estimate (A^2) from the log-linear fit of the shell-mean
+    // merged intensity against 1/d^2 (CalcGlobalWilsonB) - the analogue of XDS's "WILSON LINE ... B=".
+    // Diagnostic only; not used in scaling. NaN when not determined.
+    double wilson_b = NAN;
+    double wilson_b_correlation = NAN;
+
+    // Radiation-damage monitor (rotation only): the relative Debye-Waller B change measured from the first
+    // to the last frame of the run (A^2; positive = high-resolution intensity fades with dose = damage) and
+    // the per-batch relative-B curve it was derived from. Measured before any decay/relative-B correction is
+    // applied, so it reports how much radiation damage was present. Diagnostic; NaN / empty for stills or
+    // when not determined. batch_deg is the rotation width per batch of the curve.
+    double radiation_damage_delta_b = NAN;
+    std::vector<float> radiation_damage_b_batch;
+    double radiation_damage_batch_deg = 0.0;
 };
 
 
@@ -109,7 +129,7 @@ class MergeOnTheFly {
     // observations), so it inflates sigma without biasing the inverse-variance weights -
     // using the per-observation I_i instead would over-weight down-fluctuated points.
     std::unordered_map<uint64_t, float> error_model_mean_I;
-    [[nodiscard]] float CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl_key) const;
+    [[nodiscard]] float CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl_key, float partiality) const;
 
     // Optional per-observation outlier rejection: drop observations whose corrected
     // intensity lies more than reject_nsigma error-model sigmas from the reflection's
@@ -147,6 +167,16 @@ public:
     // order (or threading) of AddImage calls - not a draw from a shared RNG in call order.
     void AddImage(const IntegrationOutcome& outcome, int64_t image_id, bool cc_mask = false);
 
+    // Detector-plane modulation (flat-field) correction for the STILLS path: fit a smooth multiplicative
+    // factor over where a reflection lands on the detector (predicted x,y) against the merged reference and
+    // fold it into each reflection's image_scale_corr, so the following AddImage merge (and the error model)
+    // see the corrected scale. A 16x16 grid, cross-validated (fit even images, score the held-out odd
+    // equivalents by a sigma-independent R-meas-like metric) so it is a no-op when the systematic is absent
+    // or the data too sparse. The stills analogue of RotationScaleMerge::RefineModulation - the stills
+    // ScaleOnTheFly/MergeOnTheFly path otherwise has no correction surfaces. Mutates `outcomes`; call after
+    // scaling and before RefineErrorModel/AddImage. Returns the applied held-out gain fraction (0 = no-op).
+    double RefineModulation(std::vector<IntegrationOutcome> &outcomes);
+
     // Per-crystal CC1/2-delta rejection (CrystFEL deltaCChalf): returns a per-image flag
     // marking images whose removal would raise CC1/2 by a low-side outlier amount
     // (deltaCChalf < mean - nsigma*stddev). Skip the flagged images when merging.

image_analysis/scale_merge/ReindexAmbiguity.cpp

@@ -10,6 +10,36 @@
 
 #include "HKLKey.h"
 
+namespace {
+    constexpr size_t MIN_REFLECTIONS = 20;
+
+    struct BestReindex {
+        gemmi::Op op = gemmi::Op::identity();
+        bool is_identity = true;
+        double score = 0.0;
+        double identity_score = 0.0;
+    };
+
+    // Among identity (the baseline, scored `identity_score`) and `ops`, return the operator with the
+    // highest `score_op(op)`. A non-finite score never wins (too little overlap to decide). Shared by the
+    // stills per-image resolver and the rotation post-merge ChooseReindex.
+    BestReindex PickBestReindex(double identity_score, const std::vector<gemmi::Op> &ops,
+                                const std::function<double(const gemmi::Op &)> &score_op) {
+        BestReindex best;
+        best.identity_score = identity_score;
+        best.score = identity_score;
+        for (const auto &op : ops) {
+            const double sc = score_op(op);
+            if (std::isfinite(sc) && (!std::isfinite(best.score) || sc > best.score)) {
+                best.score = sc;
+                best.op = op;
+                best.is_identity = false;
+            }
+        }
+        return best;
+    }
+}
+
 std::vector<gemmi::Op> ReindexAmbiguityOperators(const UnitCell &cell, int space_group_number,
                                                  double max_obliquity_deg) {
     const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_number(space_group_number);
@@ -37,21 +67,18 @@ ReindexChoice ChooseReindex(const std::vector<MergedReflection> &merged,
                             const UnitCell &cell, int space_group_number,
                             const std::function<double(const std::vector<MergedReflection> &)> &score,
                             double max_obliquity_deg) {
-    ReindexChoice best;
-    best.identity_score = score(merged);
-    best.score = best.identity_score;
-
     const auto ops = ReindexAmbiguityOperators(cell, space_group_number, max_obliquity_deg);
-    best.n_candidates = 1 + static_cast<int>(ops.size());
-    for (const auto &op : ops) {
-        const double s = score(ReindexReflections(merged, op));
-        if (s > best.score) {
-            best.score = s;
-            best.op = op;
-            best.is_identity = false;
-        }
-    }
-    return best;
+    const BestReindex best = PickBestReindex(
+        score(merged), ops,
+        [&](const gemmi::Op &op) { return score(ReindexReflections(merged, op)); });
+
+    ReindexChoice choice;
+    choice.op = best.op;
+    choice.is_identity = best.is_identity;
+    choice.score = best.score;
+    choice.identity_score = best.identity_score;
+    choice.n_candidates = 1 + static_cast<int>(ops.size());
+    return choice;
 }
 
 double ReferenceIntensityCC(const std::vector<MergedReflection> &merged,
@@ -87,3 +114,66 @@ double ReferenceIntensityCC(const std::vector<MergedReflection> &merged,
     const double vy = n * syy - sy * sy;
     return (vx > 0 && vy > 0) ? cov / std::sqrt(vx * vy) : 0.0;
 }
+
+ReindexAmbiguityResolver::ReindexAmbiguityResolver(const DiffractionExperiment &x,
+                                                   const std::vector<MergedReflection> &reference)
+    : s(x.GetScalingSettings()),
+      hkl_key_generator(s.GetMergeFriedel(), x.GetSpaceGroupNumber().value_or(1)) {
+    for (const auto &r : reference)
+        reference_data[hkl_key_generator(r)] = r.I;
+    if (x.GetUnitCell().has_value() && x.GetSpaceGroupNumber().has_value())
+        ops = ReindexAmbiguityOperators(*x.GetUnitCell(), static_cast<int>(*x.GetSpaceGroupNumber()));
+}
+
+bool ReindexAmbiguityResolver::Accept(const Reflection &r) const {
+    if (r.on_ice_ring)   // ice-contaminated intensity would bias the correlation; keep it out
+        return false;
+    return AcceptReflection(r, s.GetHighResolutionLimit_A());
+}
+
+double ReindexAmbiguityResolver::ReferenceCC(const std::vector<Reflection> &reflections,
+                                             const gemmi::Op &op) const {
+    double sx = 0, sy = 0, sxx = 0, syy = 0, sxy = 0;
+    size_t n = 0;
+    for (const auto &r : reflections) {
+        if (!Accept(r) || r.partiality < s.GetMinPartiality())
+            continue;
+        if (!std::isfinite(r.I) || !std::isfinite(r.sigma) || r.sigma <= 0.0f || r.partiality <= 0.0f)
+            continue;
+        const gemmi::Op::Miller h = op.apply_to_hkl({{r.h, r.k, r.l}});
+        const auto it = reference_data.find(hkl_key_generator(h[0], h[1], h[2]));
+        if (it == reference_data.end())
+            continue;
+        const double x = static_cast<double>(r.I) * r.rlp / r.partiality;
+        const double y = it->second;
+        if (!std::isfinite(x) || !std::isfinite(y))
+            continue;
+        sx += x; sy += y; sxx += x * x; syy += y * y; sxy += x * y;
+        ++n;
+    }
+    if (n < MIN_REFLECTIONS)
+        return NAN;
+    const double nd = static_cast<double>(n);
+    const double cov = sxy - sx * sy / nd;
+    const double vx = sxx - sx * sx / nd;
+    const double vy = syy - sy * sy / nd;
+    return (vx > 0 && vy > 0) ? cov / std::sqrt(vx * vy) : NAN;
+}
+
+// Serial stills index each crystal in one of the merohedrally-equivalent hands at random; pick, for this
+// image alone, the reindexing whose intensities correlate best with the external reference and apply it.
+void ReindexAmbiguityResolver::Resolve(std::vector<Reflection> &reflections) const {
+    if (ops.empty())
+        return;
+
+    const BestReindex best = PickBestReindex(
+        ReferenceCC(reflections, gemmi::Op::identity()), ops,
+        [&](const gemmi::Op &op) { return ReferenceCC(reflections, op); });
+    if (best.is_identity)
+        return;
+
+    for (auto &r : reflections) {
+        const gemmi::Op::Miller h = best.op.apply_to_hkl({{r.h, r.k, r.l}});
+        r.h = h[0]; r.k = h[1]; r.l = h[2];
+    }
+}

image_analysis/scale_merge/ReindexAmbiguity.h

@@ -4,10 +4,13 @@
 #pragma once
 
 #include <functional>
+#include <map>
 #include <vector>
 
 #include "gemmi/symmetry.hpp"
 
+#include "HKLKey.h"
+#include "../../common/DiffractionExperiment.h"
 #include "../../common/Reflection.h"
 #include "../../common/UnitCell.h"
 
@@ -50,3 +53,32 @@ ReindexChoice ChooseReindex(const std::vector<MergedReflection> &merged,
 double ReferenceIntensityCC(const std::vector<MergedReflection> &merged,
                             const std::vector<MergedReflection> &reference,
                             int space_group_number);
+
+// Per-image resolver for serial stills. Each crystal is indexed independently in one of the
+// merohedrally-equivalent hands at random, so the ambiguity has to be broken per image (unlike rotation
+// data, which resolve it once for the whole merged set with ChooseReindex above). Holds the external
+// reference intensities and the crystal's twin-law operators, and reindexes one image's reflections into
+// the hand best-correlated with the reference. This runs in the per-image pipeline at integration time,
+// decoupled from scaling.
+class ReindexAmbiguityResolver {
+    const ScalingSettings s;
+    const HKLKeyGenerator hkl_key_generator;
+    std::map<HKLKey, double> reference_data;
+    // Twin-law reindexing operators (identity excluded). Empty = no ambiguity (holohedral crystal) or the
+    // cell/space group is unknown, in which case Resolve() is a no-op.
+    std::vector<gemmi::Op> ops;
+
+    bool Accept(const Reflection &r) const;
+    // Pearson correlation of this image's partiality/Lorentz-corrected intensities against the reference,
+    // matched by ASU index under `op`. Scale-invariant (the unknown per-image scale cancels). NaN when
+    // too few reflections match.
+    double ReferenceCC(const std::vector<Reflection> &reflections, const gemmi::Op &op) const;
+
+public:
+    // The reference must be an external, correctly-handed dataset (not the data's own running merge).
+    ReindexAmbiguityResolver(const DiffractionExperiment &x, const std::vector<MergedReflection> &reference);
+
+    // Reindex this image's reflections in place into the hand best-correlated with the reference (done
+    // once per image, for good). No-op when there is no ambiguity.
+    void Resolve(std::vector<Reflection> &reflections) const;
+};

image_analysis/scale_merge/RotationScaleMerge.cpp

@@ -11,6 +11,7 @@
 #include <future>
 #include <limits>
 #include <random>
+#include <unordered_map>
 
 #include <gemmi/reciproc.hpp>
 #include <gemmi/symmetry.hpp>
@@ -171,6 +172,8 @@ RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment,
     // so a no-op when their systematic is absent). Decoupled from the stills-only -B / RefineB flag.
     refine_decay_b = s.GetCorrectionSurfaces();
     absorption_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
+    modulation_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
+    relative_b_deg = s.GetRelativeBDegrees();   // opt-in per-batch relative-B (0 = off)
     scaling_iter = std::max(1, scaling_iterations);
     resolution_cutoff_method = s.GetResolutionCutoff();
     resolution_cc_target = s.GetResolutionCCTarget();
@@ -553,9 +556,10 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
     // noise / aggregate structure (not real per-reflection decay) would amplify into a high-res
     // mis-correction and worsen the symmetry-equivalent scatter that sets ISa. An opt-in correction must
     // never add noise where its systematic is absent (a low-dose reference set), so the slope is
-    // CROSS-VALIDATED: fit on even-frame fulls and score the held-out odd-frame equivalent chi^2, and vice
-    // versa. Real decay generalizes across the split (both held-out scores improve); a noise slope does
-    // not. Apply the full-data slope only if the mean held-out chi^2 strictly improves; else it is a no-op.
+    // CROSS-VALIDATED: fit on even-frame fulls and score the held-out odd-frame equivalent agreement (a
+    // sigma-independent Rmeas-like metric), and vice versa. Real decay generalizes across the split (both
+    // held-out scores improve); a noise slope does not. Apply the full-data slope only if the mean held-out
+    // agreement strictly improves; else it is a no-op.
     if (fulls.empty() || n_frames <= 0)
         return;
     const double fcenter = 0.5 * n_frames;
@@ -592,9 +596,14 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
         // strong damage (slope = 2 dB/dframe, so +-1.0 admits total relative-B up to ~n_frames/2 A^2).
         return (var > 0.0) ? std::clamp((Sxy - Sx * Sy / Sw) / var, -1.0, 1.0) : 0.0;
     };
-    // Mean studentized squared deviation over the subset {frame&1 == parity} when `slope` is applied,
-    // scored against that subset's own reference (no leakage). Lower = tighter equivalents.
-    auto subset_chi2 = [&](int parity, double slope) -> double {
+    // Rmeas-like (sigma-INDEPENDENT) disagreement of the equivalents over the subset {frame&1 == parity}
+    // when `slope` is applied, scored against that subset's own reference (no leakage): sum|Is-Iref|/sum|Iref|.
+    // Scoring on the studentized deviation instead would let the slope "improve" the held-out chi^2 by
+    // reshaping sigma - it scales sigma up on late / high-angle frames, down-weighting them - without
+    // actually tightening the intensities; a fractional metric cannot be gamed that way. This mirrors the
+    // absorption / modulation gate in ApplyCellSurface (which was switched off studentized chi^2 for the
+    // same reason); the reference is still the inverse-variance mean.
+    auto subset_disagreement = [&](int parity, double slope) -> double {
         std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
         for (const auto &o : fulls) {
             if (!usable(o) || (o.frame & 1) != parity) continue;
@@ -603,17 +612,15 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
             const double w = 1.0 / (sc * sc);
             sw[o.group] += w; swI[o.group] += w * Is;
         }
-        double sum = 0.0; size_t n = 0;
+        double num = 0.0, den = 0.0;
         for (const auto &o : fulls) {
             if (!usable(o) || (o.frame & 1) != parity || sw[o.group] <= 0.0) continue;
             const double Iref = swI[o.group] / sw[o.group];
             const double Is = static_cast<double>(o.I) * o.corr * decay_factor(o, slope);
-            const double sc = static_cast<double>(o.sigma) * o.corr * decay_factor(o, slope);
-            if (!std::isfinite(Iref) || !(sc > 0.0)) continue;
-            const double dd = (Is - Iref) / sc;
-            sum += dd * dd; ++n;
+            if (!std::isfinite(Iref) || Iref <= 0.0) continue;
+            num += std::abs(Is - Iref); den += Iref;
         }
-        return n > 0 ? sum / static_cast<double>(n) : 0.0;
+        return den > 0.0 ? num / den : 0.0;
     };
 
     // A meaningful radiation-damage relative-B accrues several A^2 across a dataset. Below a physical floor
@@ -630,8 +637,8 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
     }
     // And require the slope to cross-validate by a clear margin: fit on even frames, score the held-out
     // odd equivalents (and vice versa). Real damage generalizes; a fluke does not.
-    const double base = subset_chi2(1, 0.0) + subset_chi2(0, 0.0);
-    const double gain = base - (subset_chi2(1, fit_slope(0)) + subset_chi2(0, fit_slope(1)));
+    const double base = subset_disagreement(1, 0.0) + subset_disagreement(0, 0.0);
+    const double gain = base - (subset_disagreement(1, fit_slope(0)) + subset_disagreement(0, fit_slope(1)));
     if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
         logger.Info("Decay correction: not cross-validated (dB = {:.2f} A^2, held-out gain {:.1f}%, skipped)",
                     total_delta_B, 100.0 * gain / std::max(base, 1e-30));
@@ -644,6 +651,236 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
                 0.5 * slope * n_frames, 0.5 * slope);
 }
 
+std::vector<double> RotationScaleMerge::SolveCurvatureSmoothedB(const std::vector<double> &num,
+                                                               const std::vector<double> &den) const {
+    // Solve for a SMOOTH per-batch relative-B from the per-batch normal-equation data (num_c = sum w s^2 y,
+    // den_c = sum w s^4). A per-batch regression alone (b_c = num_c/den_c) over-fits: a genuine relative-B is
+    // a low-curvature function of dose / orientation, but an unconstrained fit chases resolution-correlated
+    // noise into a rough, oscillating curve. So minimise data-fidelity PLUS a second-difference (curvature)
+    // penalty  sum_c den_c (b_c - num_c/den_c)^2 + mu * sum_c (b_{c-1} - 2 b_c + b_{c+1})^2, which leaves a
+    // straight RAMP untouched but suppresses oscillation, solving the small SPD normal system A b = num,
+    // A = diag(den + lambda) + mu * D^T D, by Gauss-Seidel (n_batch <= a few tens). Not anchored - callers set
+    // the gauge (zero-mean for the correction, low-dose reference for the monitor).
+    constexpr double RELATIVE_B_MAX = 50.0;          // A^2, clamp a single batch's relative-B (generous guard)
+    constexpr double RELATIVE_B_CURVATURE_K = 8.0;   // curvature penalty weight (x the median batch leverage)
+    const int n_batch = static_cast<int>(num.size());
+    std::vector<double> dsorted(den);
+    std::sort(dsorted.begin(), dsorted.end());
+    const double med_den = dsorted.empty() ? 0.0 : dsorted[dsorted.size() / 2];
+    const double lambda = 0.1 * med_den;                 // ridge ~ 10% of the median batch leverage
+    const double mu = RELATIVE_B_CURVATURE_K * med_den;  // curvature penalty
+    std::vector<double> A(static_cast<size_t>(n_batch) * n_batch, 0.0);
+    auto Aat = [&](int i, int j) -> double & { return A[static_cast<size_t>(i) * n_batch + j]; };
+    for (int c = 0; c < n_batch; ++c) Aat(c, c) = den[c] + lambda;
+    constexpr int st[3] = {-1, 0, 1};
+    constexpr double sv[3] = {1.0, -2.0, 1.0};   // second-difference stencil
+    for (int c = 1; c < n_batch - 1; ++c)
+        for (int a = 0; a < 3; ++a)
+            for (int b2 = 0; b2 < 3; ++b2)
+                Aat(c + st[a], c + st[b2]) += mu * sv[a] * sv[b2];
+    std::vector<double> b(n_batch, 0.0);
+    for (int it = 0; it < 100; ++it)
+        for (int c = 0; c < n_batch; ++c) {
+            if (!(Aat(c, c) > 0.0)) continue;
+            double s = num[c];
+            for (int j = 0; j < n_batch; ++j)
+                if (j != c) s -= Aat(c, j) * b[j];
+            b[c] = std::clamp(s / Aat(c, c), -RELATIVE_B_MAX, RELATIVE_B_MAX);
+        }
+    return b;
+}
+
+std::vector<double> RotationScaleMerge::FitRelativeBCurve(int n_groups, int n_batch, int frames_per_batch,
+                                                         int gparity) const {
+    // Fit one relative-B per batch over the ASU-group subset {group&1 == gparity} (gparity < 0 = all fulls),
+    // against an inverse-variance reference built from that same subset (no leakage). Per batch the model is
+    // y = ln(I_ref / I_obs) = b * s^2 through the origin (the resolution-flat intercept is the per-frame G,
+    // already applied), so the data term is b_c = sum(w s^2 y) / sum(w s^2 s^2); SolveCurvatureSmoothedB then
+    // smooths it. Finally subtract the leverage-weighted mean: a constant B is a global Wilson-B degenerate
+    // with overall scale, and cannot change equivalent agreement (within one ASU group every equivalent sits
+    // at the same s^2, so only b VARYING between batches moves the merge). Drives the per-batch correction.
+    auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
+    auto usable = [&](const Obs &o) {
+        return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
+               && o.partiality >= min_partiality;
+    };
+    auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
+
+    std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
+    for (const auto &o : fulls) {
+        if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity)) continue;
+        const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
+        sw[o.group] += w; swI[o.group] += w * static_cast<double>(o.I) * o.corr;
+    }
+    std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
+    for (const auto &o : fulls) {
+        if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity) || sw[o.group] <= 0.0) continue;
+        const double Iref = swI[o.group] / sw[o.group];
+        const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
+        if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
+        const double w = (Is / sc) * (Is / sc);
+        const double s2 = s2_of(o.d), y = std::log(Iref / Is);
+        num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
+    }
+    std::vector<double> b = SolveCurvatureSmoothedB(num, den);
+    double dw = 0.0, dbw = 0.0;
+    for (int c = 0; c < n_batch; ++c) { dw += den[c]; dbw += den[c] * b[c]; }
+    const double bmean = dw > 0.0 ? dbw / dw : 0.0;
+    for (auto &v : b) v -= bmean;
+    return b;
+}
+
+void RotationScaleMerge::MeasureRadiationDamageB(int n_groups) {
+    // Radiation-damage MONITOR (report-only): measure the per-batch relative Debye-Waller B on the scaled
+    // fulls BEFORE any correction, so it captures how the high-resolution scattering power fades with
+    // accumulated dose. Crucially the reference is the PRISTINE LOW-DOSE crystal (first ~10% of frames), not
+    // the whole-run mean: a group measured both early and late then contributes ln(I_early / I_late) vs s^2,
+    // whose slope is the ADDED B - an absolute, correctly-signed damage read (positive = high-res fades).
+    // A whole-run-mean reference inverts on heavily damaged data (the surviving late high-res reflections
+    // look strong against a mean dominated by the frames where they had already faded). Stores the first->last
+    // relative-B change (the headline "B-factor difference") and the per-batch curve; never touches corr.
+    rad_damage_delta_b = std::numeric_limits<double>::quiet_NaN();
+    rad_damage_b_batch.clear();
+    rad_damage_batch_deg = 0.0;
+    if (fulls.empty() || n_frames <= 0)
+        return;
+    const auto gon = x.GetGoniometer();
+    const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
+    if (!(osc_deg > 1e-6))
+        return;
+    constexpr double MONITOR_BATCH_DEG = 10.0;   // rotation width per batch for the damage curve
+    const int frames_per_batch = std::max(1, static_cast<int>(std::lround(MONITOR_BATCH_DEG / osc_deg)));
+    const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
+    if (n_batch < 2)   // a single batch is a global Wilson-B (degenerate with scale) - nothing to monitor
+        return;
+    auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
+    auto usable = [&](const Obs &o) {
+        return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
+               && o.partiality >= min_partiality;
+    };
+    auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
+
+    // Low-dose reference: inverse-variance group means over the first ~10% of frames (>= 1 batch).
+    const int ref_frames = std::max(frames_per_batch, n_frames / 10);
+    std::vector<double> sw0(n_groups, 0.0), swI0(n_groups, 0.0);
+    for (const auto &o : fulls) {
+        if (!usable(o) || o.frame >= ref_frames) continue;
+        const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
+        sw0[o.group] += w; swI0[o.group] += w * static_cast<double>(o.I) * o.corr;
+    }
+    // Per batch: slope of ln(I_ref0 / I_obs) vs s^2, over groups also seen in the low-dose reference.
+    std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
+    for (const auto &o : fulls) {
+        if (!usable(o) || sw0[o.group] <= 0.0) continue;
+        const double Iref = swI0[o.group] / sw0[o.group];
+        const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
+        if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
+        const double w = (Is / sc) * (Is / sc);
+        const double s2 = s2_of(o.d), y = std::log(Iref / Is);
+        num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
+    }
+    std::vector<double> b = SolveCurvatureSmoothedB(num, den);
+    // Anchor so the reference batches read ~0 (b = added B relative to the low-dose start).
+    const int n_ref_batch = std::max(1, ref_frames / frames_per_batch);
+    double ref_mean = 0.0;
+    for (int c = 0; c < n_ref_batch && c < n_batch; ++c) ref_mean += b[c];
+    ref_mean /= std::min(n_ref_batch, n_batch);
+    for (auto &v : b) v -= ref_mean;
+    // Headline: the first->last relative-B change from a least-squares linear trend of the (smoothed) curve -
+    // robust to a single noisy end batch, unlike raw endpoints. Positive = high-res fades with dose = damage.
+    double sx = 0, sy = 0, sxx = 0, sxy = 0;
+    for (int c = 0; c < n_batch; ++c) { sx += c; sy += b[c]; sxx += double(c) * c; sxy += double(c) * b[c]; }
+    const double det = n_batch * sxx - sx * sx;
+    const double slope = det > 0.0 ? (n_batch * sxy - sx * sy) / det : 0.0;
+    rad_damage_delta_b = slope * (n_batch - 1);
+    rad_damage_batch_deg = MONITOR_BATCH_DEG;
+    rad_damage_b_batch.resize(n_batch);
+    for (int c = 0; c < n_batch; ++c) rad_damage_b_batch[c] = static_cast<float>(b[c]);
+}
+
+void RotationScaleMerge::RefineRelativeB(int n_groups) {
+    // RefineDecay removes the AVERAGE radiation-damage falloff as a single global relative-B slope, but the
+    // relative scattering power drifts NON-monotonically across a run (absorption path as the crystal
+    // rotates, slippage, dose bursts, a partly-shadowed wedge). XDS/AIMLESS model this with a per-BATCH
+    // relative-B on top of the per-batch scale. Here, on the fulls after RefineDecay: fit the smoothed
+    // per-batch relative-B (FitRelativeBCurve) and apply exp(b_batch * s^2). Same anti-overfit posture as
+    // RefineDecay - a physical spread floor and a cross-validated held-out gain - but the CV split is by ASU
+    // GROUP parity, not frame parity: a per-batch parameter owns whole frames, so it cannot be scored on a
+    // held-out FRAME; splitting the equivalents instead tests whether a batch's B generalises to reflections
+    // it was not fit on, which is exactly the over-fit risk of a many-parameter model.
+    if (fulls.empty() || n_frames <= 0 || relative_b_deg <= 0.0)
+        return;
+    const auto gon = x.GetGoniometer();
+    const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
+    if (!(osc_deg > 1e-6))
+        return;
+    const int frames_per_batch = std::max(1, static_cast<int>(std::lround(relative_b_deg / osc_deg)));
+    const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
+    if (n_batch < 2)   // a single batch is a global Wilson-B (degenerate with overall scale) - nothing relative
+        return;
+    auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
+    auto usable = [&](const Obs &o) {
+        return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
+               && o.partiality >= min_partiality;
+    };
+    auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
+    auto b_factor = [&](const Obs &o, const std::vector<double> &b) {
+        return std::exp(b[batch_of(o)] * s2_of(o.d));
+    };
+    // Held-out disagreement over the group subset {group&1 == gparity} with `b` applied: the same
+    // sigma-INDEPENDENT Rmeas-like metric as RefineDecay (sum|Is-Iref|/sum|Iref|), so a relative-B cannot
+    // "win" by reshaping sigma. Reference built from the scored subset (no leakage).
+    auto subset_disagreement = [&](int gparity, const std::vector<double> &b) -> double {
+        std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
+        for (const auto &o : fulls) {
+            if (!usable(o) || (o.group & 1) != gparity) continue;
+            const double f = b_factor(o, b);
+            const double Is = static_cast<double>(o.I) * o.corr * f, sc = static_cast<double>(o.sigma) * o.corr * f;
+            const double w = 1.0 / (sc * sc);
+            sw[o.group] += w; swI[o.group] += w * Is;
+        }
+        double num = 0.0, den = 0.0;
+        for (const auto &o : fulls) {
+            if (!usable(o) || (o.group & 1) != gparity || sw[o.group] <= 0.0) continue;
+            const double Iref = swI[o.group] / sw[o.group];
+            const double Is = static_cast<double>(o.I) * o.corr * b_factor(o, b);
+            if (!std::isfinite(Iref) || Iref <= 0.0) continue;
+            num += std::abs(Is - Iref); den += Iref;
+        }
+        return den > 0.0 ? num / den : 0.0;
+    };
+
+    // A relative-B whose peak-to-peak is below a couple A^2 is negligible; "correcting" it only spreads
+    // equivalents (same s^2, different batch) into extra scatter - net harmful, exactly like RefineDecay's
+    // sub-floor slope. (RELATIVE_B_MIN_SPREAD ~ DECAY_MIN_DELTA_B.)
+    constexpr double RELATIVE_B_MIN_SPREAD = 2.0;   // A^2, minimum peak-to-peak relative-B to engage
+    const std::vector<double> b_all = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, -1);
+    const double bmin = *std::min_element(b_all.begin(), b_all.end());
+    const double bmax = *std::max_element(b_all.begin(), b_all.end());
+    if (bmax - bmin < RELATIVE_B_MIN_SPREAD) {
+        logger.Info("Relative-B: negligible variation ({} batches, peak-to-peak {:.2f} A^2 < {:.1f}, skipped)",
+                    n_batch, bmax - bmin, RELATIVE_B_MIN_SPREAD);
+        return;
+    }
+    // Cross-validate: fit on even ASU groups, score the held-out odd equivalents (and vice-versa). Real
+    // per-batch structure generalises across the equivalent split; over-fit per-batch noise does not.
+    const std::vector<double> zero(n_batch, 0.0);
+    const double base = subset_disagreement(0, zero) + subset_disagreement(1, zero);
+    const std::vector<double> b_even = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 0);
+    const std::vector<double> b_odd = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 1);
+    const double gain = base - (subset_disagreement(1, b_even) + subset_disagreement(0, b_odd));
+    if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
+        logger.Info("Relative-B: not cross-validated ({} batches, peak-to-peak {:.2f} A^2, held-out gain {:.1f}%, skipped)",
+                    n_batch, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
+        return;
+    }
+    for (auto &o : fulls)
+        if (usable(o))
+            o.corr = static_cast<float>(o.corr * b_factor(o, b_all));
+    logger.Info("Relative-B: {} batches of ~{:.1f} deg, peak-to-peak {:.2f} A^2, held-out gain {:.1f}% (cross-validated)",
+                n_batch, relative_b_deg, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
+}
+
 void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
     // Absorption / path-length: a smooth multiplicative factor over the diffracted-beam direction in the
     // goniometer (crystal) frame. Each full's lab diffracted direction (from its predicted detector
@@ -661,7 +898,6 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
         return;
 
     constexpr int NB = 8;
-    const int ncell = NB * NB;
     std::vector<int32_t> cell(fulls.size(), -1);
     for (size_t i = 0; i < fulls.size(); ++i) {
         const Obs &o = fulls[i];
@@ -673,14 +909,46 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
         const int iy = std::clamp(static_cast<int>((u.y + 1.0f) * 0.5f * NB), 0, NB - 1);
         cell[i] = ix * NB + iy;
     }
+    ApplyCellSurface(cell, NB * NB, n_iter, n_groups, "Absorption (goniometer-frame 8x8)");
+}
 
+void RotationScaleMerge::RefineModulation(int n_iter, int n_groups) {
+    // Detector-plane modulation (flat-field): a smooth multiplicative factor over the position where a
+    // reflection lands on the detector (predicted px, py). It absorbs detector-response and geometric
+    // systematics that vary across the detector plane; symmetry-equivalents of one reflection land at
+    // different positions as the crystal rotates, over-determining the surface. Because it is a property of
+    // the detector + beam (not the rotation), the same correction concept transfers to stills.
+    if (fulls.empty())
+        return;
+    float pxmin = std::numeric_limits<float>::infinity(), pxmax = -pxmin, pymin = pxmin, pymax = -pxmin;
+    for (const Obs &o : fulls) {
+        if (!std::isfinite(o.px) || !std::isfinite(o.py)) continue;
+        pxmin = std::min(pxmin, o.px); pxmax = std::max(pxmax, o.px);
+        pymin = std::min(pymin, o.py); pymax = std::max(pymax, o.py);
+    }
+    if (!(pxmax > pxmin) || !(pymax > pymin))
+        return;
+    constexpr int NB = 16;
+    const float sx = NB / (pxmax - pxmin), sy = NB / (pymax - pymin);
+    std::vector<int32_t> cell(fulls.size(), -1);
+    for (size_t i = 0; i < fulls.size(); ++i) {
+        const Obs &o = fulls[i];
+        if (!std::isfinite(o.px) || !std::isfinite(o.py)) continue;
+        const int ix = std::clamp(static_cast<int>((o.px - pxmin) * sx), 0, NB - 1);
+        const int iy = std::clamp(static_cast<int>((o.py - pymin) * sy), 0, NB - 1);
+        cell[i] = ix * NB + iy;
+    }
+    ApplyCellSurface(cell, NB * NB, n_iter, n_groups, "Modulation (detector-frame 16x16)");
+}
+
+void RotationScaleMerge::ApplyCellSurface(const std::vector<int32_t> &cell, int ncell, int n_iter,
+                                          int n_groups, const char *name) {
     auto usable = [&](const Obs &o) {
         return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && o.partiality >= min_partiality;
     };
-    // Fit the per-cell absorption factor over the subset {frame&1 == parity} (parity < 0 = all fulls),
-    // n_iter alternating rounds against that subset's own reference (Tikhonov pull to 1, gauge-fixed to a
-    // den-weighted geometric mean of 1 so it never drifts the overall scale). Returns the accumulated
-    // per-cell factor A[cell].
+    // Fit the per-cell factor over the subset {frame&1 == parity} (parity < 0 = all fulls), n_iter
+    // alternating rounds against that subset's own reference (Tikhonov pull to 1, gauge-fixed to a
+    // den-weighted geometric mean of 1 so it never drifts the overall scale). Returns the per-cell factor.
     auto fit_surface = [&](int parity) -> std::vector<double> {
         std::vector<double> A(ncell, 1.0);
         for (int it = 0; it < n_iter; ++it) {
@@ -725,39 +993,40 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
             const double sc = static_cast<double>(o.sigma) * o.corr * a, w = 1.0 / (sc * sc);
             sw[o.group] += w; swI[o.group] += w * Is;
         }
-        double sum = 0.0; size_t n = 0;
+        // Rmeas-like (sigma-INDEPENDENT) agreement of the held-out equivalents: sum|Is - Iref| / sum|Iref|.
+        // Scoring on the studentized deviation instead lets a surface "improve" the held-out chi^2 by
+        // reshaping sigma (via corr) without tightening the actual intensities - which on mis-indexed / bad
+        // data passes cross-validation yet worsens Rmeas. A fractional metric cannot be gamed that way.
+        double num = 0.0, den = 0.0;
         for (size_t i = 0; i < fulls.size(); ++i) {
             const Obs &o = fulls[i];
             if (!usable(o) || cell[i] < 0 || (o.frame & 1) != parity || sw[o.group] <= 0.0) continue;
             const double a = A[cell[i]], Is = static_cast<double>(o.I) * o.corr * a;
-            const double sc = static_cast<double>(o.sigma) * o.corr * a;
             const double Iref = swI[o.group] / sw[o.group];
-            if (!std::isfinite(Iref) || !(sc > 0.0)) continue;
-            const double dd = (Is - Iref) / sc;
-            sum += dd * dd; ++n;
+            if (!std::isfinite(Iref) || Iref <= 0.0) continue;
+            num += std::abs(Is - Iref); den += Iref;
         }
-        return n > 0 ? sum / static_cast<double>(n) : 0.0;
+        return den > 0.0 ? num / den : 0.0;
     };
 
     // Cross-validate: fit the surface on even frames and score the held-out odd equivalents (and vice
-    // versa). A real absorption surface generalizes; an over-fit one (few obs / cell) does not. Apply only
-    // if the held-out equivalent chi^2 improves by a clear margin - so the correction can never worsen the
-    // data or act on marginal noise.
+    // versa). A real surface generalizes; an over-fit one (few obs / cell) does not. Apply only if the
+    // held-out equivalent chi^2 improves by a clear margin - so the correction can never worsen the data.
     const std::vector<double> ident(ncell, 1.0);
     const std::vector<double> A_even = fit_surface(0), A_odd = fit_surface(1);
     const double base = score(1, ident) + score(0, ident);
     const double gain = base - (score(1, A_even) + score(0, A_odd));
     if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
-        logger.Info("Absorption correction: not cross-validated (held-out gain {:.1f}%, skipped)",
-                    100.0 * gain / std::max(base, 1e-30));
+        logger.Info("{} correction: not cross-validated (held-out gain {:.1f}%, skipped)",
+                    name, 100.0 * gain / std::max(base, 1e-30));
         return;
     }
     const std::vector<double> A = fit_surface(-1);
     for (size_t i = 0; i < fulls.size(); ++i)
         if (cell[i] >= 0)
             fulls[i].corr = static_cast<float>(fulls[i].corr * A[cell[i]]);
-    logger.Info("Absorption correction: goniometer-frame {}x{} surface (cross-validated, held-out gain {:.1f}%)",
-                NB, NB, 100.0 * gain / std::max(base, 1e-30));
+    logger.Info("{} correction: cross-validated, held-out gain {:.1f}%",
+                name, 100.0 * gain / std::max(base, 1e-30));
 }
 
 void RotationScaleMerge::ComputeSmoothGWindow(const std::vector<double> &g, int window,
@@ -909,7 +1178,7 @@ void RotationScaleMerge::Combine() {
             if (dump != nullptr)
                 *dump << full.h << ' ' << full.k << ' ' << full.l << ' ' << full.I << ' '
                       << full.sigma << ' ' << d << ' ' << n_frames_event << ' ' << sum_partiality << ' '
-                      << static_cast<int>(peak_frame) << '\n';
+                      << static_cast<int>(peak_frame) << ' ' << full.px << ' ' << full.py << '\n';
         }
         return ev.size();
     };
@@ -922,7 +1191,9 @@ void RotationScaleMerge::Combine() {
         std::ofstream dump;
         if (!observation_dump_path.empty()) {
             dump.open(observation_dump_path);
-            dump << "# h k l I sigma d n_frames captured_fraction\n";
+            // Columns are enough to rescale/merge the combined fulls in an external program: peak_frame
+            // is a per-full batch (ZD) and px/py the predicted detector position (XD/YD).
+            dump << "# h k l I sigma d n_frames captured_fraction peak_frame px py\n";
         }
         fulls.reserve(n_run);
         for (int r = 0; r < n_run; ++r)
@@ -1207,9 +1478,51 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
                 fit_ab(std::move(kept));
         }
     }
+    // Asymptotic I/sigma. ISa is by definition the I -> infinity limit of the signal-to-noise, i.e. the
+    // reproducibility of the strongest reflections (Diederichs, Acta Cryst. D66 (2010), 733-740). The
+    // (a, b) fit above spans the whole intensity range, and a mild excess of scatter at intermediate
+    // intensity raises its systematic term `b`, so 1/b understates that limit. Read it instead directly
+    // from the strong equivalents: for each well-measured reflection group the counting-subtracted
+    // fractional scatter of its symmetry mates estimates the systematic term, and the robust median over
+    // strong groups is the asymptote. The I/sigma threshold that selects "strong" is relaxed on weak /
+    // radiation-damaged data that has few strong reflections (rather than fall back to the higher
+    // whole-range b); it falls back only when even the relaxed set is too small. Host-side over the
+    // merged fulls, so CPU and GPU agree.
+    double error_model_b_asymptotic = error_model_b;
+    if (error_model_active) {
+        struct GroupScatter { double sum = 0, sum_sq = 0, sum_var = 0; int n = 0; };
+        std::vector<GroupScatter> gs(n_groups);
+        for (const auto &o : fulls) {
+            if (!usable_merge(o)) continue;
+            const double I_corr = static_cast<double>(o.I) * o.corr;
+            const double sigma_corr = static_cast<double>(o.sigma) * o.corr;
+            auto &g = gs[o.group];
+            g.sum += I_corr; g.sum_sq += I_corr * I_corr; g.sum_var += sigma_corr * sigma_corr; ++g.n;
+        }
+        // Per-group counting-subtracted fractional systematic variance, paired with the group's I/sigma.
+        std::vector<std::pair<double, double>> group_scatter;   // (systematic b^2, I/sigma)
+        for (const auto &g : gs) {
+            if (g.n < 5) continue;
+            const double mean = g.sum / g.n;
+            const double counting = g.sum_var / g.n;
+            if (mean <= 0.0 || counting <= 0.0) continue;
+            const double variance = (g.sum_sq - g.sum * g.sum / g.n) / (g.n - 1);
+            group_scatter.push_back({std::max(variance - counting, 0.0) / (mean * mean),
+                                     mean / std::sqrt(counting)});
+        }
+        auto asymptote_above = [&](double snr_min, size_t min_groups) -> double {
+            std::vector<double> b2;
+            for (const auto &[b2_value, snr] : group_scatter)
+                if (snr >= snr_min) b2.push_back(b2_value);
+            return b2.size() >= min_groups ? std::sqrt(median_of(b2)) : 0.0;
+        };
+        double b_asy = asymptote_above(20.0, 100);   // tight threshold on data that supports it
+        if (b_asy <= 0.0) b_asy = asymptote_above(10.0, 50);   // relaxed for weak / damaged data
+        if (b_asy > 0.0) error_model_b_asymptotic = b_asy;
+    }
     if (error_model_active)
         logger.Info("Error model: a={:.3f} b={:.3f} ISa={:.1f} chi2={:.2f}", error_model_a, error_model_b,
-                    error_model_b > 0 ? 1.0 / error_model_b : 0.0, error_model_chi2);
+                    error_model_b_asymptotic > 0 ? 1.0 / error_model_b_asymptotic : 0.0, error_model_chi2);
 
     auto corrected_sigma = [&](float I_corr, float sigma_corr, int g) -> float {
         if (!error_model_active) return sigma_corr;
@@ -1266,7 +1579,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
         }
     // ---- Export merged reflections (+ resolution-shell R-free flags). ----
     Result result;
-    result.isa = error_model_b > 0 ? 1.0 / error_model_b : 0.0;
+    result.isa = error_model_b_asymptotic > 0 ? 1.0 / error_model_b_asymptotic : 0.0;
     std::vector<double> merged_I(n_groups, NAN);
     float d_min = std::numeric_limits<float>::max(), d_max = 0.0f;
     for (int g = 0; g < n_groups; ++g) {
@@ -1275,14 +1588,16 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
         MergedReflection mr{};
         mr.h = group_h[g]; mr.k = group_k[g]; mr.l = group_l[g];
         mr.I = static_cast<float>(a.swI / a.sw);
-        mr.sigma = SigmaWithSystematicFloor(1.0 / std::sqrt(a.sw), mr.I, error_model_b);
+        // The systematic floor caps a high-multiplicity merged sigma at the reproducibility that the
+        // strongest reflections actually reach, i.e. the asymptotic term, so merged I/sigma approaches ISa.
+        mr.sigma = SigmaWithSystematicFloor(1.0 / std::sqrt(a.sw), mr.I, error_model_b_asymptotic);
         mr.I_half[0] = mr.I_half[1] = NAN;
         mr.sigma_half[0] = mr.sigma_half[1] = NAN;
         mr.d = a.d;
         if (a.nh[0] + a.nh[1] > 0 && a.swh[0] > 0.0 && a.swh[1] > 0.0) {
             for (int i = 0; i < 2; ++i) {
                 mr.I_half[i] = static_cast<float>(a.swIh[i] / a.swh[i]);
-                mr.sigma_half[i] = SigmaWithSystematicFloor(1.0 / std::sqrt(a.swh[i]), mr.I_half[i], error_model_b);
+                mr.sigma_half[i] = SigmaWithSystematicFloor(1.0 / std::sqrt(a.swh[i]), mr.I_half[i], error_model_b_asymptotic);
             }
         }
         if (!std::isfinite(a.d) || a.d <= 0.0f) continue;
@@ -1301,7 +1616,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
         result.merged, d_min_limit, resolution_cutoff_method, resolution_cc_target, for_search, logger);
 
     AssignRfreeFlags(result.merged, x.GetSpaceGroupNumber().value_or(1), rfree_fraction);
-    ApplyFrenchWilson(result.merged, x.GetSpaceGroupNumber().value_or(1));
+    // French-Wilson (F, and F(+)/F(-) from the anomalous split) is deferred until after the anomalous
+    // accumulator below has attached I(+)/I(-), so the two hands get their amplitudes in one pass.
 
     if (reject_count > 0)
         logger.Info("Merge outlier rejection: dropped {} observations", reject_count);
@@ -1399,6 +1715,55 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
         rmeas_num_all += factor * r.sum_abs_dev; rmeas_den_all += r.sum_I;
     }
 
+    // ---- Anomalous split (always, even when the merge is Friedel-averaged): for each acentric
+    // reflection, the inverse-variance I(+)/I(-) from the SAME scaled fulls (so it never touches the
+    // Friedel-merged IMEAN, scaling or error model above). Lets I(+)/I(-) and F(+)/F(-) be written by
+    // default without scaling anomalously; a reflection with only one mate, or a centric, is left
+    // without the split. Skipped for the P1 search pass, which has no use for it. ----
+    struct AnomExport { float Ip = NAN, sIp = NAN, Im = NAN, sIm = NAN; };
+    std::unordered_map<uint64_t, AnomExport> anom_export;
+    // SigAno = <|I(+)-I(-)|>/<sigma(I(+)-I(-))> accumulated on the standard report shells (num=sum|dI|,
+    // den=sum sigma(dI)); mmCIF pdbx_absDiff_over_sigma_anomalous. NaN'd below when there is no split.
+    std::vector<double> sig_num(n_shells, 0.0), sig_den(n_shells, 0.0);
+    double sig_num_all = 0.0, sig_den_all = 0.0;
+    size_t sig_n = 0;
+    if (!for_search) {
+        const int sg_num = x.GetSpaceGroupNumber().value_or(1);
+        const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, sg_num);
+        const gemmi::GroupOps gops = gemmi::find_spacegroup_by_number(sg_num)->operations();
+        struct AnomAcc { double swI[2] = {}; double sw[2] = {}; int32_t h = 0, k = 0, l = 0; float d = NAN; };  // [hand] 0=I(+) 1=I(-)
+        std::unordered_map<uint64_t, AnomAcc> anom;
+        anom.reserve(result.merged.size() * 2 + 1);
+        for (const auto &o : fulls) {
+            if (!usable_merge(o)) continue;
+            if (rejected_obs[&o - fulls.data()]) continue;   // outlier-rejected in the merge (CPU path)
+            const HKLKey ak = anom_keygen(o.h, o.k, o.l);
+            const int hand = ak.plus ? 0 : 1;
+            const float I_corr = o.I * o.corr;
+            const float sigma_corr = corrected_sigma(I_corr, o.sigma * o.corr, o.group);
+            if (!(sigma_corr > 0.0f) || !std::isfinite(sigma_corr)) continue;
+            const double w = 1.0 / (static_cast<double>(sigma_corr) * sigma_corr);
+            AnomAcc &a = anom[HKLKey{ak.h, ak.k, ak.l, true}.pack()];
+            a.h = ak.h; a.k = ak.k; a.l = ak.l; a.d = o.d;
+            a.swI[hand] += w * static_cast<double>(I_corr); a.sw[hand] += w;
+        }
+        for (const auto &[fkey, a] : anom) {
+            // Centrics have I(+)=I(-) by symmetry; leave them without an anomalous split.
+            if (gops.is_reflection_centric(gemmi::Op::Miller{a.h, a.k, a.l})) continue;
+            AnomExport ex;
+            if (a.sw[0] > 0.0) { ex.Ip = static_cast<float>(a.swI[0] / a.sw[0]); ex.sIp = static_cast<float>(1.0 / std::sqrt(a.sw[0])); }
+            if (a.sw[1] > 0.0) { ex.Im = static_cast<float>(a.swI[1] / a.sw[1]); ex.sIm = static_cast<float>(1.0 / std::sqrt(a.sw[1])); }
+            if (std::isfinite(ex.Ip) || std::isfinite(ex.Im)) anom_export[fkey] = ex;
+            if (a.sw[0] > 0.0 && a.sw[1] > 0.0 && a.d > 0.0f) {
+                const double abs_di = std::fabs(a.swI[0] / a.sw[0] - a.swI[1] / a.sw[1]);
+                const double sig_di = std::sqrt(1.0 / a.sw[0] + 1.0 / a.sw[1]);
+                const auto shell = shells.GetShell(a.d);
+                if (shell && *shell >= 0 && *shell < n_shells) { sig_num[*shell] += abs_di; sig_den[*shell] += sig_di; }
+                sig_num_all += abs_di; sig_den_all += sig_di; ++sig_n;
+            }
+        }
+    }
+
     MergeStatistics &out = result.statistics;
     out.shells.resize(n_shells);
     for (int s = 0; s < n_shells; ++s) {
@@ -1413,6 +1778,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
         ss.cc_half = sa[s].cc_half.GetCC();
         ss.cc_ref = NAN;
         ss.r_meas = rmeas_den[s] > 0.0 ? rmeas_num[s] / rmeas_den[s] : NAN;
+        ss.abs_diff_over_sigma_anomalous = sig_den[s] > 0.0 ? sig_num[s] / sig_den[s] : NAN;
     }
     auto &overall = out.overall;
     overall.d_min = sd_min; overall.d_max = sd_max;
@@ -1427,6 +1793,33 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
     overall.cc_half = cc_half_overall.GetCC();
     overall.cc_ref = NAN;
     overall.r_meas = rmeas_den_all > 0.0 ? rmeas_num_all / rmeas_den_all : NAN;
+    overall.abs_diff_over_sigma_anomalous = sig_den_all > 0.0 ? sig_num_all / sig_den_all : NAN;
+    if (std::isfinite(overall.abs_diff_over_sigma_anomalous))
+        logger.Info("Anomalous signal SigAno = {:.2f} ({} acentric pairs)",
+                    overall.abs_diff_over_sigma_anomalous, sig_n);
+
+    // Radiation-damage monitor (measured before any correction by MeasureRadiationDamageB): carry the
+    // first->last relative-B change and the per-batch curve into the reported statistics / mmCIF.
+    out.radiation_damage_delta_b = rad_damage_delta_b;
+    out.radiation_damage_b_batch = rad_damage_b_batch;
+    out.radiation_damage_batch_deg = rad_damage_batch_deg;
+
+    // Attach the per-reflection anomalous split so the writer can emit I(+)/I(-) by default (each merged
+    // reflection maps to its Friedel-ASU key; in an anomalous merge both mates map to the same key).
+    if (!anom_export.empty()) {
+        const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, x.GetSpaceGroupNumber().value_or(1));
+        for (auto &r : result.merged) {
+            const HKLKey ak = anom_keygen(r.h, r.k, r.l);
+            const auto it = anom_export.find(HKLKey{ak.h, ak.k, ak.l, true}.pack());
+            if (it == anom_export.end()) continue;
+            r.I_plus = it->second.Ip; r.sigma_plus = it->second.sIp;
+            r.I_minus = it->second.Im; r.sigma_minus = it->second.sIm;
+        }
+    }
+
+    // French-Wilson amplitudes for IMEAN and (now that they are attached) each Bijvoet hand.
+    ApplyFrenchWilson(result.merged, x.GetSpaceGroupNumber().value_or(1));
+
     logger.Info("Merge complete ({} unique reflections)", result.merged.size());
     return result;
 }
@@ -1600,11 +1993,20 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
     //         the merged reference. Cheap host loops over the downloaded fulls; applied only on the final
     //         in-symmetry merge, never the P1 space-group search pass (corrections there add risk and can
     //         perturb the symmetry determination). ---
-    [[maybe_unused]] const bool corrections = !for_search && (refine_decay_b || absorption_iter > 0);
+    [[maybe_unused]] const bool corrections =
+        !for_search && (refine_decay_b || absorption_iter > 0 || modulation_iter > 0 || relative_b_deg > 0.0);
+    // Radiation-damage monitor: measure the per-batch relative-B on the scaled fulls BEFORE any correction
+    // (report-only; captures the full damage signature, not a residual). Skipped on the P1 search pass.
+    if (!for_search)
+        MeasureRadiationDamageB(n_groups);
     if (!for_search && refine_decay_b)
         RefineDecay(n_groups);
+    if (!for_search && relative_b_deg > 0.0)
+        RefineRelativeB(n_groups);   // per-batch relative-B on top of the single decay slope
     if (!for_search && absorption_iter > 0)
         RefineAbsorption(absorption_iter, n_groups);
+    if (!for_search && modulation_iter > 0)
+        RefineModulation(modulation_iter, n_groups);
 #ifdef JFJOCH_USE_CUDA
     // The corrections mutate the host fulls' corr; when the merge runs on the resident (GPU) fulls, push
     // the corrected corr back to the device so the merge reads it.

image_analysis/scale_merge/RotationScaleMerge.h

@@ -4,6 +4,7 @@
 #pragma once
 
 #include <cstdint>
+#include <limits>
 #include <optional>
 #include <vector>
 
@@ -106,6 +107,8 @@ private:
     bool scale_fulls = true;
     bool refine_decay_b = false;   // per-time-block Debye-Waller decay correction (radiation damage)
     int  absorption_iter = 0;      // >0: fit a goniometer-frame absorption surface over this many iterations
+    int  modulation_iter = 0;      // >0: fit a detector-plane modulation (flat-field) surface, this many iterations
+    double relative_b_deg = 0.0;   // >0: fit a per-batch relative-B (batch width in deg); 0 = off
     double mosaicity_deg = 0.1;
     float ice_half_width_q = 0.0f;
     // Automatic high-resolution cutoff for the written reflections + reported shells (post-merge; the
@@ -141,6 +144,12 @@ private:
     // written back for the per-image scaling table. Empty if there is no per-frame mosaicity.
     std::vector<float> mos_smooth;
 
+    // Radiation-damage monitor (measured by MeasureRadiationDamageB on the scaled fulls before any decay
+    // correction; report-only, copied into the result statistics by MergeAndStats). NaN / empty until set.
+    double rad_damage_delta_b = std::numeric_limits<double>::quiet_NaN();  // relative-B first->last (A^2)
+    std::vector<float> rad_damage_b_batch;                                 // per-batch relative-B curve (A^2)
+    double rad_damage_batch_deg = 0.0;                                     // rotation width per batch (deg)
+
     // Working per-group arrays (sized to the current group count; reused).
     std::vector<int32_t> group_h, group_k, group_l;
 
@@ -197,7 +206,38 @@ private:
     // physical total-dB floor). RefineAbsorption fits a smooth factor over the diffracted-beam direction in
     // the goniometer frame (path-length / absorption; negligible at hard X-rays, matters at low energy).
     void RefineDecay(int n_groups);
+    // Solve a smooth per-batch relative-B from the per-batch normal equations (num_c=sum w s^2 y,
+    // den_c=sum w s^4): data-fidelity + a second-difference (curvature) penalty, by Gauss-Seidel. Returns the
+    // un-anchored curve; the caller sets the gauge. Shared by the correction and the radiation-damage monitor.
+    std::vector<double> SolveCurvatureSmoothedB(const std::vector<double> &num,
+                                                const std::vector<double> &den) const;
+    // Fit a smoothed per-batch relative-B curve (A^2 per batch) on the fulls over the ASU-group subset
+    // {group&1==gparity} (gparity<0 = all): the weighted s^2 slope of ln(Iref/Iobs) per batch against a
+    // subset-global reference, smoothed and zero-mean-anchored. Drives the per-batch correction.
+    std::vector<double> FitRelativeBCurve(int n_groups, int n_batch, int frames_per_batch, int gparity) const;
+    // Radiation-damage MONITOR (report-only): measure the per-batch relative-B on the scaled fulls before
+    // any decay correction and store the first->last relative-B change + the per-batch curve on this object
+    // (copied into the result statistics by MergeAndStats, then printed / logged / written to the mmCIF).
+    void MeasureRadiationDamageB(int n_groups);
+    // Per-batch relative-B, applied after RefineDecay: the single decay slope removes the average
+    // radiation-damage falloff, but the relative scattering power drifts NON-monotonically across a run
+    // (absorption path, crystal slippage, dose bursts). Refine one relative Debye-Waller B per batch
+    // (FitRelativeBCurve), anchored to zero mean (the constant part is a global Wilson-B, degenerate with
+    // overall scale). Guarded by a physical peak-to-peak floor and cross-validated by ASU-GROUP parity (a
+    // per-batch parameter cannot be scored on a held-out FRAME the batch owns; splitting the equivalents
+    // tests whether a batch's B generalises to reflections it was not fit on). Opt-in (--relative-b).
+    void RefineRelativeB(int n_groups);
     void RefineAbsorption(int n_iter, int n_groups);
+    // Detector-plane modulation (flat-field): the same cross-validated surface fit as absorption, but the
+    // cell is the predicted detector position (px, py) instead of the goniometer-frame direction. Corrects
+    // detector-response / geometric systematics that vary with where a reflection lands; because it lives
+    // in the detector frame (not tied to the rotation) the same correction concept applies to stills.
+    void RefineModulation(int n_iter, int n_groups);
+    // Shared engine for the correction surfaces: given a per-full cell assignment (cell[i] in [0,ncell), or
+    // <0 to skip), fit a Tikhonov-regularised multiplicative factor per cell against the merged reference,
+    // cross-validate on even/odd frames, and fold it into corr only if the held-out equivalents improve.
+    void ApplyCellSurface(const std::vector<int32_t> &cell, int ncell, int n_iter, int n_groups,
+                          const char *name);
 
     // Sort `fulls` by peak frame and (re)build fulls_frame_start/count (the per-frame CSR the scale-fulls
     // step slices). Shared by the CPU Combine tail and the GPU combine path.

image_analysis/scale_merge/ScaleOnTheFly.h

@@ -9,6 +9,7 @@
 #include "../IntegrationOutcome.h"
 
 #include <map>
+#include <vector>
 
 
 struct ScaleOnTheFlyResult {

image_analysis/scale_merge/ScalingResult.cpp

@@ -40,6 +40,9 @@ void ScalingResult::SaveToFile(const std::string &filename) {
                               "Cannot open " + img_path + " for writing");
     }
 
+    // Header so the columns are self-describing (lines starting with '#' are comments for gnuplot/numpy).
+    img_file << "# image_number scale_G bfactor_Ang2 mosaicity_deg wedge_deg cc_to_merge cc_n\n";
+
     for (size_t i = 0; i < image_scale_g.size(); ++i) {
         img_file << i
                 << " " << image_scale_g[i]

image_analysis/scale_merge/SearchSpaceGroup.cpp

@@ -222,6 +222,14 @@ SearchSpaceGroupResult SearchSpaceGroup(
         }
     }
 
+    // Overlap guard (Stage A / correlation only): drop the extreme resolution-normalised-E tail, which
+    // on a two-lattice crystal is the one-sided overlap contamination that poisons the operator CC.
+    // See SearchSpaceGroupOptions::max_e_squared_for_cc. Absences (pass_absence) keep the full range.
+    if (opt.max_e_squared_for_cc > 0.0)
+        for (size_t i = 0; i < n; ++i)
+            if (pass_cc[i] && Esq[i] > opt.max_e_squared_for_cc)
+                pass_cc[i] = false;
+
     std::unordered_map<HKLKey, int, HKLKeyHash> key_to_index;
     key_to_index.reserve(n * 2);
     for (size_t i = 0; i < n; ++i)
@@ -374,7 +382,7 @@ SearchSpaceGroupResult SearchSpaceGroup(
     // has to grow to swallow it (mirroring the merge error model's b / ISa collapse). This isolates
     // the systematic part of the scatter, which the fixed-sigma chi^2 ratio cannot: a genuine but
     // imperfectly-scaled high-symmetry merge and a twin can share a chi^2 ratio (~2) yet differ
-    // sharply here (cubic Ins_I_3 b x1.04 vs twin Ins_H_2 b x1.6).
+    // sharply here (a genuine cubic step b x1.04 vs a merohedral twin b x1.6).
     auto merge_systematic_b = [&](const std::vector<gemmi::Op>& rotations) -> double {
         struct Acc { double sw = 0.0, swI = 0.0; int n = 0; };
         std::unordered_map<HKLKey, Acc, HKLKeyHash> grp;
@@ -447,24 +455,24 @@ SearchSpaceGroupResult SearchSpaceGroup(
     for (const auto& c : pg_cands) {
         // A genuine symmetry operator merges equivalent reflections, so it barely changes the reduced
         // chi^2 relative to the best subgroup - across the whole rotation-test battery every correct
-        // point group stays within ~1.7x, even on weak or badly-integrated data (F432 chi2_ref 8.3 ->
-        // 1.15; Thau P41212 -> 1.71). A twin law or pseudo-symmetry forces non-equivalent reflections
-        // together, so its ratio is markedly higher (Ins_H_2's twin 2-fold: R3 3.02 -> R32 6.07, ratio
-        // 2.01). max_merge_chi2_ratio sits between the two. (An earlier log10(chi2_ref) widening
+        // point group stays within ~1.7x, even on weak or badly-integrated data (a cubic F432 chi2_ref
+        // 8.3 -> 1.15; a tetragonal P41212 -> 1.71). A twin law or pseudo-symmetry forces non-equivalent
+        // reflections together, so its ratio is markedly higher (a merohedral twin 2-fold: R3 3.02 ->
+        // R32 6.07, ratio 2.01). max_merge_chi2_ratio sits between the two. (An earlier log10(chi2_ref) widening
         // compensated for an under-calibrated error model that inflated real-symmetry ratios with data
         // weakness; the variance-floor fix removed that inflation, and the widening now only let the
         // twin through, so it is gone.)
         bool consistent = c.pg->rotations.empty() || !std::isfinite(c.chi2) ||
                           !std::isfinite(chi2_ref) || c.chi2 <= chi2_ref * opt.max_merge_chi2_ratio;
 
-        // Rescue a genuine high-symmetry merge whose chi^2 lands just past the ratio bound because its
-        // data are imperfectly scaled (see max_merge_chi2_rescue). The systematic-error test tells it
-        // apart from a twin: promote only if the extra intensity-proportional error b, relative to the
-        // largest confirmed subgroup (by rotation-set inclusion), stayed within max_systematic_b_ratio -
-        // a genuine step barely moves it, a twin's balloons.
-        if (!consistent && std::isfinite(c.chi2) && std::isfinite(chi2_ref)
-            && c.chi2 <= chi2_ref * opt.max_merge_chi2_rescue) {
-            double parent_b = -1.0;
+        // Systematic-error test vs the largest confirmed subgroup (by rotation-set inclusion): merging
+        // under a genuine operator gains multiplicity without intensity-proportional disagreement, so the
+        // merge error model's b barely moves; a merohedral twin forces non-equivalent reflections together
+        // and b balloons. It both RESCUES a genuine step whose chi^2 drifts just past the ratio bound
+        // (imperfectly scaled data) and VETOES a twin whose chi^2 now looks self-consistent but whose b
+        // balloons - the chi^2 ratio alone no longer separates them.
+        double parent_b = -1.0;
+        if (!c.pg->rotations.empty()) {
             int parent_order = 0;
             for (const auto& s : pg_cands)
                 if (s.order < c.order && s.order > parent_order
@@ -473,9 +481,29 @@ SearchSpaceGroupResult SearchSpaceGroup(
                     parent_order = s.order;
                     parent_b = s.b_extra;
                 }
-            if (parent_b > 1e-4 && c.b_extra <= parent_b * opt.max_systematic_b_ratio)
-                consistent = true;
         }
+        // The chi^2 ratio is only trustworthy when the error model is calibrated. When even the best
+        // subgroup's reduced chi^2 (chi2_ref) is far above 1 - weak, low-resolution data whose merged
+        // sigmas are badly under-estimated - the ratio grows with point-group order for genuine high
+        // symmetry too and wrongly rejects it (a true weak F432 reaches ratio ~14). The systematic-b test
+        // re-fits its own error, so it stays valid under a broken sigma model: a genuine step's b barely
+        // moves (b-ratio ~1) while a twin's balloons. So once chi2_ref shows the error model is unreliable,
+        // a promotion is rescued on the b-test alone (subject to the balloon veto below); otherwise the
+        // rescue is confined to the narrow chi^2 band just past the ratio bound.
+        const bool miscalibrated = std::isfinite(chi2_ref) && chi2_ref > opt.chi2_ref_reliable;
+        if (!consistent && parent_b > 1e-4 && c.b_extra <= parent_b * opt.max_systematic_b_ratio
+            && (miscalibrated || (std::isfinite(c.chi2) && std::isfinite(chi2_ref)
+                                  && c.chi2 <= chi2_ref * opt.max_merge_chi2_rescue)))
+            consistent = true;
+        // Veto a chi^2-passing promotion whose b clearly ballooned (above the largest genuine step, below a
+        // twin); a genuine but imperfectly-scaled high-symmetry merge stays under the bound and is untouched.
+        // The parent b is floored (min_systematic_b_for_veto) so a near-zero parent on excellent data cannot
+        // fabricate a huge ratio out of a still-tiny absolute b (a genuine 422 at b=0.05 over a 222 parent at
+        // b=0.008 is not a twin - a real twin drives b to ~0.19 regardless).
+        if (consistent && parent_b > 1e-4
+            && c.b_extra > std::max(parent_b, opt.min_systematic_b_for_veto) * opt.max_systematic_b_veto)
+            consistent = false;
+
         if (!consistent)
             continue;
         if (c.order > best_pg_order || (c.order == best_pg_order && c.min_class_cc > best_pg_min_cc)) {
@@ -517,7 +545,9 @@ SearchSpaceGroupResult SearchSpaceGroup(
         // a large, correct centering-absent set hide a few strong screw violations and over-claim
         // screw axes (e.g. I4_132 on I432 data).
         int centering_absent = 0, centering_violations = 0;
+        double centering_absent_sum = 0;
         int screw_absent = 0, screw_violations = 0;
+        int present_strong = 0;
 
         for (size_t i = 0; i < n; ++i) {
             if (!pass_absence[i])
@@ -534,6 +564,7 @@ SearchSpaceGroupResult SearchSpaceGroup(
                 s.absent_observed += 1;
                 absent_sum += IoverSigma[i];
                 centering_absent += 1;
+                centering_absent_sum += IoverSigma[i];
                 if (present) { s.absent_violations += 1; centering_violations += 1; }
             } else if (gops.is_systematically_absent(hkl)) {
                 s.absent_observed += 1;
@@ -543,6 +574,7 @@ SearchSpaceGroupResult SearchSpaceGroup(
             } else {
                 present_n += 1;
                 present_sum += IoverSigma[i];
+                if (present) present_strong += 1;
             }
         }
 
@@ -551,8 +583,37 @@ SearchSpaceGroupResult SearchSpaceGroup(
         if (present_n > 0)
             s.present_mean_i_over_sigma = present_sum / present_n;
 
-        const bool centering_ok = centering_absent == 0 ||
-            centering_violations <= opt.max_absent_violation_fraction * centering_absent;
+        // Centering is judged by class STRENGTH, not a per-reflection violation count. A real centering
+        // cancels structure factors, so its absent class is systematically weak - its mean signed
+        // I/sigma sits well below the present class - regardless of noise or obverse/reverse twinning;
+        // a false centering leaves the "absent" class as strong as the present one (mean ratio ~1). The
+        // count-of-strong-violations gate is brittle on noisy/twinned data, where enough genuinely-absent
+        // reflections randomly clear I/sigma>3 to trip the 10% bound though the class is 3-4x weaker (a
+        // true R3 at 13.5% violations, absent 1.7 vs present 6.0). The mean is well-determined here
+        // because a centering-absent class holds a third-to-half of all reflections. Screws keep the
+        // count gate: their predicted-absent class is a handful of axial reflections, too few to average.
+        const double present_mean = present_n > 0 ? present_sum / present_n : 0.0;
+        const double centering_absent_mean =
+            centering_absent > 0 ? centering_absent_sum / centering_absent : 0.0;
+        // The centering-absent class proves itself weak in either of two floor-independent ways; a
+        // FALSE centering (absent as strong as present) fails both:
+        //  (1) mean signed I/sigma well below the present class, OR
+        //  (2) its strong-reflection RATE well below the present class's own strong rate.
+        // (2) is needed because weak / low-energy data carry a positive intensity floor (background /
+        // profile leakage) that lifts <I/s>abs to ~1.5-2.3 even for genuinely extinct reflections; when
+        // the present class is itself weak (small present_mean) that additive floor inflates the mean
+        // ratio past the bound and hides a real centering - e.g. an I-centred cubic crystal at low
+        // energy, whose true I-centering sat at ratio ~0.57. Normalising the violation count by the
+        // present class's own strong rate cancels the shared floor and stays reliable on weak data
+        // (both rates shrink together).
+        const double present_strong_rate =
+            present_n > 0 ? static_cast<double>(present_strong) / present_n : 0.0;
+        const double centering_violation_rate =
+            centering_absent > 0 ? static_cast<double>(centering_violations) / centering_absent : 0.0;
+        const bool centering_ok = centering_absent == 0
+            || (present_n > 0 && centering_absent_mean <= opt.max_absent_present_ratio * present_mean)
+            || (present_strong_rate > 0.0
+                && centering_violation_rate <= opt.max_absent_present_ratio * present_strong_rate);
         const bool screw_ok = screw_absent == 0 ||
             screw_violations <= opt.max_absent_violation_fraction * screw_absent;
         s.consistent = centering_ok && screw_ok;
@@ -561,8 +622,17 @@ SearchSpaceGroupResult SearchSpaceGroup(
 
     // A candidate is eligible when its absences are confirmed and there are enough of them to
     // trust (the symmorphic group, with no absences, is always eligible as the fallback). Rank
-    // eligible candidates by how many absences they explain - the screw/centering content that is
-    // both real and maximal wins, instead of defaulting to the symmorphic group.
+    // eligible candidates by how many absences they GENUINELY explain - absent_observed minus the
+    // violations, not the gross count. A false super-centering over-claims: F222 on a C222 crystal
+    // predicts every C absence (all genuinely weak) PLUS a block of C-present reflections it wrongly
+    // calls absent, so its gross count is larger yet its net count only equals C222's. Its diluted
+    // absent class (many true zeros + a strong block) also slips under the strength/rate gate, so the
+    // gate cannot veto it alone; netting the violations puts the two level, and the fewer-violations
+    // and lower-number tie-breaks then keep the honest, less-centred C222. The ranking is symmetric:
+    // on a genuine F222 crystal F explains strictly more weak absences and still wins.
+    auto net_absent = [](const SpaceGroupCandidateScore& s) {
+        return s.absent_observed - s.absent_violations;
+    };
     auto eligible = [&](const SpaceGroupCandidateScore& s) {
         return s.consistent && (s.absent_observed == 0 || s.absent_observed >= opt.min_absent_observed);
     };
@@ -570,16 +640,24 @@ SearchSpaceGroupResult SearchSpaceGroup(
               [&](const SpaceGroupCandidateScore& a, const SpaceGroupCandidateScore& b) {
                   if (eligible(a) != eligible(b))
                       return eligible(a);
-                  if (a.absent_observed != b.absent_observed)
-                      return a.absent_observed > b.absent_observed;
-                  // Tie (e.g. I23 vs I2_13, indistinguishable by absences): lower space-group number.
+                  if (net_absent(a) != net_absent(b))
+                      return net_absent(a) > net_absent(b);
+                  if (a.absent_violations != b.absent_violations)
+                      return a.absent_violations < b.absent_violations;   // prefer the honest, less over-claiming group
+                  // Genuinely indistinguishable (e.g. I23 vs I2_13, or an enantiomorphic pair): lower
+                  // space-group number is the representative.
                   return a.space_group.number < b.space_group.number;
               });
 
     if (!result.candidates.empty() && eligible(result.candidates.front())) {
-        const int best_absent = result.candidates.front().absent_observed;
+        // Alternatives are only the candidates with the SAME absence signature - identical absent AND
+        // violation counts - as the winner: the enantiomorphic / origin-ambiguous partners the data
+        // truly cannot separate. A super-centering that nets the same count but over-claims differs in
+        // its violation count and is therefore not reported as an equal alternative.
+        const int sel_absent = result.candidates.front().absent_observed;
+        const int sel_violations = result.candidates.front().absent_violations;
         for (auto& s : result.candidates) {
-            if (!eligible(s) || s.absent_observed != best_absent)
+            if (!eligible(s) || s.absent_observed != sel_absent || s.absent_violations != sel_violations)
                 continue;
             s.selected = true;
             if (!result.best_space_group.has_value())

image_analysis/scale_merge/SearchSpaceGroup.h

@@ -65,6 +65,15 @@ struct SearchSpaceGroupOptions {
     // keep weak reflections - that is where the screw-axis signal lives).
     double min_i_over_sigma = 0.0;
 
+    // Drop reflections STRONGER than this resolution-normalised E^2 = I/<I>(shell) from the correlation
+    // stage only. A second lattice deposits intensity on one reciprocal position but not its symmetry
+    // mate, so an overlap-contaminated reflection is a one-sided E^2 outlier that poisons an operator's
+    // I(h)/I(Rh) correlation (it flipped a pseudo-merohedral P2_1 case to P1 under -A: excluding the E^2>9 tail, ~0.4%
+    // of reflections, lifted the 2-fold CC 0.33->0.53 back over the gate). Clean Wilson-distributed data
+    // almost never reaches E^2=9 (P(E^2>9) ~ 0.01-0.3%), so this removes essentially nothing there and
+    // only trims the overlap tail. 0 disables. Absences are unaffected (they need the weak tail).
+    double max_e_squared_for_cc = 9.0;
+
     // --- Stage A: point group ---
     // A rotation is accepted as a real symmetry when its I(h)/I(Rh) correlation reaches this over
     // at least min_pairs_per_operator independent pairs.
@@ -78,25 +87,52 @@ struct SearchSpaceGroupOptions {
     // candidate. A false operator forces non-equivalent reflections together so they disagree by many
     // sigma and chi^2 blows up; a real one leaves it ~flat even when the operator CC is only moderate.
     // Calibrated on the rotation-test battery: every correct point group stays within ~1.7x the best
-    // subgroup even on weak / badly-integrated data (worst real case Thau P41212 at 1.71), while a twin
-    // law or pseudo-symmetry lands clearly higher (Ins_H_2's R3->R32 twin 2-fold at 2.01). 1.85 sits
+    // subgroup even on weak / badly-integrated data (worst real case a P41212 at 1.71), while a twin
+    // law or pseudo-symmetry lands clearly higher (a merohedral R3->R32 twin 2-fold at 2.01). 1.85 sits
     // between the two, so a partial merohedral twin is kept in its true lower symmetry (R3), not
     // over-promoted to the holohedral R32.
     double max_merge_chi2_ratio = 1.85;
 
     // A genuine high-symmetry merge can drift just past max_merge_chi2_ratio when its data are only
     // imperfectly scaled: each real symmetry step then adds a little systematic scatter, so a weakly-
-    // scaled cubic like Ins_I_3 lands at ratio ~2.0 - right where a merohedral twin (Ins_H_2's R3->R32
-    // at ~1.95) also lands, so the chi^2 ratio alone cannot separate them. A candidate whose chi^2 is
+    // scaled cubic case lands at ratio ~2.0 - right where a merohedral twin (an R3->R32 case at ~1.95)
+    // also lands, so the chi^2 ratio alone cannot separate them. A candidate whose chi^2 is
     // only this far past the best subgroup is therefore rescued if the SYSTEMATIC part of the extra
     // scatter stayed small (max_systematic_b_ratio): merging under a genuine operator gains multiplicity
     // without intensity-proportional disagreement, so the merge error model's b barely moves, whereas a
-    // twin forces non-equivalent reflections together and b balloons (Ins_I_3 b x1.6 vs Ins_H_2 x2.2,
-    // measured against the largest confirmed subgroup). Both bounds sit in the gap; the rescue only ever
+    // twin forces non-equivalent reflections together and b balloons (a genuine cubic step b x1.6 vs a
+    // merohedral twin x2.2, measured against the largest confirmed subgroup). Both bounds sit in the gap; the rescue only ever
     // promotes, and only in this narrow chi^2 band.
     double max_merge_chi2_rescue = 2.30;
     double max_systematic_b_ratio = 1.90;
 
+    // Veto bound. The systematic-b test above is otherwise only a rescue - it can promote a chi^2-borderline
+    // genuine step but never demote a chi^2-passing one. A merohedral twin whose within-orbit scatter looks
+    // self-consistent (chi^2 ratio below max_merge_chi2_ratio) therefore slips through on chi^2 alone even
+    // though merging its non-equivalent reflections balloons b. So a chi^2-passing high-symmetry promotion
+    // whose b, relative to the largest confirmed subgroup, exceeds this bound is vetoed and kept in its true
+    // lower symmetry. Calibrated on the rotation-test battery, where the largest genuine step (a P422
+    // tetragonal) sits at b-ratio ~1.8 and a merohedral R3->R32 twin at ~2.6 - an empty gap - so 2.0 keeps
+    // every genuine high-symmetry merge (they are never demoted) while catching the twin. The veto only ever
+    // keeps a clearly-ballooned promotion down, never promotes.
+    double max_systematic_b_veto = 2.0;
+
+    // Floor on the parent's systematic-b when forming the veto ratio. On excellent data a genuine merge's
+    // b is near zero (ISa well above 20), so even a small, harmless absolute increase gives a huge b-ratio
+    // and the veto would wrongly reject real high symmetry (a true tetragonal 422 whose parent 222 sits at
+    // b=0.008 and 422 at b=0.049 - both tiny - reads as a 6x balloon). A real twin, by contrast, drives b
+    // to a large ABSOLUTE value (~0.19). Flooring the parent b here makes the ratio meaningful only once b
+    // is a non-negligible fraction of I; below the floor the increase is treated as noise, not a twin.
+    double min_systematic_b_for_veto = 0.05;
+
+    // Above this reduced chi^2 for the best subgroup (chi2_ref), the merged error model is treated as
+    // badly miscalibrated (weak, low-resolution data whose sigmas are far too small): the fixed-sigma
+    // chi^2 ratio then grows with point-group order for genuine high symmetry too and can no longer
+    // arbitrate, so a promotion is confirmed on the systematic-b test alone (which re-fits its own error
+    // and stays valid). A well-calibrated merge sits near 1; 3.0 (sigmas ~1.7x too small) marks the point
+    // where the ratio stops being trustworthy. The balloon veto still guards against a twin.
+    double chi2_ref_reliable = 3.0;
+
     // --- Stage B: space group (screw axes / centering) ---
     bool determine_space_group = true;     // false: stop at the symmorphic representative
 
@@ -112,10 +148,19 @@ struct SearchSpaceGroupOptions {
     // (e.g. the monoclinic 2_1: 0k0-odd at ~1% of 0k0-even). 0 disables the gate (I/sigma only).
     double present_e_squared = 0.3;
 
-    // A candidate's absence conditions are accepted when at most this fraction of the reflections
-    // it predicts absent are in fact strongly present.
+    // A candidate's SCREW/glide absence conditions are accepted when at most this fraction of the
+    // reflections it predicts absent are in fact strongly present.
     double max_absent_violation_fraction = 0.10;
 
+    // A CENTERING is accepted when its systematically-absent class is this much weaker than the present
+    // class - mean signed I/sigma of the centering-absent reflections <= this fraction of the present
+    // mean. A real centering cancels structure factors so its absent class sits near zero (ratio ~0-0.3
+    // across the test battery) even on noisy or obverse/reverse-twinned data; a false centering leaves it
+    // as strong as the present class (ratio ~1.0). 0.5 separates the two with wide margin. This strength
+    // test replaces a per-reflection violation-count gate for centering, which was brittle when noise
+    // pushed genuinely-absent reflections over I/sigma>3 (a true R3 was lost at 13.5% violations).
+    double max_absent_present_ratio = 0.5;
+
     // Need at least this many observed reflections in the predicted-absent class before a
     // screw/centering is claimed (guards against deciding from a handful of reflections).
     int min_absent_observed = 8;

image_analysis/scale_merge/TwinningAnalysis.cpp

@@ -191,7 +191,7 @@ TwinningAnalysisResult AnalyzeTwinning(const std::vector<MergedReflection>& merg
         result.estimated_twin_fraction = (1.0 - std::sqrt(std::max(0.0, 2.0 * m - 3.0))) / 2.0;
     }
     // Either indicator dropping clearly below its untwinned value is suspicious - but only where a twin
-    // law can actually exist. In a holohedral Laue class (e.g. lysozyme's 422) no merohedral twinning is
+    // law can actually exist. In a holohedral Laue class (e.g. 422) no merohedral twinning is
     // possible, so a low <|L|> is a statistical artefact (correlated near-neighbours) rather than a
     // twin, and must not be flagged.
     result.merohedral_twinning_possible = MerohedralTwinningPossible(space_group);