v1.0.0-rc.160 (#70)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/geom_refinement/XtalOptimizer.cpp

@@ -5,201 +5,24 @@
 #include <Eigen/Dense>
 
 #include "XtalOptimizer.h"
+#include "XtalResidual.h"
 #include "ceres/ceres.h"
 #include "ceres/rotation.h"
 #include "LatticeReduction.h"
 
-struct XtalResidual {
-    XtalResidual(double x, double y,
-                 double lambda,
-                 double pixel_size,
-                 double rot3,
-                 double angle_rad,
-                 double exp_h, double exp_k,
-                 double exp_l,
-                 gemmi::CrystalSystem symmetry)
-        : obs_x(x), obs_y(y),
-          inv_lambda(1.0/lambda),
-          pixel_size(pixel_size),
-          rot3(rot3),
-          exp_h(exp_h),
-          exp_k(exp_k),
-          exp_l(exp_l),
-          angle_rad(angle_rad),
-          symmetry(symmetry) {
-        if (std::fabs(lambda) < 1e-6)
-            throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
-                "Lambda cannot be close to zero");
-    }
-
+// Soft header prior on ONE beam-centre component (the spindle-parallel, gauge-weak one). Residual = w*(b - b0);
+// the caller sets w so the prior behaves like a sigma-pixel restraint that competes with the (unit-weight)
+// positional residuals - strong enough to pin the gauge direction, negligible in the well-constrained one.
+struct BeamComponentPrior {
+    BeamComponentPrior(int component, double b0, double weight)
+        : component(component), b0(b0), weight(weight) {}
     template<typename T>
-    bool operator()(const T *const beam,
-                    const T *const distance_mm,
-                    const T *const detector_rot,
-                    const T *const rotation_axis,
-                    const T *const p0,
-                    const T *const p1,
-                    const T *const p2,
-                    T *residual) const {
-        // PyFAI convention: poni_rot = Rz(-rot3) * Rx(-rot2) * Ry(+rot1).
-        // detector_rot[0] = rot1, detector_rot[1] = rot2 are refined; rot3 is fixed
-        // (e.g. from a PONI import) and baked in here as a constant so that a non-zero
-        // rot3 is not silently dropped during refinement.
-        const T rot1 = detector_rot[0];
-        const T rot2 = detector_rot[1];
-
-        // Ry(+rot1): rotation around Y-axis
-        const T c1 = ceres::cos(rot1);
-        const T s1 = ceres::sin(rot1);
-
-        // Rx(-rot2): rotation around X-axis with inverted sign (PyFAI left-handed)
-        const T c2 = ceres::cos(rot2);
-        const T s2 = ceres::sin(rot2);
-
-        // Rz(-rot3): rotation around Z (beam); constant, identity when rot3 == 0
-        const T c3 = T(cos(rot3));
-        const T s3 = T(sin(rot3));
-
-        // Detector coordinates in mm
-        const T det_x = (T(obs_x) - beam[0]) * T(pixel_size);
-        const T det_y = (T(obs_y) - beam[1]) * T(pixel_size);
-        const T det_z = T(distance_mm[0]);
-
-        // Apply Ry(rot1) first: rotate around Y
-        const T t1_x = c1 * det_x + s1 * det_z;
-        const T t1_y = det_y;
-        const T t1_z = -s1 * det_x + c1 * det_z;
-
-        // Then apply Rx(-rot2): rotate around X
-        const T t2_x = t1_x;
-        const T t2_y = c2 * t1_y + s2 * t1_z;
-        const T t2_z = -s2 * t1_y + c2 * t1_z;
-
-        // Then apply Rz(-rot3): rotate around Z (beam)
-        const T x = c3 * t2_x + s3 * t2_y;
-        const T y = -s3 * t2_x + c3 * t2_y;
-        const T z = t2_z;
-
-        // convert to recip space
-        const T lab_norm = ceres::sqrt(x * x + y * y + z * z);
-        const T inv_norm = T(1) / lab_norm;
-
-        T recip_raw[3];
-        recip_raw[0] = x * inv_norm * T(inv_lambda);
-        recip_raw[1] = y * inv_norm * T(inv_lambda);
-        recip_raw[2] = (z * inv_norm - T(1.0)) * T(inv_lambda);
-
-        // Apply goniometer "back-to-start" rotation:
-        // brings observed reciprocal from image orientation into reference crystal frame
-        const T aa_back[3] = {
-            T(angle_rad) * rotation_axis[0],
-            T(angle_rad) * rotation_axis[1],
-            T(angle_rad) * rotation_axis[2]
-        };
-
-        T recip_obs[3];
-        ceres::AngleAxisRotatePoint(aa_back, recip_raw, recip_obs);
-
-        const Eigen::Matrix<T, 3, 1> e_obs_recip(recip_obs[0], recip_obs[1], recip_obs[2]);
-
-        // Build unit cell lengths and B (convention: columns are a, b, c prior to global rotation)
-        Eigen::Matrix<T, 3, 1> e_uc_len = Eigen::Matrix<T, 3, 1>::Zero();
-        Eigen::Matrix<T, 3, 3> B = Eigen::Matrix<T, 3, 3>::Identity();
-
-        if (symmetry == gemmi::CrystalSystem::Hexagonal) {
-            e_uc_len << p1[0], p1[0], p1[2];
-            B(0, 1) = T(-0.5); // cos(120)
-            B(1, 1) = T(sqrt(3.0) / 2.0); // sin(120)
-        } else if (symmetry == gemmi::CrystalSystem::Orthorhombic) {
-            e_uc_len << p1[0], p1[1], p1[2];
-        } else if (symmetry == gemmi::CrystalSystem::Tetragonal) {
-            e_uc_len << p1[0], p1[0], p1[2];
-        } else if (symmetry == gemmi::CrystalSystem::Cubic) {
-            e_uc_len << p1[0], p1[0], p1[0];
-        } else if (symmetry == gemmi::CrystalSystem::Monoclinic) {
-            // Unique axis b: alpha = gamma = 90°, beta free (angle between a and c)
-            e_uc_len << p1[0], p1[1], p1[2];
-            B(0, 2) = ceres::cos(p2[0]);
-            B(2, 2) = ceres::sin(p2[0]);
-        } else {
-            // Triclinic: p1 = (a,b,c), p2 = (alpha, beta, gamma) in radians
-            const T ca = ceres::cos(p2[0]);
-            const T cb = ceres::cos(p2[1]);
-            const T cg = ceres::cos(p2[2]);
-            const T sg = ceres::sin(p2[2]);
-
-            e_uc_len << p1[0], p1[1], p1[2];
-
-            B(0, 0) = T(1);
-            B(1, 0) = T(0);
-            B(2, 0) = T(0);
-            B(0, 1) = cg;
-            B(1, 1) = sg;
-            B(2, 1) = T(0);
-
-            // c vector components:
-            const T cx = cb;
-            const T cy = (ca - cb * cg) / sg;
-            const T v = T(1) - cx * cx - cy * cy;
-            const T cz = (v >= T(0)) ? ceres::sqrt(v) : T(0);
-
-            B(0, 2) = cx;
-            B(1, 2) = cy;
-            B(2, 2) = cz;
-        }
-
-        // Build unrotated direct lattice columns: (B * D), then rotate them by p0.
-        // This avoids AngleAxisToRotationMatrix + matrix multiplications.
-        const T L0 = e_uc_len[0];
-        const T L1 = e_uc_len[1];
-        const T L2 = e_uc_len[2];
-
-        T col0_unrot[3] = {B(0, 0) * L0, B(1, 0) * L0, B(2, 0) * L0};
-        T col1_unrot[3] = {B(0, 1) * L1, B(1, 1) * L1, B(2, 1) * L1};
-        T col2_unrot[3] = {B(0, 2) * L2, B(1, 2) * L2, B(2, 2) * L2};
-
-        T col0_rot[3], col1_rot[3], col2_rot[3];
-        ceres::AngleAxisRotatePoint(p0, col0_unrot, col0_rot);
-        ceres::AngleAxisRotatePoint(p0, col1_unrot, col1_rot);
-        ceres::AngleAxisRotatePoint(p0, col2_unrot, col2_rot);
-
-        const Eigen::Matrix<T, 3, 1> A(col0_rot[0], col0_rot[1], col0_rot[2]);
-        const Eigen::Matrix<T, 3, 1> Bv(col1_rot[0], col1_rot[1], col1_rot[2]);
-        const Eigen::Matrix<T, 3, 1> C(col2_rot[0], col2_rot[1], col2_rot[2]);
-
-        const Eigen::Matrix<T, 3, 1> BxC = Bv.cross(C);
-        const Eigen::Matrix<T, 3, 1> CxA = C.cross(A);
-        const Eigen::Matrix<T, 3, 1> AxB = A.cross(Bv);
-
-        const T V = A.dot(BxC);
-        const T invV = T(1) / V;
-
-        const Eigen::Matrix<T, 3, 1> Astar = BxC * invV;
-        const Eigen::Matrix<T, 3, 1> Bstar = CxA * invV;
-        const Eigen::Matrix<T, 3, 1> Cstar = AxB * invV;
-
-        const T h = T(exp_h);
-        const T k = T(exp_k);
-        const T l = T(exp_l);
-
-        const Eigen::Matrix<T, 3, 1> e_pred_recip = Astar * h + Bstar * k + Cstar * l;
-
-        residual[0] = e_obs_recip[0] - e_pred_recip[0];
-        residual[1] = e_obs_recip[1] - e_pred_recip[1];
-        residual[2] = e_obs_recip[2] - e_pred_recip[2];
-
+    bool operator()(const T *const beam, T *residual) const {
+        residual[0] = T(weight) * (beam[component] - T(b0));
         return true;
     }
-
-    const double obs_x, obs_y;
-    const double inv_lambda;
-    const double pixel_size;
-    const double rot3;
-    const double exp_h;
-    const double exp_k;
-    const double exp_l;
-    const double angle_rad;
-    gemmi::CrystalSystem symmetry;
+    int component;
+    double b0, weight;
 };
 
 struct XtalResidualRotationOnlyPrecomp {
@@ -391,6 +214,29 @@ bool XtalOptimizerInternal(XtalOptimizerData &data,
 
         if (!data.refine_beam_center)
             problem.SetParameterBlockConstant(beam);
+        else if (data.axis) {
+            // Gauge handling (single-axis rotation): rotating the whole experiment about the spindle leaves every
+            // spot position unchanged, so the beam-centre component PARALLEL to the spindle is a null/gauge-weak
+            // direction. Refining it freely lets it wander (~+3 px) and absorb centroid systematics into a wrong
+            // beam that the co-refined orientation keeps position-consistent. Rather than freeze it (the beam
+            // does drift - it is only LaB6-monitored to ~a few px), RESTRAIN it toward the header with a soft
+            // prior: the gauge direction has ~zero data sensitivity so the prior pins it near the header, while a
+            // real, well-supported drift can still overcome it. The spindle is along a detector axis in standard
+            // geometry, so restrain the dominant of X / Y.
+            const Coord spindle = data.axis->GetAxis();
+            const int parallel = (std::fabs(spindle.x) >= std::fabs(spindle.y)) ? 0 : 1;
+            // Weight so the prior is a sigma_px-pixel restraint that competes with the positional residuals.
+            // k = d|recip|/d(beam_px) ~ pixel/(distance*lambda) [A^-1/px]; scaling by sqrt(#residuals) makes the
+            // prior's curvature ~ (1/9) of the well-constrained-data curvature at sigma_px=3, i.e. data wins the
+            // perpendicular direction, the prior wins the gauge one.
+            constexpr double sigma_px = 3.0;
+            const double k = data.geom.GetPixelSize_mm() / (distance_mm * data.geom.GetWavelength_A());
+            const double w = k * std::sqrt(static_cast<double>(problem.NumResidualBlocks())) / sigma_px;
+            problem.AddResidualBlock(
+                new ceres::AutoDiffCostFunction<BeamComponentPrior, 1, 2>(
+                    new BeamComponentPrior(parallel, beam[parallel], w)),
+                nullptr, beam);
+        }
 
         if (!data.refine_detector_angles) {
             problem.SetParameterBlockConstant(detector_rot);