v1.0.0-rc.166 (#76)

* `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands.
* `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion.
* Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants.
* `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing.
* A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed.
* `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing.
* Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences.
* The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to.
* The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after.
* The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution.
* `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have.
* Twinning is no longer reported when the L-test contradicts it.
* The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's.
* `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots.
* The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area.

Reviewed-on: #76
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/geom_refinement/XtalOptimizer.cpp

@@ -14,16 +14,18 @@
 // Soft header prior on ONE beam-centre component (the spindle-parallel, gauge-weak one). Residual = w*(b - b0);
 // the caller sets w so the prior behaves like a sigma-pixel restraint that competes with the (unit-weight)
 // positional residuals - strong enough to pin the gauge direction, negligible in the well-constrained one.
-struct BeamComponentPrior {
-    BeamComponentPrior(int component, double b0, double weight)
-        : component(component), b0(b0), weight(weight) {}
+// Soft restraint on one direction of a two-component block: g.(p - p0), weighted. Used for the beam
+// centre and for the detector tilt, which are the same gauge seen twice (see the gauge block below),
+// so they take the same direction g and cannot disagree about it.
+struct GaugeDirectionPrior {
+    GaugeDirectionPrior(double gx, double gy, double p0, double weight)
+        : gx(gx), gy(gy), p0(p0), weight(weight) {}
     template<typename T>
-    bool operator()(const T *const beam, T *residual) const {
-        residual[0] = T(weight) * (beam[component] - T(b0));
+    bool operator()(const T *const p, T *residual) const {
+        residual[0] = T(weight) * (T(gx) * p[0] + T(gy) * p[1] - T(p0));
         return true;
     }
-    int component;
-    double b0, weight;
+    double gx, gy, p0, weight;
 };
 
 struct XtalResidualRotationOnlyPrecomp {
@@ -137,6 +139,16 @@ bool XtalOptimizerInternal(XtalOptimizerData &data,
                            const float tolerance,
                            const int num_threads) {
     try {
+        // A coplanar basis has no reciprocal cell: 1/V is infinite, every predicted reciprocal vector
+        // comes out NaN, and Ceres fails on the very first evaluation - after dumping the offending
+        // block to stderr. There is nothing for the refinement to recover here, so refuse the lattice
+        // before the problem is built rather than let the solver discover it. The check has to be on
+        // the vectors: this close to flat, float cell angles no longer carry even the SIGN of the
+        // metric determinant, and the triclinic branch of XtalResidual then clamps c into the a-b
+        // plane and divides by the zero volume that makes.
+        if (data.latt.VolumeFraction() < MIN_BASIS_VOLUME_FRACTION)
+            return false;
+
         Coord vec0 = data.latt.Vec0();
         Coord vec1 = data.latt.Vec1();
         Coord vec2 = data.latt.Vec2();
@@ -272,7 +284,8 @@ bool XtalOptimizerInternal(XtalOptimizerData &data,
                                             cos_rot3, sin_rot3,
                                             angle_rad,
                                             h, k, l,
-                                            data.crystal_system);
+                                            data.crystal_system,
+                                            data.geom.GetOrientation());
 
                 // Ceres has no per-residual weight; ScaledLoss(nullptr, a) multiplies the squared
                 // residual by the constant a, i.e. it applies a weight of sqrt(a) to the residual.
@@ -307,6 +320,71 @@ bool XtalOptimizerInternal(XtalOptimizerData &data,
         if (problem.NumResidualBlocks() < data.min_spots)
             return false;
 
+        // The gauge direction of a single-axis rotation experiment - parallel to the spindle - written
+        // once, for both of the parameter pairs it applies to. The two need it in DIFFERENT frames and
+        // that is the whole difficulty:
+        //
+        //   beam[0]/beam[1] are PIXEL columns and rows. The pixel axes reach the laboratory through
+        //   det_matrix = PoniRotMatrix * DetectorOrientation::Matrix(), so on a quarter turn of 1 or 3
+        //   the pixel X axis IS the laboratory Y axis. Comparing the goniometer vector's laboratory
+        //   components against a beam index is therefore only right when that orientation is the
+        //   identity; elsewhere it pins the determined component and frees the gauge one. Project the
+        //   spindle onto the pixel axes' own laboratory images instead - exact for any orientation,
+        //   any tilt and a spindle at any angle, and equal to picking the dominant component when the
+        //   orientation is the identity and the spindle lies along a detector axis.
+        //
+        //   detector_rot[0]/[1] are rotations about the LABORATORY y and x axes (see PoniRotMatrix),
+        //   applied outside that orientation matrix, and they move the direct beam along laboratory x
+        //   and y respectively by D/pixel per radian. So the tilt's gauge combination is the spindle's
+        //   own laboratory x and y components, with no orientation in it.
+        //
+        // Same spindle, same physical direction, each in the frame its parameters live in.
+        double gauge_beam_x = 0.0, gauge_beam_y = 0.0;
+        double gauge_rot_x = 0.0, gauge_rot_y = 0.0;
+        if (data.axis) {
+            const Coord spindle = data.axis->GetAxis().Normalize();
+            const Coord fast = data.geom.GetFastAxis();
+            const Coord slow = data.geom.GetSlowAxis();
+            const double bx = spindle * fast, by = spindle * slow;
+            const double bn = std::hypot(bx, by);
+            if (bn > 0.0) {
+                gauge_beam_x = bx / bn;
+                gauge_beam_y = by / bn;
+            }
+            const double rn = std::hypot(spindle.x, spindle.y);
+            if (rn > 0.0) {
+                gauge_rot_x = spindle.x / rn;
+                gauge_rot_y = spindle.y / rn;
+            }
+        }
+        // Weight so a gauge prior is a sigma_px-pixel restraint that competes with the positional
+        // residuals. k = d|recip|/d(beam_px) ~ pixel/(distance*lambda) [A^-1/px]; scaling by
+        // sqrt(#residuals) makes the prior's curvature ~ (1/9) of the well-constrained-data curvature
+        // at sigma_px=3, i.e. data wins the perpendicular direction, the prior wins the gauge one.
+        // Note what that scaling means: the prior's curvature grows with the number of spots exactly
+        // as the data's does, so the split it picks between two aliased parameters is the same however
+        // much data the stage has. More frames, a longer sweep or a later stage cannot break it.
+        constexpr double sigma_px = 3.0;
+        // The tilt's budgets, in those same direct-beam pixels: one for the spindle-parallel
+        // combination and one for the perpendicular one. Zero means no restraint at all, so which
+        // component is held and which is refined is these two numbers and nothing else.
+        //
+        // The parallel one is TIGHTER than the beam's on purpose: the data determine the SUM of the
+        // two, so with equal budgets the shift splits evenly and half of a beam-centre error still
+        // arrives as an angle (measured: the coupling to the starting beam centre falls only from
+        // 79% to 41% of one-for-one at equal budgets, and to 8% at this one). The detector tilt is a
+        // property of the mounting, re-measured when the detector is calibrated; the beam centre
+        // drifts between runs. When both ends of an alias have to be restrained, the tighter
+        // restraint belongs on the one that moves less.
+        //
+        // The perpendicular one is free. That is the arrangement the data support today: it is the
+        // component whose conditioning tracks the 2theta the fit reaches, i.e. the one the data speak
+        // about, while the parallel one's does not move with 2theta at all.
+        constexpr double SIGMA_TILT_PARALLEL_PX = 1.0;
+        constexpr double SIGMA_TILT_PERPENDICULAR_PX = 0.0;
+        const double gauge_w = data.geom.GetPixelSize_mm() / (distance_mm * data.geom.GetWavelength_A())
+                               * std::sqrt(effective_spots) / sigma_px;
+
         if (!data.refine_beam_center)
             problem.SetParameterBlockConstant(beam);
         else if (data.axis) {
@@ -316,20 +394,11 @@ bool XtalOptimizerInternal(XtalOptimizerData &data,
             // beam that the co-refined orientation keeps position-consistent. Rather than freeze it (the beam
             // does drift - it is only LaB6-monitored to ~a few px), RESTRAIN it toward the header with a soft
             // prior: the gauge direction has ~zero data sensitivity so the prior pins it near the header, while a
-            // real, well-supported drift can still overcome it. The spindle is along a detector axis in standard
-            // geometry, so restrain the dominant of X / Y.
-            const Coord spindle = data.axis->GetAxis();
-            const int parallel = (std::fabs(spindle.x) >= std::fabs(spindle.y)) ? 0 : 1;
-            // Weight so the prior is a sigma_px-pixel restraint that competes with the positional residuals.
-            // k = d|recip|/d(beam_px) ~ pixel/(distance*lambda) [A^-1/px]; scaling by sqrt(#residuals) makes the
-            // prior's curvature ~ (1/9) of the well-constrained-data curvature at sigma_px=3, i.e. data wins the
-            // perpendicular direction, the prior wins the gauge one.
-            constexpr double sigma_px = 3.0;
-            const double k = data.geom.GetPixelSize_mm() / (distance_mm * data.geom.GetWavelength_A());
-            const double w = k * std::sqrt(effective_spots) / sigma_px;
+            // real, well-supported drift can still overcome it.
             problem.AddResidualBlock(
-                new ceres::AutoDiffCostFunction<BeamComponentPrior, 1, 2>(
-                    new BeamComponentPrior(parallel, beam[parallel], w)),
+                new ceres::AutoDiffCostFunction<GaugeDirectionPrior, 1, 2>(
+                    new GaugeDirectionPrior(gauge_beam_x, gauge_beam_y,
+                                            gauge_beam_x * beam[0] + gauge_beam_y * beam[1], gauge_w)),
                 nullptr, beam);
         }
 
@@ -344,10 +413,60 @@ bool XtalOptimizerInternal(XtalOptimizerData &data,
                     problem.SetParameterLowerBound(detector_rot, i, detector_rot[i] - rot_range);
                     problem.SetParameterUpperBound(detector_rot, i, detector_rot[i] + rot_range);
                 }
+                // The same gauge as the beam prior above, described a second time: the tilt moves the
+                // direct beam exactly as the beam centre does, at D/pixel px per radian, so leaving
+                // its gauge combination free lets a beam-centre error the prior refuses to absorb
+                // reappear as an angle - measured at 0.072 deg per pixel of the STARTING beam centre,
+                // against a geometric one-for-one of 0.080, while the refined beam never leaves its
+                // anchor by more than a quarter of a pixel.
+                //
+                // Restraining it does not make the tilt a measurement, and nothing here should be read
+                // that way. In THIS fit the restrained component carries no information of its own:
+                // the crystal orientation is refined alongside it and absorbs the difference, so it
+                // ends up as accurate as the file's beam centre and no more. The free component does
+                // carry information, and is separately known to sit ~0.06 deg from a powder
+                // calibration on one measured detector, which is many times its formal error - so a
+                // single crystal's tilt is not a number to feed back into a file. What this buys is
+                // that a beam-centre error is no longer laundered into a reported angle.
+                //
+                // "In this fit" is the load-bearing part: a later stage that FREEZES the orientation
+                // has no such compensator, and whether the parallel component is measurable there is a
+                // different question with a different answer. This restraint is local to the fit that
+                // co-refines the orientation and does not speak for any other.
+                if (data.axis) {
+                    const double lever = distance_mm / data.geom.GetPixelSize_mm();
+                    // Parallel first, then the perpendicular direction (-gy, gx). Both go through the
+                    // same restraint, so swapping which one is held is a change to the two budgets.
+                    const double dirs[2][2] = {{gauge_rot_x, gauge_rot_y}, {-gauge_rot_y, gauge_rot_x}};
+                    const double budget[2] = {SIGMA_TILT_PARALLEL_PX, SIGMA_TILT_PERPENDICULAR_PX};
+                    for (int i = 0; i < 2; ++i) {
+                        if (budget[i] <= 0.0)
+                            continue;
+                        problem.AddResidualBlock(
+                            new ceres::AutoDiffCostFunction<GaugeDirectionPrior, 1, 2>(
+                                new GaugeDirectionPrior(dirs[i][0], dirs[i][1],
+                                                        dirs[i][0] * detector_rot[0]
+                                                            + dirs[i][1] * detector_rot[1],
+                                                        gauge_w * (sigma_px / budget[i]) * lever)),
+                            nullptr, detector_rot);
+                    }
+                }
             }
 
             if (!data.refine_rotation_axis) {
                 problem.SetParameterBlockConstant(rot_vec);
+            } else {
+                // Only the DIRECTION of the goniometer axis is a parameter. The residual applies
+                // angle_rad * |rot_vec|, so a free three-vector also fits a rotation SCALE - which
+                // GoniometerAxis::Axis() then normalises away, leaving the candidate scored by
+                // RotationIndexer::accumulate() under a rotation model the fit did not use. Measured
+                // over the corpus, that length reached 1.2 % and the fit/score disagreement a whole
+                // degree of goniometer angle. It is not a usable measurement either: on synthetic
+                // data it recovers 54 % of a known scale error, repeated first passes on one dataset
+                // disagree with each other in SIGN, and on the one dataset with a real 1.3 % stage
+                // fault it comes out negative. The rotation scale is measured properly, once, with
+                // four gates and a jackknife, in PostRefine.
+                problem.SetManifold(rot_vec, new ceres::SphereManifold<3>);
             }
 
             if (!data.refine_unit_cell) {
@@ -473,6 +592,11 @@ bool XtalOptimizerRotationOnly(XtalOptimizerData &data,
                                const std::vector<SpotToSave> &spots,
                                const float tolerance) {
     try {
+        // Same refusal as XtalOptimizerInternal: the residual here is built from Astar/Bstar/Cstar,
+        // which divide by the cell volume, so a coplanar basis makes every one of them infinite.
+        if (data.latt.VolumeFraction() < MIN_BASIS_VOLUME_FRACTION)
+            return false;
+
         // Parameter: angle-axis for the extra rotation. Identity == {0,0,0}.
         double rot_aa[3] = {0.0, 0.0, 0.0};