v1.0.0-rc.166 (#76)

* `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands.
* `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion.
* Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants.
* `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing.
* A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed.
* `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing.
* Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences.
* The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to.
* The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after.
* The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution.
* `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have.
* Twinning is no longer reported when the L-test contradicts it.
* The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's.
* `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots.
* The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area.

Reviewed-on: #76
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/geom_refinement/PostRefine.cpp

@@ -20,6 +20,20 @@
 
 namespace {
 
+// How far post-refinement's step B may move the beam centre away from a value something else already
+// believes. The bound is not about how large a real correction can be - it is the backstop for a fit
+// corrupted by something every cross-validation fold shares (a second lattice, most often), which the
+// relative "the held-out residual improved" gate cannot see. The absolute size of the move is what
+// separates that from a genuine header correction.
+//
+// It is measured from whichever of the nominal centre and the run's own MEASUREMENT of the centre
+// (PostRefineSettings::measured_beam_px) is nearer. Anchored on the nominal centre alone it caps the
+// correction at exactly the header's own error, which is the opposite of its job: the header is most
+// worth correcting when it is most wrong. Measured on a rotation crystal whose header centre is 22.1 px
+// out - a value the same run's beam-centre check had already placed to +-0.30 px and then discarded -
+// the old bound rejected a fit that improved the held-out positional residual nine-fold.
+constexpr double BEAM_BOUND_PXL = 15.0;
+
 // One integrated partial, flattened across all images. Kept as narrow as the sort and the event
 // split allow: on a large cell this array is gigabytes, and the scatter and every level of the
 // per-bucket sort move all of it. The goniometer angle is not stored - it is a function of the
@@ -329,6 +343,7 @@ PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome
             const double rot3 = nominal_geom.GetPoniRot3_rad();
             // Same for every observation, so taken once here rather than per residual.
             const double cos_rot3 = std::cos(rot3), sin_rot3 = std::sin(rot3);
+            const DetectorOrientation orientation = nominal_geom.GetOrientation();
             const double pixel_mm = nominal_geom.GetPixelSize_mm();
             const double det_rot[2] = {nominal_geom.GetPoniRot1_rad(), nominal_geom.GetPoniRot2_rad()};
             const UnitCell r0 = reference_latt.GetUnitCell();
@@ -661,7 +676,7 @@ PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome
                     if (!in(pp->h, pp->k, pp->l, s)) continue;
                     XtalResidual r(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, cos_rot3, sin_rot3,
                                    angle_rad(pp->img),
-                                   pp->h, pp->k, pp->l, sys);
+                                   pp->h, pp->k, pp->l, sys, orientation);
                     double resid[3] = {0, 0, 0};
                     r(beam, dist, det_rot, rot_vec, p0, p1, p2, resid);
                     c += resid[0]*resid[0] + resid[1]*resid[1] + resid[2]*resid[2]; ++n;
@@ -683,7 +698,7 @@ PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome
                     p.AddResidualBlock(new ceres::AutoDiffCostFunction<XtalResidualBeamDistance, 3, 2, 1>(
                         new XtalResidualBeamDistance(
                             XtalResidual(pp->obs_x, pp->obs_y, lambda_l, pixel_mm, cos_rot3, sin_rot3,
-                                         angle_rad(pp->img), pp->h, pp->k, pp->l, sys),
+                                         angle_rad(pp->img), pp->h, pp->k, pp->l, sys, orientation),
                             fc, p0)),
                         new ceres::CauchyLoss(0.02), beam, dist);
                 }
@@ -692,8 +707,14 @@ PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome
                 // above bakes those five blocks in rather than declaring them and freezing them, so there
                 // is nothing left to hold constant here.
                 p.SetParameterLowerBound(dist, 0, dist0 * 0.95); p.SetParameterUpperBound(dist, 0, dist0 * 1.05);
-                for (int j = 0; j < 2; ++j) { p.SetParameterLowerBound(beam, j, beam[j] - 15.0);
-                                              p.SetParameterUpperBound(beam, j, beam[j] + 15.0); }
+                // The box has to reach everywhere the gate below would accept, or the gate is never the
+                // thing that decides: a fit pinned at a box face lands exactly ON the bound and is then
+                // refused for being there.
+                for (int j = 0; j < 2; ++j) {
+                    const double m = settings.measured_beam_px ? (*settings.measured_beam_px)[j] : beam[j];
+                    p.SetParameterLowerBound(beam, j, std::min(beam[j], m) - BEAM_BOUND_PXL);
+                    p.SetParameterUpperBound(beam, j, std::max(beam[j], m) + BEAM_BOUND_PXL);
+                }
                 ceres::Solver::Options o; o.linear_solver_type = ceres::DENSE_QR; o.max_num_iterations = 60;
                 o.num_threads = std::max(1, settings.num_threads); o.logging_type = ceres::LoggingType::SILENT;
                 ceres::Solver::Summary sum; ceres::Solve(o, &p, &sum);
@@ -715,15 +736,34 @@ PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome
                 // degeneracy) far off. The absolute size of the move discriminates a genuine header correction
                 // from that failure far better than the absolute residual, which real marginal (noisy / iced)
                 // data shares with the multi-lattice case.
+                const double from_nominal = std::hypot(beam_fit[0] - beam_x0, beam_fit[1] - beam_y0);
+                const double from_measured = settings.measured_beam_px
+                        ? std::hypot(beam_fit[0] - (*settings.measured_beam_px)[0],
+                                     beam_fit[1] - (*settings.measured_beam_px)[1])
+                        : std::numeric_limits<double>::infinity();
                 const bool in_bounds = std::fabs(dist_fit - dist0) < 0.01 * dist0
-                                       && std::hypot(beam_fit[0] - beam_x0, beam_fit[1] - beam_y0) < 15.0;
+                                       && std::min(from_nominal, from_measured) < BEAM_BOUND_PXL;
                 result.detector_refined = convB && cvB_ref < 0.98 * cvB_nom && in_bounds;
                 if (result.detector_refined) { double bo[2]; solve_detector(ALL, bo, dist); beam[0] = bo[0]; beam[1] = bo[1]; }
+                // Name which test refused it. Three different things reject here and the geometry that
+                // comes out is the same in all three, so a run that silently keeps its header geometry
+                // says nothing about whether the fit was bad, the improvement too small, or the move
+                // too large for the bound - which is the one case where the number worth reading is the
+                // one that was thrown away.
+                const char *verdict =
+                        result.detector_refined      ? "COMMIT"
+                        : !convB                     ? "reject (the fit did not converge; kept nominal detector)"
+                        : !(cvB_ref < 0.98 * cvB_nom) ? "reject (the held-out residual did not improve enough; "
+                                                        "kept nominal detector)"
+                        : std::fabs(dist_fit - dist0) >= 0.01 * dist0
+                                                     ? "reject (the distance moved more than 1 %; kept nominal detector)"
+                                                     : "reject (the beam moved further than the bound from every "
+                                                       "centre anything believes; kept nominal detector)";
                 logger.Info("Post-refine GEOM step B (distance/beam): dist {:.3f} -> {:.3f} mm, beam "
                             "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), held-out pos {:.3e} -> {:.3e} => {}",
                             dist0, result.detector_refined ? dist : dist0, beam_x0, beam_y0,
                             result.detector_refined ? beam[0] : beam_x0, result.detector_refined ? beam[1] : beam_y0,
-                            cvB_nom, cvB_ref, result.detector_refined ? "COMMIT" : "reject (kept nominal detector)");
+                            cvB_nom, cvB_ref, verdict);
             } else {
                 logger.Info("Post-refine GEOM step B: only {} positional observations - skipped", n_obs);
             }