rugnux: a canonical pass the geometry walk will replace stops early

Every round the geometry walk takes ends in a pass that writes the files
again, so the canonical pass before it spent its space-group search, merges,
analyses, model validation and files on an answer nobody reads. The walk's
first question needs only that pass's post-refinement and, for a move of less
than a step, two indexing probes. The speculative probe now runs both beside
the merge - at the fit's geometry as before, then at the geometry in hand,
which the walk otherwise indexed after the pass (a memo hit where the pass's
own first pass left one) - and GeometryWalkTakesRound reads the walk's answer
from the same first-pass memos the walk's probes then read. The canonical pass
asks it, without waiting, before its space-group search (stops there; not
with a _process.h5 open) and again at the superseded point before the
analyses. An answer not yet in leaves the pass to finish as before, so only
timing depends on when the probes finish; the walk itself is unchanged.

Exactness E (not F): merged data and all output files identical to rc175
e9d4f2221 on every walking set, 2 runs each; decisions identical. Fired:
before the search on 7ph1, 5m17, 8pqd; at the superseded point on 2xfw, 6zr0,
9t6s; on 6zqy the probes were not in by either point in these runs; 8xtf's
canonical pass is starved and never merges.

Wall (time -v, base -> new, rep1 / rep2, shared machine):
  5m17  65.9->57.0 / 68.5->56.7
  8pqd  49.8->43.0 / 50.0->43.1
  7ph1  51.1->48.9 / 51.1->45.3
  6zr0  31.1->22.7 / 24.4->21.1
  2xfw  101.2->66.7 / 48.2->48.2 (rep1 base noisy)
  6zqy  27.3->26.1 / 26.8->25.7
  controls myob_x10sa, cytc_x10sa, 11if, kdp_x10sa_20keV, 8a1a: no change
  beyond machine noise

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi

rugnux/Rugnux.cpp

@@ -397,22 +397,28 @@ void Rugnux::StartSpeculativeGeometryProbe(std::shared_ptr<Rugnux> run, const st
     run->prepass_detector_geometry_ = prepass_detector_geometry_;
     run->speculative_probe_.reset();
     run->speculate_geometry_probe_ = false;
-    run->experiment_.BeamX_pxl(geometry[0]).BeamY_pxl(geometry[1]).DetectorDistance_mm(geometry[2])
-                    .PoniRot1_rad(geometry[3]).PoniRot2_rad(geometry[4]);
     run->indexing_probe_only_ = true;
     run->postrefine_probe_ = false;
     run->postrefine_probe_only_ = false;
     run->rerun_if_starved_ = false;
+    // The walk's other probe, at the geometry in hand, on a copy of its own: where this pass's own
+    // first pass has not left its memo the walk would index there too, after the pass.
+    auto hand = std::make_shared<Rugnux>(*run);
+    run->experiment_.BeamX_pxl(geometry[0]).BeamY_pxl(geometry[1]).DetectorDistance_mm(geometry[2])
+                    .PoniRot1_rad(geometry[3]).PoniRot2_rad(geometry[4]);
     auto sp = std::make_shared<SpeculativeProbe>();
     sp->run = run;
+    sp->hand = hand;
     // A probe that fails leaves no memo, and the walk's own probe then runs it again and meets the
     // failure itself. It runs beside the merge of the pass that started it, on the same card: a merge
     // that does not fit beside it waits for it to end (GPUWorkBeside).
-    sp->done = std::async(std::launch::async, [run] {
+    sp->done = std::async(std::launch::async, [run, hand] {
         GPUWorkBeside beside;
-        try {
-            run->RunPipeline(nullptr, /*write_output=*/false, /*geometry_prepass=*/false);
-        } catch (const std::exception &) {
+        for (const auto &r : {run, hand}) {
+            try {
+                r->RunPipeline(nullptr, /*write_output=*/false, /*geometry_prepass=*/false);
+            } catch (const std::exception &) {
+            }
         }
     });
     speculative_probe_ = std::move(sp);
@@ -432,6 +438,8 @@ void Rugnux::CollectSpeculativeProbe() {
     speculative_probe_->done.wait();
     for (const auto &m : speculative_probe_->run->first_pass_memo_)
         AddFirstPassMemo(m);
+    for (const auto &m : speculative_probe_->hand->first_pass_memo_)
+        AddFirstPassMemo(m);
     speculative_probe_.reset();
 }
 
@@ -491,7 +499,11 @@ std::vector<double> Rugnux::SpotFindingKey(const DiffractionExperiment &x,
 }
 
 std::vector<double> Rugnux::FirstPassInputKey() const {
-    std::vector<double> k = ExperimentKey();
+    return FirstPassInputKey(experiment_);
+}
+
+std::vector<double> Rugnux::FirstPassInputKey(const DiffractionExperiment &e) const {
+    std::vector<double> k = ExperimentKey(e);
     // Run state the first pass reads besides the experiment.
     k.insert(k.end(), {static_cast<double>(lattice_angles_demoted_), prepass_rotation_scale_.value_or(-1.0f),
                        static_cast<double>(force_rotation_result_.has_value()),

rugnux/Rugnux.h

@@ -851,14 +851,17 @@ class Rugnux {
     [[nodiscard]] std::vector<double> SpotFindingKey(const DiffractionExperiment &x,
                                                      const AzimuthalIntegrationMapping &mapping) const;
     void AddFirstPassMemo(const FirstPassMemo &memo);
-    // The geometry walk's next indexing probe, started on a copy of the run as soon as the canonical
-    // pass has post-refined - it depends on nothing that pass does afterwards - so that it runs beside
-    // the pass's scaling, merge and output. It leaves its evidence in the copy's first-pass memo, and
-    // the walk's own probe takes it from there through the usual key check (CollectSpeculativeProbe).
+    // The geometry walk's next indexing probes - at the fit's geometry, then at the geometry in hand
+    // (a memo hit where the pass's own first pass left one) - started on copies of the run as soon as
+    // the canonical pass has post-refined - they depend on nothing that pass does afterwards - so that
+    // they run beside the pass's scaling, merge and output. They leave their evidence in the copies'
+    // first-pass memos, and the walk's own probes take it from there through the usual key check
+    // (CollectSpeculativeProbe).
     // RunAllPasses sets speculate_geometry_probe_ on the passes whose post-refinement the walk will
     // probe with an indexing pass.
     struct SpeculativeProbe {
         std::shared_ptr<Rugnux> run;
+        std::shared_ptr<Rugnux> hand;   // the walk's probe at the geometry in hand, after run's
         std::future<void> done;
     };
     std::shared_ptr<SpeculativeProbe> speculative_probe_;
@@ -869,6 +872,11 @@ class Rugnux {
     void StartSpeculativeGeometryProbe(std::shared_ptr<Rugnux> run, const std::array<float, 5> &geometry);
     void CollectSpeculativeProbe();
     [[nodiscard]] std::vector<double> FirstPassInputKey() const;
+    // The same, with another experiment in place of the run's.
+    [[nodiscard]] std::vector<double> FirstPassInputKey(const DiffractionExperiment &e) const;
+    // Whether the geometry walk after the canonical pass will take a round, asked by that pass from
+    // what it already has in hand; `fit` is its post-refinement running beside the merge (RefinedPasses).
+    [[nodiscard]] bool GeometryWalkTakesRound(const ProcessResult &result, const std::future<void> &fit) const;
     // The part of FirstPassInputKey that is the experiment and the spot-finding settings; without the
     // goniometer's angles where rotation_angles is false (SpotFindingKey).
     [[nodiscard]] std::vector<double> ExperimentKey(bool rotation_angles = true) const {

rugnux/RugnuxPasses.cpp

@@ -727,6 +727,45 @@ RotationScaleWalk Rugnux::WalkRotationScaleBeside(const ProcessResult &pass1, Ro
     }
 }
 
+// The geometry walk's first question (see RefinedPasses), asked by the canonical pass before its
+// space-group search and again before its analyses: every round the walk takes ends in a pass that
+// writes the files again, so a canonical pass the walk is going to replace only spends its merge,
+// model validation and files on an answer nobody reads. The question needs the pass's own
+// post-refinement and, for a move of less than a step, the walk's two indexing probes, which the
+// speculative probe runs beside the merge (or the pass's own first pass has run) - taken from the
+// first-pass memos the walk's probes read them from, so the answer is the walk's own. Nothing waits
+// for it: an answer not yet in leaves the pass to finish as it always did.
+bool Rugnux::GeometryWalkTakesRound(const ProcessResult &result, const std::future<void> &fit) const {
+    const auto ready = [](const std::future<void> &f) {
+        return !f.valid() || f.wait_for(std::chrono::seconds(0)) == std::future_status::ready;
+    };
+    if (!speculate_geometry_probe_ || cancelled_ || !ready(fit) || !prepass_detector_geometry_
+        || !result.post_refine || !result.post_refine->detector_refined)
+        return false;
+    if (result.post_refine->large_move)
+        return true;
+    if (config_.verify_first_pass_memo || !speculative_probe_ || !ready(speculative_probe_->done))
+        return false;
+    const auto evidence_at = [&](const std::array<float, 5> &g) -> std::optional<ValidationSpotEvidence> {
+        DiffractionExperiment e = experiment_;
+        e.SetSpaceGroup(user_fixed_sg_);   // as RunPipeline starts every pass
+        e.BeamX_pxl(g[0]).BeamY_pxl(g[1]).DetectorDistance_mm(g[2]).PoniRot1_rad(g[3]).PoniRot2_rad(g[4]);
+        const std::vector<double> key = FirstPassInputKey(e);
+        const std::vector<FirstPassMemo> *memos[] = {&first_pass_memo_, &speculative_probe_->run->first_pass_memo_,
+                                                     &speculative_probe_->hand->first_pass_memo_};
+        for (const auto *list : memos)
+            for (const auto &m : *list)
+                if (m.key == key)
+                    return m.evidence;
+        return std::nullopt;
+    };
+    const auto at_hand = evidence_at({experiment_.GetBeamX_pxl(), experiment_.GetBeamY_pxl(),
+                                      experiment_.GetDetectorDistance_mm(), experiment_.GetPoniRot1_rad(),
+                                      experiment_.GetPoniRot2_rad()});
+    const auto at_fit = evidence_at(*prepass_detector_geometry_);
+    return at_hand && at_fit && ValidationEvidencePrefers(*at_hand, *at_fit);
+}
+
 // The refined passes of the rotation two-pass, from the second pass on: everything after the
 // rotation-scale walk (see RunAllPasses). Empty where they ran beside the walk and it overturned them.
 std::optional<ProcessResult> Rugnux::RefinedPasses(RugnuxObserver *observer, ProcessResult pass1,

rugnux/RugnuxScaleMerge.cpp

@@ -1243,6 +1243,14 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) {
     // above triclinic - i.e. a promotion was made, which is what makes a later "the Laue class is
     // holohedral so there is no twin law" conclusion circular.
     bool promoted_point_group = false;
+    // A canonical pass the geometry walk is going to replace stops here, where the answer is
+    // already in (GeometryWalkTakesRound); it is asked again below, before the analyses. Not with a
+    // _process.h5 open: that file is written whole by every pass that opens it.
+    if (!writer && GeometryWalkTakesRound(result, p.postrefine_ahead)) {
+        logger.Info("The geometry walk will re-integrate at the geometry the post-refinement asks for, "
+                    "so this pass stops before its space-group search");
+        return true;
+    }
     TimingMark search_mark(logger, "space-group search");
     if (search_space_group) {
         SearchSpaceGroupOptions sg_opts;
@@ -2775,14 +2783,16 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) {
     // the files are rewritten by the pass that supersedes this one, and the one place that touches
     // experiment_ - the model frame - is undone by the SetSpaceGroup at the top of RunPipeline.
     //
-    // A refined pass run beside the rotation-scale walk is superseded too where the walk overturns it
-    // (see RunAllPasses); it waits here for the walk's answer.
+    // So is a canonical pass the geometry walk is going to replace, where that answer is in by now
+    // (GeometryWalkTakesRound). A refined pass run beside the rotation-scale walk is superseded too
+    // where the walk overturns it (see RunAllPasses); it waits here for the walk's answer.
     superseded =
             (!cancelled_ && quality_guard_pass1_ && quality_guard_pass1_->has_merge_statistics
              && ((bragg_adaptive_ && bkg_starved_fraction_
                   && *bkg_starved_fraction_ > spot_width::BKG_STARVED_MAX_FRACTION)
                  || !RefinedPassIsWorse(*quality_guard_pass1_, result).empty()))
             || (lattice_arm_check_ && lattice_arm_check_->lost)   // only with a _process.h5 open, see above
+            || GeometryWalkTakesRound(result, p.postrefine_ahead)
             || (rotation_scale_check_ && rotation_scale_check_->Overturns());
     if (superseded) {
         logger.Info("This pass is going to be re-run, so it makes no report and writes no files.");