rugnux: take the decisions before pass 2 from the first part of a rotation sweep

The pre-scan sample (beam-stop shadow, defective pixels, spot width, the background beam centre)
and the first pass's frames (the spread and wedge schemes, the validation frames, the prefetch) now
come from [start, PrepassEnd) - the part of the sweep the geometry pre-pass integrates - instead of
the whole sweep. Rugnux::PrepassEnd is the one place that range is defined, for the pre-pass and for
these decisions alike, so any change to the pre-pass range (such as a minimum rotation) moves both.
Stills keep the whole range.

With rugnux following a collection while it is written, nothing before pass 2 then waits for the
second part of the sweep: the pre-scan, both first passes and the pre-pass run while it is
collected, and pass 2 starts reading as soon as they are done. An offline run takes the same
decisions from the same frames, so the streamed result still equals the offline one by md5 (in-house
JUNGFRAU 4M thaumatin replayed at one data file per second, also at -N 8 and with the replay paused
and resumed): 2.4 s from the last data file to the result, against 8.0 s for the offline run.

Result-changing. p.mtz md5 at af3ea9cca -> here (with the 120 deg minimum of the previous commit),
GPU: myob de2ab820 -> 9bdd6916, cytc 3e5d2f91 -> 62e744a1, thau bc5e3de3 -> 3ec88c34,
kdp cf83ee5d -> cdc2c068, 8a1a 50ab659e -> 309ea9a8, 8qaw d3287922 -> c2849d52; CPU: myob 3f9d12d8 ->
18c7ac2f, cytc d30c9969 -> 40e17d97, thau 4cdbdd94 -> f14e52da, kdp f1d801f3 -> eb930714. Sweeps of
120 deg or less are unchanged by this commit (their early range is the whole sweep). Needs the
battery before it is merged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi

docs/CPU_DATA_ANALYSIS_INDEXING.md

@@ -212,7 +212,7 @@ The lookup can also land *short*: near the Niggli type-I/type-II boundary the ch
 
 The rotation first pass refines the **twelve** best candidate lattices rather than four. The pre-refinement indexed fraction is an unreliable ranking, so a correct cell can sit below several degenerate ones and never be refined at all.
 
-For rotation data the first pass therefore refines the constrained cell *and* an unconstrained (triclinic) one from the same spots — which it finds itself, over a sample spread across the sweep, rather than reading what the acquisition wrote — and settles the two on how many of a fixed set of validation frames each actually indexes. The bar is a clear majority, not a margin. An unconstrained refinement holds no cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it indexes a few more — a percentage margin therefore demotes real lattices (measured: a genuine $I$-centred orthorhombic to $P1$). Only a constrained cell that fails outright while its unconstrained cell works distinguishes a false promotion. The intensities settle the space group later regardless (§13).
+For rotation data the first pass therefore refines the constrained cell *and* an unconstrained (triclinic) one from the same spots — which it finds itself, over a sample spread across the first part of the sweep (the part the geometry pre-pass integrates), rather than reading what the acquisition wrote — and settles the two on how many of a fixed set of validation frames each actually indexes. The bar is a clear majority, not a margin. An unconstrained refinement holds no cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it indexes a few more — a percentage margin therefore demotes real lattices (measured: a genuine $I$-centred orthorhombic to $P1$). Only a constrained cell that fails outright while its unconstrained cell works distinguishes a false promotion. The intensities settle the space group later regardless (§13).
 
 Two further hypotheses are weighed at the same point, both by default. Where two first-pass answers
 have primitive volumes in a small integer ratio (2–4×) and tie on the validation frames, the frame
@@ -315,7 +315,7 @@ So every round is scored on the **widest** gate — the population the first pas
 
 The refinement above (§7.2) runs per image against that image's spots. For rotation data an additional **post-refinement** (on by default; `--rotation-no-postrefine` disables it) improves the detector distance, beam centre and crystal cell/axis using **all** frames at once, then re-integrates:
 
-1. **Pass 1** integrates, scales and merges at the header geometry — the first half of the sweep by default, but at least 120° of it, so a sweep up to 120° is pre-passed whole (`--prepass-fraction`, `1` for all of it). The first-pass indexing has already sampled the whole sweep; what pass 1 adds is integrated reflections for the post-refinement below, and half a sweep holds plenty of them while costing half the time. Its merge, post-refinement and smoothed mosaicity describe a sweep of those frames alone.
+1. **Pass 1** integrates, scales and merges at the header geometry — the first half of the sweep by default, but at least 120° of it, so a sweep up to 120° is pre-passed whole (`--prepass-fraction`, `1` for all of it). The first-pass indexing and the pre-scan have sampled the same part of the sweep, so nothing before pass 2 reads the rest of it - which is what lets a run that follows a collection still being written do all of this while the rest is collected; what pass 1 adds is integrated reflections for the post-refinement below, and half a sweep holds plenty of them while costing half the time. Its merge, post-refinement and smoothed mosaicity describe a sweep of those frames alone.
 2. From pass-1's integrated reflections, the crystal and the detector are refined together over all its frames (Ceres, robust loss) in **one joint fit**, against both residuals at once:
    - the **positional** detector↔reciprocal residual at each partial's observed spot, and
    - a distance-independent **Ewald excitation** residual at each reflection's observed rocking centroid $\phi_\mathrm{obs}$.

docs/RUGNUX_ADVANCED.md

@@ -342,7 +342,7 @@ For very long axes the direction grid's angular resolution binds well below 500
 | `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) |
 | `--force-rotation-lattice <vec>` | Force rotation lattice (9 floats, Å), skipping the first pass |
 | `--rotation-no-postrefine` | Rotation: disable the default-on two-pass geometry post-refine (see the rotation section) |
-| `--prepass-fraction <f>` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep, but at least 120° of it (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame |
+| `--prepass-fraction <f>` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep, but at least 120° of it (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame. The pre-scan and the first-pass indexing sample the same part of the sweep |
 | `--refine-geometry[=N\|off]` | Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills with a reference cell (`-C` / `-z`), `=off` disables |
 | `--index-ice-rings[=on\|off]` | Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: **off**; no effect without `--detect-ice-rings`, which does the flagging) |
 

docs/RUGNUX_OVERVIEW.md

@@ -8,7 +8,7 @@ rotation run with the defaults; stills differences are at the end.
 ([What Rugnux reads](RUGNUX_FORMATS.md)). A goniometer axis makes it a rotation run, none makes it
 serial stills — nothing is asked of the user.
 
-**Pre-scan.** A projection of frames spread over the sweep (60 by default) finds the beam-stop shadow and masks
+**Pre-scan.** A projection of frames (60 by default) spread over the first part of the sweep, the part the geometry pre-pass integrates, finds the beam-stop shadow and masks
 it ([§1.5](CPU_DATA_ANALYSIS_IMAGE.md)), masks pixels its frames show to be defective
 ([§1.6](CPU_DATA_ANALYSIS_IMAGE.md)), measures the beam centre from the isotropy of the
 scattered background and compares it with the file's ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)), and

rugnux/Rugnux.cpp

@@ -636,10 +636,16 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
     if (full && config_.refine_geometry.has_value())
         RefineStillsGeometry(start_image, end_image, images_to_process, observer);
 
+    // The geometry pre-pass integrates only the first part of the sweep (pass 2 reads it all), and on a
+    // rotation sweep the decisions before pass 2 read only that part too (PrepassEnd).
+    const int prepass_end = PrepassEnd(start_image, end_image);
+    const int prepass_images = (prepass_end - start_image + config_.stride - 1) / config_.stride;
+    const int early_images = experiment_.IsRotationIndexing() ? prepass_images : images_to_process;
+
     // After any geometry refinement, so the shadow is found about the beam centre actually used,
     // and before the azimuthal mapping and the output mask below, which both read pixel_mask_.
     if (config_.detect_beam_stop.has_value() || config_.adaptive_integration_radius)
-        PreScan(start_image, images_to_process,
+        PreScan(start_image, early_images,
                 std::min(config_.detect_beam_stop.value_or(BEAM_CENTER_PROJECTION_IMAGES),
                          HotPixelFinder::MAX_SAMPLED_FRAMES), observer);
 
@@ -828,8 +834,6 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
 
     const auto start_time = std::chrono::steady_clock::now();
 
-    const int prepass_end = PrepassEnd(start_image, end_image);
-    const int prepass_images = (prepass_end - start_image + config_.stride - 1) / config_.stride;
     const int loop_end = geometry_prepass ? prepass_end : end_image;
 
     // Declared here rather than where they are first filled, so that the phases below can share them

rugnux/Rugnux.h

@@ -815,6 +815,13 @@ class Rugnux {
     // the defective pixels, and the recorded spot width that sets the integration radius
     // (config_.adaptive_integration_radius).
     void PreScan(int start_image, int images_to_process, int frame_count, RugnuxObserver *observer);
+    // The first part of a rotation sweep, [start_image, PrepassEnd): what the geometry pre-pass
+    // integrates, and the only frames the decisions taken before the canonical pass read - the
+    // pre-scan sample (mask, spot width, background beam centre) and the first pass's schemes and
+    // validation frames. So those decisions are made from frames that exist while the rest of the
+    // sweep is still being collected, and an offline run makes them from the same frames. It is the
+    // first config_.prepass_fraction of the sweep, but at least PREPASS_MIN_ROTATION_DEG (Rugnux.cpp).
+    [[nodiscard]] int PrepassEnd(int start_image, int end_image) const;
     // The pre-scan's defective-pixel mask (HotPixelFinder), read on `sample` with the rings drawn
     // about `ring_centre` where the background measured one.
     void MaskDefectivePixels(int start_image, const std::vector<int> &sample,
@@ -845,10 +852,6 @@ class Rugnux {
     // geometry_prepass == true stops after post-refining the detector geometry from the first integration
     // and applies it (+ the refined lattice) to experiment_, so the next pass re-integrates with it.
     ProcessResult RunPipeline(RugnuxObserver *observer, bool write_output, bool geometry_prepass);
-    // The end of the first part of a rotation sweep, [start_image, PrepassEnd), that the geometry
-    // pre-pass integrates: the first config_.prepass_fraction of it, but at least
-    // PREPASS_MIN_ROTATION_DEG (Rugnux.cpp).
-    [[nodiscard]] int PrepassEnd(int start_image, int end_image) const;
     // The phases of RunPipeline that live in files of their own; each reads and writes RunPipeline's
     // locals through `p` (RugnuxPipeline.h).
     // The per-image loop over the sweep (azimuthal integration, or the full per-image analysis), what is

rugnux/RugnuxFirstPass.cpp

@@ -195,7 +195,8 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     auto &observer = p.observer;
     auto &geometry_prepass = p.geometry_prepass;
     auto &start_image = p.start_image;
-    auto &images_to_process = p.images_to_process;
+    // Every frame the first pass reads is in the first part of the sweep (Rugnux::PrepassEnd).
+    const int early_images = p.prepass_images;
     auto &dataset = p.dataset;
     auto &mapping = p.mapping;
     auto &indexer_pool = p.indexer_pool;
@@ -420,7 +421,7 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     // sub-lattice from an under-sampled scheme) indexes few frames, while the true cell indexes
     // many - this is the discriminator that lets the best scheme win.
     const std::vector<int> validation = select_equally_spaced_image_ordinals(
-        images_to_process, std::min(images_to_process, 60));
+        early_images, std::min(early_images, 60));
 
     const float index_tol = experiment_.GetIndexingSettings().GetTolerance();
 
@@ -600,16 +601,16 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     // Sampling schemes, tried in order of preference. We index each and keep the one that indexes
     // the most frames; the schemes cover complementary failure modes and no single one is best for
     // every crystal:
-    //  - "spread": images spread over the whole rotation - full angular range, coarse stride; the
-    //    long-standing default, best for typical crystals.
+    //  - "spread": images spread over the first part of the rotation (PrepassEnd) - wide angular range,
+    //    coarse stride; the long-standing default, best for typical crystals.
     //  - "wedge": consecutive frames from the clean start - native stride keeps each reflection's
     //    rocking curve continuous across frames, so the FFT can trace a long axis whose fine
     //    reciprocal spacing the coarse spread cannot resolve (e.g. an F-cubic cell viewed down
     //    [111]); bounded by the accumulation cap to the least radiation-damaged early wedge.
-    std::vector<int> all_consecutive(images_to_process);
+    std::vector<int> all_consecutive(early_images);
     std::iota(all_consecutive.begin(), all_consecutive.end(), 0);
     const std::vector<int> spread_ordinals = select_equally_spaced_image_ordinals(
-        images_to_process, config_.rotation_indexing_image_count);
+        early_images, config_.rotation_indexing_image_count);
     const std::vector<std::pair<std::string, std::vector<int>>> schemes = {
         {"spread", spread_ordinals},
         {"wedge",  all_consecutive},
@@ -1129,7 +1130,7 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     {
         std::vector<int> upfront = spread_ordinals;
         upfront.insert(upfront.end(), validation.begin(), validation.end());
-        for (int i = 0; i < std::min(SPOT_PREFETCH_CHUNK, images_to_process); i++)
+        for (int i = 0; i < std::min(SPOT_PREFETCH_CHUNK, early_images); i++)
             upfront.push_back(i);
         std::sort(upfront.begin(), upfront.end());
         upfront.erase(std::unique(upfront.begin(), upfront.end()), upfront.end());

rugnux/RugnuxPreScan.cpp

@@ -170,10 +170,11 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
     if (!want_shadow && !want_width && !want_hot)
         return;
 
-    // The sample is taken from the middle of the sweep, both ends left out: an even spread over
-    // [0, images_to_process) always picks the very first and the very last image, and those are the
-    // ones a shutter out of step spoils - the last carries under 90 % of the sample's median counts
-    // on 12 of 38 regression sweeps and is essentially unexposed on 5.
+    // The sample is taken from the frames given - on a rotation sweep its first part (PrepassEnd) -
+    // both ends left out: an even spread over [0, images_to_process) always picks the very first and
+    // the very last image, and those are the ones a shutter out of step spoils - the last carries
+    // under 90 % of the sample's median counts on 12 of 38 regression sweeps and is essentially
+    // unexposed on 5.
     const int margin = std::min(PRESCAN_END_MARGIN_IMAGES, images_to_process / 10);
     auto shadow_sample = select_equally_spaced_image_ordinals(images_to_process - 2 * margin,
                                                               frame_count);

rugnux/rugnux_cli.cpp

@@ -148,7 +148,7 @@ void print_usage() {
     std::cout << "   --force-rotation-lattice <vec>   Force rotation indexer with external lattice (in Angstrom) : \"a0x,a0y,a0z,a1x,a1y,a1z,a2x,a2y,a2z\" (9 floats, skips first pass)" << std::endl;
     std::cout << "   --rotation-scale <k>             Goniometer rotation scale: the stage turned k times the angle stored in the file (the commanded one). Applied to both passes; overrides the fitted correction" << std::endl;
     std::cout << "   --rotation-no-postrefine         Disable the (default-on) two-pass rotation post-refine (post-refine detector distance/beam + cell/axis, then re-integrate; only the refined pass is written, as the canonical <prefix>_* output)" << std::endl;
-    std::cout << "   --prepass-fraction <f>           Integrate the first fraction f of the sweep, but at least 120 deg of it, in the two-pass rotation geometry pre-pass (0.05-1, default 0.5; 1 = the whole sweep); pass 2 reads every frame" << std::endl;
+    std::cout << "   --prepass-fraction <f>           Integrate the first fraction f of the sweep, but at least 120 deg of it, in the two-pass rotation geometry pre-pass (0.05-1, default 0.5; 1 = the whole sweep); pass 2 reads every frame. The pre-scan and the first-pass indexing sample the same part of the sweep" << std::endl;
     std::cout << "   -X, --indexing-algorithm <txt>   Indexing algorithm (FFBIDX|FFT|FFTW|Auto|None)" << std::endl;
     std::cout << "   -S, --space-group <num|symbol>   Space group number (96) or symbol (P43212) - for indexing and scaling" << std::endl;
     std::cout << "   -C, --unit-cell <cell>           Fix reference unit cell: \"a,b,c,alpha,beta,gamma\"" << std::endl;