Merge branch 'pp-angle' into rc175

docs/CPU_DATA_ANALYSIS_INDEXING.md

@@ -212,7 +212,7 @@ The lookup can also land *short*: near the Niggli type-I/type-II boundary the ch
 
 The rotation first pass refines the **twelve** best candidate lattices rather than four. The pre-refinement indexed fraction is an unreliable ranking, so a correct cell can sit below several degenerate ones and never be refined at all.
 
-For rotation data the first pass therefore refines the constrained cell *and* an unconstrained (triclinic) one from the same spots — which it finds itself, over a sample spread across the sweep, rather than reading what the acquisition wrote — and settles the two on how many of a fixed set of validation frames each actually indexes. The bar is a clear majority, not a margin. An unconstrained refinement holds no cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it indexes a few more — a percentage margin therefore demotes real lattices (measured: a genuine $I$-centred orthorhombic to $P1$). Only a constrained cell that fails outright while its unconstrained cell works distinguishes a false promotion. The intensities settle the space group later regardless (§13).
+For rotation data the first pass therefore refines the constrained cell *and* an unconstrained (triclinic) one from the same spots — which it finds itself, over a sample spread across the first part of the sweep (the part the geometry pre-pass integrates), rather than reading what the acquisition wrote — and settles the two on how many of a fixed set of validation frames each actually indexes. The bar is a clear majority, not a margin. An unconstrained refinement holds no cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it indexes a few more — a percentage margin therefore demotes real lattices (measured: a genuine $I$-centred orthorhombic to $P1$). Only a constrained cell that fails outright while its unconstrained cell works distinguishes a false promotion. The intensities settle the space group later regardless (§13).
 
 Two further hypotheses are weighed at the same point, both by default. Where two first-pass answers
 have primitive volumes in a small integer ratio (2–4×) and tie on the validation frames, the frame
@@ -315,7 +315,7 @@ So every round is scored on the **widest** gate — the population the first pas
 
 The refinement above (§7.2) runs per image against that image's spots. For rotation data an additional **post-refinement** (on by default; `--rotation-no-postrefine` disables it) improves the detector distance, beam centre and crystal cell/axis using **all** frames at once, then re-integrates:
 
-1. **Pass 1** integrates, scales and merges at the header geometry — the first half of the sweep by default (`--prepass-fraction`, `1` for all of it). The first-pass indexing has already sampled the whole sweep; what pass 1 adds is integrated reflections for the post-refinement below, and half a sweep holds plenty of them while costing half the time. Its merge, post-refinement and smoothed mosaicity describe a sweep of those frames alone.
+1. **Pass 1** integrates, scales and merges at the header geometry — the first half of the sweep by default, but at least 120° of it, so a sweep up to 120° is pre-passed whole (`--prepass-fraction`, `1` for all of it). The first-pass indexing and the pre-scan have sampled the same part of the sweep, so nothing before pass 2 reads the rest of it - which is what lets a run that follows a collection still being written do all of this while the rest is collected; what pass 1 adds is integrated reflections for the post-refinement below, and half a sweep holds plenty of them while costing half the time. Its merge, post-refinement and smoothed mosaicity describe a sweep of those frames alone.
 2. From pass-1's integrated reflections, the crystal and the detector are refined together over all its frames (Ceres, robust loss) in **one joint fit**, against both residuals at once:
    - the **positional** detector↔reciprocal residual at each partial's observed spot, and
    - a distance-independent **Ewald excitation** residual at each reflection's observed rocking centroid $\phi_\mathrm{obs}$.

docs/RUGNUX_ADVANCED.md

@@ -342,7 +342,7 @@ For very long axes the direction grid's angular resolution binds well below 500
 | `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) |
 | `--force-rotation-lattice <vec>` | Force rotation lattice (9 floats, Å), skipping the first pass |
 | `--rotation-no-postrefine` | Rotation: disable the default-on two-pass geometry post-refine (see the rotation section) |
-| `--prepass-fraction <f>` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame |
+| `--prepass-fraction <f>` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep, but at least 120° of it (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame. The pre-scan and the first-pass indexing sample the same part of the sweep |
 | `--refine-geometry[=N\|off]` | Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills with a reference cell (`-C` / `-z`), `=off` disables |
 | `--index-ice-rings[=on\|off]` | Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: **off**; no effect without `--detect-ice-rings`, which does the flagging) |
 

docs/RUGNUX_OVERVIEW.md

@@ -8,7 +8,7 @@ rotation run with the defaults; stills differences are at the end.
 ([What Rugnux reads](RUGNUX_FORMATS.md)). A goniometer axis makes it a rotation run, none makes it
 serial stills — nothing is asked of the user.
 
-**Pre-scan.** A projection of frames spread over the sweep (60 by default) finds the beam-stop shadow and masks
+**Pre-scan.** A projection of frames (60 by default) spread over the first part of the sweep, the part the geometry pre-pass integrates, finds the beam-stop shadow and masks
 it ([§1.5](CPU_DATA_ANALYSIS_IMAGE.md)), masks pixels its frames show to be defective
 ([§1.6](CPU_DATA_ANALYSIS_IMAGE.md)), measures the beam centre from the isotropy of the
 scattered background and compares it with the file's ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)), and
@@ -35,7 +35,7 @@ nothing, gives an axis harmonic of the measured centre's lattice, or loses to it
 are merged ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)). A failed pass triggers the discrete rescues — the
 rotation-axis sign, the first-pass depth ladder, the long-axis rescue — before anything is given up on.
 
-**First integration pass.** At the geometry in the file, every frame of the first half of the sweep
+**First integration pass.** At the geometry in the file, every frame of the first half of the sweep, at least 120° of it
 (`--prepass-fraction`) is predicted
 ([§8](CPU_DATA_ANALYSIS_INTEGRATION.md)) and profile-fit integrated
 ([§9](CPU_DATA_ANALYSIS_INTEGRATION.md)); partials are combined into fulls, scaled and merged

docs/RUGNUX_REPORT.md

@@ -129,7 +129,7 @@ grep '^JFJOCH_DATASET_SETTINGS=' out_report.txt | cut -d' ' -f2- > geometry.json
 ```
 
 **Which pass.** A rotation run integrates twice — once at the geometry in the input file (the first
-half of the sweep, `--prepass-fraction`), then the whole sweep again at the post-refined geometry — and can integrate a third time if a guard rejects the second pass.
+half of the sweep but at least 120°, `--prepass-fraction`), then the whole sweep again at the post-refined geometry — and can integrate a third time if a guard rejects the second pass.
 There is **one** report, for the pass that became the canonical output, and `PASS=` /
 `PASS_DECISION=` (`--developer`) in section 1 say which pass that is and on what evidence, so no number in the file
 is ambiguous about which geometry produced it.

docs/RUGNUX_TUTORIAL.md

@@ -109,7 +109,7 @@ resolution) sharpens both the space-group search and the error model. To tune th
 treated as independent stills use `--force-still`.
 
 By default a rotation run also **post-refines the geometry** in a second pass: the first pass
-integrates and merges the first half of the sweep at the header geometry (`--prepass-fraction 1` for all of it), then the detector distance + beam centre and the crystal
+integrates and merges the first half of the sweep (at least 120° of it) at the header geometry (`--prepass-fraction 1` for all of it), then the detector distance + beam centre and the crystal
 cell / rotation-axis are refined against the merged fulls (cross-validated; the distance moves 1 % a
 step and a larger move is ratified by re-indexing, and the gauge-weak beam centre stays within 15 px of
 the centre the first pass ran at or of the run's own measured centre, whichever is nearer), and the second pass

rugnux/Rugnux.cpp

@@ -133,6 +133,9 @@ namespace {
     // Images the pre-scan reads when the beam-stop pre-pass is off and only the spot width is wanted;
     // with it on, that pre-pass's image count is used and one sample serves both.
     constexpr int BEAM_CENTER_PROJECTION_IMAGES = 60;
+
+    // The least rotation the two-pass geometry pre-pass integrates, whatever --prepass-fraction asks.
+    constexpr double PREPASS_MIN_ROTATION_DEG = 120.0;
 }
 
 namespace rugnux_internal {
@@ -541,6 +544,16 @@ std::vector<double> Rugnux::FirstPassInputKey() const {
     return k;
 }
 
+int Rugnux::PrepassEnd(int start_image, int end_image) const {
+    int end = start_image + static_cast<int>(std::lround((end_image - start_image) * config_.prepass_fraction));
+    // At least PREPASS_MIN_ROTATION_DEG of rotation, so a narrow sweep is pre-passed whole. The angle is
+    // the file's, so every pass of a run picks the same frames, before and after a rotation-scale fit.
+    if (const auto g = reader_.GetDataset()->experiment.GetGoniometer(); g && g->GetIncrement_deg() != 0.0f)
+        end = std::max(end, start_image + static_cast<int>(
+                std::lround(PREPASS_MIN_ROTATION_DEG / std::abs(g->GetIncrement_deg()))));
+    return std::min(end, end_image);
+}
+
 ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, bool geometry_prepass) {
     Logger logger("Rugnux");
     ProcessResult result;
@@ -628,10 +641,16 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
     if (full && config_.refine_geometry.has_value())
         RefineStillsGeometry(start_image, end_image, images_to_process, observer);
 
+    // The geometry pre-pass integrates only the first part of the sweep (pass 2 reads it all), and on a
+    // rotation sweep the decisions before pass 2 read only that part too (PrepassEnd).
+    const int prepass_end = PrepassEnd(start_image, end_image);
+    const int prepass_images = (prepass_end - start_image + config_.stride - 1) / config_.stride;
+    const int early_images = experiment_.IsRotationIndexing() ? prepass_images : images_to_process;
+
     // After any geometry refinement, so the shadow is found about the beam centre actually used,
     // and before the azimuthal mapping and the output mask below, which both read pixel_mask_.
     if (config_.detect_beam_stop.has_value() || config_.adaptive_integration_radius)
-        PreScan(start_image, images_to_process,
+        PreScan(start_image, early_images,
                 std::min(config_.detect_beam_stop.value_or(BEAM_CENTER_PROJECTION_IMAGES),
                          HotPixelFinder::MAX_SAMPLED_FRAMES), observer);
 
@@ -820,10 +839,6 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
 
     const auto start_time = std::chrono::steady_clock::now();
 
-    // The geometry pre-pass integrates only the first prepass_fraction of the sweep (pass 2 reads it all).
-    const int prepass_end = start_image
-            + static_cast<int>(std::lround((end_image - start_image) * config_.prepass_fraction));
-    const int prepass_images = (prepass_end - start_image + config_.stride - 1) / config_.stride;
     const int loop_end = geometry_prepass ? prepass_end : end_image;
 
     // Declared here rather than where they are first filled, so that the phases below can share them

rugnux/Rugnux.h

@@ -821,6 +821,13 @@ class Rugnux {
     // the defective pixels, and the recorded spot width that sets the integration radius
     // (config_.adaptive_integration_radius).
     void PreScan(int start_image, int images_to_process, int frame_count, RugnuxObserver *observer);
+    // The first part of a rotation sweep, [start_image, PrepassEnd): what the geometry pre-pass
+    // integrates, and the only frames the decisions taken before the canonical pass read - the
+    // pre-scan sample (mask, spot width, background beam centre) and the first pass's schemes and
+    // validation frames. So those decisions are made from frames that exist while the rest of the
+    // sweep is still being collected, and an offline run makes them from the same frames. It is the
+    // first config_.prepass_fraction of the sweep, but at least PREPASS_MIN_ROTATION_DEG (Rugnux.cpp).
+    [[nodiscard]] int PrepassEnd(int start_image, int end_image) const;
     // The pre-scan's defective-pixel mask (HotPixelFinder), read on `sample` with the rings drawn
     // about `ring_centre` where the background measured one.
     void MaskDefectivePixels(int start_image, const std::vector<int> &sample,

rugnux/RugnuxFirstPass.cpp

@@ -204,7 +204,8 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     auto &observer = p.observer;
     auto &geometry_prepass = p.geometry_prepass;
     auto &start_image = p.start_image;
-    auto &images_to_process = p.images_to_process;
+    // Every frame the first pass reads is in the first part of the sweep (Rugnux::PrepassEnd).
+    const int early_images = p.prepass_images;
     auto &dataset = p.dataset;
     auto &mapping = p.mapping;
     auto &indexer_pool = p.indexer_pool;
@@ -432,7 +433,7 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     // sub-lattice from an under-sampled scheme) indexes few frames, while the true cell indexes
     // many - this is the discriminator that lets the best scheme win.
     const std::vector<int> validation = select_equally_spaced_image_ordinals(
-        images_to_process, std::min(images_to_process, 60));
+        early_images, std::min(early_images, 60));
 
     const float index_tol = experiment_.GetIndexingSettings().GetTolerance();
 
@@ -612,16 +613,16 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     // Sampling schemes, tried in order of preference. We index each and keep the one that indexes
     // the most frames; the schemes cover complementary failure modes and no single one is best for
     // every crystal:
-    //  - "spread": images spread over the whole rotation - full angular range, coarse stride; the
-    //    long-standing default, best for typical crystals.
+    //  - "spread": images spread over the first part of the rotation (PrepassEnd) - wide angular range,
+    //    coarse stride; the long-standing default, best for typical crystals.
     //  - "wedge": consecutive frames from the clean start - native stride keeps each reflection's
     //    rocking curve continuous across frames, so the FFT can trace a long axis whose fine
     //    reciprocal spacing the coarse spread cannot resolve (e.g. an F-cubic cell viewed down
     //    [111]); bounded by the accumulation cap to the least radiation-damaged early wedge.
-    std::vector<int> all_consecutive(images_to_process);
+    std::vector<int> all_consecutive(early_images);
     std::iota(all_consecutive.begin(), all_consecutive.end(), 0);
     const std::vector<int> spread_ordinals = select_equally_spaced_image_ordinals(
-        images_to_process, config_.rotation_indexing_image_count);
+        early_images, config_.rotation_indexing_image_count);
     const std::vector<std::pair<std::string, std::vector<int>>> schemes = {
         {"spread", spread_ordinals},
         {"wedge",  all_consecutive},
@@ -1141,7 +1142,7 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) {
     {
         std::vector<int> upfront = spread_ordinals;
         upfront.insert(upfront.end(), validation.begin(), validation.end());
-        for (int i = 0; i < std::min(SPOT_PREFETCH_CHUNK, images_to_process); i++)
+        for (int i = 0; i < std::min(SPOT_PREFETCH_CHUNK, early_images); i++)
             upfront.push_back(i);
         std::sort(upfront.begin(), upfront.end());
         upfront.erase(std::unique(upfront.begin(), upfront.end()), upfront.end());

rugnux/RugnuxPreScan.cpp

@@ -170,10 +170,11 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
     if (!want_shadow && !want_width && !want_hot)
         return;
 
-    // The sample is taken from the middle of the sweep, both ends left out: an even spread over
-    // [0, images_to_process) always picks the very first and the very last image, and those are the
-    // ones a shutter out of step spoils - the last carries under 90 % of the sample's median counts
-    // on 12 of 38 regression sweeps and is essentially unexposed on 5.
+    // The sample is taken from the frames given - on a rotation sweep its first part (PrepassEnd) -
+    // both ends left out: an even spread over [0, images_to_process) always picks the very first and
+    // the very last image, and those are the ones a shutter out of step spoils - the last carries
+    // under 90 % of the sample's median counts on 12 of 38 regression sweeps and is essentially
+    // unexposed on 5.
     const int margin = std::min(PRESCAN_END_MARGIN_IMAGES, images_to_process / 10);
     auto shadow_sample = select_equally_spaced_image_ordinals(images_to_process - 2 * margin,
                                                               frame_count);

rugnux/rugnux_cli.cpp

@@ -148,7 +148,7 @@ void print_usage() {
     std::cout << "   --force-rotation-lattice <vec>   Force rotation indexer with external lattice (in Angstrom) : \"a0x,a0y,a0z,a1x,a1y,a1z,a2x,a2y,a2z\" (9 floats, skips first pass)" << std::endl;
     std::cout << "   --rotation-scale <k>             Goniometer rotation scale: the stage turned k times the angle stored in the file (the commanded one). Applied to both passes; overrides the fitted correction" << std::endl;
     std::cout << "   --rotation-no-postrefine         Disable the (default-on) two-pass rotation post-refine (post-refine detector distance/beam + cell/axis, then re-integrate; only the refined pass is written, as the canonical <prefix>_* output)" << std::endl;
-    std::cout << "   --prepass-fraction <f>           Integrate the first fraction f of the sweep in the two-pass rotation geometry pre-pass (0.05-1, default 0.5; 1 = the whole sweep); pass 2 reads every frame" << std::endl;
+    std::cout << "   --prepass-fraction <f>           Integrate the first fraction f of the sweep, but at least 120 deg of it, in the two-pass rotation geometry pre-pass (0.05-1, default 0.5; 1 = the whole sweep); pass 2 reads every frame. The pre-scan and the first-pass indexing sample the same part of the sweep" << std::endl;
     std::cout << "   -X, --indexing-algorithm <txt>   Indexing algorithm (FFBIDX|FFT|FFTW|Auto|None)" << std::endl;
     std::cout << "   -S, --space-group <num|symbol>   Space group number (96) or symbol (P43212) - for indexing and scaling" << std::endl;
     std::cout << "   -C, --unit-cell <cell>           Fix reference unit cell: \"a,b,c,alpha,beta,gamma\"" << std::endl;