diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index 899ce9ba6..4fa756e0d 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -212,7 +212,7 @@ The lookup can also land *short*: near the Niggli type-I/type-II boundary the ch The rotation first pass refines the **twelve** best candidate lattices rather than four. The pre-refinement indexed fraction is an unreliable ranking, so a correct cell can sit below several degenerate ones and never be refined at all. -For rotation data the first pass therefore refines the constrained cell *and* an unconstrained (triclinic) one from the same spots — which it finds itself, over a sample spread across the sweep, rather than reading what the acquisition wrote — and settles the two on how many of a fixed set of validation frames each actually indexes. The bar is a clear majority, not a margin. An unconstrained refinement holds no cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it indexes a few more — a percentage margin therefore demotes real lattices (measured: a genuine $I$-centred orthorhombic to $P1$). Only a constrained cell that fails outright while its unconstrained cell works distinguishes a false promotion. The intensities settle the space group later regardless (§13). +For rotation data the first pass therefore refines the constrained cell *and* an unconstrained (triclinic) one from the same spots — which it finds itself, over a sample spread across the first part of the sweep (the part the geometry pre-pass integrates), rather than reading what the acquisition wrote — and settles the two on how many of a fixed set of validation frames each actually indexes. The bar is a clear majority, not a margin. An unconstrained refinement holds no cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it indexes a few more — a percentage margin therefore demotes real lattices (measured: a genuine $I$-centred orthorhombic to $P1$). Only a constrained cell that fails outright while its unconstrained cell works distinguishes a false promotion. The intensities settle the space group later regardless (§13). Two further hypotheses are weighed at the same point, both by default. Where two first-pass answers have primitive volumes in a small integer ratio (2–4×) and tie on the validation frames, the frame @@ -315,7 +315,7 @@ So every round is scored on the **widest** gate — the population the first pas The refinement above (§7.2) runs per image against that image's spots. For rotation data an additional **post-refinement** (on by default; `--rotation-no-postrefine` disables it) improves the detector distance, beam centre and crystal cell/axis using **all** frames at once, then re-integrates: -1. **Pass 1** integrates, scales and merges at the header geometry — the first half of the sweep by default (`--prepass-fraction`, `1` for all of it). The first-pass indexing has already sampled the whole sweep; what pass 1 adds is integrated reflections for the post-refinement below, and half a sweep holds plenty of them while costing half the time. Its merge, post-refinement and smoothed mosaicity describe a sweep of those frames alone. +1. **Pass 1** integrates, scales and merges at the header geometry — the first half of the sweep by default, but at least 120° of it, so a sweep up to 120° is pre-passed whole (`--prepass-fraction`, `1` for all of it). The first-pass indexing and the pre-scan have sampled the same part of the sweep, so nothing before pass 2 reads the rest of it - which is what lets a run that follows a collection still being written do all of this while the rest is collected; what pass 1 adds is integrated reflections for the post-refinement below, and half a sweep holds plenty of them while costing half the time. Its merge, post-refinement and smoothed mosaicity describe a sweep of those frames alone. 2. From pass-1's integrated reflections, the crystal and the detector are refined together over all its frames (Ceres, robust loss) in **one joint fit**, against both residuals at once: - the **positional** detector↔reciprocal residual at each partial's observed spot, and - a distance-independent **Ewald excitation** residual at each reflection's observed rocking centroid $\phi_\mathrm{obs}$. diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 4b3956f5c..2a42120c3 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -342,7 +342,7 @@ For very long axes the direction grid's angular resolution binds well below 500 | `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) | | `--force-rotation-lattice ` | Force rotation lattice (9 floats, Å), skipping the first pass | | `--rotation-no-postrefine` | Rotation: disable the default-on two-pass geometry post-refine (see the rotation section) | -| `--prepass-fraction ` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame | +| `--prepass-fraction ` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep, but at least 120° of it (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame. The pre-scan and the first-pass indexing sample the same part of the sweep | | `--refine-geometry[=N\|off]` | Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills with a reference cell (`-C` / `-z`), `=off` disables | | `--index-ice-rings[=on\|off]` | Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: **off**; no effect without `--detect-ice-rings`, which does the flagging) | diff --git a/docs/RUGNUX_OVERVIEW.md b/docs/RUGNUX_OVERVIEW.md index b2276095f..3feeed413 100644 --- a/docs/RUGNUX_OVERVIEW.md +++ b/docs/RUGNUX_OVERVIEW.md @@ -8,7 +8,7 @@ rotation run with the defaults; stills differences are at the end. ([What Rugnux reads](RUGNUX_FORMATS.md)). A goniometer axis makes it a rotation run, none makes it serial stills — nothing is asked of the user. -**Pre-scan.** A projection of frames spread over the sweep (60 by default) finds the beam-stop shadow and masks +**Pre-scan.** A projection of frames (60 by default) spread over the first part of the sweep, the part the geometry pre-pass integrates, finds the beam-stop shadow and masks it ([§1.5](CPU_DATA_ANALYSIS_IMAGE.md)), masks pixels its frames show to be defective ([§1.6](CPU_DATA_ANALYSIS_IMAGE.md)), measures the beam centre from the isotropy of the scattered background and compares it with the file's ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)), and @@ -35,7 +35,7 @@ nothing, gives an axis harmonic of the measured centre's lattice, or loses to it are merged ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)). A failed pass triggers the discrete rescues — the rotation-axis sign, the first-pass depth ladder, the long-axis rescue — before anything is given up on. -**First integration pass.** At the geometry in the file, every frame of the first half of the sweep +**First integration pass.** At the geometry in the file, every frame of the first half of the sweep, at least 120° of it (`--prepass-fraction`) is predicted ([§8](CPU_DATA_ANALYSIS_INTEGRATION.md)) and profile-fit integrated ([§9](CPU_DATA_ANALYSIS_INTEGRATION.md)); partials are combined into fulls, scaled and merged diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 5fdeb5899..ffaacd498 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -129,7 +129,7 @@ grep '^JFJOCH_DATASET_SETTINGS=' out_report.txt | cut -d' ' -f2- > geometry.json ``` **Which pass.** A rotation run integrates twice — once at the geometry in the input file (the first -half of the sweep, `--prepass-fraction`), then the whole sweep again at the post-refined geometry — and can integrate a third time if a guard rejects the second pass. +half of the sweep but at least 120°, `--prepass-fraction`), then the whole sweep again at the post-refined geometry — and can integrate a third time if a guard rejects the second pass. There is **one** report, for the pass that became the canonical output, and `PASS=` / `PASS_DECISION=` (`--developer`) in section 1 say which pass that is and on what evidence, so no number in the file is ambiguous about which geometry produced it. diff --git a/docs/RUGNUX_TUTORIAL.md b/docs/RUGNUX_TUTORIAL.md index 1dc615133..213fe8db5 100644 --- a/docs/RUGNUX_TUTORIAL.md +++ b/docs/RUGNUX_TUTORIAL.md @@ -109,7 +109,7 @@ resolution) sharpens both the space-group search and the error model. To tune th treated as independent stills use `--force-still`. By default a rotation run also **post-refines the geometry** in a second pass: the first pass -integrates and merges the first half of the sweep at the header geometry (`--prepass-fraction 1` for all of it), then the detector distance + beam centre and the crystal +integrates and merges the first half of the sweep (at least 120° of it) at the header geometry (`--prepass-fraction 1` for all of it), then the detector distance + beam centre and the crystal cell / rotation-axis are refined against the merged fulls (cross-validated; the distance moves 1 % a step and a larger move is ratified by re-indexing, and the gauge-weak beam centre stays within 15 px of the centre the first pass ran at or of the run's own measured centre, whichever is nearer), and the second pass diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 79ad0f48a..919861e05 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -133,6 +133,9 @@ namespace { // Images the pre-scan reads when the beam-stop pre-pass is off and only the spot width is wanted; // with it on, that pre-pass's image count is used and one sample serves both. constexpr int BEAM_CENTER_PROJECTION_IMAGES = 60; + + // The least rotation the two-pass geometry pre-pass integrates, whatever --prepass-fraction asks. + constexpr double PREPASS_MIN_ROTATION_DEG = 120.0; } namespace rugnux_internal { @@ -541,6 +544,16 @@ std::vector Rugnux::FirstPassInputKey() const { return k; } +int Rugnux::PrepassEnd(int start_image, int end_image) const { + int end = start_image + static_cast(std::lround((end_image - start_image) * config_.prepass_fraction)); + // At least PREPASS_MIN_ROTATION_DEG of rotation, so a narrow sweep is pre-passed whole. The angle is + // the file's, so every pass of a run picks the same frames, before and after a rotation-scale fit. + if (const auto g = reader_.GetDataset()->experiment.GetGoniometer(); g && g->GetIncrement_deg() != 0.0f) + end = std::max(end, start_image + static_cast( + std::lround(PREPASS_MIN_ROTATION_DEG / std::abs(g->GetIncrement_deg())))); + return std::min(end, end_image); +} + ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, bool geometry_prepass) { Logger logger("Rugnux"); ProcessResult result; @@ -628,10 +641,16 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b if (full && config_.refine_geometry.has_value()) RefineStillsGeometry(start_image, end_image, images_to_process, observer); + // The geometry pre-pass integrates only the first part of the sweep (pass 2 reads it all), and on a + // rotation sweep the decisions before pass 2 read only that part too (PrepassEnd). + const int prepass_end = PrepassEnd(start_image, end_image); + const int prepass_images = (prepass_end - start_image + config_.stride - 1) / config_.stride; + const int early_images = experiment_.IsRotationIndexing() ? prepass_images : images_to_process; + // After any geometry refinement, so the shadow is found about the beam centre actually used, // and before the azimuthal mapping and the output mask below, which both read pixel_mask_. if (config_.detect_beam_stop.has_value() || config_.adaptive_integration_radius) - PreScan(start_image, images_to_process, + PreScan(start_image, early_images, std::min(config_.detect_beam_stop.value_or(BEAM_CENTER_PROJECTION_IMAGES), HotPixelFinder::MAX_SAMPLED_FRAMES), observer); @@ -820,10 +839,6 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b const auto start_time = std::chrono::steady_clock::now(); - // The geometry pre-pass integrates only the first prepass_fraction of the sweep (pass 2 reads it all). - const int prepass_end = start_image - + static_cast(std::lround((end_image - start_image) * config_.prepass_fraction)); - const int prepass_images = (prepass_end - start_image + config_.stride - 1) / config_.stride; const int loop_end = geometry_prepass ? prepass_end : end_image; // Declared here rather than where they are first filled, so that the phases below can share them diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index eff26a3e7..97b91a4de 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -821,6 +821,13 @@ class Rugnux { // the defective pixels, and the recorded spot width that sets the integration radius // (config_.adaptive_integration_radius). void PreScan(int start_image, int images_to_process, int frame_count, RugnuxObserver *observer); + // The first part of a rotation sweep, [start_image, PrepassEnd): what the geometry pre-pass + // integrates, and the only frames the decisions taken before the canonical pass read - the + // pre-scan sample (mask, spot width, background beam centre) and the first pass's schemes and + // validation frames. So those decisions are made from frames that exist while the rest of the + // sweep is still being collected, and an offline run makes them from the same frames. It is the + // first config_.prepass_fraction of the sweep, but at least PREPASS_MIN_ROTATION_DEG (Rugnux.cpp). + [[nodiscard]] int PrepassEnd(int start_image, int end_image) const; // The pre-scan's defective-pixel mask (HotPixelFinder), read on `sample` with the rings drawn // about `ring_centre` where the background measured one. void MaskDefectivePixels(int start_image, const std::vector &sample, diff --git a/rugnux/RugnuxFirstPass.cpp b/rugnux/RugnuxFirstPass.cpp index 3cbd61e8b..11eb9898d 100644 --- a/rugnux/RugnuxFirstPass.cpp +++ b/rugnux/RugnuxFirstPass.cpp @@ -204,7 +204,8 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) { auto &observer = p.observer; auto &geometry_prepass = p.geometry_prepass; auto &start_image = p.start_image; - auto &images_to_process = p.images_to_process; + // Every frame the first pass reads is in the first part of the sweep (Rugnux::PrepassEnd). + const int early_images = p.prepass_images; auto &dataset = p.dataset; auto &mapping = p.mapping; auto &indexer_pool = p.indexer_pool; @@ -432,7 +433,7 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) { // sub-lattice from an under-sampled scheme) indexes few frames, while the true cell indexes // many - this is the discriminator that lets the best scheme win. const std::vector validation = select_equally_spaced_image_ordinals( - images_to_process, std::min(images_to_process, 60)); + early_images, std::min(early_images, 60)); const float index_tol = experiment_.GetIndexingSettings().GetTolerance(); @@ -612,16 +613,16 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) { // Sampling schemes, tried in order of preference. We index each and keep the one that indexes // the most frames; the schemes cover complementary failure modes and no single one is best for // every crystal: - // - "spread": images spread over the whole rotation - full angular range, coarse stride; the - // long-standing default, best for typical crystals. + // - "spread": images spread over the first part of the rotation (PrepassEnd) - wide angular range, + // coarse stride; the long-standing default, best for typical crystals. // - "wedge": consecutive frames from the clean start - native stride keeps each reflection's // rocking curve continuous across frames, so the FFT can trace a long axis whose fine // reciprocal spacing the coarse spread cannot resolve (e.g. an F-cubic cell viewed down // [111]); bounded by the accumulation cap to the least radiation-damaged early wedge. - std::vector all_consecutive(images_to_process); + std::vector all_consecutive(early_images); std::iota(all_consecutive.begin(), all_consecutive.end(), 0); const std::vector spread_ordinals = select_equally_spaced_image_ordinals( - images_to_process, config_.rotation_indexing_image_count); + early_images, config_.rotation_indexing_image_count); const std::vector>> schemes = { {"spread", spread_ordinals}, {"wedge", all_consecutive}, @@ -1141,7 +1142,7 @@ bool Rugnux::FirstPassRotationIndexing(PipelineLocals &p) { { std::vector upfront = spread_ordinals; upfront.insert(upfront.end(), validation.begin(), validation.end()); - for (int i = 0; i < std::min(SPOT_PREFETCH_CHUNK, images_to_process); i++) + for (int i = 0; i < std::min(SPOT_PREFETCH_CHUNK, early_images); i++) upfront.push_back(i); std::sort(upfront.begin(), upfront.end()); upfront.erase(std::unique(upfront.begin(), upfront.end()), upfront.end()); diff --git a/rugnux/RugnuxPreScan.cpp b/rugnux/RugnuxPreScan.cpp index 12f9cccd3..30bc7ae46 100644 --- a/rugnux/RugnuxPreScan.cpp +++ b/rugnux/RugnuxPreScan.cpp @@ -170,10 +170,11 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru if (!want_shadow && !want_width && !want_hot) return; - // The sample is taken from the middle of the sweep, both ends left out: an even spread over - // [0, images_to_process) always picks the very first and the very last image, and those are the - // ones a shutter out of step spoils - the last carries under 90 % of the sample's median counts - // on 12 of 38 regression sweeps and is essentially unexposed on 5. + // The sample is taken from the frames given - on a rotation sweep its first part (PrepassEnd) - + // both ends left out: an even spread over [0, images_to_process) always picks the very first and + // the very last image, and those are the ones a shutter out of step spoils - the last carries + // under 90 % of the sample's median counts on 12 of 38 regression sweeps and is essentially + // unexposed on 5. const int margin = std::min(PRESCAN_END_MARGIN_IMAGES, images_to_process / 10); auto shadow_sample = select_equally_spaced_image_ordinals(images_to_process - 2 * margin, frame_count); diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 962c17e45..90dab4842 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -148,7 +148,7 @@ void print_usage() { std::cout << " --force-rotation-lattice Force rotation indexer with external lattice (in Angstrom) : \"a0x,a0y,a0z,a1x,a1y,a1z,a2x,a2y,a2z\" (9 floats, skips first pass)" << std::endl; std::cout << " --rotation-scale Goniometer rotation scale: the stage turned k times the angle stored in the file (the commanded one). Applied to both passes; overrides the fitted correction" << std::endl; std::cout << " --rotation-no-postrefine Disable the (default-on) two-pass rotation post-refine (post-refine detector distance/beam + cell/axis, then re-integrate; only the refined pass is written, as the canonical _* output)" << std::endl; - std::cout << " --prepass-fraction Integrate the first fraction f of the sweep in the two-pass rotation geometry pre-pass (0.05-1, default 0.5; 1 = the whole sweep); pass 2 reads every frame" << std::endl; + std::cout << " --prepass-fraction Integrate the first fraction f of the sweep, but at least 120 deg of it, in the two-pass rotation geometry pre-pass (0.05-1, default 0.5; 1 = the whole sweep); pass 2 reads every frame. The pre-scan and the first-pass indexing sample the same part of the sweep" << std::endl; std::cout << " -X, --indexing-algorithm Indexing algorithm (FFBIDX|FFT|FFTW|Auto|None)" << std::endl; std::cout << " -S, --space-group Space group number (96) or symbol (P43212) - for indexing and scaling" << std::endl; std::cout << " -C, --unit-cell Fix reference unit cell: \"a,b,c,alpha,beta,gamma\"" << std::endl;