Find the first pass's spots on every worker

The first pass of rotation indexing feeds two sampling schemes and a validation set,
and it found their spots one frame at a time on one CUDA stream. It reads 271 to 402
frames depending on the crystal, once per pass, and there are two passes. On a 16M-pixel
dataset that is 6.7 s of a 30 s run at a mean occupancy of 1.9 threads out of 48.

The comment said this was to keep the spot cache from depending on scheduling. The
guarantee is stronger than that and survives: a frame's spots are a pure function of
that frame, so whatever order the workers finish in, the cache ends up holding exactly
what the serial loop put there. The feed order and the point at which the consecutive
scheme stops accumulating are untouched, so the schemes see the same frames as before.

The phase was never CPU-bound, which is why this works: its one thread was spin-waiting
in the driver on the same six full-image passes, about 5.5 ms a frame.

One engine per worker for the whole pass, not per batch. An engine on a 16M-pixel
detector costs ~0.13 s to build - a first attempt rebuilt them per batch and came out
SLOWER than the serial loop, four batches of eight engines against ~1.5 s of frame work.

Two smaller things, in the same per-image path:

The strong-pixel flagging kernel reads four pixels at a time and issues one atomicOr per
four instead of per pixel. Same values, confirmed end to end.

The beam-stop pre-scan sized its per-worker accumulators for the worker count rather than
for the workers that use them. Two int64 arrays per pixel per shard, eight shards: 2.3 GB
allocated and cleared on a 16M-pixel detector for a path the GPU never reads. They are
allocated on use now, which is most of what made that phase look like eight workers with
two of them running.

Measured, three A/B pairs with the order alternated, whole run: 16M-pixel 3600-frame
42.8 s -> 38.8 s, 16M-pixel 1800-frame 38.7 s -> 36.1 s, both at two GPUs; a 4M-pixel
control 23.3 s -> 23.1 s. First-pass indexing itself -44 % and -36 %. The two smaller
datasets merge byte-identically over six runs each; the largest is not bit-reproducible
in the unmodified binary either, and stayed inside that spread.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
jungfrau
2026-08-17 18:06:03 -04:00
co-authored by Claude Opus 5
parent a22a51372e
commit debd73c0bd
4 changed files with 167 additions and 33 deletions
+2 -1
View File
@@ -3,7 +3,8 @@
### Unreleased
* **rugnux: scaling and merging are much faster on crystals that integrate far beyond the resolution they merge at.** Observations outside the scaling resolution range are dropped as they are ingested instead of being scaled, combined and post-refined first, and the geometry post-refinement fits on a bounded sample of them. On a large-cell dataset merging at less than half the resolution its detector reaches, a run drops from 68 s to 37 s; merged statistics are unchanged.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector rather than the reflections that happen to be present, so whether it is applied no longer depends on how far integration reached. On a crystal where it was being refused, merged R_meas improves by 4 percentage points.
* rugnux: the beam-stop pre-scan reads and accumulates its frames on several threads instead of one. On a 16M-pixel rotation dataset it drops from 7.9 s to 4.8 s; the shadow it finds is unchanged.
* rugnux: the first-pass rotation indexing finds its spots on every worker rather than one, which is worth most on large detectors - on a 16M-pixel dataset that phase drops by 44% and the whole run by 9%. The lattice it picks is unchanged.
* rugnux: the beam-stop pre-scan reads and accumulates its frames on several threads instead of one. On a 16M-pixel rotation dataset it drops from 7.9 s to 4.8 s; the shadow it finds is unchanged. Its per-worker accumulators are now allocated only where they are used, saving 2.3 GB of host memory on a 16M-pixel detector.
### 1.0.0-rc.161
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.