The pre-scan read its sample of frames in a plain serial loop: one thread
did the HDF5 read, the decompression and the full-detector accumulation for
every frame. The cost is fixed per frame rather than per dataset, so it grew
straight with detector area - measured at 0.9 s on a 2M-pixel detector and
7.9 s on a 16M-pixel one, where it was 11% of the whole run with 47 of 48
cores idle.
Frames are now read on several workers. ShadowFinder keeps one projection per
worker so nothing is locked while an image is added, and the projections are
summed when the mask is read; the sums and counts are integers, so the result
does not depend on how the frames were spread over the workers. A shard that
never counted a pixel is skipped when the maxima are merged - it holds 0,
which would otherwise beat a genuinely negative maximum.
Worker count is capped (PRESCAN_MAX_WORKERS): a shard costs 20 bytes per
pixel, and the accumulation is memory-bound, so a handful of workers already
saturates it.
The beam-centre spot pool is stitched together in sample order after the
workers join, so frame numbering and the spot list are what the serial read
produced regardless of how the workers interleaved. A frame still joins the
pool only if it could be read.
ShadowFinder::AddImage took its decompression scratch buffer BY VALUE, so the
caller's buffer stayed empty and every frame allocated and zero-filled a fresh
full-size uncompressed image (72 MB on a 16M-pixel detector) and freed it
again. It takes a reference now, and each worker reuses one buffer.
Measured on a 16M-pixel rotation dataset: pre-scan 7.9 s -> 4.8 s, whole run
69.2 s -> 65.1 s. Results are unchanged - same shadow pixel count, same space
group, same merged reflection count and merging statistics on both a 16M and a
2M-pixel dataset, and the beam-centre path still commits the same centre.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
