The pre-scan decompressed its frames on the host and folded them into a
per-pixel projection there. On a 16M-pixel detector that is 60 frames of 72 MB
to decompress and 20 bytes per pixel to read and write back per frame - about
40 GB of memory traffic - and it was the whole cost of the phase once the mask
was no longer the bottleneck.
Only the compressed chunk crosses PCIe now. BSLZ4DecoderGPU already exposes the
raw decoded bytes (Decode(), the path its own tests use), which is what this
needs: the projection is defined on the RAW STORED COUNTS with the pixel type's
sentinel skipped, not on the preprocessed image, so nothing here goes through
the preprocessor. Sums, maxima and counts are integers, so the device result is
identical to the host's rather than merely close.
Frames are folded in batches of four. The fold reads and writes the whole
accumulator whatever the batch holds, so per frame it was spending most of the
bandwidth on the accumulator rather than on the data; four is where that stops
mattering, and every frame beyond it is another full frame of device memory,
which costs more in cudaMalloc - device-synchronizing - than it saves.
The accumulator is built on a thread of its own. It allocates and clears
several hundred megabytes, and doing that in the constructor stalled the caller
before it had read its first frame.
Frames the device cannot take - anything but bitshuffle+LZ4 - still go to a host
shard, so a run mixing compressions needs no second code path, and a build
without CUDA is unchanged.
RotationScaleMergeGPU set the CUDA device in its constructor and never put it
back. CUDA's current device is per-thread, so that silently re-pinned the
calling thread for the rest of its life, and the destructor freed several
gigabytes against whatever device happened to be current by then - CudaDevicePtr
records no device of its own. Every entry point now sets the device on entry and
restores it on exit.
ParallelFor/ParallelChunks moved to common/ParallelFor.h; two files had copies
and a third wants them.
Measured on a 16M-pixel rotation dataset: pre-scan 4.78 s -> 2.37 s -> ~2.0 s,
shadow unchanged at 139126 pixels (22143 on a 2M-pixel dataset). Full 24-crystal
battery: same space group on all 24, none failed, 15m32s -> 14m49s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
