Find the first pass's spots on every worker
The first pass of rotation indexing feeds two sampling schemes and a validation set, and it found their spots one frame at a time on one CUDA stream. It reads 271 to 402 frames depending on the crystal, once per pass, and there are two passes. On a 16M-pixel dataset that is 6.7 s of a 30 s run at a mean occupancy of 1.9 threads out of 48. The comment said this was to keep the spot cache from depending on scheduling. The guarantee is stronger than that and survives: a frame's spots are a pure function of that frame, so whatever order the workers finish in, the cache ends up holding exactly what the serial loop put there. The feed order and the point at which the consecutive scheme stops accumulating are untouched, so the schemes see the same frames as before. The phase was never CPU-bound, which is why this works: its one thread was spin-waiting in the driver on the same six full-image passes, about 5.5 ms a frame. One engine per worker for the whole pass, not per batch. An engine on a 16M-pixel detector costs ~0.13 s to build - a first attempt rebuilt them per batch and came out SLOWER than the serial loop, four batches of eight engines against ~1.5 s of frame work. Two smaller things, in the same per-image path: The strong-pixel flagging kernel reads four pixels at a time and issues one atomicOr per four instead of per pixel. Same values, confirmed end to end. The beam-stop pre-scan sized its per-worker accumulators for the worker count rather than for the workers that use them. Two int64 arrays per pixel per shard, eight shards: 2.3 GB allocated and cleared on a 16M-pixel detector for a path the GPU never reads. They are allocated on use now, which is most of what made that phase look like eight workers with two of them running. Measured, three A/B pairs with the order alternated, whole run: 16M-pixel 3600-frame 42.8 s -> 38.8 s, 16M-pixel 1800-frame 38.7 s -> 36.1 s, both at two GPUs; a 4M-pixel control 23.3 s -> 23.1 s. First-pass indexing itself -44 % and -36 %. The two smaller datasets merge byte-identically over six runs each; the largest is not bit-reproducible in the unmodified binary either, and stayed inside that spread. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
a22a51372e
commit
debd73c0bd
@@ -302,16 +302,12 @@ ShadowFinder::ShadowFinder(const DiffractionExperiment &experiment, const PixelM
|
||||
#endif
|
||||
}
|
||||
|
||||
// A shard's accumulators are allocated when a frame is first added to it, not here: with a GPU they
|
||||
// are never used at all, and on a 16 Mpx detector eight of them are 2.9 GB to allocate and clear -
|
||||
// which measured 0.8 s of the pre-scan, all of it wasted.
|
||||
void ShadowFinder::SetShardCount(size_t n) {
|
||||
const size_t npixels = static_cast<size_t>(width) * height;
|
||||
shards.clear();
|
||||
shards.resize(std::max<size_t>(1, n));
|
||||
for (auto &p : shards) {
|
||||
p.max_value.assign(npixels, 0);
|
||||
p.sum_value.assign(npixels, 0);
|
||||
p.valid_count.assign(npixels, 0);
|
||||
p.frames = 0;
|
||||
}
|
||||
}
|
||||
|
||||
ShadowFinder::Projection ShadowFinder::Reduce() const {
|
||||
@@ -376,7 +372,8 @@ ShadowFinder::Projection ShadowFinder::Reduce() const {
|
||||
}
|
||||
}
|
||||
#endif
|
||||
for (const auto &p : shards)
|
||||
for (const auto &p : shards) {
|
||||
if (p.frames == 0) continue; // never used, and its accumulators were never allocated
|
||||
for (size_t i = lo; i < hi; i++) {
|
||||
if (p.valid_count[i] == 0)
|
||||
continue;
|
||||
@@ -385,6 +382,7 @@ ShadowFinder::Projection ShadowFinder::Reduce() const {
|
||||
out.sum_value[i] += p.sum_value[i];
|
||||
out.valid_count[i] += p.valid_count[i];
|
||||
}
|
||||
}
|
||||
}));
|
||||
}
|
||||
for (auto &f : futures) f.get();
|
||||
@@ -441,6 +439,12 @@ void ShadowFinder::AddImage(const DataMessage &data, std::vector<uint8_t> &buffe
|
||||
#endif
|
||||
|
||||
Projection &p = shards[shard];
|
||||
if (p.max_value.empty()) {
|
||||
const size_t npixels = static_cast<size_t>(width) * height;
|
||||
p.max_value.assign(npixels, 0);
|
||||
p.sum_value.assign(npixels, 0);
|
||||
p.valid_count.assign(npixels, 0);
|
||||
}
|
||||
const auto ptr = data.image.GetUncompressedPtr(buffer);
|
||||
switch (data.image.GetMode()) {
|
||||
case CompressedImageMode::Int8: Add(reinterpret_cast<const int8_t *>(ptr), p); break;
|
||||
|
||||
Reference in New Issue
Block a user