mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases 67 Wiki Activity

bragg_integration: grow the GPU reflection arrays with slack

Browse Source 
EnsureCapacity resized its 13 device arrays to exactly the current image's
predicted-reflection count, so every image that set a new record freed and
reallocated all of them. cudaMalloc and cudaFree take a device-wide lock in the
CUDA driver, so those images stalled every other worker: sampling the worker
threads during the per-image loop found 21-24 of 32 parked in cuMemAlloc_v2 or
cuMemFree_v2, all called from this one function, and the running maximum makes
32 workers do far more allocator work than one does.

Grow by half again instead. All transfers and kernel launches are sized by the
per-image reflection count rather than by the capacity, and the member is
already documented as holding at least that many, so over-allocating changes no
result. On an 18 Mpx rotation set the integration stage drops from 1.37 to
1.25 ms per image at 32 workers; merged statistics, error model and adopted
space group are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
leonarski_f
2026-08-03 01:02:07 +02:00
co-authored by Claude Opus 5
parent 6e4c0ce202
commit 13aa20a528
1 changed files with 22 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/bragg_integration/BraggIntegrationEngineGPU.cu
+22 -18
View File
@@ -434,25 +434,29 @@ BraggIntegrationEngineGPU::BraggIntegrationEngineGPU(const DiffractionExperiment
void BraggIntegrationEngineGPU::EnsureCapacity(size_t n) {
if (n <= capacity)
return;
d_px_x = CudaDevicePtr<float>(n);
d_px_y = CudaDevicePtr<float>(n);
d_d = CudaDevicePtr<float>(n);
d_cx = CudaDevicePtr<int>(n);
d_cy = CudaDevicePtr<int>(n);
d_I = CudaDevicePtr<float>(n);
d_sigma = CudaDevicePtr<float>(n);
d_bkg = CudaDevicePtr<float>(n);
d_obs_x = CudaDevicePtr<float>(n);
d_obs_y = CudaDevicePtr<float>(n);
d_ok = CudaDevicePtr<uint8_t>(n);
d_strong = CudaDevicePtr<uint8_t>(n);
d_has_obs = CudaDevicePtr<uint8_t>(n);
// Grow with slack. cudaMalloc/cudaFree take a device-wide lock in the driver, so growing to
// exactly n makes every image that sets a new reflection-count record stall all other workers.
const size_t new_capacity = std::max(n, capacity + capacity / 2);
h_px_x.resize(n); h_px_y.resize(n); h_d.resize(n);
h_I.resize(n); h_sigma.resize(n); h_bkg.resize(n);
h_obs_x.resize(n); h_obs_y.resize(n);
h_ok.resize(n); h_has_obs.resize(n);
capacity = n;
d_px_x = CudaDevicePtr<float>(new_capacity);
d_px_y = CudaDevicePtr<float>(new_capacity);
d_d = CudaDevicePtr<float>(new_capacity);
d_cx = CudaDevicePtr<int>(new_capacity);
d_cy = CudaDevicePtr<int>(new_capacity);
d_I = CudaDevicePtr<float>(new_capacity);
d_sigma = CudaDevicePtr<float>(new_capacity);
d_bkg = CudaDevicePtr<float>(new_capacity);
d_obs_x = CudaDevicePtr<float>(new_capacity);
d_obs_y = CudaDevicePtr<float>(new_capacity);
d_ok = CudaDevicePtr<uint8_t>(new_capacity);
d_strong = CudaDevicePtr<uint8_t>(new_capacity);
d_has_obs = CudaDevicePtr<uint8_t>(new_capacity);
h_px_x.resize(new_capacity); h_px_y.resize(new_capacity); h_d.resize(new_capacity);
h_I.resize(new_capacity); h_sigma.resize(new_capacity); h_bkg.resize(new_capacity);
h_obs_x.resize(new_capacity); h_obs_y.resize(new_capacity);
h_ok.resize(new_capacity); h_has_obs.resize(new_capacity);
capacity = new_capacity;
}
std::vector<Reflection> BraggIntegrationEngineGPU::Run(const ImagePreprocessorBuffer &image,