mx/Jungfraujoch
3
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 81 Wiki Activity

Pinned host memory from a process-wide pool; the compressed chunk uploaded from pinned staging

Browse Source 
The GPU engines are built per worker per step, and each took its page-locked host buffers with
cudaHostAlloc / cudaHostRegister and gave them back with cudaFreeHost / cudaHostUnregister. Those
calls map and lock pages under the process's memory-map lock, so all workers queue behind each
other: on the 5080 workstation a myoglobin run made ~3400 of them (6-22 thread-s, depending on the
load), 9gdj ~10400 (124 thread-s); on a four-socket host they cost 20-30 ms each.

- CudaHostPtr takes its memory from a pool: a freed buffer is kept in a power-of-two size class for
  the next engine, up to 512 MiB in all; a buffer above 64 MiB is never kept. Rugnux::Run hands the
  pool back at its end.
- PinnedVector (std::vector with the pool's allocator) replaces the registered vectors of
  AdaptiveSpotFinderGPU and ImagePreprocessorGPU; the Bragg predictors download into a pooled
  pinned buffer and copy into their output, instead of registering it.
- BSLZ4DecoderGPU copies the compressed chunk into a pooled pinned buffer before the upload, so it
  is one DMA instead of the driver's pageable staging (9gdj: 163 GB pageable H2D, 52 s of copy
  time, -> 147 GB pinned, 13 s).

Measured with nsys on the workstation: myob 3398 -> 1396 calls (674 of them the release at the
end), cytc 2986 -> 1270, 9gdj 10428 -> 1748 (124 -> 3.7 thread-s). Output identical (p.hkl md5 on
myob, cytc, 9gdj; CI tier battery).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
This commit is contained in:
leonarski_f
2026-10-11 10:37:23 +02:00
co-authored by Claude Opus 5.5
parent 54f8e07591
commit 09c41d067d
13 changed files with 175 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/image_preprocessing/ImagePreprocessorGPU.h
+1 -2
View File
@@ -23,8 +23,7 @@ class ImagePreprocessorGPU : public ImagePreprocessor {
CudaDevicePtr<uint8_t> gpu_decompressed_image;
CudaDevicePtr<ImageStatistics> gpu_stats;
std::vector<ImageStatistics> cpu_stats;
CudaRegisteredVector<ImageStatistics> cpu_stats_reg;
PinnedVector<ImageStatistics> cpu_stats;
CudaRegisteredVector<uint8_t> input_reg; // page-locks the caller's decompression buffer
// The uncompressed-image counterpart of input_reg: the region PinInputRegion last page-locked.
const void *pinned_input = nullptr;