Jungfraujoch/NUMA_GPU_REVIEW.md

NUMA / GPU usage — current state, goals, proposed changes (for review)

Working note to agree on direction before modifying ImageBuffer / NUMAHWPolicy and the GPU-dispatch logic. Nothing here is implemented yet. File:line anchors are from branch 2606-pixel-refine.

1. Current state (the mind map)

1a. ImageBuffer — the big RAM ring buffer

• One instance, a member of the receiver: JFJochReceiverService::image_buffer (receiver/JFJochReceiverService.h:21), sized image_buffer_MiB from broker config (broker/jfjoch_broker.cpp:104 → JFJochReceiverService ctor receiver/JFJochReceiverService.cpp:15, image_buffer(send_buffer_size_MiB*1024*1024)). This is the 150–200 GB allocation.
• Allocated + zeroed in the ctor (common/ImageBuffer.cpp): numa_alloc_interleaved if libnuma, else std::malloc; then a single-threaded memset to pre-fault every page (deliberate — kills first-use page-fault latency in the hot path). Happens once at broker startup.
• Producers/consumers: receiver/decompression threads write frames into slots; consumers are preview/TIFF/JPEG/HTTP retrieval (GetImage) and the ZMQ/file sender. Access is random and unpinned (any thread → any slot).
• Not used by jfjoch_process or jfjoch_viewer — they read HDF5 through the reader, never instantiate ImageBuffer. So this buffer is broker/receiver-only (it only needs to compile for the viewer).

1b. NUMAHWPolicy — bundles three concerns per worker thread

Built from the broker config numa_policy string (e.g. n2g2, n8g4, n8g4_hbm) into a table of NUMABinding{cpu_node, mem_node, gpu}; GetBinding(thread) = bindings[thread % nbindings] (round-robin, common/NUMAHWPolicy.cpp:54). Bind(thread) does all three at once (common/NUMAHWPolicy.cpp:61):

1. CPU pin — RunOnNode / numa_run_on_node
2. Memory bind — MemOnNode / numa_set_membind (on n8g4_hbm, mem_node = i+8 → binds to HBM nodes)
3. GPU select — SelectGPU / set_gpu (= cudaSetDevice, not libnuma)

Call sites:

• receiver/JFJochReceiverFPGA.cpp:180/217/264 — data-stream threads RunOnNode(FPGA's NUMA node) (device-locality pin, the NIC-era idea applied to the FPGA card).
• receiver/JFJochReceiverFPGA.cpp:299, receiver/JFJochReceiverLite.cpp:234 — analysis worker threads numa_policy.Bind(threadid) → cpu + mem + GPU per the policy table.
• image_analysis/indexing/IndexerThreadPool.cpp:34 — each indexer thread SelectGPUAndItsNUMA(threadid % gpu_count) (GPU round-robin + that GPU's own NUMA node; independent of the numa_policy table).

libnuma is used in exactly three files: NUMAHWPolicy.cpp (the above), ImageBuffer.cpp (numa_alloc_interleaved/numa_free), and CUDAWrapper.cpp (one numa_node lookup).

1c. GPU dispatch — and why jfjoch_process underuses GPUs

• get_gpu_count() = cudaGetDeviceCount() (common/CUDAWrapper.*), so it already honours CUDA_VISIBLE_DEVICES. All dispatch is % gpu_count.
• Broker/receiver: worker threads spread over GPUs via numa_policy.Bind (the gpu field) + the indexer pool via threadid % gpu_count. → uses all visible GPUs.
• jfjoch_process: its worker lambda (tools/jfjoch_process.cpp:849, launched nthreads times) constructs MXAnalysisWithoutFPGA but never calls Bind/SelectGPU, and MXAnalysisWithoutFPGA itself does not select a device (image_analysis/MXAnalysisWithoutFPGA.cpp:38 just builds GPU engines on the current device). → all per-image preprocessing / spot-finding / azimuthal integration run on GPU 0. Only the indexer pool spreads. This is the root cause of "jfjoch_process doesn't use all GPUs."

2. Goals

• G1 — multiple brokers, disjoint GPUs. Run >1 jfjoch_broker on one machine, each confined to a subset of GPUs, with the code transparently using "all it can see" (no hard-coded indices). Pure workload control, no security requirement.
• G2 — jfjoch_process should use all visible GPUs, not just GPU 0.
• G3 — drop the libnuma dependency if it doesn't cost real performance (annoying dep; also a blocker for the long-term Windows/MSVC viewer).
• G4 — reassess whether NUMA CPU/mem pinning is still worth it given the FPGA pipeline (DMA into kernel-mmap'd buffers, negligible IRQ traffic) rather than the old network-RX model.

3. Proposed changes

• G1 (zero code): launch each broker under CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=<subset> jfjoch_broker …. get_gpu_count()/% gpu_count already do the rest. Action item: document this (deployment note) and set CUDA_DEVICE_ORDER=PCI_BUS_ID so indices are stable across boots.

• G2 (DONE): added pin_gpu() to CUDAWrapper — a process-wide round-robin counter (counter++ % get_gpu_count(), no thread id needed, no-op when no GPU). The jfjoch_process worker calls it once before building MXAnalysisWithoutFPGA, so each worker's CUDA streams/engines land on a distinct device. Caller-agnostic and reusable by other thread pools later.

• G3 / ImageBuffer: replace numa_alloc_interleaved + single-threaded memset with plain malloc + a parallel first-touch memset (N threads, unpinned). Threads spread by the scheduler → balanced placement ≈ interleave for random access, and faster startup, and no libnuma. Safe here because the buffer is 30–40 % of RAM (first-touch spills to the other node if one fills; no OOM; just check vm.zone_reclaim_mode == 0). Deterministic per-page interleave (if ever needed under tight RAM) is a raw mbind(MPOL_INTERLEAVE) syscall — still libnuma-free.

• G3 / G4 / NUMAHWPolicy: split the bundled concerns:

  • CPU pin (RunOnNode) — likely drop for the FPGA path (G4). If ever wanted back, use sched_setaffinity (no libnuma).
  • GPU select (SelectGPU/SelectGPUAndItsNUMA) — keep; already cudaSetDevice, no libnuma.
  • Memory bind (MemOnNode) — drop. (HBM is out of scope: the only Xeon MAX box didn't pay off and isn't worth special-casing, so no mbind path is needed — treat every host as plain multi-socket.)
  • CUDAWrapper numa_node — read the GPU PCIe device's /sys/.../numa_node instead of libnuma.
  • Result: NUMA_LIBRARY leaves the CMake entirely.

4. Open questions / to validate before deleting anything

• G4 is empirical. A/B at production frame rate (pinning on vs off): sustained throughput, dropped frames, latency jitter. The reasoning predicts "no regression," but measure on one real box first — production systems are currently tuned around this.
• Confirm vm.zone_reclaim_mode is 0 on the broker hosts (else first-touch reclaims locally before spilling → latency stalls).
• Parallel first-touch placement is approximate (depends on the scheduler spreading the zeroing threads); fine with RAM headroom, but note it's not the guaranteed 50/50 of mbind interleave.

5. Suggested order (low-risk first)

1. G1 — document CUDA_VISIBLE_DEVICES launch (no code).
2. G2 — per-worker GPU pin in jfjoch_process. DONE (pin_gpu()).
3. ImageBuffer parallel first-touch (drops one libnuma user, helps startup; stands alone).
4. G4 A/B on a real broker; if clean, drop CPU pinning.
5. NUMAHWPolicy simplify (keep GPU select, drop CPU pin + mem bind) + CUDAWrapper sysfs → remove NUMA_LIBRARY from CMake.