@@ -0,0 +1,115 @@
# NUMA / GPU usage — current state, goals, proposed changes (for review)
Working note to agree on direction **before ** modifying `ImageBuffer` / `NUMAHWPolicy` and the
GPU-dispatch logic. Nothing here is implemented yet. File:line anchors are from branch
`2606-pixel-refine` .
## 1. Current state (the mind map)
### 1a. `ImageBuffer` — the big RAM ring buffer
- **One instance**, a member of the receiver: `JFJochReceiverService::image_buffer`
(`receiver/JFJochReceiverService.h:21` ), sized `image_buffer_MiB` from broker config
(`broker/jfjoch_broker.cpp:104` → `JFJochReceiverService` ctor
`receiver/JFJochReceiverService.cpp:15` , `image_buffer(send_buffer_size_MiB*1024*1024)` ).
This is the 150–
- **Allocated + zeroed in the ctor** (`common/ImageBuffer.cpp` ): `numa_alloc_interleaved` if libnuma,
else `std::malloc` ; then a **single-threaded `memset` ** to pre-fault every page (deliberate — kills
first-use page-fault latency in the hot path). Happens once at broker startup.
- **Producers/consumers**: receiver/decompression threads write frames into slots; consumers are
preview/TIFF/JPEG/HTTP retrieval (`GetImage` ) and the ZMQ/file sender. Access is random and
unpinned (any thread → any slot).
- **Not used by `jfjoch_process` or `jfjoch_viewer` ** — they read HDF5 through the reader, never
instantiate `ImageBuffer` . So this buffer is **broker/receiver-only ** (it only needs to * compile *
for the viewer).
### 1b. `NUMAHWPolicy` — bundles three concerns per worker thread
Built from the broker config `numa_policy` string (e.g. `n2g2` , `n8g4` , `n8g4_hbm` ) into a table of
`NUMABinding{cpu_node, mem_node, gpu}` ; `GetBinding(thread) = bindings[thread % nbindings]`
(round-robin, `common/NUMAHWPolicy.cpp:54` ). `Bind(thread)` does **all three ** at once
(`common/NUMAHWPolicy.cpp:61` ):
1. **CPU pin ** — `RunOnNode` / `numa_run_on_node`
2. **Memory bind ** — `MemOnNode` / `numa_set_membind` (on `n8g4_hbm` , `mem_node = i+8` → binds to HBM nodes)
3. **GPU select ** — `SelectGPU` / `set_gpu` (= `cudaSetDevice` , **not ** libnuma)
Call sites:
- `receiver/JFJochReceiverFPGA.cpp:180/217/264` — data-stream threads `RunOnNode(FPGA's NUMA node)`
(device-locality pin, the NIC-era idea applied to the FPGA card).
- `receiver/JFJochReceiverFPGA.cpp:299` , `receiver/JFJochReceiverLite.cpp:234` — analysis worker
threads `numa_policy.Bind(threadid)` → cpu + mem + **GPU ** per the policy table.
- `image_analysis/indexing/IndexerThreadPool.cpp:34` — each indexer thread
`SelectGPUAndItsNUMA(threadid % gpu_count)` (GPU round-robin + that GPU's own NUMA node;
independent of the `numa_policy` table).
libnuma is used in exactly three files: `NUMAHWPolicy.cpp` (the above), `ImageBuffer.cpp`
(`numa_alloc_interleaved` /`numa_free` ), and `CUDAWrapper.cpp` (one `numa_node` lookup).
### 1c. GPU dispatch — and why `jfjoch_process` underuses GPUs
- `get_gpu_count()` = `cudaGetDeviceCount()` (`common/CUDAWrapper.*` ), so it already honours
* * `CUDA_VISIBLE_DEVICES` **. All dispatch is `% gpu_count` .
- **Broker/receiver**: worker threads spread over GPUs via `numa_policy.Bind` (the `gpu` field) +
the indexer pool via `threadid % gpu_count` . → uses all visible GPUs.
- **`jfjoch_process` **: its worker lambda (`tools/jfjoch_process.cpp:849` , launched `nthreads` times)
constructs `MXAnalysisWithoutFPGA` but **never calls `Bind`/`SelectGPU` ** , and
`MXAnalysisWithoutFPGA` itself does not select a device (`image_analysis/MXAnalysisWithoutFPGA.cpp:38`
just builds GPU engines on the * current * device). → all per-image preprocessing / spot-finding /
azimuthal integration run on **GPU 0 ** . Only the indexer pool spreads. **This is the root cause of
"`jfjoch_process` doesn't use all GPUs."**
## 2. Goals
- **G1 — multiple brokers, disjoint GPUs.** Run >1 `jfjoch_broker` on one machine, each confined to a
subset of GPUs, with the code transparently using "all it can see" (no hard-coded indices). Pure
workload control, no security requirement.
- **G2 — `jfjoch_process` should use all visible GPUs**, not just GPU 0.
- **G3 — drop the libnuma dependency** if it doesn't cost real performance (annoying dep; also a
blocker for the long-term Windows/MSVC viewer).
- **G4 — reassess whether NUMA CPU/mem pinning is still worth it** given the FPGA pipeline (DMA into
kernel-mmap'd buffers, negligible IRQ traffic) rather than the old network-RX model.
## 3. Proposed changes
- **G1 (zero code):** launch each broker under `CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VISIBLE_DEVICES=<subset> jfjoch_broker …`. ` get_gpu_count()`/` % gpu_count` already do the rest.
Action item: **document this** (deployment note) and set ` CUDA_DEVICE_ORDER=PCI_BUS_ID` so indices
are stable across boots.
- **G2 (DONE):** added ` pin_gpu()` to ` CUDAWrapper` — a process-wide round-robin counter
(` counter++ % get_gpu_count()`, no thread id needed, no-op when no GPU). The ` jfjoch_process` worker
calls it once before building ` MXAnalysisWithoutFPGA`, so each worker's CUDA streams/engines land on
a distinct device. Caller-agnostic and reusable by other thread pools later.
- **G3 / ` ImageBuffer`:** replace ` numa_alloc_interleaved` + single-threaded ` memset` with **plain
` malloc` + a parallel first-touch ` memset`** (N threads, unpinned). Threads spread by the scheduler
→ balanced placement ≈ interleave for random access, *and* faster startup, *and* no libnuma. Safe
here because the buffer is 30–
just check ` vm.zone_reclaim_mode == 0`). Deterministic per-page interleave (if ever needed under
tight RAM) is a raw ` mbind(MPOL_INTERLEAVE)` syscall — still libnuma-free.
- **G3 / G4 / ` NUMAHWPolicy`:** split the bundled concerns:
- **CPU pin** (` RunOnNode`) — likely **drop** for the FPGA path (G4). If ever wanted back, use
` sched_setaffinity` (no libnuma).
- **GPU select** (` SelectGPU`/` SelectGPUAndItsNUMA`) — **keep**; already ` cudaSetDevice`, no libnuma.
- **Memory bind** (` MemOnNode`) — **drop.** (HBM is out of scope: the only Xeon MAX box didn't pay
off and isn't worth special-casing, so no ` mbind` path is needed — treat every host as plain
multi-socket.)
- **` CUDAWrapper` ` numa_node`** — read the GPU PCIe device's ` /sys/.../numa_node` instead of libnuma.
- Result: ` NUMA_LIBRARY` leaves the CMake entirely.
## 4. Open questions / to validate before deleting anything
- **G4 is empirical.** A/B at production frame rate (pinning on vs off): sustained throughput, dropped
frames, latency jitter. The reasoning predicts "no regression," but measure on one real box first —
production systems are currently tuned around this.
- Confirm ` vm.zone_reclaim_mode` is ` 0` on the broker hosts (else first-touch reclaims locally before
spilling → latency stalls).
- Parallel first-touch placement is *approximate* (depends on the scheduler spreading the zeroing
threads); fine with RAM headroom, but note it's not the guaranteed 50/50 of ` mbind` interleave.
## 5. Suggested order (low-risk first)
1. **G1** — document ` CUDA_VISIBLE_DEVICES` launch (no code).
2. ~~**G2** — per-worker GPU pin in ` jfjoch_process`.~~ **DONE** (` pin_gpu()`).
3. **ImageBuffer** parallel first-touch (drops one libnuma user, helps startup; stands alone).
4. **G4 A/B** on a real broker; if clean, drop CPU pinning.
5. **NUMAHWPolicy** simplify (keep GPU select, drop CPU pin + mem bind) + ` CUDAWrapper` sysfs
→ remove ` NUMA_LIBRARY` from CMake.
Claude Opus 4.8