jfjoch_process: spread per-image GPU work across all visible GPUs

The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA
device, so all per-image preprocessing/spot-finding/azimuthal integration ran on
GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a
process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op
without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before
building the analysis resources so their CUDA streams/engines land on distinct
devices.

Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU
dispatch with goals and a staged plan (multi-broker GPU isolation via
CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

NUMA_GPU_REVIEW.md

@@ -0,0 +1,115 @@
+# NUMA / GPU usage — current state, goals, proposed changes (for review)
+
+Working note to agree on direction **before** modifying `ImageBuffer` / `NUMAHWPolicy` and the
+GPU-dispatch logic. Nothing here is implemented yet. File:line anchors are from branch
+`2606-pixel-refine`.
+
+## 1. Current state (the mind map)
+
+### 1a. `ImageBuffer` — the big RAM ring buffer
+- **One instance**, a member of the receiver: `JFJochReceiverService::image_buffer`
+  (`receiver/JFJochReceiverService.h:21`), sized `image_buffer_MiB` from broker config
+  (`broker/jfjoch_broker.cpp:104` → `JFJochReceiverService` ctor
+  `receiver/JFJochReceiverService.cpp:15`, `image_buffer(send_buffer_size_MiB*1024*1024)`).
+  This is the 150–200 GB allocation.
+- **Allocated + zeroed in the ctor** (`common/ImageBuffer.cpp`): `numa_alloc_interleaved` if libnuma,
+  else `std::malloc`; then a **single-threaded `memset`** to pre-fault every page (deliberate — kills
+  first-use page-fault latency in the hot path). Happens once at broker startup.
+- **Producers/consumers**: receiver/decompression threads write frames into slots; consumers are
+  preview/TIFF/JPEG/HTTP retrieval (`GetImage`) and the ZMQ/file sender. Access is random and
+  unpinned (any thread → any slot).
+- **Not used by `jfjoch_process` or `jfjoch_viewer`** — they read HDF5 through the reader, never
+  instantiate `ImageBuffer`. So this buffer is **broker/receiver-only** (it only needs to *compile*
+  for the viewer).
+
+### 1b. `NUMAHWPolicy` — bundles three concerns per worker thread
+Built from the broker config `numa_policy` string (e.g. `n2g2`, `n8g4`, `n8g4_hbm`) into a table of
+`NUMABinding{cpu_node, mem_node, gpu}`; `GetBinding(thread) = bindings[thread % nbindings]`
+(round-robin, `common/NUMAHWPolicy.cpp:54`). `Bind(thread)` does **all three** at once
+(`common/NUMAHWPolicy.cpp:61`):
+1. **CPU pin** — `RunOnNode` / `numa_run_on_node`
+2. **Memory bind** — `MemOnNode` / `numa_set_membind` (on `n8g4_hbm`, `mem_node = i+8` → binds to HBM nodes)
+3. **GPU select** — `SelectGPU` / `set_gpu` (= `cudaSetDevice`, **not** libnuma)
+
+Call sites:
+- `receiver/JFJochReceiverFPGA.cpp:180/217/264` — data-stream threads `RunOnNode(FPGA's NUMA node)`
+  (device-locality pin, the NIC-era idea applied to the FPGA card).
+- `receiver/JFJochReceiverFPGA.cpp:299`, `receiver/JFJochReceiverLite.cpp:234` — analysis worker
+  threads `numa_policy.Bind(threadid)` → cpu + mem + **GPU** per the policy table.
+- `image_analysis/indexing/IndexerThreadPool.cpp:34` — each indexer thread
+  `SelectGPUAndItsNUMA(threadid % gpu_count)` (GPU round-robin + that GPU's own NUMA node;
+  independent of the `numa_policy` table).
+
+libnuma is used in exactly three files: `NUMAHWPolicy.cpp` (the above), `ImageBuffer.cpp`
+(`numa_alloc_interleaved`/`numa_free`), and `CUDAWrapper.cpp` (one `numa_node` lookup).
+
+### 1c. GPU dispatch — and why `jfjoch_process` underuses GPUs
+- `get_gpu_count()` = `cudaGetDeviceCount()` (`common/CUDAWrapper.*`), so it already honours
+  **`CUDA_VISIBLE_DEVICES`**. All dispatch is `% gpu_count`.
+- **Broker/receiver**: worker threads spread over GPUs via `numa_policy.Bind` (the `gpu` field) +
+  the indexer pool via `threadid % gpu_count`. → uses all visible GPUs.
+- **`jfjoch_process`**: its worker lambda (`tools/jfjoch_process.cpp:849`, launched `nthreads` times)
+  constructs `MXAnalysisWithoutFPGA` but **never calls `Bind`/`SelectGPU`**, and
+  `MXAnalysisWithoutFPGA` itself does not select a device (`image_analysis/MXAnalysisWithoutFPGA.cpp:38`
+  just builds GPU engines on the *current* device). → all per-image preprocessing / spot-finding /
+  azimuthal integration run on **GPU 0**. Only the indexer pool spreads. **This is the root cause of
+  "`jfjoch_process` doesn't use all GPUs."**
+
+## 2. Goals
+
+- **G1 — multiple brokers, disjoint GPUs.** Run >1 `jfjoch_broker` on one machine, each confined to a
+  subset of GPUs, with the code transparently using "all it can see" (no hard-coded indices). Pure
+  workload control, no security requirement.
+- **G2 — `jfjoch_process` should use all visible GPUs**, not just GPU 0.
+- **G3 — drop the libnuma dependency** if it doesn't cost real performance (annoying dep; also a
+  blocker for the long-term Windows/MSVC viewer).
+- **G4 — reassess whether NUMA CPU/mem pinning is still worth it** given the FPGA pipeline (DMA into
+  kernel-mmap'd buffers, negligible IRQ traffic) rather than the old network-RX model.
+
+## 3. Proposed changes
+
+- **G1 (zero code):** launch each broker under `CUDA_DEVICE_ORDER=PCI_BUS_ID
+  CUDA_VISIBLE_DEVICES=<subset> jfjoch_broker …`. `get_gpu_count()`/`% gpu_count` already do the rest.
+  Action item: **document this** (deployment note) and set `CUDA_DEVICE_ORDER=PCI_BUS_ID` so indices
+  are stable across boots.
+
+- **G2 (DONE):** added `pin_gpu()` to `CUDAWrapper` — a process-wide round-robin counter
+  (`counter++ % get_gpu_count()`, no thread id needed, no-op when no GPU). The `jfjoch_process` worker
+  calls it once before building `MXAnalysisWithoutFPGA`, so each worker's CUDA streams/engines land on
+  a distinct device. Caller-agnostic and reusable by other thread pools later.
+
+- **G3 / `ImageBuffer`:** replace `numa_alloc_interleaved` + single-threaded `memset` with **plain
+  `malloc` + a parallel first-touch `memset`** (N threads, unpinned). Threads spread by the scheduler
+  → balanced placement ≈ interleave for random access, *and* faster startup, *and* no libnuma. Safe
+  here because the buffer is 30–40 % of RAM (first-touch spills to the other node if one fills; no OOM;
+  just check `vm.zone_reclaim_mode == 0`). Deterministic per-page interleave (if ever needed under
+  tight RAM) is a raw `mbind(MPOL_INTERLEAVE)` syscall — still libnuma-free.
+
+- **G3 / G4 / `NUMAHWPolicy`:** split the bundled concerns:
+  - **CPU pin** (`RunOnNode`) — likely **drop** for the FPGA path (G4). If ever wanted back, use
+    `sched_setaffinity` (no libnuma).
+  - **GPU select** (`SelectGPU`/`SelectGPUAndItsNUMA`) — **keep**; already `cudaSetDevice`, no libnuma.
+  - **Memory bind** (`MemOnNode`) — **drop.** (HBM is out of scope: the only Xeon MAX box didn't pay
+    off and isn't worth special-casing, so no `mbind` path is needed — treat every host as plain
+    multi-socket.)
+  - **`CUDAWrapper` `numa_node`** — read the GPU PCIe device's `/sys/.../numa_node` instead of libnuma.
+  - Result: `NUMA_LIBRARY` leaves the CMake entirely.
+
+## 4. Open questions / to validate before deleting anything
+
+- **G4 is empirical.** A/B at production frame rate (pinning on vs off): sustained throughput, dropped
+  frames, latency jitter. The reasoning predicts "no regression," but measure on one real box first —
+  production systems are currently tuned around this.
+- Confirm `vm.zone_reclaim_mode` is `0` on the broker hosts (else first-touch reclaims locally before
+  spilling → latency stalls).
+- Parallel first-touch placement is *approximate* (depends on the scheduler spreading the zeroing
+  threads); fine with RAM headroom, but note it's not the guaranteed 50/50 of `mbind` interleave.
+
+## 5. Suggested order (low-risk first)
+
+1. **G1** — document `CUDA_VISIBLE_DEVICES` launch (no code).
+2. ~~**G2** — per-worker GPU pin in `jfjoch_process`.~~ **DONE** (`pin_gpu()`).
+3. **ImageBuffer** parallel first-touch (drops one libnuma user, helps startup; stands alone).
+4. **G4 A/B** on a real broker; if clean, drop CPU pinning.
+5. **NUMAHWPolicy** simplify (keep GPU select, drop CPU pin + mem bind) + `CUDAWrapper` sysfs
+   → remove `NUMA_LIBRARY` from CMake.

common/CUDAWrapper.cpp

@@ -11,6 +11,8 @@ int32_t get_gpu_count() {
 
 void set_gpu(int32_t dev_id) {}
 
+void pin_gpu() {}
+
 int get_gpu_numa_node(int dev_id) {
     return -1;
 }

common/CUDAWrapper.cu

@@ -2,6 +2,7 @@
 // SPDX-License-Identifier: GPL-3.0-only
 
 #include <fstream>
+#include <atomic>
 
 #include "CUDAWrapper.h"
 #include "JFJochException.h"
@@ -38,6 +39,13 @@ void set_gpu(int32_t dev_id) {
     }
 }
 
+void pin_gpu() {
+    static std::atomic<uint32_t> counter{0};
+    auto dev_count = get_gpu_count();
+    if (dev_count > 0)
+        set_gpu(counter.fetch_add(1) % dev_count);
+}
+
 // Return CUDA device PCI Bus ID as "domain:bus:device.function", e.g., "0000:65:00.0"
 static std::string get_cuda_device_pci_bus_id(int dev_id) {
     // CUDA API provides cudaDeviceGetPCIBusId

common/CUDAWrapper.h

@@ -8,3 +8,8 @@
 int32_t get_gpu_count();
 void set_gpu(int32_t dev_id);
 int get_gpu_numa_node(int dev_id);
+
+// Pin the calling thread to the next GPU in round-robin order, using a process-wide counter
+// (counter++ % get_gpu_count()). Call once per thread; no thread id needed. No-op when no GPU
+// is visible. Honours CUDA_VISIBLE_DEVICES via get_gpu_count().
+void pin_gpu();

tools/jfjoch_process.cpp

@@ -25,6 +25,7 @@
 #include "../common/print_license.h"
 #include "../image_analysis/MXAnalysisWithoutFPGA.h"
 #include "../image_analysis/indexing/IndexerFactory.h"
+#include "../common/CUDAWrapper.h"
 #include "../writer/FileWriter.h"
 #include "../image_analysis/IndexAndRefine.h"
 #include "../receiver/JFJochReceiverPlots.h"
@@ -847,6 +848,11 @@ int main(int argc, char **argv) {
     std::atomic<int> finished_count = 0;
 
     auto worker = [&](int thread_id) {
+        // Spread the per-image GPU pipeline across all visible GPUs (round-robin).
+        // Must run before building the analysis resources so their CUDA streams/engines are
+        // created on this thread's device.
+        pin_gpu();
+
         // Thread-local analysis resources
         MXAnalysisWithoutFPGA analysis(experiment, mapping, pixel_mask, indexer);