cc925b2668
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m30s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m13s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m45s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m13s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m51s
Build Packages / build:rpm (rocky8) (push) Successful in 8m29s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 9m31s
Build Packages / build:rpm (rocky9) (push) Successful in 9m42s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 8m47s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m23s
Build Packages / Generate python client (push) Successful in 19s
Build Packages / Build documentation (push) Successful in 38s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 6m18s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m4s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m35s
Build Packages / DIALS test (push) Successful in 10m39s
Build Packages / Unit tests (push) Successful in 1h24m58s
NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are placed device-local by the kernel (dma_alloc_coherent), the big RAM ring buffer is random-access (first-touch handles placement), and GPU work is already spread across all visible devices. So drop the pinning entirely and with it libnuma. - Delete NUMAHWPolicy; the only concern worth keeping - GPU selection - is done directly via pin_gpu() (round-robin over visible GPUs) in the indexer pool and the Lite analysis threads. CPU-only threads (FPGA acquire/pedestal/summation/frame-transform) no longer bind anything. - Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA used it. - numa_policy broker setting is deprecated and ignored (kept in the API for backward compatibility; warns once on startup). - Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake. - Docs: drop the NUMA dependency, remove the numa_policy config example, and document running multiple brokers on disjoint GPUs via CUDA_VISIBLE_DEVICES. - Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
48 lines
1.3 KiB
Plaintext
48 lines
1.3 KiB
Plaintext
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include <fstream>
|
|
#include <atomic>
|
|
|
|
#include "CUDAWrapper.h"
|
|
#include "JFJochException.h"
|
|
|
|
inline void cuda_err(cudaError_t val) {
|
|
if (val != cudaSuccess)
|
|
throw JFJochException(JFJochExceptionCategory::GPUCUDAError, cudaGetErrorString(val));
|
|
}
|
|
|
|
int32_t get_gpu_count() {
|
|
int device_count;
|
|
cudaError_t val = cudaGetDeviceCount(&device_count);
|
|
switch (val) {
|
|
case cudaSuccess:
|
|
return device_count;
|
|
case cudaErrorNoDevice:
|
|
case cudaErrorInsufficientDriver:
|
|
return 0;
|
|
default:
|
|
throw JFJochException(JFJochExceptionCategory::GPUCUDAError, cudaGetErrorString(val));
|
|
}
|
|
|
|
}
|
|
|
|
void set_gpu(int32_t dev_id) {
|
|
auto dev_count = get_gpu_count();
|
|
|
|
// Ignore if no GPU present
|
|
if (dev_count > 0) {
|
|
if ((dev_id < 0) || (dev_id >= dev_count))
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Device ID cannot be negative");
|
|
|
|
cuda_err(cudaSetDevice(dev_id));
|
|
}
|
|
}
|
|
|
|
void pin_gpu() {
|
|
static std::atomic<uint32_t> counter{0};
|
|
auto dev_count = get_gpu_count();
|
|
if (dev_count > 0)
|
|
set_gpu(counter.fetch_add(1) % dev_count);
|
|
}
|