leonarski_fandClaude Opus 5.5
3c5ff3f5df
With -N left to choose, rugnux now confines the process (main thread, the CPUs RestoreThreadAffinity hands back, and what AvailableCpus counts) to the CPUs of the packages its cards hang off, and sizes its threads to them. Nothing changes where the allowed CPUs sit in one package, there is no card or a card's NUMA node is unknown, or the cards' packages already hold every allowed CPU. A mask set from outside (taskset, numactl, a cpuset) is kept: the decision works within it. An explicit -N keeps the old behaviour. On a 48-CPU server with four cards on two of its four NUMA nodes, a rotation run took 24.5-24.7 s confined to those nodes' 24 CPUs against 27.7-27.9 s on all 48 (scaling and merging 8.1 -> 6.2 s), and two larger sets moved 95-99 -> 84-87 s and 131-135 -> 113-115 s. On a single-socket machine this is a no-op. The decision itself (CpusOfGpuPackages) is a pure function, tested on synthetic 4-package, 2-package hyper-threaded, cards-on-both, no-card and restricted-mask topologies. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
93 lines
2.1 KiB
C++
93 lines
2.1 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "CUDAWrapper.h"
|
|
|
|
#include <condition_variable>
|
|
#include <mutex>
|
|
|
|
// Build-independent: the CUDA build gets get_gpu_names() from CUDAWrapper.cu, the CPU-only build from
|
|
// the stub below, and this collapses whichever list came back. Four identical cards read better as
|
|
// "4x <name>" than as the same name four times, and a mixed machine keeps one group per model.
|
|
std::string get_gpu_description() {
|
|
const auto names = get_gpu_names();
|
|
|
|
std::string out;
|
|
for (size_t i = 0; i < names.size();) {
|
|
size_t n = 1;
|
|
while (i + n < names.size() && names[i + n] == names[i])
|
|
n++;
|
|
if (!out.empty())
|
|
out += ", ";
|
|
if (n > 1)
|
|
out += std::to_string(n) + "x ";
|
|
out += names[i];
|
|
i += n;
|
|
}
|
|
return out;
|
|
}
|
|
|
|
namespace {
|
|
std::mutex gpu_work_beside_mutex;
|
|
std::condition_variable gpu_work_beside_done;
|
|
int gpu_work_beside = 0;
|
|
}
|
|
|
|
GPUWorkBeside::GPUWorkBeside() {
|
|
std::lock_guard lock(gpu_work_beside_mutex);
|
|
++gpu_work_beside;
|
|
}
|
|
|
|
GPUWorkBeside::~GPUWorkBeside() {
|
|
{
|
|
std::lock_guard lock(gpu_work_beside_mutex);
|
|
--gpu_work_beside;
|
|
}
|
|
gpu_work_beside_done.notify_all();
|
|
}
|
|
|
|
bool wait_for_gpu_work_beside(std::chrono::seconds timeout) {
|
|
std::unique_lock lock(gpu_work_beside_mutex);
|
|
return gpu_work_beside_done.wait_for(lock, timeout, [] { return gpu_work_beside == 0; });
|
|
}
|
|
|
|
#ifndef JFJOCH_USE_CUDA
|
|
|
|
int32_t get_gpu_count() {
|
|
return 0;
|
|
}
|
|
|
|
std::vector<std::string> get_gpu_names() {
|
|
return {};
|
|
}
|
|
|
|
size_t get_gpu_total_memory(int32_t) {
|
|
return 0;
|
|
}
|
|
|
|
void set_gpu(int32_t dev_id) {}
|
|
|
|
void pin_gpu() {}
|
|
|
|
void pin_gpu(int32_t dev_id) {}
|
|
|
|
ScopedGpuPin::ScopedGpuPin(int32_t) {}
|
|
|
|
ScopedGpuPin::~ScopedGpuPin() {}
|
|
|
|
void enable_gpu_numa_binding() {}
|
|
|
|
std::vector<int> get_gpu_numa_nodes() { return {}; }
|
|
|
|
void set_gpu_blocking_sync() {}
|
|
|
|
void create_gpu_contexts(int32_t) {}
|
|
|
|
void release_gpu_contexts() {}
|
|
|
|
void cuda_clear_error() {}
|
|
|
|
void cuda_throw_if_context_lost() {}
|
|
|
|
#endif
|