leonarski_fandClaude Opus 5.5
b43844ae06
The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two
threads that drive the merging tail were not: the quality guard took its card with set_gpu(1),
and the main engine runs on the pipeline's own thread, which goes on to everything else. On a
multi-socket server their pageable copies to the card went through a remote socket.
RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost,
IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous
device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding
covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads
the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use
std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not
squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses
pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask
themselves; the new test checks both that and that a pinned caller keeps its own mask.
No flag, no environment variable. On one NUMA node (this workstation) only the device setting
happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base =
rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa,
cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs),
-N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise,
e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for
multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
83 lines
4.0 KiB
C++
83 lines
4.0 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <chrono>
|
|
#include <cstdint>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
int32_t get_gpu_count();
|
|
|
|
// Names of the visible GPUs, in device order and one entry per device, so repeated cards repeat.
|
|
// Empty without CUDA and on a machine with no device, which is also what get_gpu_count() == 0 says.
|
|
std::vector<std::string> get_gpu_names();
|
|
|
|
// The same list collapsed for a person: "4x NVIDIA A100-SXM4-80GB", or several such groups separated
|
|
// by ", " on a mixed machine. Empty when no GPU is visible.
|
|
std::string get_gpu_description();
|
|
void set_gpu(int32_t dev_id);
|
|
|
|
// Pin the calling thread to the next GPU in round-robin order, using a process-wide counter
|
|
// (counter++ % get_gpu_count()). Call once per thread; no thread id needed. No-op when no GPU
|
|
// is visible. Honours CUDA_VISIBLE_DEVICES via get_gpu_count().
|
|
void pin_gpu();
|
|
|
|
// From here on, pin_gpu() also keeps the calling thread on the CPUs of the NUMA node its GPU is
|
|
// attached to (ThreadAffinity.h) - on a machine with one node, or without CUDA, nothing changes.
|
|
// Off unless a program asks for it.
|
|
void enable_gpu_numa_binding();
|
|
|
|
// pin_gpu() onto a given device: set_gpu(dev_id), and the NUMA binding above where it is enabled.
|
|
// For a worker that takes its card by index rather than round-robin.
|
|
void pin_gpu(int32_t dev_id);
|
|
|
|
// pin_gpu(dev_id) for as long as the object lives; then the calling thread gets back the device and
|
|
// the CPUs it had before. For a thread that drives a card only for a while - the merging engine, whose
|
|
// calls come from a thread that goes on to other work - where a pin for the rest of its life would
|
|
// carry the card's node onto everything that thread does, and starts, afterwards.
|
|
class ScopedGpuPin {
|
|
public:
|
|
explicit ScopedGpuPin(int32_t dev_id);
|
|
~ScopedGpuPin();
|
|
ScopedGpuPin(const ScopedGpuPin &) = delete;
|
|
ScopedGpuPin &operator=(const ScopedGpuPin &) = delete;
|
|
private:
|
|
int32_t prev_device = -1; // -1: nothing to give back
|
|
std::vector<int> prev_cpus; // empty: the CPUs were not changed
|
|
};
|
|
|
|
// Have every GPU's host threads BLOCK in a synchronisation (cudaDeviceScheduleBlockingSync) instead
|
|
// of spinning on a core until the device finishes. Must be called before anything creates a CUDA
|
|
// context; a device that already has one keeps the flags it was created with. No-op without CUDA.
|
|
void set_gpu_blocking_sync();
|
|
|
|
// Drop the error CUDA has recorded for the calling thread. Call it where a CUDA failure has been
|
|
// HANDLED - a device route that fell back to the host, an indexing attempt whose failure was turned
|
|
// into a result - because the error otherwise stays as the thread's last error and the next
|
|
// cuda_err(cudaGetLastError()) after some later kernel launch reports it, over work that went fine.
|
|
// A sticky error (an illegal access, say) is not cleared by this, and nothing here pretends it is:
|
|
// the context is gone in that case and every later call fails on its own. No-op without CUDA.
|
|
void cuda_clear_error();
|
|
|
|
// Throw if the CUDA context is lost - a sticky error (an illegal access, say) that every later call
|
|
// on this device will fail with. For a device route about to fall back to the host: there is no
|
|
// fallback from that, and falling back only moves the failure somewhere less clear. A handled,
|
|
// non-sticky failure passes. No-op without CUDA.
|
|
void cuda_throw_if_context_lost();
|
|
|
|
// GPU work that runs beside the main line of a process and gives all of its device memory back when it
|
|
// ends: a probe pass run on a copy of the run while the run itself scales and merges. Hold one for as
|
|
// long as that work runs. Build-independent - it only counts.
|
|
class GPUWorkBeside {
|
|
public:
|
|
GPUWorkBeside();
|
|
~GPUWorkBeside();
|
|
GPUWorkBeside(const GPUWorkBeside &) = delete;
|
|
GPUWorkBeside &operator=(const GPUWorkBeside &) = delete;
|
|
};
|
|
|
|
// Wait until no GPUWorkBeside is held, for at most `timeout`. True once none is - at once if none was.
|
|
bool wait_for_gpu_work_beside(std::chrono::seconds timeout);
|