leonarski_fandClaude Opus 5.5
6936b04d5f
cuFFT ships its transforms as PTX for every card (cuFFT 11 and 12 alike), so each plan has the driver compile kernels, and a second plan of the same shape compiles them again. From the driver's JIT cache that costs ~0.5 ms a plan; without it 45-130 ms, under cuFFT's process-wide lock - which, over the ~100 plans of molecular replacement, matches the stage pattern of the slow MR in the production broker. The cache is unusable in more ways than CUDA_CACHE_DISABLE: with no HOME at all (a systemd system service without User= gets none, and the driver does not fall back to the passwd entry), on a read-only or unwritable directory, on a filesystem whose fcntl locks fail (NFS without locking) - each measured as slow as no cache - or on a slow network filesystem (each plan opens and locks the cache ~10 times). - common/CudaJitCache: the path the driver uses (CUDA_CACHE_PATH, else $HOME/.nv/ComputeCache), its filesystem type, whether it is writable, whether its index opens read-write and takes the driver's fcntl read lock, CUDA_CACHE_DISABLE and a CUDA_CACHE_MAXSIZE below 64 MiB. One line: info when fine, a warning with the cause and the remedy (CUDA_CACHE_PATH on a local disk) when not, or when a plan takes over 10 ms. - WarmUpCuFFT returns the fastest of three 64^3 R2C plan makes after a first one that pays the initialisation (and a cold-cache compile): ~2 ms in all with a cache, ~0.3 s without. - jfjoch_broker runs it after its version line (0.15 s at start, cuFFT init and card 0's context included); rugnux on its existing background warm-up thread. - CUDAMemHelpers.h: the modules are PTX on every platform, not only on Windows. No result changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
27 lines
1.3 KiB
C++
27 lines
1.3 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <string>
|
|
|
|
#include "Logger.h"
|
|
|
|
// cuFFT ships its transforms as PTX (cuFFT 11 and 12 alike, for every card), so every plan it makes has
|
|
// the driver compile kernels - and makes them again for a second plan of the same shape. The driver keeps
|
|
// what it compiled in its JIT cache, CUDA_CACHE_PATH or else $HOME/.nv/ComputeCache: from there a plan
|
|
// takes about 1 ms, without it 60-130 ms, all under cuFFT's process-wide lock. A run that makes a hundred
|
|
// plans (molecular replacement) is then seconds slower, with nothing else to show for it.
|
|
struct CudaJitCache {
|
|
std::string path; // where the driver keeps it; empty when it keeps none
|
|
std::string filesystem; // the type of the filesystem it is on (Linux only)
|
|
std::string problem; // why the driver cannot use it, or only slowly; empty when none is seen
|
|
};
|
|
|
|
CudaJitCache InspectCudaJitCache();
|
|
|
|
// One line for a program's start: the cache, and `plan_ms` - how long one cuFFT plan took to make once
|
|
// cuFFT was initialised (WarmUpCuFFT). A warning, saying what to do, when the cache is unusable or a plan
|
|
// is slow. Nothing when plan_ms is NaN: there is no GPU, so no cuFFT.
|
|
void LogCudaJitCache(Logger &logger, double plan_ms);
|