jfjoch_broker and rugnux check the CUDA JIT cache at start and warn when cuFFT plans are slow
cuFFT ships its transforms as PTX for every card (cuFFT 11 and 12 alike), so each plan has the driver compile kernels, and a second plan of the same shape compiles them again. From the driver's JIT cache that costs ~0.5 ms a plan; without it 45-130 ms, under cuFFT's process-wide lock - which, over the ~100 plans of molecular replacement, matches the stage pattern of the slow MR in the production broker. The cache is unusable in more ways than CUDA_CACHE_DISABLE: with no HOME at all (a systemd system service without User= gets none, and the driver does not fall back to the passwd entry), on a read-only or unwritable directory, on a filesystem whose fcntl locks fail (NFS without locking) - each measured as slow as no cache - or on a slow network filesystem (each plan opens and locks the cache ~10 times). - common/CudaJitCache: the path the driver uses (CUDA_CACHE_PATH, else $HOME/.nv/ComputeCache), its filesystem type, whether it is writable, whether its index opens read-write and takes the driver's fcntl read lock, CUDA_CACHE_DISABLE and a CUDA_CACHE_MAXSIZE below 64 MiB. One line: info when fine, a warning with the cause and the remedy (CUDA_CACHE_PATH on a local disk) when not, or when a plan takes over 10 ms. - WarmUpCuFFT returns the fastest of three 64^3 R2C plan makes after a first one that pays the initialisation (and a cold-cache compile): ~2 ms in all with a cache, ~0.3 s without. - jfjoch_broker runs it after its version line (0.15 s at start, cuFFT init and card 0's context included); rugnux on its existing background warm-up thread. - CUDAMemHelpers.h: the modules are PTX on every platform, not only on Windows. No result changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
This commit is contained in:
1 parent
02e5bc387f
commit
6936b04d5f
8 files changed
+188
-11
No files matched your search
@@ -3,6 +3,9 @@
|
||||
|
||||
#include "IndexerThreadPool.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <map>
|
||||
#include <tuple>
|
||||
|
||||
@@ -18,13 +21,27 @@
|
||||
#include "FFTIndexerCPU.h"
|
||||
#endif
|
||||
|
||||
void WarmUpCuFFT() {
|
||||
double WarmUpCuFFT() {
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
if (get_gpu_count() == 0)
|
||||
return;
|
||||
cufftHandle plan = 0;
|
||||
if (cufftPlan1d(&plan, 1024, CUFFT_C2C, 1) == CUFFT_SUCCESS)
|
||||
return NAN;
|
||||
// The first plan pays the initialisation, and compiles its kernels if the JIT cache does not have them
|
||||
// yet; the next ones show what a plan costs from then on. The fastest of three, so that a stall
|
||||
// beside (another card's context being made) is not taken for a slow plan.
|
||||
double fastest_ms = INFINITY;
|
||||
for (int i = 0; i < 4; i++) {
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
cufftHandle plan = 0;
|
||||
if (cufftPlan3d(&plan, 64, 64, 64, CUFFT_R2C) != CUFFT_SUCCESS)
|
||||
return NAN;
|
||||
cufftDestroy(plan);
|
||||
const double ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count();
|
||||
if (i > 0)
|
||||
fastest_ms = std::min(fastest_ms, ms);
|
||||
}
|
||||
return fastest_ms;
|
||||
#else
|
||||
return NAN;
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user