jfjoch_broker and rugnux check the CUDA JIT cache at start and warn when cuFFT plans are slow

cuFFT ships its transforms as PTX for every card (cuFFT 11 and 12 alike), so each plan has the driver
compile kernels, and a second plan of the same shape compiles them again. From the driver's JIT cache
that costs ~0.5 ms a plan; without it 45-130 ms, under cuFFT's process-wide lock - which, over the
~100 plans of molecular replacement, matches the stage pattern of the slow MR in the production broker.

The cache is unusable in more ways than CUDA_CACHE_DISABLE: with no HOME at all (a systemd system
service without User= gets none, and the driver does not fall back to the passwd entry), on a read-only
or unwritable directory, on a filesystem whose fcntl locks fail (NFS without locking) - each measured
as slow as no cache - or on a slow network filesystem (each plan opens and locks the cache ~10 times).

- common/CudaJitCache: the path the driver uses (CUDA_CACHE_PATH, else $HOME/.nv/ComputeCache), its
  filesystem type, whether it is writable, whether its index opens read-write and takes the driver's
  fcntl read lock, CUDA_CACHE_DISABLE and a CUDA_CACHE_MAXSIZE below 64 MiB. One line: info when fine,
  a warning with the cause and the remedy (CUDA_CACHE_PATH on a local disk) when not, or when a plan
  takes over 10 ms.
- WarmUpCuFFT returns the fastest of three 64^3 R2C plan makes after a first one that pays the
  initialisation (and a cold-cache compile): ~2 ms in all with a cache, ~0.3 s without.
- jfjoch_broker runs it after its version line (0.15 s at start, cuFFT init and card 0's context
  included); rugnux on its existing background warm-up thread.
- CUDAMemHelpers.h: the modules are PTX on every platform, not only on Windows.

No result changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-20 01:37:10 +02:00
1 parent 02e5bc387f
commit 6936b04d5f
8 files changed
+188 -11

No files matched your search

+21 -4
View File
@@ -3,6 +3,9 @@
#include "IndexerThreadPool.h"
#include <algorithm>
#include <chrono>
#include <cmath>
#include <map>
#include <tuple>
@@ -18,13 +21,27 @@
#include "FFTIndexerCPU.h"
#endif
void WarmUpCuFFT() {
double WarmUpCuFFT() {
#ifdef JFJOCH_USE_CUDA
if (get_gpu_count() == 0)
return;
cufftHandle plan = 0;
if (cufftPlan1d(&plan, 1024, CUFFT_C2C, 1) == CUFFT_SUCCESS)
return NAN;
// The first plan pays the initialisation, and compiles its kernels if the JIT cache does not have them
// yet; the next ones show what a plan costs from then on. The fastest of three, so that a stall
// beside (another card's context being made) is not taken for a slow plan.
double fastest_ms = INFINITY;
for (int i = 0; i < 4; i++) {
const auto t0 = std::chrono::steady_clock::now();
cufftHandle plan = 0;
if (cufftPlan3d(&plan, 64, 64, 64, CUFFT_R2C) != CUFFT_SUCCESS)
return NAN;
cufftDestroy(plan);
const double ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count();
if (i > 0)
fastest_ms = std::min(fastest_ms, ms);
}
return fastest_ms;
#else
return NAN;
#endif
}