leonarski_fandClaude Opus 5.5
ddf41e3a18
A service started by systemd without User= runs as root with no HOME, so the driver keeps no JIT cache and every cuFFT plan is compiled again (50-130 ms each, under cuFFT's process-wide lock) - seconds of MR in the broker. Before anything touches CUDA, the broker and rugnux now point CUDA_CACHE_PATH at /var/tmp/jfjoch-cuda-cache-<uid> when the default cache is missing or unusable and the site has set no CUDA_CACHE_PATH. The directory is used only if it is a real directory of this user's, mode 0700; otherwise the start-up warning stays as before. Measured without HOME: a cuFFT plan 51 ms -> 0.5 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
33 lines
1.8 KiB
C++
33 lines
1.8 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <string>
|
|
|
|
#include "Logger.h"
|
|
|
|
// cuFFT ships its transforms as PTX (cuFFT 11 and 12 alike, for every card), so every plan it makes has
|
|
// the driver compile kernels - and makes them again for a second plan of the same shape. The driver keeps
|
|
// what it compiled in its JIT cache, CUDA_CACHE_PATH or else $HOME/.nv/ComputeCache: from there a plan
|
|
// takes about 1 ms, without it 60-130 ms, all under cuFFT's process-wide lock. A run that makes a hundred
|
|
// plans (molecular replacement) is then seconds slower, with nothing else to show for it.
|
|
struct CudaJitCache {
|
|
std::string path; // where the driver keeps it; empty when it keeps none
|
|
std::string filesystem; // the type of the filesystem it is on (Linux only)
|
|
std::string problem; // why the driver cannot use it, or only slowly; empty when none is seen
|
|
};
|
|
|
|
CudaJitCache InspectCudaJitCache();
|
|
|
|
// Where the driver would keep no usable cache - typically a service run as root by systemd without User=,
|
|
// so with no HOME - and CUDA_CACHE_PATH is not set, point CUDA_CACHE_PATH at a directory of this user's
|
|
// own, /var/tmp/jfjoch-cuda-cache-<uid> (0700). The driver reads it once, when it initialises, so this
|
|
// must run before anything touches CUDA. Linux only; a CUDA_CACHE_PATH set by the site is left alone.
|
|
void UsePrivateCudaJitCacheIfNeeded();
|
|
|
|
// One line for a program's start: the cache, and `plan_ms` - how long one cuFFT plan took to make once
|
|
// cuFFT was initialised (WarmUpCuFFT). A warning, saying what to do, when the cache is unusable or a plan
|
|
// is slow. Nothing when plan_ms is NaN: there is no GPU, so no cuFFT.
|
|
void LogCudaJitCache(Logger &logger, double plan_ms);
|