GPU or CPU decided up front from the card's total memory, never from what is free

- Scaling: the 80%-of-card check runs right after the device ingest's sort and cut, where the
  observation and distinct-hkl counts are first known, instead of after the geometry smoothing;
  same counts, same decision, but a run that goes to the CPU no longer does the device smoothing
  (6.9 s on an 8 GB card for a 33 M observation set) or holds its memory meanwhile.
- Rigid body: engines budgeted on a quarter of the card's total (the validation's plan), not
  min(total/4, free - 1 GB). A busy card no longer sends the rigid body to the CPU (measured on an
  8 GB card: ~80 s and a different fit); engine allocations wait for GPU work beside the run, and
  an engine that cannot be set up fails the validation (RigidBodyGPUFailure) instead of falling
  back to the CPU.
- Beam-centre FFT: GPU when its footprint is at most half of the card's total, otherwise CPU,
  logged; no catch-and-rescore on the CPU after a GPU failure.
- FitPerFrameG reserves its per-frame vector (CPU scaling waited on the heap lock on Windows).
- get_gpu_free_memory(): cudaMemGetInfo, except on an integrated GPU under Linux, where free also
  counts MemAvailable less a tenth of the total. Used only where free memory is waited on
  (cuda_memory_available) or quoted in out-of-memory messages; it decides nothing.
  The two card-filling [CUDAMemHelpers] tests skip on integrated GPUs.
- Out-of-memory messages point at rugnux --no-gpu instead of "the CPU build".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-14 21:28:38 +02:00
1 parent f96ee13629
commit 5de7189643
23 files changed
+143 -79

No files matched your search

+30
View File
@@ -17,6 +17,14 @@
#include "../image_analysis/indexing/CUDAMemHelpers.h"
#include "../image_analysis/indexing/IndexerThreadPool.h"
// Whether the current GPU shares the host's memory (GB10, Jetson). A test that fills the card until an
// allocation fails would there fill the host's RAM - every process on the machine runs out with it.
static bool gpu_is_integrated() {
int device = 0, integrated = 0;
return cudaGetDevice(&device) == cudaSuccess
&& cudaDeviceGetAttribute(&integrated, cudaDevAttrIntegrated, device) == cudaSuccess && integrated;
}
// The broker starts fresh worker threads for every data collection. A stream per thread ever started
// is a device-memory leak that ends a long-running broker, so threads that follow one another have to
// end up on the same stream.
@@ -134,6 +142,8 @@ TEST_CASE("CudaPinnedPool_ReusedByTheNextEngine", "[CUDAMemHelpers]") {
TEST_CASE("IndexerThreadPool_FFTIndexerWaitsForGPUWorkBeside", "[CUDAMemHelpers]") {
if (get_gpu_count() == 0)
SKIP("No CUDA GPU present");
if (gpu_is_integrated())
SKIP("Fills the card, which on an integrated GPU is the host's memory");
IndexingSettings settings;
settings.Algorithm(IndexingAlgorithmEnum::FFT).IndexingThreads(1);
@@ -212,6 +222,8 @@ TEST_CASE("IndexerThreadPool_FFTIndexerWaitsForGPUWorkBeside", "[CUDAMemHelpers]
TEST_CASE("CudaDevicePtr_AfterWorkBesideWaitsForItsMemory", "[CUDAMemHelpers]") {
if (get_gpu_count() == 0)
SKIP("no GPU");
if (gpu_is_integrated())
SKIP("Fills the card, which on an integrated GPU is the host's memory");
std::vector<CudaDevicePtr<uint8_t>> held;
const auto fill = [&] {
for (size_t chunk = size_t(256) << 20; chunk >= (size_t(1) << 20); chunk /= 2)
@@ -239,6 +251,24 @@ TEST_CASE("CudaDevicePtr_AfterWorkBesideWaitsForItsMemory", "[CUDAMemHelpers]")
cuda_trim_allocation_pool();
}
// The free memory waited on is cudaMemGetInfo's on a discrete card. On one that shares the host's memory
// (GB10, Jetson) it also counts the page cache the host hands back, less a tenth of the card for the host:
// never less than cudaMemGetInfo's. Allocations between the readings move them, hence the 1 GB slack.
TEST_CASE("GetGpuFreeMemory_IntegratedCountsThePageCache", "[CUDAMemHelpers]") {
if (get_gpu_count() == 0)
SKIP("No CUDA GPU present");
constexpr double SLACK = 1e9;
size_t free_bytes = 0, total_bytes = 0;
REQUIRE(get_gpu_free_memory(free_bytes, total_bytes));
size_t cuda_free = 0, cuda_total = 0;
REQUIRE(cudaMemGetInfo(&cuda_free, &cuda_total) == cudaSuccess);
CHECK(total_bytes == cuda_total);
CHECK(free_bytes <= total_bytes);
CHECK(double(free_bytes) >= double(cuda_free) - SLACK);
if (!gpu_is_integrated())
CHECK(double(free_bytes) == Catch::Approx(double(cuda_free)).margin(SLACK));
}
#endif
// A merge that finds the card full waits for the GPU work running beside it: until the last one