GPU or CPU decided up front from the card's total memory, never from what is free
- Scaling: the 80%-of-card check runs right after the device ingest's sort and cut, where the observation and distinct-hkl counts are first known, instead of after the geometry smoothing; same counts, same decision, but a run that goes to the CPU no longer does the device smoothing (6.9 s on an 8 GB card for a 33 M observation set) or holds its memory meanwhile. - Rigid body: engines budgeted on a quarter of the card's total (the validation's plan), not min(total/4, free - 1 GB). A busy card no longer sends the rigid body to the CPU (measured on an 8 GB card: ~80 s and a different fit); engine allocations wait for GPU work beside the run, and an engine that cannot be set up fails the validation (RigidBodyGPUFailure) instead of falling back to the CPU. - Beam-centre FFT: GPU when its footprint is at most half of the card's total, otherwise CPU, logged; no catch-and-rescore on the CPU after a GPU failure. - FitPerFrameG reserves its per-frame vector (CPU scaling waited on the heap lock on Windows). - get_gpu_free_memory(): cudaMemGetInfo, except on an integrated GPU under Linux, where free also counts MemAvailable less a tenth of the total. Used only where free memory is waited on (cuda_memory_available) or quoted in out-of-memory messages; it decides nothing. The two card-filling [CUDAMemHelpers] tests skip on integrated GPUs. - Out-of-memory messages point at rugnux --no-gpu instead of "the CPU build". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f96ee13629
commit
5de7189643
23 files changed
+143
-79
No files matched your search
@@ -17,6 +17,14 @@
|
||||
#include "../image_analysis/indexing/CUDAMemHelpers.h"
|
||||
#include "../image_analysis/indexing/IndexerThreadPool.h"
|
||||
|
||||
// Whether the current GPU shares the host's memory (GB10, Jetson). A test that fills the card until an
|
||||
// allocation fails would there fill the host's RAM - every process on the machine runs out with it.
|
||||
static bool gpu_is_integrated() {
|
||||
int device = 0, integrated = 0;
|
||||
return cudaGetDevice(&device) == cudaSuccess
|
||||
&& cudaDeviceGetAttribute(&integrated, cudaDevAttrIntegrated, device) == cudaSuccess && integrated;
|
||||
}
|
||||
|
||||
// The broker starts fresh worker threads for every data collection. A stream per thread ever started
|
||||
// is a device-memory leak that ends a long-running broker, so threads that follow one another have to
|
||||
// end up on the same stream.
|
||||
@@ -134,6 +142,8 @@ TEST_CASE("CudaPinnedPool_ReusedByTheNextEngine", "[CUDAMemHelpers]") {
|
||||
TEST_CASE("IndexerThreadPool_FFTIndexerWaitsForGPUWorkBeside", "[CUDAMemHelpers]") {
|
||||
if (get_gpu_count() == 0)
|
||||
SKIP("No CUDA GPU present");
|
||||
if (gpu_is_integrated())
|
||||
SKIP("Fills the card, which on an integrated GPU is the host's memory");
|
||||
|
||||
IndexingSettings settings;
|
||||
settings.Algorithm(IndexingAlgorithmEnum::FFT).IndexingThreads(1);
|
||||
@@ -212,6 +222,8 @@ TEST_CASE("IndexerThreadPool_FFTIndexerWaitsForGPUWorkBeside", "[CUDAMemHelpers]
|
||||
TEST_CASE("CudaDevicePtr_AfterWorkBesideWaitsForItsMemory", "[CUDAMemHelpers]") {
|
||||
if (get_gpu_count() == 0)
|
||||
SKIP("no GPU");
|
||||
if (gpu_is_integrated())
|
||||
SKIP("Fills the card, which on an integrated GPU is the host's memory");
|
||||
std::vector<CudaDevicePtr<uint8_t>> held;
|
||||
const auto fill = [&] {
|
||||
for (size_t chunk = size_t(256) << 20; chunk >= (size_t(1) << 20); chunk /= 2)
|
||||
@@ -239,6 +251,24 @@ TEST_CASE("CudaDevicePtr_AfterWorkBesideWaitsForItsMemory", "[CUDAMemHelpers]")
|
||||
cuda_trim_allocation_pool();
|
||||
}
|
||||
|
||||
// The free memory waited on is cudaMemGetInfo's on a discrete card. On one that shares the host's memory
|
||||
// (GB10, Jetson) it also counts the page cache the host hands back, less a tenth of the card for the host:
|
||||
// never less than cudaMemGetInfo's. Allocations between the readings move them, hence the 1 GB slack.
|
||||
TEST_CASE("GetGpuFreeMemory_IntegratedCountsThePageCache", "[CUDAMemHelpers]") {
|
||||
if (get_gpu_count() == 0)
|
||||
SKIP("No CUDA GPU present");
|
||||
constexpr double SLACK = 1e9;
|
||||
size_t free_bytes = 0, total_bytes = 0;
|
||||
REQUIRE(get_gpu_free_memory(free_bytes, total_bytes));
|
||||
size_t cuda_free = 0, cuda_total = 0;
|
||||
REQUIRE(cudaMemGetInfo(&cuda_free, &cuda_total) == cudaSuccess);
|
||||
CHECK(total_bytes == cuda_total);
|
||||
CHECK(free_bytes <= total_bytes);
|
||||
CHECK(double(free_bytes) >= double(cuda_free) - SLACK);
|
||||
if (!gpu_is_integrated())
|
||||
CHECK(double(free_bytes) == Catch::Approx(double(cuda_free)).margin(SLACK));
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
// A merge that finds the card full waits for the GPU work running beside it: until the last one
|
||||
|
||||
Reference in new issue
Block a user