Indexer failures are thrown, not read as "no lattice"; GPU memory errors say what to do
A CUDA out-of-memory in an indexer worker was turned into an IndexerResult carrying only the message string. The rotation indexer then returned no lattice, the first pass treated that as a genuine negative and the rescues acted on it: in a stress run (two rugnux on one card) a run had both first-pass schemes fail with out-of-memory, the axis-sign rescue compared 0 frames at the flipped sign with the -1 "no lattice" sentinel, adopted the flipped sign, and the beam-centre rescue then moved the centre - a failure of the machine used as evidence about the crystal. The CUDA category was lost on the way, so the fail-fast check (IsFatalResourceError) never saw it. - IndexerResult::error is now the std::exception_ptr the worker caught. RotationIndexer::RunIndexing rethrows it before touching any state, so nothing is memoised for a failed run and the exception keeps its category: rugnux stops with the CUDA message at once (verified with a forced indexer failure: exit 1 after 1.9 s, "CUDA (GPU) error (...)"). FirstPass::indexer_error, RotationIndexer::GetIndexerError and RequireLattice's machine-failure branch are gone. - The beam-centre arm in RunAllPasses rethrows fatal resource errors like its sibling catches; it had caught the out-of-memory and carried on at the file's centre. - IndexAndRefine keeps per-frame indexer failures apart from frames that did not index (count + first exception). The frame stays "not attempted" as before; the receiver logs the count and the first error at the end of a collection (no cancel), rugnux's image pass and stills geometry refinement rethrow it and stop. A rotation-indexer failure inside IndexAndRefine (online rotation indexing, FinalizeRotationIndexing) is recorded the same way, so a live collection is not cancelled by it. - A failed device allocation or cuFFT plan now says how much was asked, how much of the card is free, and - when the card is merely full - that another program is using the GPU or the data set needs more than the card has, with the CPU remedy; previously "Failed to allocate device memory (out of memory)" / "Failed to create CUFFT plan with cufftPlanMany". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a897b08a7d
commit
c6f6396e46
17 files changed
+131
-58
No files matched your search
@@ -29,7 +29,9 @@ TEST_CASE("CudaDevicePtr_FailedAllocationLeavesErrorUntilCleared", "[CUDAMemHelp
|
||||
SKIP("No CUDA GPU present");
|
||||
|
||||
cuda_clear_error();
|
||||
REQUIRE_THROWS(CudaDevicePtr<uint8_t>(std::numeric_limits<size_t>::max(), CudaAlloc::Synchronous));
|
||||
// Said as what it is, with the remedy, not as a bare CUDA status.
|
||||
REQUIRE_THROWS_WITH(CudaDevicePtr<uint8_t>(std::numeric_limits<size_t>::max(), CudaAlloc::Synchronous),
|
||||
Catch::Matchers::ContainsSubstring("another program is using this GPU"));
|
||||
|
||||
// Work that goes fine in between changes nothing: the error is still waiting.
|
||||
CudaDevicePtr<uint8_t> fine(1 << 16);
|
||||
|
||||
Reference in new issue
Block a user