Indexer failures are thrown, not read as "no lattice"; GPU memory errors say what to do

A CUDA out-of-memory in an indexer worker was turned into an IndexerResult carrying only the
message string. The rotation indexer then returned no lattice, the first pass treated that as a
genuine negative and the rescues acted on it: in a stress run (two rugnux on one card) a run had
both first-pass schemes fail with out-of-memory, the axis-sign rescue compared 0 frames at the
flipped sign with the -1 "no lattice" sentinel, adopted the flipped sign, and the beam-centre rescue
then moved the centre - a failure of the machine used as evidence about the crystal. The CUDA
category was lost on the way, so the fail-fast check (IsFatalResourceError) never saw it.

- IndexerResult::error is now the std::exception_ptr the worker caught. RotationIndexer::RunIndexing
  rethrows it before touching any state, so nothing is memoised for a failed run and the exception
  keeps its category: rugnux stops with the CUDA message at once (verified with a forced indexer
  failure: exit 1 after 1.9 s, "CUDA (GPU) error (...)"). FirstPass::indexer_error,
  RotationIndexer::GetIndexerError and RequireLattice's machine-failure branch are gone.
- The beam-centre arm in RunAllPasses rethrows fatal resource errors like its sibling catches; it
  had caught the out-of-memory and carried on at the file's centre.
- IndexAndRefine keeps per-frame indexer failures apart from frames that did not index (count +
  first exception). The frame stays "not attempted" as before; the receiver logs the count and the
  first error at the end of a collection (no cancel), rugnux's image pass and stills geometry
  refinement rethrow it and stop. A rotation-indexer failure inside IndexAndRefine (online rotation
  indexing, FinalizeRotationIndexing) is recorded the same way, so a live collection is not
  cancelled by it.
- A failed device allocation or cuFFT plan now says how much was asked, how much of the card is
  free, and - when the card is merely full - that another program is using the GPU or the data set
  needs more than the card has, with the CPU remedy; previously "Failed to allocate device memory
  (out of memory)" / "Failed to create CUFFT plan with cufftPlanMany".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-10 18:56:00 +02:00
1 parent a897b08a7d
commit c6f6396e46
17 files changed
+131 -58

No files matched your search

+3 -1
View File
@@ -29,7 +29,9 @@ TEST_CASE("CudaDevicePtr_FailedAllocationLeavesErrorUntilCleared", "[CUDAMemHelp
SKIP("No CUDA GPU present");
cuda_clear_error();
REQUIRE_THROWS(CudaDevicePtr<uint8_t>(std::numeric_limits<size_t>::max(), CudaAlloc::Synchronous));
// Said as what it is, with the remedy, not as a bare CUDA status.
REQUIRE_THROWS_WITH(CudaDevicePtr<uint8_t>(std::numeric_limits<size_t>::max(), CudaAlloc::Synchronous),
Catch::Matchers::ContainsSubstring("another program is using this GPU"));
// Work that goes fine in between changes nothing: the error is still waiting.
CudaDevicePtr<uint8_t> fine(1 << 16);