Stop on a lost CUDA context instead of falling back to the host decoder
A sticky error (an illegal address, say) is reported by whichever worker synchronises next - usually
the bslz4 device decode - and MXAnalysisWithoutFPGA::Analyze and Rugnux::MaskDefectivePixels then
logged "falling back to host" and carried on, although the context is gone and the run fails anyway,
later and less clearly. cuda_throw_if_context_lost() now throws a GPUCUDAError ("CUDA device
unusable after an unrecoverable error: ...") there first, which the image loops already treat as
fatal (IsFatalResourceError), like a CUDA out-of-memory.
The last error cannot tell: once cudaGetLastError() has returned a sticky error, it and
cudaPeekAtLastError() both report success. cudaFree(nullptr) frees nothing and does not
synchronise, but returns the sticky error (checked: after an illegal-address kernel it returns
cudaErrorIllegalAddress; with only a pending out-of-memory it returns success), so a handled,
non-sticky decode failure still falls back - MXAnalysis_HandledDeviceDecodeFailureLeavesNoError
passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
This commit is contained in:
@@ -117,7 +117,8 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
|
||||
// The device route must never be the reason a frame fails: whatever it could not handle, the
|
||||
// host decoder gets its turn. If the data really is bad the host throws too and the caller
|
||||
// sees the same error it saw before any of this existed - but a GPU-side problem costs speed
|
||||
// rather than the acquisition.
|
||||
// rather than the acquisition. Unless the context is lost, which the host route cannot survive.
|
||||
cuda_throw_if_context_lost();
|
||||
spdlog::warn("Device decoding failed ({}), falling back to host decompression", e.what());
|
||||
decoded_on_device = false;
|
||||
// Handled here, so it must not stay behind as the thread's last error: the host route below
|
||||
|
||||
Reference in New Issue
Block a user