leonarski_fandClaude Opus 5
373fda05a4
rc.172 stopped two CUDA failures from staying behind as the thread's last error, both inside
CUDAMemHelpers.h: a pooled allocation that fell back to cudaMalloc, and a stream that could not be
created. The reason those mattered was general, and four other places have it too.
Nothing but cudaGetLastError() takes an error away. A later successful call neither clears it nor
returns it - measured: after a cudaMalloc that cannot be served, cudaStreamSynchronize,
cudaMemcpyAsync and a small cudaMalloc all report success, and the "out of memory" is still waiting
for whoever calls cudaGetLastError() next. In this code that is the check after a kernel launch. So
wherever a CUDA failure is caught and the work carries on by another route, the error it left is
reported over the route that went fine:
- the device decoder in MXAnalysisWithoutFPGA::Analyze, whose failure is answered by decompressing
on the host - and the host route runs kernels of its own. This is the broker's path, and it is
the pair of log lines a long-running broker showed, one warning naming the device route and one
fatal error naming nothing.
- the indexer worker, which turns a failed attempt into a result and goes on to the next frame -
on the same thread, so one failure would read as an indexer that never works again.
- the beam stop's GPU accumulate and the FFT beam-centre capture, both of which fall back to the
host.
Each discharges it now, through cuda_clear_error() in CUDAWrapper (a no-op without CUDA). A sticky
error is not cleared by this and nothing here pretends otherwise.
tests/CudaHandledErrorTest.cpp: that a failed allocation leaves an error which survives later
successful work until it is cleared, and that an image whose device decode fails comes out of
Analyze whole and leaves nothing behind for the next one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
44 lines
1.1 KiB
C++
44 lines
1.1 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "CUDAWrapper.h"
|
|
|
|
// Build-independent: the CUDA build gets get_gpu_names() from CUDAWrapper.cu, the CPU-only build from
|
|
// the stub below, and this collapses whichever list came back. Four identical cards read better as
|
|
// "4x <name>" than as the same name four times, and a mixed machine keeps one group per model.
|
|
std::string get_gpu_description() {
|
|
const auto names = get_gpu_names();
|
|
|
|
std::string out;
|
|
for (size_t i = 0; i < names.size();) {
|
|
size_t n = 1;
|
|
while (i + n < names.size() && names[i + n] == names[i])
|
|
n++;
|
|
if (!out.empty())
|
|
out += ", ";
|
|
if (n > 1)
|
|
out += std::to_string(n) + "x ";
|
|
out += names[i];
|
|
i += n;
|
|
}
|
|
return out;
|
|
}
|
|
|
|
#ifndef JFJOCH_USE_CUDA
|
|
|
|
int32_t get_gpu_count() {
|
|
return 0;
|
|
}
|
|
|
|
std::vector<std::string> get_gpu_names() {
|
|
return {};
|
|
}
|
|
|
|
void set_gpu(int32_t dev_id) {}
|
|
|
|
void pin_gpu() {}
|
|
|
|
void cuda_clear_error() {}
|
|
|
|
#endif
|