MR score-engine cache and merge buffers: no hang, no doubled wait; near-tie doc follows offline decide-late
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 3m24s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m14s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 14m43s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 19m8s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 19m30s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 21m44s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 22m47s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 24m6s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 25m47s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 27m47s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 19m48s
Build Packages / Generate python client (push) Successful in 30s
Build Packages / Build documentation (push) Successful in 1m9s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 21m10s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 22m34s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 23m52s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 18m52s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 23m41s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 19m32s
Build Packages / Unit tests (push) Successful in 1h25m53s
Build Packages / Validation Battery (push) Successful in 31m9s
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 3m24s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m14s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 14m43s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 19m8s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 19m30s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 21m44s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 22m47s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 24m6s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 25m47s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 27m47s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 19m48s
Build Packages / Generate python client (push) Successful in 30s
Build Packages / Build documentation (push) Successful in 1m9s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 21m10s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 22m34s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 23m52s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 18m52s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 23m41s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 19m32s
Build Packages / Unit tests (push) Successful in 1h25m53s
Build Packages / Validation Battery (push) Successful in 31m9s
- MolecularReplacement: an engine whose making throws anything (std::bad_alloc, a gemmi error), not only a JFJochException, is refused and its reservation given back, so a thread waiting for the same engine is woken instead of waiting forever. - RotationScaleMergeGPU::Alloc waits for the GPU work beside itself (GPU_BUSY_TIMEOUT), so its buffers are allocated plainly Synchronous; SynchronousAfterWorkBeside added a second 10-minute wait in front of it. - CPU_DATA_ANALYSIS_INDEXING: an offline run decides its near ties on the whole sweep from the start; only a live run (broker URL or --simulate-live) decides early and re-checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -230,15 +230,20 @@ also runs (rotation only, and not when a cell was given — `-C` already lowers
|
||||
and its answer is adopted only when it is an integer sub-cell of the standing one and ties or beats
|
||||
it on the validation frames. Everywhere else the standard pass's answer stands.
|
||||
|
||||
**A near tie is decided early and checked again late.** The first pass decides on validation frames
|
||||
from the start of the sweep, so it can act at once - also while a collection is still being written.
|
||||
Where one of its decisions was a near tie that later frames could tip (the lattice, the indexing
|
||||
path, the beam centre, the spot-finding settings), the pass is run again once every frame is in, on a
|
||||
copy of the run taken as it began and deciding on the rest of the sweep too, and only where that
|
||||
re-check ends elsewhere is the whole run made again with every near tie decided late; a run that found
|
||||
no lattice after carrying a near tie is made again likewise. The answer is therefore the one deciding
|
||||
late throughout gives, at the cost of the re-checks (measured: a median of 0.9 s on the sets that had
|
||||
one).
|
||||
**A near tie is decided on every frame - from the start offline, late on a live run.** A run on a
|
||||
stored file has the whole sweep from the start, so its first pass decides each near tie that later
|
||||
frames could tip (the lattice, the indexing path, the beam centre, the spot-finding settings) on
|
||||
validation frames from the whole sweep at once, with nothing to check again. A live run - one following
|
||||
a collection that is still being written (`rugnux <master.h5> <broker URL>`, or `--simulate-live` on a
|
||||
stored file) - cannot wait for frames that do not exist yet: its first pass decides on validation
|
||||
frames from the start of the sweep, so it can act at once. Where one of those decisions was a near tie,
|
||||
the pass is run again once every frame is in, on a copy of the run taken as it began and deciding on
|
||||
the rest of the sweep too, and only where that re-check ends elsewhere is the whole run made again with
|
||||
every near tie decided late; a run that found no lattice after carrying a near tie is made again
|
||||
likewise. Both give the answer that deciding late throughout gives; the live run pays for it with the
|
||||
re-checks (measured: a median of 0.9 s on the sets that had one). The checks taken on the
|
||||
pre-pass frames - the geometry walk's refits, the metric-symmetry arm and the pre-pass spot-crowding count - stay on those frames
|
||||
in both modes, and their near ties are re-checked once every frame is in on both.
|
||||
|
||||
**A first pass that ends with no usable lattice tries a leaner, shallower one.** A crystal sitting in
|
||||
a crystalline powder floods the pass with spots that are not its own — measured, 2112 spots a frame
|
||||
|
||||
@@ -1186,10 +1186,12 @@ struct RotationScaleMergeGPU::Impl {
|
||||
//
|
||||
// The failed allocation is handled here, so its error must not stay behind as this thread's last
|
||||
// error: the next kernel launch's cudaGetLastError() would report it over work that went fine.
|
||||
//
|
||||
// Plain Synchronous, not ALLOC: this function waits for the work beside itself, with its own limit.
|
||||
template <typename T>
|
||||
CudaDevicePtr<T> Alloc(size_t n) const {
|
||||
try {
|
||||
return CudaDevicePtr<T>(n, ALLOC);
|
||||
return CudaDevicePtr<T>(n, CudaAlloc::Synchronous);
|
||||
} catch (const JFJochException &) {
|
||||
cuda_throw_if_context_lost(); // not memory: waiting would only hide the real error
|
||||
cuda_clear_error();
|
||||
@@ -1197,7 +1199,7 @@ struct RotationScaleMergeGPU::Impl {
|
||||
if (!wait_for_gpu_work_beside(GPU_BUSY_TIMEOUT))
|
||||
ThrowGPUBusy(n_obs);
|
||||
try {
|
||||
return CudaDevicePtr<T>(n, ALLOC);
|
||||
return CudaDevicePtr<T>(n, CudaAlloc::Synchronous);
|
||||
} catch (const JFJochException &) {
|
||||
ThrowOutOfGPUMemory(n_obs, n * sizeof(T));
|
||||
}
|
||||
|
||||
@@ -324,7 +324,7 @@ public:
|
||||
points = sf->Reflections();
|
||||
bytes = sf->DeviceBytes() + ModelScaleGPU::DeviceBytes(points) + 2 * points * sizeof(std::complex<float>);
|
||||
quarter = sf->DeviceTotalMemory() / 4;
|
||||
} catch (const JFJochException &) {
|
||||
} catch (const std::exception &) {
|
||||
cuda_clear_error();
|
||||
sf.reset();
|
||||
}
|
||||
@@ -352,7 +352,7 @@ public:
|
||||
engine->fmask = CudaDevicePtr<std::complex<float>>(points);
|
||||
engine->sf = std::move(sf);
|
||||
set_gpu(device);
|
||||
} catch (const JFJochException &) {
|
||||
} catch (const std::exception &) {
|
||||
engine.reset();
|
||||
cuda_clear_error();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user