v1.0.0-rc.172 #82
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mx/Jungfraujoch#82
Reference in New Issue
Block a user
jfjoch_brokercancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection.--scaling-iterationsis now the cap on that loop (default 100).<prefix>_detector.jpg, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and<prefix>_plot.txt, one row per image.When the all-observation and Lorentz-filtered search merges disagree, the higher point group is adopted. The losing arm's answer was dropped, so a promotion one arm refused on its twin gates (H ratio, added-operator R) went through with nothing in the output saying a gate had fired - and the report printed the adopted group's failing H ratio ("2.02 against a bound of 1.85") as if it were a margin. Seen on an H3 partial twin (5j23), refused on both twin gates by the all-observation merge and promoted to R32 by the filtered one. Now the losing arm's refusal (group and reason), or the fact that it never confirmed the operators, is logged as a warning and written to the report (SPACE_GROUP_ARM_OVERRIDE). A bound the adopted group exceeds is printed as "EXCEEDS its bound - that gate fired and was overridden", for the H ratio and for the best-operator R. The comment that claimed both arms keep their vetoes now describes the rule as implemented. The rule itself is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Every map -> structure-factor transform on the model path - Fcalc from the model density and Fmask from the bulk-solvent mask, in the fit, the rigid-body evaluations (two per evaluation, ~120 evaluations per placement), the frame probe and the stills model reference - went through gemmi's vendored pocketfft, single-threaded. They now go through MapToFPhi (rugnux/ModelFFT.{h,cpp}): an FFTW r2c (fftw3f, already fetched and linked) planned with the guru interface on the same layout gemmi uses (u fastest, w halved), scaled by V/N, into the same FPhiGrid, so gemmi's prepare_asu_data() extracts the reflections exactly as before. gemmi's vendored code is not modified. The F -> map transforms of the output maps are left on gemmi. Plans are FFTW_ESTIMATE only (MEASURE times candidates and could choose differently between runs), made under a mutex - FFTW's planner is not thread-safe, and the null replicates and the parallel Jacobian call this concurrently - cached per grid size for the life of the process, and executed with fftwf_execute_dft_r2c on fftwf_malloc buffers (new-array execution needs the planning arrays' alignment, which a std::vector does not promise). NOT bit-identical with pocketfft: the two libraries order their arithmetic differently, so structure factors move in the last float bits and everything downstream (scale fit, rigid body, R-factors, maps) can move in the last printed digit. Verify by tolerance, separately from the parallelisation commits: MODEL_* keys equal to printed precision and the same MODEL_FIT / hand / indexing decisions on the audit set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByATA long-running broker began cancelling every data collection with Device decoding failed (CUDA (GPU) error (out of memory)), falling back to host decompression CUDA (GPU) error (out of memory) while nvidia-smi showed the cards less than a fifth full. Two defects, both in CUDAMemHelpers.h and both from the pooled allocator (cudaMallocAsync) that came with rc.162. The leak. cuda_allocation_stream() kept one stream per (thread, device) in a thread_local map of raw cudaStream_t and never destroyed them. That was written against rugnux, where the worker threads live as long as the process. The broker starts fresh std::async threads for every data collection - 16 or 64 of them - so every collection left that many streams behind. Measured: 0.56 MB of device memory per leaked stream, linear to 4928 streams, never returned, with nothing on the host side growing. The RAII wrapper (CudaStream) was there but not used at this site, and using it as-is - a stream destroyed when its thread exits - would not have been safe: ShadowFinder builds its GPU accumulator on a throw-away std::async thread and frees it from another thread long after, and that free is ordered on the allocating thread's stream. So the streams are still never destroyed, but a thread now only borrows one: CudaStream objects live in a process-wide per-device idle list, a thread takes one on first use and hands it back when it exits. Their number is bounded by the threads that were ever alive at once instead of by the threads ever started. The list itself is deliberately leaked, so that nothing calls into CUDA during static destruction. Replaying the broker's pattern against the real header, 60 collections of 64 threads: before 3840 streams, 260 -> 2424 MB of device memory after 64 streams, 260 -> 358 MB The stale error. Every helper here throws a named message, yet the log carried the raw CUDA string, so the failure came through a cuda_err() and not from an allocation. CudaDevicePtr falls back to cudaMalloc when cudaMallocAsync fails, silently - but the failed call stays behind as the thread's last error, and the cudaGetLastError() that follows the next kernel launch reports it. The buffers were all allocated; the frame was lost anyway, once on the device-decode route (caught, hence the warning) and once more on the host fallback (fatal). The pooled attempt failing, and a stream that cannot be created, are both handled by falling back, so both now clear the error they leave. What finite resource the production cards ran out of at under 4 GB used was not established - no cap on the number of streams was found up to 4928 on the card this was measured on. The leak is the only thing on this path that grows with uptime. tests/CUDAMemHelpersTest.cpp: later threads end up on the same stream, concurrent threads on different ones, a buffer is freed cleanly after its allocating thread has exited (and another has borrowed its stream), and a pool that cannot serve a request leaves no error behind - the last by capping a memory pool at 4 MB so that the pooled attempt fails and the fallback succeeds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>--model and --report-resolution are both processing-inert (merged reflections bit-identical with and without them), so the bare/model and bare/xds variant pairs collapse into one command per arm: open: rugnux -o p --no-export-unmerged --model <deposited coords> <input> (no --model where there is no deposited model) inhouse/private: rugnux -o p --no-export-unmerged [-A] --report-resolution <reference d_min>,<XDS d_max> <input> Every row is rugnux's own result (own cut vs the reference, as `bare` was), and on the XDS arms also carries the REFRES_* table, scored against CORRECT.LP's completeness, multiplicity, R_meas, CC1/2 and ISa, with ISa and R_meas ratio plots and a like-for-like table; REFRES_SHELLS_PAST_LIMIT > 0 marks the row as coverage (rugnux's cut coarser than the reference). Nothing is forced on the processing any more. On the open arm the space group is scored on the data's own determination (SOHNCKE_SPACE_GROUP where SPACE_GROUP_ENANTIOMORPH is ASSUMED_FROM_MODEL), the model's label kept as sg_label; R_FREE/R_WORK/CC_MODEL /MODEL_FIT are trend fields (placement-only, own free set); REFMAC stays opt-in. results.json schema 3: one row per set, no variant/first_read; older runs are read through their bare rows (open-arm model R-factors folded in), so compare and report keep working against them. Every time is now a first read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The full-resolution merge-degradation gate divided the added operators' mean intensity-weighted R by the smallest R anywhere in the crystal. That denominator is an extreme order statistic, so it has a pole: one unusually clean operator condemns every other genuine one. It is not a rare accident. A rotation about an axis near the spindle maps a reflection onto one recorded at nearly the same detector position, so the lab-frame systematics cancel for that operator alone; measured over the rotation corpus the spread WITHIN a genuine group reaches 2.0-2.6x, the whole width of the old bound of 2.0. On a weak cubic crystal whose near-spindle 3-fold read 0.074 against 0.16-0.22 for its ten other genuine operators, every one of them was refused, the search kept only the group generated by the reference operator itself (which reads 1.00 by construction), and the same 222 hypothesis on the same merge read 2.45 under a cubic enumeration and 1.02 under an orthorhombic one - a statistic that moves with which OTHER operators were enumerated. The added operators are now placed on the scale the crystal itself defines, between the R of unrelated reflections and the R of the best-agreeing operator: contrast = (random_pairing_r - r_added) / (random_pairing_r - global_best_operator_r) random_pairing_r is new: the same intensity-weighted R over shell-matched pairs of reflections no symmetry relates, measured on the merge (deterministic, no RNG). Both ends are hypothesis-free, and the form has no pole - an unusually clean reference widens the denominator by a few per cent instead of driving the divisor towards zero. The test abstains where the best-agreeing operator is itself no better than half way to unrelated reflections, i.e. where the merge holds no clean end to measure from, as it already abstains when there is no second operator at all. Calibrated over 149 rotation datasets with a known answer, read as the gate reads it: genuine promotions reach down to 0.77 and the worst false one reads 0.695, so the bound is 0.72. It keeps every refusal the old ratio made on that corpus except the weak cubic one, which now promotes to its cubic group; the twins that read above the genuine range are refused by the H gate and the twin-immune zone, as before. The reported value, the point-group report line, the finalist ledger and the gate-fired-and-was-overridden note all follow the contrast. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByATrc.172 stopped two CUDA failures from staying behind as the thread's last error, both inside CUDAMemHelpers.h: a pooled allocation that fell back to cudaMalloc, and a stream that could not be created. The reason those mattered was general, and four other places have it too. Nothing but cudaGetLastError() takes an error away. A later successful call neither clears it nor returns it - measured: after a cudaMalloc that cannot be served, cudaStreamSynchronize, cudaMemcpyAsync and a small cudaMalloc all report success, and the "out of memory" is still waiting for whoever calls cudaGetLastError() next. In this code that is the check after a kernel launch. So wherever a CUDA failure is caught and the work carries on by another route, the error it left is reported over the route that went fine: - the device decoder in MXAnalysisWithoutFPGA::Analyze, whose failure is answered by decompressing on the host - and the host route runs kernels of its own. This is the broker's path, and it is the pair of log lines a long-running broker showed, one warning naming the device route and one fatal error naming nothing. - the indexer worker, which turns a failed attempt into a result and goes on to the next frame - on the same thread, so one failure would read as an indexer that never works again. - the beam stop's GPU accumulate and the FFT beam-centre capture, both of which fall back to the host. Each discharges it now, through cuda_clear_error() in CUDAWrapper (a no-op without CUDA). A sticky error is not cleared by this and nothing here pretends otherwise. tests/CudaHandledErrorTest.cpp: that a failed allocation leaves an error which survives later successful work until it is cleared, and that an image whose device decode fails comes out of Analyze whole and leaves nothing behind for the next one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT