With the main image pass already waited for, a simulated 8 GB card with 7.0 GB free (the display's
share taken) still ran out of memory: the rotation-scale walk's index-only probe and the speculative
geometry probe ran beside each other during the merge, and the walk's cuFFT plan failed. On a small
card a probe pass now also waits, before it starts, for every other probe pass to end. Scheduling
only; nothing changes on a 16 GB card.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a 16 GB card a probe pass (GPUWorkBeside) runs its image work beside the main line's image pass
(GPUImagePass). Measured on a 16 Mpx detector with nsys, the two together held 8.6 GB of device
buffers (two sets of 24 per-worker 67 MB image buffers plus five 0.6 GB FFT indexers), so on an
8 GB card the refined-geometry pass ran out of memory ("Allocating GPU memory (53.0 MB) failed").
Below 12 GB of card memory each of the two now waits, before it starts, for the other kind to end.
Only when each runs depends on the card, never what it computes; neither waits once running, so
nothing waits in a circle. On a 16 GB card nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
hide_gpus() in CUDAWrapper makes get_gpu_count() answer 0 without calling
cudaGetDeviceCount, so every GPU decision in the process (indexer Auto ->
FFTW, preprocessing/spot finding/decode, pre-scan, scaling engine, model
validation, MR) takes the path it takes on a machine without a GPU, and no
CUDA context is created. rugnux reads --no-gpu in main, ahead of the parser,
because set_gpu_blocking_sync() already asks CUDA for the devices. The
banner says 'CPU only (--no-gpu)'. The 'run it on the CPU' advice in the
out-of-GPU-memory messages and in the install docs now names --no-gpu
instead of CUDA_VISIBLE_DEVICES=.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Several pipeline copies of one run build FFT indexer pools on the card at once - the
rotation-scale walk's indexing probes start as the output pass begins its image loop,
and the speculative geometry probe runs beside the merge - and each indexer is a cuFFT
plan plus 0.6-0.9 GB of buffers. Construction had no admission control: a cufftPlanMany
or device allocation that found the card full failed straight through, and since
IndexerResult carries the exception that ends the run. With 7.2 GB of the card held by
another process, a small-molecule rotation set failed 4 of 4 runs this way although
each pass fits on its own.
Construction now does what the merge's allocations do: hand back the pool's unused
memory, wait for the work beside it to end, and build again. Who waits for whom is
fixed so that nothing waits in a circle:
- the main line waits for the probes (GPUWorkBeside, as the merge already does);
- a probe waits for the main line's image pass (new GPUImagePass), never for a probe,
itself included. The image pass waits for no probe, so it ends.
The walk's indexing probes now hold a GPUWorkBeside too (they are exactly that work),
and a run carries gpu_work_beside_ so its pools know which side they are on. Only when
an indexer is built changes, never what it computes; with nothing to wait for it still
fails at once.
Same run with the 7.2 GB hog: 5/5 complete (also at 7.6 and 8.0 GB), p.hkl identical
to the unhogged run; a card that cannot hold the run still fails in 0.5 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
With -N left to choose, rugnux now confines the process (main thread, the CPUs
RestoreThreadAffinity hands back, and what AvailableCpus counts) to the CPUs of
the packages its cards hang off, and sizes its threads to them. Nothing changes
where the allowed CPUs sit in one package, there is no card or a card's NUMA
node is unknown, or the cards' packages already hold every allowed CPU. A mask
set from outside (taskset, numactl, a cpuset) is kept: the decision works
within it. An explicit -N keeps the old behaviour.
On a 48-CPU server with four cards on two of its four NUMA nodes, a rotation
run took 24.5-24.7 s confined to those nodes' 24 CPUs against 27.7-27.9 s on
all 48 (scaling and merging 8.1 -> 6.2 s), and two larger sets moved
95-99 -> 84-87 s and 131-135 -> 113-115 s. On a single-socket machine this is
a no-op.
The decision itself (CpusOfGpuPackages) is a pure function, tested on
synthetic 4-package, 2-package hyper-threaded, cards-on-both, no-card and
restricted-mask topologies.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Exact: P4 p.hkl identical to the base on 39 noise-subset sets; concurrent arms p.hkl md5 and MTZ data identical
to P4 on 12 sets at default -N and -N 8 (incl. 5 sets where the arms disagree). Side arm on card 2 with 3+ cards,
else on the last card when 2 x observations x 256 B fits its total memory (decided up front).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
set_gpu_blocking_sync() called cudaSetDevice() on every card, and since
CUDA 12 that creates the card's primary context: ~0.18 s per card on the
workstation (measured, RTX 5080), one after another, before WALL_TIME
starts. On the 4-card server that is most of the 1.73 s per run spent
outside WALL_TIME (vs 0.37 s on one card).
- The blocking-sync flag is now set with the driver's
cuDevicePrimaryCtxSetFlags, taken through cudaGetDriverEntryPointByVersion
(no link against libcuda, so a CUDA build still starts without a
driver). It creates no context; contexts created later carry the flag
(cudaGetDeviceFlags 0xc, as before; user CPU time unchanged, so no
spinning).
- Card 0's context is created by the existing cuFFT warm-up thread, beside
the file open and pre-scan; cards 1..N-1 get theirs on a new background
thread per card (create_gpu_contexts), as the pre-scan uses card 0 only.
- At exit the contexts are destroyed by one thread per card
(release_gpu_contexts, cudaDeviceReset) after the run has destroyed
everything that used them, instead of serially by the runtime's exit
handler. Microbenchmark: an explicit reset costs the same as the
implicit teardown on one card (175 ms), so no change on one card.
Which card does what is unchanged; only when contexts are made/destroyed.
Measured on the 1-card workstation (load 7-16, 2 repeats x 2 sets plus
2 earlier pairs on myob): process start -> first log line median 0.39 ->
0.18 s; last log line -> exit 0.18 -> 0.20 s (one context either way);
elapsed - WALL_TIME median 0.53 -> 0.37 s; WALL_TIME unchanged (context
creation now overlaps the input read). The multi-card gain (3 serial
context creations at start, 4 serial destroys at exit) cannot be measured
here. p.hkl md5 identical on myob_x10sa, cytc_x10sa, 2xfw, 6zqy -N 8;
all runs exit 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The space-group search made its two P1 merges one after the other on one
engine: the Lorentz-filtered one, its search, then the all-observation one and
its search. Since every merge now starts from the same state, the
all-observation merge is made on a copy of the engine (CloneIngest: the host
half of the ingest copied, the device half staged on the copy's card), started
right after the ingest and read where the search needs it, so it runs beside the
P1 merge, the re-scaled search merge and the first arm. Its search then runs
beside the first arm's.
Placement: from three cards up the copy gets card 2 to itself (card 0 has the
main engine, card 1 the quality guard's). With one or two cards it shares the
last card, and only where two engines of this size fit the card's total memory
(DEVICE_BYTES_PER_OBSERVATION = 256; two engines measured at ~150 B/observation
each), so the choice never depends on what is free at the moment. On a 16 GB
card that admits sets up to ~32M observations; larger ones keep the serial
order. Nothing of the copy reaches the run before the first arm says the arm is
wanted: its log, its result fields and its per-frame write-back onto the
outcomes are taken at the point where the serial merge would have been made. A
copy that failed is merged again in series.
Results are bit-identical to the serial order (p.hkl md5 and MTZ data), at
default -N and -N 8.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two
threads that drive the merging tail were not: the quality guard took its card with set_gpu(1),
and the main engine runs on the pipeline's own thread, which goes on to everything else. On a
multi-socket server their pageable copies to the card went through a remote socket.
RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost,
IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous
device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding
covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads
the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use
std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not
squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses
pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask
themselves; the new test checks both that and that a pinned caller keeps its own mask.
No flag, no environment variable. On one NUMA node (this workstation) only the device setting
happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base =
rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa,
cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs),
-N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise,
e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for
multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
* jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports.
* jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls.
* Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results.
* Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable.
* Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate.
* Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do.
* Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence.
* Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags.
* Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check.
* Rugnux: Clear error messages when a data set needs more GPU or host memory than is available.
Reviewed-on: #83
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection.
* Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100).
* Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused.
* Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image.
Reviewed-on: #82
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-).
* `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands.
* `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`.
* `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed.
* `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name.
* A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log.
* The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed.
* The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`.
* rugnux says which GPUs it can see before it starts processing.
* `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`.
* `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged.
* A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`.
* `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200.
* A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`.
* `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`.
* The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before.
Reviewed-on: #75
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
leonarski_fandClaude Opus 5.5