v1.0.0-rc.177 #87

Merged

leonarski_f merged 11 commits from rc177 into main 2026-10-19 16:07:31 +02:00

Conversation 0 Commits 11 Files Changed 183

+797 -445

leonarski_f commented 2026-10-19 16:07:16 +02:00

Owner

Copy link

Copy source

• jfjoch_broker: Faster live Rugnux analysis, with the GPU indexers prepared at the start of a collection and kept between collections; Rugnux no longer writes .ccp4 maps (the map coefficients are in <prefix>_maps.mtz).

* jfjoch_broker: Faster live Rugnux analysis, with the GPU indexers prepared at the start of a collection and kept between collections; Rugnux no longer writes `.ccp4` maps (the map coefficients are in `<prefix>_maps.mtz`).

leonarski_f added 11 commits 2026-10-19 16:07:16 +02:00

rugnux: no .ccp4 map files - the maps are given as coefficients in _maps.mtz only ... 80ed784d1d

A model validation no longer writes <prefix>_2fofc.ccp4, _fofc.ccp4 and _anom.ccp4 (nor their _mr
twins); <prefix>_maps.mtz, which Coot and PyMOL open as maps, is what is written. With the files
goes everything that only existed for them: ModelMaps keeps the coefficients alone, the
keep_map_grids switch is gone from the validation schedule and the rugnux config, and the
mFo-DFc map is no longer transformed to real space at all (nothing read it). The 2mFo-DFc and
anomalous maps are still made, for the mean atom density and the anomalous-site read-out.

The anomalous difference map is now in no output file: _maps.mtz carries no anomalous columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

PickBest: the file centre's indexing starts before the work beside it, so it overlaps the beam-centre capture wait ... a955bee0fc

With two or more GPUs StartCentreSpeculation joins the pre-scan's beam-centre capture; that join
used to come before the file centre's schemes were indexed, so on the live server (capture 2.1 s)
the file centre and the short-axis pass waited for it. The file centre's indexing now runs on its
own thread while alongside() runs, and the short-axis pass is started before the centre
speculation. Each indexer reads only what it was fed, so the results are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

BeamCenterFFT: the 99.9th percentile and median of the projection by radix select instead of nth_element ... b3039245f9

PrepareImage copied the finite pixels and ran two nth_element over them (18 M pixels on a 16M
detector, ~200 ms single-threaded). Two counting passes over the float bits (upper, then lower
16 bits of an order-preserving key) give the same k-th values in ~55 ms without the copy. Values
are identical; only the sign of a zero picked as a quantile may differ, which the clipping maps
to the same prepared image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

rugnux live: the lattice probe runs without the beam-centre check ... 2828c359cd

The probe reports only the lattice and indexing rate on the first 60 degrees and decides nothing
for the run. Its beam-centre capture and second first pass at the measured centre loaded the GPUs
beside the main run's pre-scan for a centre nobody reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

Model validation: warm up on the GPU when the broker gets a model ... 0ff6442765

WarmUpModelValidation runs one throwaway validation of the uploaded model against its own
calculated intensities on a background thread (holding a GPUWorkBeside), so kernels, cuFFT
kernels and contexts on every card the null's replicates use are loaded before the collection's
validation. ValidateAgainstModel waits for a warm-up still running, so a real validation never
shares the cards with it and computes what it computes without one (test
ModelValidation_WarmUpChangesNothing; CLI run with and without warm-up: identical p.mtz,
p_maps.mtz, report and validation lines). The broker logs the warm-up's time at upload, which
measures the first-use cost on the machine directly (local RTX 5080: 0.88 s first, 0.78 s second).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

Merge branches 'rc177-no-ccp4', 'live5s-a2' and 'live5s-a5' into rc177-live 13113b1751

rugnux live: keep FFT indexers and cuFFT plans for the broker's lifetime, built at START ... 19ee20fbe8

jfjoch_broker opened no FFTPlanReuse, so every cuFFT plan of a live run was made on first use and
destroyed after it, and every pass (first pass, measured-centre arm, short-axis pool, pass 2) built
its GPU FFT indexers afresh. Now:
- FFTIndexerReuse: while one lives, a pool worker borrows a kept GPU FFT indexer of its card and
  construction settings instead of building one, and gives it back when it ends or is released.
- The rugnux receiver holds an FFTIndexerReuse and an FFTPlanReuse for the broker's lifetime.
- At START, beside the run (which is waiting for its first frames), WarmUpRugnuxLiveGPU builds on
  every card the first-pass and short-axis indexers and the beam-centre capture's plan pair.
The CLI opens no FFTIndexerReuse and is unchanged. Results identical: CLI p.hkl md5 equal on two
sets (CLI and --simulate-live --fast-live); broker replay, 3 collections each, MTZ data identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

Merge branch 'live5s-a1' into rc177-live b5d666af0e

1.0.0-rc.177 ... 2841312c86

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

broker rugnux live: no nice - the analysis and the probe run at the broker's own priority ... c289e288df

The rugnux live thread, its lattice probe and the process-wide ParallelFor pool were lowered to nice 10
while a collection ran (and the pool raised back afterwards where the process had the right to). That
is removed: the analysis runs at the broker's priority, and WorkerPool::SetNice with the thread-id
bookkeeping it needed goes with it.

Measured on a workstation (model validation, 10 rigid bodies, under 32 busy threads at nice 0): nice 10
made the host-sync-heavy GPU work 4-15x slower (6.5 s at nice 0 vs 24.7 s at nice 10 with blocking
sync); on the server the post-collection tail ran at nice 10 though the forward threads are idle then.
Results unchanged (p.hkl md5 identical, simulated live).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

rugnux: the model's first validation starts beside the analyses instead of after them ... 7e5e4c54e4

Validation reads of the final merge only its French-Wilson amplitudes, which the analyses
made last, from the anisotropy tensor. That tensor is a fit to the merged intensities alone
(the unmerged observations only give its error bar), so AnisotropyPriorHKL computes it from
the merge without the rest of the analysis, bit for bit what AnisotropyTensorHKL of the full
analysis gives (test). The first ValidateAgainstModel now runs from the start of the analyses
on a copy of the merge given those same amplitudes, on the caller's card (get_gpu), with its
log held and replayed where it was printed before; nothing it decides is acted on earlier.
The model's preparation (census wait, indexer pool, GPU hold, merge engine release) moves
with it.

Measured, --simulate-live 10 --fast-live, 4axt model, RTX 5080, analyses + validation steps:
strong lysozyme 0.91 -> 0.63 s, reference lysozyme 1.02 -> 0.66 s (validation beside the
analyses 0.60-0.62 s vs 0.39-0.45 s alone, CPU shared; analyses 0.53 -> 0.62 s). The
analyses were 80% waiting for the anisotropy (0.43 of 0.53 s), so starting on the full
analysis' tensor would have gained ~0.1 s. p.hkl, p_maps.mtz identical, p.mtz data identical,
all validation numbers identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

leonarski_f merged commit 7ed32b899f into main 2026-10-19 16:07:31 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) mx-gitea perl_d (perl_d) takaba_k (takaba_k) wang_m (wang_m)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#87