From 645d1cc1bbe4a9bda280471b71a2ae3e5b6589e0 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 13 Oct 2026 07:41:37 +0200 Subject: [PATCH 01/10] CHANGELOG: rc.175 says --model falls back to molecular replacement and takes predicted models; viewer refresh The --mr line now says what the unified --model flow does for a user: a predicted model is accepted, and a model the rigid body does not place goes to molecular replacement, with --mr forcing it. One viewer line for the Files panel refresh (F5 and the poll) and the combo boxes that no longer reset during live collection. Owner-requested edit. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3 --- docs/CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index dee2cb9b5..16fa1f934 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,8 +7,9 @@ * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `_twin_.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. -* Rugnux: Experimental `--mr` places a predicted or experimental model by molecular replacement to settle space group, hand and twinning, writing a different answer as `_mr.mtz`. +* Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `_mr.mtz`. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. +* jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * Fixes: The TCP image pusher no longer blocks the broker's receiver, the master file appears only after its data files, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. ### 1.0.0-rc.174 From 463cd8eec357ad6ec92e3c51c717d357ab2abf4a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 13 Oct 2026 07:54:56 +0200 Subject: [PATCH 02/10] CBF byte-offset decoder: plain stores instead of SSE streaming stores The host decoder wrote each pixel with _mm_stream_si32 and ended with _mm_sfence, to keep the decoded frame out of the cache on its way to the GPU by DMA. The GPU now decodes CBF itself, so the host decoder only serves the CPU engines and the tests, and the project rule is no SIMD intrinsics. Plain stores; the values written are the same. CBFDecoderGPU_MatchesHost, CBFDecoderGPU_ShortStreamIsRefused, ImagePreprocessorGPU_StoredEncodingsMatchCPU and the CBF reader tests pass. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3 --- compression/CBFByteOffset.cpp | 29 +---------------------------- 1 file changed, 1 insertion(+), 28 deletions(-) diff --git a/compression/CBFByteOffset.cpp b/compression/CBFByteOffset.cpp index f618e4e02..2e37db681 100644 --- a/compression/CBFByteOffset.cpp +++ b/compression/CBFByteOffset.cpp @@ -6,11 +6,6 @@ #include #include -#if defined(__SSE2__) || defined(_M_X64) || defined(_M_AMD64) -# include -# define MINICBF_STREAM_STORE 1 -#endif - #include "../common/JFJochException.h" namespace { @@ -19,26 +14,6 @@ int16_t ReadI16(const uint8_t *p) { int16_t v; std::memcpy(&v, p, 2); return v; int32_t ReadI32(const uint8_t *p) { int32_t v; std::memcpy(&v, p, 4); return v; } int64_t ReadI64(const uint8_t *p) { int64_t v; std::memcpy(&v, p, 8); return v; } -// Nothing on this side reads the decoded image back: it goes to the GPU by DMA, or a CPU engine -// walks it once and drops it. An ordinary store therefore pays twice over - it reads every cache -// line before overwriting it, and it evicts 72 MB of live cache to make room - so the pixels go -// straight past the cache. Measured on an 18 Mpx frame: 27 ms of ordinary stores against 11 ms of -// these. The bytes written are the same either way; the fence is what makes them visible to the -// copy that follows. -void StorePixel(int32_t *out, int32_t value) { -#ifdef MINICBF_STREAM_STORE - _mm_stream_si32(out, value); -#else - *out = value; -#endif -} - -void StoreFence() { -#ifdef MINICBF_STREAM_STORE - _mm_sfence(); -#endif -} - } // namespace // The x-CBF_BYTE_OFFSET scheme: a running value, each pixel stored as a delta in the smallest @@ -75,12 +50,10 @@ void DecodeCBFByteOffset(const uint8_t *data, size_t size, int32_t *out, size_t } value += delta; - StorePixel(out + written, static_cast(value)); + out[written] = static_cast(value); written++; } - StoreFence(); - if (written != n_pixels) throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "miniCBF byte-offset stream ended after " + std::to_string(written) From e53c3f50cda14560725ab4a1ec0ae66967111619 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 13 Oct 2026 07:54:56 +0200 Subject: [PATCH 03/10] CalcBraggPredictionTest: skip the GPU cases on a machine without a card None of the eight GPU test cases in the file (BraggPredictionGPU*, the CPU/GPU consistency cases and BraggPredictionRotGPU_DeviceOrderMatchesHost) checked for a device, so a CUDA build on a machine without one failed them instead of skipping. Same guard as the other GPU tests. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3 --- tests/CalcBraggPredictionTest.cpp | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/tests/CalcBraggPredictionTest.cpp b/tests/CalcBraggPredictionTest.cpp index 465a6d4d7..d9ef3bb74 100644 --- a/tests/CalcBraggPredictionTest.cpp +++ b/tests/CalcBraggPredictionTest.cpp @@ -525,6 +525,7 @@ TEST_CASE("BraggPredictionRot_ColumnRangeKeepsEveryReflection", "[portable]") { } #ifdef JFJOCH_USE_CUDA +#include "../common/CUDAWrapper.h" #include "../image_analysis/bragg_prediction/BraggPredictionGPU.h" #include "../image_analysis/bragg_prediction/BraggPredictionRotGPU.h" #include "../image_analysis/bragg_prediction/BraggPredictionRot.h" @@ -532,6 +533,8 @@ TEST_CASE("BraggPredictionRot_ColumnRangeKeepsEveryReflection", "[portable]") { #include TEST_CASE("BraggPredictionGPU") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); DiffractionExperiment experiment(DetJF4M()); experiment.DetectorDistance_mm(100.0).BeamX_pxl(1500.0).BeamY_pxl(1000.0) .IncidentEnergy_keV(13.0); @@ -568,6 +571,8 @@ TEST_CASE("BraggPredictionGPU") { } TEST_CASE("BraggPredictionGPU_Rot1_Rot2") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); DiffractionExperiment experiment(DetJF4M()); experiment.DetectorDistance_mm(100.0).BeamX_pxl(1500.0).BeamY_pxl(1000.0) .PoniRot1_rad(2.0/180.0 * PI).PoniRot2_rad(3.0/180.0 * PI) @@ -605,6 +610,8 @@ TEST_CASE("BraggPredictionGPU_Rot1_Rot2") { } TEST_CASE("BraggPredictionGPU_systematic_absences") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); DiffractionExperiment experiment(DetJF4M()); experiment.DetectorDistance_mm(120.0).BeamX_pxl(1500.0).BeamY_pxl(1000.0) .IncidentEnergy_keV(12.0); @@ -664,6 +671,8 @@ TEST_CASE("BraggPredictionGPU_systematic_absences") { } TEST_CASE("BraggPredictionGPU_backscattering") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); DiffractionExperiment experiment(DetJF9M()); experiment.DetectorDistance_mm(120.0).BeamX_pxl(1500.0).BeamY_pxl(1000.0) .IncidentEnergy_keV(3.0/WVL_1A_IN_KEV); @@ -691,6 +700,8 @@ TEST_CASE("BraggPredictionGPU_backscattering") { } TEST_CASE("BraggPrediction_CPU_GPU_consistency_tilted") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); // Verify CPU and GPU implementations produce identical results with tilted detector DiffractionExperiment experiment(DetJF4M()); experiment.DetectorDistance_mm(100.0).BeamX_pxl(1500.0).BeamY_pxl(1000.0) @@ -761,6 +772,8 @@ TEST_CASE("BraggPrediction_CPU_GPU_consistency_tilted") { } TEST_CASE("BraggPredictionRot_CPU_GPU_consistency_tilted") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); // The rotation counterpart of the test above. Same purpose: every per-reflection quantity the // two implementations both produce has to agree - the Lorentz-polarization factor and the sensor // efficiency each in their own field, so neither can hide behind the other in a product. @@ -818,6 +831,8 @@ TEST_CASE("BraggPredictionRot_CPU_GPU_consistency_tilted") { CHECK(max_flight > 1.0f); } TEST_CASE("BraggPrediction_CPU_GPU_consistency_large_cell") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); // A frame predicting more than kPredictionCapacity: both paths grow their buffer, so both predict the // same reflections. The CPU used to stop at the capacity and return the first 20000 of the walk. const auto experiment = LargeCellExperiment(); @@ -863,6 +878,8 @@ TEST_CASE("BraggPrediction_CPU_GPU_consistency_large_cell") { } TEST_CASE("BraggPredictionRotGPU_DeviceOrderMatchesHost") { + if (get_gpu_count() == 0) + SKIP("No CUDA GPU present"); // The GPU predictor orders and selects its reflections on the device. What it returns has to be what // OrderOutput and TruncateToOutput make of the same predictions on the host: the full list in hkl, // delta_phi order, and under a cap the best-recorded ones in TruncateToOutput's order. From 5e916fe1206893938db3286436fe5c6d86e556c1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 13 Oct 2026 08:07:44 +0200 Subject: [PATCH 04/10] docs: --model places by rigid body, then molecular replacement; --mr; _mr files RUGNUX_ADVANCED's --model page described the rigid body alone. It now says how the unified flow places a model: an experimental model by the rigid body, checked against its null in the answer's or the twin hypothesis' merge, and by molecular replacement where it is not placed; a prediction straight to molecular replacement; --mr skipping the rigid body; MODEL_PLACEMENT. It says what a placed model may change in .mtz (the hand, and from the rigid body an indexing-ambiguity reindex) and that another group or the twin subgroup the model chooses goes to _mr.mtz/.cif with _mr_model.pdb/.cif. --mode scale places by the rigid body only (no molecular replacement, --mr refused); a full run does molecular replacement on serial stills too (checked: a stills run with a predicted model reports MODEL_PLACEMENT= MR). The options table gains --mr and the --model row the fallback. RUGNUX_INTEGRATION no longer says rugnux does no molecular replacement, and ACKNOWLEDGEMENT names --model beside --mr for predicted models. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3 --- docs/ACKNOWLEDGEMENT.md | 4 ++-- docs/RUGNUX_ADVANCED.md | 46 ++++++++++++++++++++++++++++++++------ docs/RUGNUX_INTEGRATION.md | 6 +++-- 3 files changed, 45 insertions(+), 11 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 83bb65981..7083f4c66 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -438,8 +438,8 @@ Problems Whose Variables Separate" (1973), SIAM J. Numer. Anal. 10, 413-432 for solving separable nonlinear least squares problems" (1975), BIT 15, 49-57 [doi:10.1007/BF01932995](https://doi.org/10.1007/BF01932995). -**Predicted models** — `--mr` (experimental) places an AlphaFold prediction (or any model) by molecular -replacement. J. Jumper, R. Evans, A. Pritzel et al., "Highly accurate protein structure prediction +**Predicted models** — `--model` (and `--mr`) place an AlphaFold prediction (or any model the rigid +body does not place) by molecular replacement. J. Jumper, R. Evans, A. Pritzel et al., "Highly accurate protein structure prediction with AlphaFold" (2021), Nature 596, 583-589 [doi:10.1038/s41586-021-03819-2](https://doi.org/10.1038/s41586-021-03819-2); the models are those of the AlphaFold Protein Structure Database: M. Varadi, S. Anyango, M. Deshpande et al., "AlphaFold diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 0ac0a01f1..d8697aef2 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -161,6 +161,35 @@ the data cell and then placed as **one rigid body**, so a deposited model from a quite isomorphous still sits where the density is. The placement runs on the GPU where there is one and on the CPU otherwise, with nothing to set; the two agree to rounding, not bit for bit. +The rigid body only moves a model that already sits near its place, so its placement is **checked**: +the model counts as placed where its fit beats the same model refitted from random orientations (the +null behind `MODEL_FIT`, below), in the answer's merge or in the twin hypothesis' one. An experimental +model the rigid body does not place — a deposited structure of another crystal form, a model of a +subgroup the data were merged above, a file with no crystal cell — is then searched for by +**molecular replacement**: as many copies as the cell holds, in every space group the merge cannot tell +apart and in the twin hypothesis' subgroup (the search and its keys are in +[Molecular replacement](RUGNUX_REPORT.md#molecular-replacement)). A **predicted** model — AlphaFold or +ModelCIF, recognised from the file's content (pLDDT in the B column) — has no placement of its own and +goes to molecular replacement straight away, trimmed to the residues of pLDDT 70 and above and, where +the AlphaFold DB's predicted-aligned-error JSON sits beside it under its own name, cut into rigid +domains. `--mr ` is `--model` with the rigid body skipped: molecular replacement straight away, +whatever the model. The report says which way the model was placed, `MODEL_PLACEMENT=` `RIGID_BODY`, +`MR` or `NOT_PLACED`; a model placed nowhere is a result, not a warning, and the reflection files are +then what a run without a model writes. Where molecular replacement placed it, the R-factors, the maps +and the placed model below are those of its placement. Molecular replacement is part of the full +`--mode mx` run, rotation and serial stills alike; `--mode scale` (below) places the model by the +rigid body alone. + +A placed model changes two things in `.mtz`/`.cif`/`.hkl` and nothing else: the +**enantiomorph** label and, placed by the rigid body, the +[indexing ambiguity](#the-indexing-ambiguity) (molecular replacement relabels the hand, past the same +anomalous check, and never reindexes). Where the model chooses **another space group** with the +answer's rotations, or the **twin hypothesis' subgroup**, the answer's files stay the run's own and that +merge is written beside them as `_mr.mtz`/`.cif` — from the merge it was chosen on, with the +twin law and fraction where it is the subgroup — together with the model as placed there, +`_mr_model.pdb`/`.cif`; the report names them (`MODEL_CHOICE_FILE_PREFIX=`, +`MODEL_CHOICE_SPACE_GROUP=`). + Because the model moves, the input file no longer describes these maps, so the model **as placed** is written as `_model.cif` — the input's chains, residues, ligands, waters, B-factors, occupancies and anisotropic *U*s, at the coordinates the maps were computed from, in the same unit @@ -177,8 +206,9 @@ in a subgroup with a twin law, written when the adopted space group may have abs [the twin hypothesis](RUGNUX_REPORT.md#the-twin-hypothesis-twin_subgroup_)) — the model is scored against that merge as well, plainly and as twinned by the law, and the maps and `_model.cif` come from whichever of the two merges the model fits better on R-free. The report says which -(`MAPS_FROM=`) and gives the R-free of both. The model only chooses the maps: `.mtz` and the -space group are the run's own either way. Both are written whenever the +(`MAPS_FROM=`) and gives the R-free of both. `.mtz` and the space group are the run's own +either way; where the model also fits that merge against its null, it has chosen the subgroup, and the +merge is written once more as `_mr.mtz`/`.cif` (above). Both are written whenever the maps are, including for a model the data rejected: the rejection is a result, and it is exactly the case where someone wants to look at the model in the density. @@ -237,10 +267,11 @@ in P1 unless `-S` says otherwise. A reference MTZ (`-z`) is accepted here on **stills** data, where it fixes the space group and cell, reports CCref and hands over its R-free flags; on rotation data the rotation scaler -declines it and the run stops with a message saying so. `--model` works here exactly as in a full -run: on rotation data, where the [indexing ambiguity](#the-indexing-ambiguity) is one choice for -the whole dataset, it can settle the enantiomorph label and the indexing — under the same fit test -as always — and the re-merged reflections are written in what it settled. What no re-merge can +declines it and the run stops with a message saying so. `--model` works here as in a full run, except +that the model is placed by the rigid body only: a model it does not place, or a prediction, is not +searched for by molecular replacement, and `--mr` is refused. On rotation data, where the +[indexing ambiguity](#the-indexing-ambiguity) is one choice for the whole dataset, it can settle the +enantiomorph label and the indexing — under the same fit test as always — and the re-merged reflections are written in what it settled. What no re-merge can repair is a **stills** `_process.h5` integrated without a reference: there the ambiguity was decided per image, at integration time, and the distinction is gone from the stored reflections. @@ -421,7 +452,8 @@ Scaling and merging: | `--scaling-iterations ` | Cap on the per-frame scaling iterations; the loop stops earlier once the scales settle, and a merge that hits the cap is flagged `SCALING_NOT_CONVERGED` (default: 100) | | `-z, --reference ` | Reference data of the same crystal form — an MTZ or a PDB structure-factor mmCIF (`-sf.cif`), gzipped or not, recognised by content; `--reference-mtz` is the same option: fixes the space group, its setting and the cell, resolves the [indexing ambiguity](#the-indexing-ambiguity), hands over the R-free set and reports CCref. Not a scale anchor | | `--reference-column