From 5f47d73cc1448a94bf7da4edbfec23c8cad92b70 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sun, 30 Aug 2026 10:51:25 +0200 Subject: [PATCH] rugnux: write the unmerged MTZ by default The unmerged export is how a run's observations reach the rest of the field. It is what aimless, pointless and careless read, it is what a head-to-head against another program's answer runs through, and handing several of these files to pointless is the only way to merge sweeps rugnux does not combine itself. Behind a flag it reached only the people who already knew the flag existed, which is the shape of defect the "rugnux with nothing else" direction asks to design out. --no-export-unmerged turns it off, and the regression batteries now pass it: the objection to defaulting it on was their disk and time cost, not the product's. Measured on three rotation crystals of 900, 1800 and 3600 frames: the file is 21, 86 and 26 MB - two to five times the merged .mtz, .cif and .hkl put together - and the write costs 0.24, 2.29 and 1.11 s, 5.5 %, 12.2 % and 3.1 % of each run's own wall time. The cost is linear in the number of observations and independent of how long processing took, so it is the largest fraction of the runs rugnux finishes fastest, not of the longest sweeps. A run's .hkl, .mtz and .cif are byte-identical with the export on and off. --export-unmerged-partials stays off: it is a second, larger artefact and a separate question. The viewer's reprocessing dialog ties the export to its merged-output switch, so it appears beside the merged files rather than beside a job that only asked for the per-image _process.h5. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f --- docs/CHANGELOG.md | 1 + docs/RUGNUX.md | 31 ++++++++++++------- rugnux/Rugnux.h | 12 ++++--- rugnux/rugnux_cli.cpp | 7 +++-- rugnux_stills_ab.py | 5 ++- rugnux_vs_xds.py | 5 ++- viewer/windows/JFJochProcessingJobsWindow.cpp | 4 +++ 7 files changed, 44 insertions(+), 21 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index e2c0875ec..bceb0158a 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,6 +1,7 @@ # Changelog ## 1.0.0 ### 1.0.0-rc.166 +* `rugnux` writes the unmerged MTZ `_unmerged.mtz` by default; `--no-export-unmerged` skips it. * `jfjoch_viewer` opens PILATUS miniCBF sweeps - naming any frame opens the whole sweep - and can run a processing job on one. * A detector whose stored image is mirrored in Y or mounted at a multiple of 90 degrees can be described as such, in the detector configuration or with `--detector-mirror-y` / `--detector-quarter-turns`, rather than having to be expressed as a detector rotation. * The rotation first pass refines twelve candidate lattices rather than four, so a correct cell that the pre-refinement ranking put fifth is still reached. diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index f8d7ccc55..40ac207cd 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -245,8 +245,8 @@ rugnux -o quicklook -e 200 dataset_master.h5 # merge as usual, but also keep the per-image file so the data can be re-merged later rugnux -o myrun --write-process-h5 dataset_master.h5 -# also write the unmerged MTZ, to scale and merge the observations with another program -rugnux -o myrun --export-unmerged dataset_master.h5 +# skip the unmerged MTZ (written by default), when only the merged data is wanted +rugnux -o myrun --no-export-unmerged dataset_master.h5 # check the merged data against a known structure: R-work / R-free and maps rugnux -o myrun --model model.pdb dataset_master.h5 @@ -401,8 +401,8 @@ reusing them would hide the spot-finding settings from the lattice search. from systematic absences** (constrained to the indexed lattice symmetry). - `_unmerged.mtz` — the integrated observations *before* merging, as an unmerged MTZ in POINTLESS's column layout, so the data can be scaled and merged by **aimless**, **pointless**, - **careless** or `iotbx.merging_statistics` instead of by rugnux. Written when `--export-unmerged` - asks for it, alongside the merged files and with `--no-merge` too. See + **careless** or `iotbx.merging_statistics` instead of by rugnux. Written by default, alongside the + merged files and with `--no-merge` too; `--no-export-unmerged` skips it. See [The unmerged export](#the-unmerged-export) below. `--export-unmerged-partials` writes `_unmerged_partials.mtz`, one row per image, instead of summing. - `_report.txt` — the **results report**: what the run determined, in a form both a person and @@ -459,13 +459,21 @@ expect. Two properties worth knowing before using it: `_unmerged.mtz` holds every integrated observation, before scaling and merging, in the column layout POINTLESS writes and **aimless**, **pointless**, **careless** and `iotbx.merging_statistics` -read. `--export-unmerged` asks for it, in `--mode mx` and `--mode scale` alike and with `--no-merge` -as well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs -an output prefix (`-o`). It is off by default because it is the largest file a run produces, larger -on a dense rotation dataset than the merged `.mtz`, `.cif` and `.hkl` put together. +read. It is written by default, in `--mode mx` and `--mode scale` alike and with `--no-merge` as +well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs an +output prefix (`-o`). It is the largest file a run produces, larger on a dense rotation dataset than +the merged `.mtz`, `.cif` and `.hkl` put together, so a run that only wants the merged numbers — a +regression battery, or a throughput pipeline — turns it off with `--no-export-unmerged`. Use it to scale the data with a different program, to have pointless give an independent opinion on -the space group, or to compare rugnux's merge against another one on identical input. +the space group, or to compare rugnux's merge against another one on identical input. Each sweep's +file is self-contained, so several of them can be handed to pointless and aimless as separate +`HKLIN`s to merge sweeps rugnux does not combine itself. + +> **Trap when combining files.** pointless silently **drops `HKLIN` files whose modification times +> are not in increasing order**, then merges what is left and prints a plausible result. Give them in +> the order they were written, or `touch` them into that order first, and check the file count in its +> log against the number given. **Columns.** `H K L M/ISYM BATCH I SIGI FRACTIONCALC XDET YDET ROT LP FLAG` — POINTLESS's own set — plus four rugnux extras, `DELPHI` (offset from the centre of the rocking curve), `ZETA` (the Lorentz @@ -1027,8 +1035,9 @@ Scaling and merging: | `--reference-column