From 0fed51f76704222e1e850d88774defa6392aa36e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 27 Aug 2026 12:24:54 +0200 Subject: [PATCH] rugnux: write the unmerged MTZ on every run, not on request _unmerged.mtz is what another scaling program needs - aimless, pointless, careless, iotbx.merging_statistics - and a run that did not ask for it up front had to be repeated to get it. It is now written on every run that has an output prefix, in --mode mx and --mode scale and with --no-merge, alongside the merged files and replacing none of them. --no-export-unmerged turns it off; --export-unmerged is still accepted and now does nothing. The summed form is the default one, as before: each rocking event's partials combined into one full, written at the batch its rocking curve is centred on. --export-unmerged-partials still additionally writes the unsummed _unmerged_partials.mtz for a program that would rather sum them itself. Both write sites gated on the export flag alone and would have written a file called "_unmerged.mtz" when there was no output prefix. That could not happen while the export was opt-in and a prefix was implied by asking for it; with the export on by default it would fire on any prefix-less run. They now use the same guard the merged writes use. Verified on the rotation test dataset: written by default, by --mode scale and under --no-merge, absent with --no-export-unmerged. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa --- docs/CHANGELOG.md | 13 +++++++++---- docs/RUGNUX.md | 26 ++++++++++++++------------ rugnux/Rugnux.cpp | 11 ++++++----- rugnux/Rugnux.h | 8 ++++---- rugnux/rugnux_cli.cpp | 22 ++++++++++++++-------- 5 files changed, 47 insertions(+), 33 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a3e5a511..e2a2a912 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog ## 1.0.0 +### 1.0.0-rc.165 +This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. + +* rugnux writes `_unmerged.mtz` on every run that produces an output prefix, instead of only when asked for it with `--export-unmerged`. It is written in `--mode mx` and `--mode scale` and with `--no-merge`, alongside the merged files and replacing none of them. `--no-export-unmerged` turns it off. +* `/start` asks the writer whether the run can be written before the detector is armed, so a run whose output file already exists, or whose output directory cannot be created, is refused up front with the writer's own message instead of failing once the detector is running. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream has no way to answer and is unchanged. +* An output data file already in the way is refused when the collection starts, not when the file is renamed into place at the end of it. +* `/wait_till_done` answers 500 with the message when a collection ended in an error - packets missing, or a writer that could not write - instead of 200. A cancelled collection and a collection that only triggered a warning still answer 200. +* The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. + ### 1.0.0-rc.164 This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. @@ -17,10 +26,6 @@ This is an UNSTABLE release. It includes many experimental features, as well as * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. -* `/start` asks the writer whether the run can be written before the detector is armed, so a run whose output file already exists, or whose output directory cannot be created, is refused up front with the writer's own message instead of failing once the detector is running. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream has no way to answer and is unchanged. -* An output data file already in the way is refused when the collection starts, not when the file is renamed into place at the end of it. -* `/wait_till_done` answers 500 with the message when a collection ended in an error - packets missing, or a writer that could not write - instead of 200. A cancelled collection and a collection that only triggered a warning still answer 200. -* The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index d8f39890..95f5ba10 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -242,8 +242,8 @@ rugnux -o quicklook -e 200 dataset_master.h5 # merge as usual, but also keep the per-image file so the data can be re-merged later rugnux -o myrun --write-process-h5 dataset_master.h5 -# also write an unmerged MTZ, to scale the data with aimless instead -rugnux -o myrun --export-unmerged dataset_master.h5 +# skip the unmerged MTZ, which is otherwise written alongside the merged files +rugnux -o myrun --no-export-unmerged dataset_master.h5 # check the merged data against a known structure: R-work / R-free and maps rugnux -o myrun --model model.pdb dataset_master.h5 @@ -396,10 +396,11 @@ reusing them would hide the spot-finding settings from the lattice search. All three carry the **refined unit cell** (from rotation indexing) and the **space group determined from systematic absences** (constrained to the indexed lattice symmetry). -- `_unmerged.mtz` — **opt-in** (`--export-unmerged`): the integrated observations *before* - merging, as an unmerged MTZ in POINTLESS's column layout, so the data can be scaled and merged by - **aimless**, **pointless**, **careless** or `iotbx.merging_statistics` instead of by rugnux. See - [The unmerged export](#the-unmerged-export) below. `--export-unmerged-partials` writes +- `_unmerged.mtz` — the integrated observations *before* merging, as an unmerged MTZ in + POINTLESS's column layout, so the data can be scaled and merged by **aimless**, **pointless**, + **careless** or `iotbx.merging_statistics` instead of by rugnux. Written by default, alongside the + merged files and with `--no-merge` too; `--no-export-unmerged` turns it off. See + [The unmerged export](#the-unmerged-export) below. `--export-unmerged-partials` additionally writes `_unmerged_partials.mtz`, one row per image, instead of summing. - `_report.txt` — the **results report**: what the run determined, in a form both a person and a beamline script can read. Always written, next to the files above. See @@ -453,10 +454,11 @@ expect. Two properties worth knowing before using it: ## The unmerged export -`--export-unmerged` writes `_unmerged.mtz`: every integrated observation, before scaling and -merging, in the column layout POINTLESS writes and **aimless**, **pointless**, **careless** and -`iotbx.merging_statistics` read. It is written in `--mode mx` and `--mode scale` alike, off by -default, and does not replace anything — rugnux still writes its own merged files in the same run. +`_unmerged.mtz` holds every integrated observation, before scaling and merging, in the column +layout POINTLESS writes and **aimless**, **pointless**, **careless** and `iotbx.merging_statistics` +read. It is written by default, in `--mode mx` and `--mode scale` alike and with `--no-merge` as +well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs an +output prefix (`-o`), and `--no-export-unmerged` turns it off. Use it to scale the data with a different program, to have pointless give an independent opinion on the space group, or to compare rugnux's merge against another one on identical input. @@ -973,8 +975,8 @@ Scaling and merging: | `--reference-column