mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 72 Wiki Activity

rugnux: write the unmerged MTZ on every run, not on request

Browse Source 
<prefix>_unmerged.mtz is what another scaling program needs - aimless,
pointless, careless, iotbx.merging_statistics - and a run that did not ask for
it up front had to be repeated to get it. It is now written on every run that
has an output prefix, in --mode mx and --mode scale and with --no-merge,
alongside the merged files and replacing none of them. --no-export-unmerged
turns it off; --export-unmerged is still accepted and now does nothing.

The summed form is the default one, as before: each rocking event's partials
combined into one full, written at the batch its rocking curve is centred on.
--export-unmerged-partials still additionally writes the unsummed
<prefix>_unmerged_partials.mtz for a program that would rather sum them itself.

Both write sites gated on the export flag alone and would have written a file
called "_unmerged.mtz" when there was no output prefix. That could not happen
while the export was opt-in and a prefix was implied by asking for it; with the
export on by default it would fire on any prefix-less run. They now use the same
guard the merged writes use.

Verified on the rotation test dataset: written by default, by --mode scale and
under --no-merge, absent with --no-export-unmerged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa
This commit is contained in:
leonarski_f
2026-08-27 12:24:54 +02:00
co-authored by Claude Opus 5
parent 5f838bdb26
commit 0fed51f767
5 changed files with 47 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CHANGELOG.md
+9 -4
View File
@@ -1,5 +1,14 @@
# Changelog
## 1.0.0
### 1.0.0-rc.165
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux writes `<prefix>_unmerged.mtz` on every run that produces an output prefix, instead of only when asked for it with `--export-unmerged`. It is written in `--mode mx` and `--mode scale` and with `--no-merge`, alongside the merged files and replacing none of them. `--no-export-unmerged` turns it off.
* `/start` asks the writer whether the run can be written before the detector is armed, so a run whose output file already exists, or whose output directory cannot be created, is refused up front with the writer's own message instead of failing once the detector is running. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream has no way to answer and is unchanged.
* An output data file already in the way is refused when the collection starts, not when the file is renamed into place at the end of it.
* `/wait_till_done` answers 500 with the message when a collection ended in an error - packets missing, or a writer that could not write - instead of 200. A cancelled collection and a collection that only triggered a warning still answer 200.
* The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before.
### 1.0.0-rc.164
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
@@ -17,10 +26,6 @@ This is an UNSTABLE release. It includes many experimental features, as well as
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
* `/start` asks the writer whether the run can be written before the detector is armed, so a run whose output file already exists, or whose output directory cannot be created, is refused up front with the writer's own message instead of failing once the detector is running. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream has no way to answer and is unchanged.
* An output data file already in the way is refused when the collection starts, not when the file is renamed into place at the end of it.
* `/wait_till_done` answers 500 with the message when a collection ended in an error - packets missing, or a writer that could not write - instead of 200. A cancelled collection and a collection that only triggered a warning still answer 200.
* The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before.
* A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
* A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires.
* `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
docs/RUGNUX.md
+14 -12
View File
@@ -242,8 +242,8 @@ rugnux -o quicklook -e 200 dataset_master.h5
# merge as usual, but also keep the per-image file so the data can be re-merged later
rugnux -o myrun --write-process-h5 dataset_master.h5
# also write an unmerged MTZ, to scale the data with aimless instead
rugnux -o myrun --export-unmerged dataset_master.h5
# skip the unmerged MTZ, which is otherwise written alongside the merged files
rugnux -o myrun --no-export-unmerged dataset_master.h5
# check the merged data against a known structure: R-work / R-free and maps
rugnux -o myrun --model model.pdb dataset_master.h5
@@ -396,10 +396,11 @@ reusing them would hide the spot-finding settings from the lattice search.
All three carry the **refined unit cell** (from rotation indexing) and the **space group determined
from systematic absences** (constrained to the indexed lattice symmetry).
- `<prefix>_unmerged.mtz` — **opt-in** (`--export-unmerged`): the integrated observations *before*
merging, as an unmerged MTZ in POINTLESS's column layout, so the data can be scaled and merged by
**aimless**, **pointless**, **careless** or `iotbx.merging_statistics` instead of by rugnux. See
[The unmerged export](#the-unmerged-export) below. `--export-unmerged-partials` writes
- `<prefix>_unmerged.mtz` — the integrated observations *before* merging, as an unmerged MTZ in
POINTLESS's column layout, so the data can be scaled and merged by **aimless**, **pointless**,
**careless** or `iotbx.merging_statistics` instead of by rugnux. Written by default, alongside the
merged files and with `--no-merge` too; `--no-export-unmerged` turns it off. See
[The unmerged export](#the-unmerged-export) below. `--export-unmerged-partials` additionally writes
`<prefix>_unmerged_partials.mtz`, one row per image, instead of summing.
- `<prefix>_report.txt` — the **results report**: what the run determined, in a form both a person and
a beamline script can read. Always written, next to the files above. See
@@ -453,10 +454,11 @@ expect. Two properties worth knowing before using it:
## The unmerged export
`--export-unmerged` writes `<prefix>_unmerged.mtz`: every integrated observation, before scaling and
merging, in the column layout POINTLESS writes and **aimless**, **pointless**, **careless** and
`iotbx.merging_statistics` read. It is written in `--mode mx` and `--mode scale` alike, off by
default, and does not replace anything — rugnux still writes its own merged files in the same run.
`<prefix>_unmerged.mtz` holds every integrated observation, before scaling and merging, in the column
layout POINTLESS writes and **aimless**, **pointless**, **careless** and `iotbx.merging_statistics`
read. It is written by default, in `--mode mx` and `--mode scale` alike and with `--no-merge` as
well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs an
output prefix (`-o`), and `--no-export-unmerged` turns it off.
Use it to scale the data with a different program, to have pointless give an independent opinion on
the space group, or to compare rugnux's merge against another one on identical input.
@@ -973,8 +975,8 @@ Scaling and merging:
| `--reference-column <label>` | Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…) |
| `--model <file.pdb>` | Validate the merged intensities against this atomic model — R-work / R-free and maps (see [Validating against a model](#validating-against-a-model-rugnux-model)). It also settles the frame the reflections are written in: the enantiomorph, and the [indexing ambiguity](#the-indexing-ambiguity) where no `-z` did. For serial stills given `-C` / `-S`, the model's structure factors become the per-image reference |
| `--write-process-h5` | Also write the (large) `_process.h5` when merging (default: only `.mtz`/`.cif`) |
| `--export-unmerged` | Also write the integrated observations as `<prefix>_unmerged.mtz`, an unmerged MTZ (POINTLESS column layout) for aimless / pointless / careless. Rotation partials are summed into one full per reflection. Intensities carry the Lorentz-polarization factor and nothing else — the partiality is not divided out and the per-image scale is not applied. Works in `--mode mx` and `--mode scale`. See [The unmerged export](#the-unmerged-export) |
| `--export-unmerged-partials` | As above, but writes `<prefix>_unmerged_partials.mtz` with each partial as its own row (one batch per image) for the reading program to sum |
| `--no-export-unmerged` | Do not write `<prefix>_unmerged.mtz`. It is written by default: an unmerged MTZ (POINTLESS column layout) of the integrated observations, for aimless / pointless / careless. Rotation partials are summed into one full per reflection. Intensities carry the Lorentz-polarization factor and nothing else — the partiality is not divided out and the per-image scale is not applied. Written in `--mode mx` and `--mode scale`, and with `--no-merge`. See [The unmerged export](#the-unmerged-export) |
| `--export-unmerged-partials` | Also write `<prefix>_unmerged_partials.mtz`, the same observations with each partial as its own row (one batch per image) for the reading program to sum |
Integration:
rugnux/Rugnux.cpp
+6 -5
View File
@@ -3336,11 +3336,12 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
}
}
// Unmerged observations (--export-unmerged). After the merge, because that is where the space
// group is settled and the file has to be written in it, but from the integrated observations
// rather than the merged ones: the partiality and the per-image scale are left for the reading
// program, which fits a scale model of its own.
if (full && !cancelled_ && write_output && !geometry_prepass && result.consensus_cell) {
// Unmerged observations. After the merge, because that is where the space group is settled and
// the file has to be written in it, but from the integrated observations rather than the merged
// ones: the partiality and the per-image scale are left for the reading program, which fits a
// scale model of its own. write_files, not write_output - without an output prefix there is
// nowhere to put it, and the name would come out as a bare "_unmerged.mtz".
if (full && !cancelled_ && write_files && !geometry_prepass && result.consensus_cell) {
if (config_.export_unmerged) {
if (observer) observer->OnPhase("Writing unmerged reflections");
const std::string path = config_.output_prefix + "_unmerged.mtz";
rugnux/Rugnux.h
+4 -4
View File
@@ -127,10 +127,10 @@ struct ProcessConfig {
// Diagnostic: if set, the -P rot3d combine writes the unmerged fulls here (for comparison vs XDS).
std::string observation_dump_path;
// Write the integrated observations as <prefix>_unmerged.mtz (--export-unmerged), for aimless /
// pointless / careless and anything else that scales unmerged data itself - each rocking event
// summed into one full.
bool export_unmerged = false;
// Write the integrated observations as <prefix>_unmerged.mtz, for aimless / pointless / careless
// and anything else that scales unmerged data itself - each rocking event summed into one full.
// On by default in the rugnux CLI (--no-export-unmerged turns it off); needs an output prefix.
bool export_unmerged = true;
// The same as <prefix>_unmerged_partials.mtz, with each partial written as its own row for the
// reader to sum (--export-unmerged-partials). Independent of the above; a run can write both.
rugnux/rugnux_cli.cpp
+14 -8
View File
@@ -164,8 +164,8 @@ void print_usage() {
std::cout << " -z, --reference-mtz <file> Reference MTZ file" << std::endl;
std::cout << " --reference-column <label> Reference MTZ column to use (default: auto - F-model, else IMEAN/I, else FP/FOBS/F)" << std::endl;
std::cout << " --model <file.pdb> After merging, validate vs this model: R-free + 2Fo-Fc/Fo-Fc maps" << std::endl;
std::cout << " --export-unmerged Also write the integrated observations as <prefix>_unmerged.mtz, an unmerged MTZ (POINTLESS column layout) for aimless / pointless / careless. On a rotation run the partials of each reflection are summed into one full, written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in FRACTIONCALC; an event that caught less of its rocking curve than --min-partiality is not written, as in the merge. Intensities carry the Lorentz-polarization factor and nothing else: the partiality is not divided out and the per-image scale is not applied at all, since those programs scale the data themselves. Works in --mode mx and --mode scale" << std::endl;
std::cout << " --export-unmerged-partials As --export-unmerged, but writes <prefix>_unmerged_partials.mtz with each partial as its own row (one batch per image) flagged for the reading program to sum, instead of summing them here" << std::endl;
std::cout << " --no-export-unmerged Do not write <prefix>_unmerged.mtz. It is written by default: an unmerged MTZ (POINTLESS column layout) of the integrated observations, for aimless / pointless / careless. On a rotation run the partials of each reflection are summed into one full, written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in FRACTIONCALC; an event that caught less of its rocking curve than --min-partiality is not written, as in the merge. Intensities carry the Lorentz-polarization factor and nothing else: the partiality is not divided out and the per-image scale is not applied at all, since those programs scale the data themselves. Written in --mode mx and --mode scale, and with --no-merge" << std::endl;
std::cout << " --export-unmerged-partials Also write <prefix>_unmerged_partials.mtz, the same observations with each partial as its own row (one batch per image) flagged for the reading program to sum, instead of summed here" << std::endl;
std::cout << std::endl;
std::cout << " Integration" << std::endl;
@@ -243,6 +243,7 @@ enum {
OPT_REFERENCE_COLUMN,
OPT_MODEL,
OPT_EXPORT_UNMERGED,
OPT_NO_EXPORT_UNMERGED,
OPT_EXPORT_UNMERGED_PARTIALS,
OPT_DUMP_OBSERVATIONS,
OPT_INTEGRATOR,
@@ -294,6 +295,7 @@ static option long_options[] = {
{"reference-column", required_argument, nullptr, OPT_REFERENCE_COLUMN},
{"model", required_argument, nullptr, OPT_MODEL},
{"export-unmerged", no_argument, nullptr, OPT_EXPORT_UNMERGED},
{"no-export-unmerged", no_argument, nullptr, OPT_NO_EXPORT_UNMERGED},
{"export-unmerged-partials", no_argument, nullptr, OPT_EXPORT_UNMERGED_PARTIALS},
{"dump-observations", required_argument, nullptr, OPT_DUMP_OBSERVATIONS},
{"space-group", required_argument, nullptr, 'S'},
@@ -634,7 +636,7 @@ static int RunRugnux(int argc, char **argv) {
std::string ref_mtz;
std::string ref_column;
std::string model_pdb; // --model: PDB to validate merged intensities against (R-free + maps)
bool export_unmerged = false; // --export-unmerged: <prefix>_unmerged.mtz
bool export_unmerged = true; // <prefix>_unmerged.mtz, off with --no-export-unmerged
bool export_unmerged_partials = false; // --export-unmerged-partials: the same, one row per image
std::string dump_observations; // diagnostic: dump unmerged -P rot3d fulls to this path
double min_partiality = 0.02;
@@ -863,6 +865,9 @@ static int RunRugnux(int argc, char **argv) {
case OPT_EXPORT_UNMERGED:
export_unmerged = true;
break;
case OPT_NO_EXPORT_UNMERGED:
export_unmerged = false;
break;
case OPT_EXPORT_UNMERGED_PARTIALS:
export_unmerged_partials = true;
break;
@@ -1649,15 +1654,16 @@ static int RunRugnux(int argc, char **argv) {
}
}
// Unmerged observations (--export-unmerged), from the integrated observations rather than the
// merged ones: the partiality and the per-image scale are left for the reading program, which
// fits a scale model of its own.
if (export_unmerged) {
// Unmerged observations, from the integrated observations rather than the merged ones: the
// partiality and the per-image scale are left for the reading program, which fits a scale
// model of its own. Without an output prefix there is nowhere to put it, and the name would
// come out as a bare "_unmerged.mtz".
if (export_unmerged && !output_prefix.empty()) {
const std::string path = output_prefix + "_unmerged.mtz";
WriteUnmergedMtzReflections(reflections, *experiment.GetUnitCell(), experiment, true, path);
logger.Info("Unmerged observations written to {}", path);
}
if (export_unmerged_partials) {
if (export_unmerged_partials && !output_prefix.empty()) {
const std::string path = output_prefix + "_unmerged_partials.mtz";
WriteUnmergedMtzReflections(reflections, *experiment.GetUnitCell(), experiment, false, path);
logger.Info("Unmerged partials written to {}", path);