mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests 1 Actions 3 Packages Projects Releases 72 Wiki Activity

rugnux: write the unmerged MTZ by default

Browse Source 
The unmerged export is how a run's observations reach the rest of the field. It is what aimless,
pointless and careless read, it is what a head-to-head against another program's answer runs
through, and handing several of these files to pointless is the only way to merge sweeps rugnux
does not combine itself. Behind a flag it reached only the people who already knew the flag
existed, which is the shape of defect the "rugnux <file> with nothing else" direction asks to
design out. --no-export-unmerged turns it off, and the regression batteries now pass it: the
objection to defaulting it on was their disk and time cost, not the product's.

Measured on three rotation crystals of 900, 1800 and 3600 frames: the file is 21, 86 and 26 MB -
two to five times the merged .mtz, .cif and .hkl put together - and the write costs 0.24, 2.29 and
1.11 s, 5.5 %, 12.2 % and 3.1 % of each run's own wall time. The cost is linear in the number of
observations and independent of how long processing took, so it is the largest fraction of the
runs rugnux finishes fastest, not of the longest sweeps. A run's .hkl, .mtz and .cif are
byte-identical with the export on and off.

--export-unmerged-partials stays off: it is a second, larger artefact and a separate question.

The viewer's reprocessing dialog ties the export to its merged-output switch, so it appears beside
the merged files rather than beside a job that only asked for the per-image _process.h5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f
This commit is contained in:
leonarski_f
2026-08-30 10:51:25 +02:00
co-authored by Claude Opus 5
parent 8e9ca1f6d2
commit 5f47d73cc1
7 changed files with 44 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CHANGELOG.md
+1
View File
@@ -1,6 +1,7 @@
# Changelog
## 1.0.0
### 1.0.0-rc.166
* `rugnux` writes the unmerged MTZ `<prefix>_unmerged.mtz` by default; `--no-export-unmerged` skips it.
* `jfjoch_viewer` opens PILATUS miniCBF sweeps - naming any frame opens the whole sweep - and can run a processing job on one.
* A detector whose stored image is mirrored in Y or mounted at a multiple of 90 degrees can be described as such, in the detector configuration or with `--detector-mirror-y` / `--detector-quarter-turns`, rather than having to be expressed as a detector rotation.
* The rotation first pass refines twelve candidate lattices rather than four, so a correct cell that the pre-refinement ranking put fifth is still reached.
docs/RUGNUX.md
+20 -11
View File
@@ -245,8 +245,8 @@ rugnux -o quicklook -e 200 dataset_master.h5
# merge as usual, but also keep the per-image file so the data can be re-merged later
rugnux -o myrun --write-process-h5 dataset_master.h5
# also write the unmerged MTZ, to scale and merge the observations with another program
rugnux -o myrun --export-unmerged dataset_master.h5
# skip the unmerged MTZ (written by default), when only the merged data is wanted
rugnux -o myrun --no-export-unmerged dataset_master.h5
# check the merged data against a known structure: R-work / R-free and maps
rugnux -o myrun --model model.pdb dataset_master.h5
@@ -401,8 +401,8 @@ reusing them would hide the spot-finding settings from the lattice search.
from systematic absences** (constrained to the indexed lattice symmetry).
- `<prefix>_unmerged.mtz` — the integrated observations *before* merging, as an unmerged MTZ in
POINTLESS's column layout, so the data can be scaled and merged by **aimless**, **pointless**,
**careless** or `iotbx.merging_statistics` instead of by rugnux. Written when `--export-unmerged`
asks for it, alongside the merged files and with `--no-merge` too. See
**careless** or `iotbx.merging_statistics` instead of by rugnux. Written by default, alongside the
merged files and with `--no-merge` too; `--no-export-unmerged` skips it. See
[The unmerged export](#the-unmerged-export) below. `--export-unmerged-partials` writes
`<prefix>_unmerged_partials.mtz`, one row per image, instead of summing.
- `<prefix>_report.txt` — the **results report**: what the run determined, in a form both a person and
@@ -459,13 +459,21 @@ expect. Two properties worth knowing before using it:
`<prefix>_unmerged.mtz` holds every integrated observation, before scaling and merging, in the column
layout POINTLESS writes and **aimless**, **pointless**, **careless** and `iotbx.merging_statistics`
read. `--export-unmerged` asks for it, in `--mode mx` and `--mode scale` alike and with `--no-merge`
as well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs
an output prefix (`-o`). It is off by default because it is the largest file a run produces, larger
on a dense rotation dataset than the merged `.mtz`, `.cif` and `.hkl` put together.
read. It is written by default, in `--mode mx` and `--mode scale` alike and with `--no-merge` as
well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs an
output prefix (`-o`). It is the largest file a run produces, larger on a dense rotation dataset than
the merged `.mtz`, `.cif` and `.hkl` put together, so a run that only wants the merged numbers — a
regression battery, or a throughput pipeline — turns it off with `--no-export-unmerged`.
Use it to scale the data with a different program, to have pointless give an independent opinion on
the space group, or to compare rugnux's merge against another one on identical input.
the space group, or to compare rugnux's merge against another one on identical input. Each sweep's
file is self-contained, so several of them can be handed to pointless and aimless as separate
`HKLIN`s to merge sweeps rugnux does not combine itself.
> **Trap when combining files.** pointless silently **drops `HKLIN` files whose modification times
> are not in increasing order**, then merges what is left and prints a plausible result. Give them in
> the order they were written, or `touch` them into that order first, and check the file count in its
> log against the number given.
**Columns.** `H K L M/ISYM BATCH I SIGI FRACTIONCALC XDET YDET ROT LP FLAG` — POINTLESS's own set —
plus four rugnux extras, `DELPHI` (offset from the centre of the rocking curve), `ZETA` (the Lorentz
@@ -1027,8 +1035,9 @@ Scaling and merging:
| `--reference-column <label>` | Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…) |
| `--model <file>` | Validate the merged intensities against this atomic model (PDB or mmCIF, gzipped or not; the format is taken from the file's content) — R-work / R-free and maps (see [Validating against a model](#validating-against-a-model-rugnux-model)). It also settles the frame the reflections are written in: the enantiomorph, and the [indexing ambiguity](#the-indexing-ambiguity) where no `-z` did. For serial stills given `-C` / `-S`, the model's structure factors become the per-image reference |
| `--write-process-h5` | Also write the (large) `_process.h5` when merging (default: only `.mtz`/`.cif`) |
| `--export-unmerged` | Write `<prefix>_unmerged.mtz`, an unmerged MTZ (POINTLESS column layout) of the integrated observations, for aimless / pointless / careless. Off by default. Rotation partials are summed into one full per reflection. Intensities carry the Lorentz-polarization factor and nothing else — the partiality is not divided out and the per-image scale is not applied. Written in `--mode mx` and `--mode scale`, and with `--no-merge`. See [The unmerged export](#the-unmerged-export) |
| `--export-unmerged-partials` | Write `<prefix>_unmerged_partials.mtz`, the same observations with each partial as its own row (one batch per image) for the reading program to sum. Independent of `--export-unmerged` |
| `--export-unmerged` | Write `<prefix>_unmerged.mtz`, an unmerged MTZ (POINTLESS column layout) of the integrated observations, for aimless / pointless / careless. **On by default** whenever there is an output prefix. Rotation partials are summed into one full per reflection. Intensities carry the Lorentz-polarization factor and nothing else — the partiality is not divided out and the per-image scale is not applied. Written in `--mode mx` and `--mode scale`, and with `--no-merge`. See [The unmerged export](#the-unmerged-export) |
| `--no-export-unmerged` | Do not write `<prefix>_unmerged.mtz`. It is the largest file a run produces, so a run whose observations are not going to another scaling program can skip it |
| `--export-unmerged-partials` | Write `<prefix>_unmerged_partials.mtz`, the same observations with each partial as its own row (one batch per image) for the reading program to sum. Off by default, and independent of `--export-unmerged` |
Integration:
rugnux/Rugnux.h
+7 -5
View File
@@ -138,13 +138,15 @@ struct ProcessConfig {
// Write the integrated observations as <prefix>_unmerged.mtz, for aimless / pointless / careless
// and anything else that scales unmerged data itself - each rocking event summed into one full.
// Off by default (--export-unmerged asks for it): the file is the largest a run produces, and a
// run that is not handing its observations to another scaling program should not pay for it.
// Needs an output prefix.
bool export_unmerged = false;
// On by default (--no-export-unmerged turns it off): it is how a run's observations reach the
// rest of the field, and a user who does not know the flag exists should still get the file.
// It is the largest file a run produces, so a batch that only wants the merged numbers turns it
// off. Needs an output prefix.
bool export_unmerged = true;
// The same as <prefix>_unmerged_partials.mtz, with each partial written as its own row for the
// reader to sum (--export-unmerged-partials). Independent of the above; a run can write both.
// reader to sum (--export-unmerged-partials). Off by default; independent of the above, and a
// run can write both.
bool export_unmerged_partials = false;
// If set, after merging, validate the merged intensities against this atomic model (PDB or
rugnux/rugnux_cli.cpp
+4 -3
View File
@@ -168,8 +168,9 @@ void print_usage() {
std::cout << " -z, --reference-mtz <file> Reference MTZ file" << std::endl;
std::cout << " --reference-column <label> Reference MTZ column to use (default: auto - F-model, else IMEAN/I, else FP/FOBS/F)" << std::endl;
std::cout << " --model <file> After merging, validate vs this atomic model: R-free, 2Fo-Fc/Fo-Fc maps, and - where the merge kept the Bijvoet split - an anomalous difference map with the strongest anomalous scatterers named by the atom they sit on. PDB or mmCIF, gzipped or not, recognised by content rather than by extension" << std::endl;
std::cout << " --export-unmerged Write <prefix>_unmerged.mtz, an unmerged MTZ (POINTLESS column layout) of the integrated observations, for aimless / pointless / careless. Off by default - the file is large, and a run that does not need it should not pay for it. On a rotation run the partials of each reflection are summed into one full, written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in FRACTIONCALC; an event that caught less of its rocking curve than --min-partiality is not written, as in the merge. Intensities carry the Lorentz-polarization factor and nothing else: the partiality is not divided out and the per-image scale is not applied at all, since those programs scale the data themselves. Written in --mode mx and --mode scale, and with --no-merge" << std::endl;
std::cout << " --export-unmerged-partials Write <prefix>_unmerged_partials.mtz, the same observations with each partial as its own row (one batch per image) flagged for the reading program to sum, instead of summed here. Independent of --export-unmerged; both can be given" << std::endl;
std::cout << " --export-unmerged Write <prefix>_unmerged.mtz, an unmerged MTZ (POINTLESS column layout) of the integrated observations, for aimless / pointless / careless. On by default, whenever there is an output prefix. On a rotation run the partials of each reflection are summed into one full, written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in FRACTIONCALC; an event that caught less of its rocking curve than --min-partiality is not written, as in the merge. Intensities carry the Lorentz-polarization factor and nothing else: the partiality is not divided out and the per-image scale is not applied at all, since those programs scale the data themselves. Written in --mode mx and --mode scale, and with --no-merge" << std::endl;
std::cout << " --no-export-unmerged Do not write <prefix>_unmerged.mtz. It is the largest file a run produces, so a run whose observations are not going to another scaling program - a regression battery, say - can skip writing it" << std::endl;
std::cout << " --export-unmerged-partials Write <prefix>_unmerged_partials.mtz, the same observations with each partial as its own row (one batch per image) flagged for the reading program to sum, instead of summed here. Off by default and independent of --export-unmerged; both can be written in one run" << std::endl;
std::cout << std::endl;
std::cout << " Integration" << std::endl;
@@ -695,7 +696,7 @@ static int RunRugnux(int argc, char **argv) {
std::string ref_mtz;
std::string ref_column;
std::string model_pdb; // --model: PDB to validate merged intensities against (R-free + maps)
bool export_unmerged = false; // --export-unmerged: write <prefix>_unmerged.mtz
bool export_unmerged = true; // on by default; --no-export-unmerged turns it off
bool export_unmerged_partials = false; // --export-unmerged-partials: the same, one row per image
std::string dump_observations; // diagnostic: dump unmerged -P rot3d fulls to this path
double min_partiality = 0.02;
rugnux_stills_ab.py
+4 -1
View File
@@ -236,7 +236,10 @@ def run_arm(ds, arm_name, arm_args, workdir, rugnux_bin, threads, timeout, reuse
cif = wd / f"{prefix}.cif"
report = wd / f"{prefix}_report.txt"
cmd = [rugnux_bin, "-o", prefix,
# --no-export-unmerged: the unmerged MTZ is written by default, is the largest file a run
# produces, and nothing here reads it - and this harness runs every dataset once per arm. It
# comes before --common-args / --arm-args, so either can still turn it back on.
cmd = [rugnux_bin, "-o", prefix, "--no-export-unmerged",
"-s", str(ds.start), "-e", str(ds.end), "-t", str(ds.stride),
"--scaling-high-resolution", f"{ds.dmin:.3f}",
"--scaling-low-resolution", f"{ds.dmax:.3f}"]
rugnux_vs_xds.py
+4 -1
View File
@@ -258,7 +258,10 @@ def run_rugnux(master, workdir, name, xds, rugnux_bin, threads, timeout, reuse,
return cif, None, None # no fresh timing when we skip the run
# rugnux always writes both MTZ and CIF now (the old --scaling-output flag was removed).
cmd = [rugnux_bin, "-o", name]
# --no-export-unmerged: the unmerged MTZ is written by default, is the largest file a run
# produces, and nothing in this comparison reads it. It comes before --extra-args, so an
# --extra-args --export-unmerged still turns it back on.
cmd = [rugnux_bin, "-o", name, "--no-export-unmerged"]
if xds.get("anomalous"):
cmd.append("-A")
if xds.get("dmin"):
viewer/windows/JFJochProcessingJobsWindow.cpp
+4
View File
@@ -363,6 +363,10 @@ ProcessConfig JFJochProcessingJobsWindow::buildConfig(const JobSpec &spec, const
? spec.prefix.toStdString() : std::string();
config.write_process_h5 = spec.save_h5;
config.write_merged = spec.save_merged;
// The unmerged MTZ is a merged-output artefact - it is what another scaling program is handed
// instead of our merge - so it follows the merged switch rather than appearing beside a run that
// only asked for the per-image _process.h5.
config.export_unmerged = spec.save_merged;
config.spot_finding = inputs.spot_finding;
if (spec.detect_beam_stop)
config.detect_beam_stop = spec.detect_beam_stop_frames;