From 762c093b10fd568a0bb4e21df1a24ab5286b477f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 2 Sep 2026 09:03:13 +0200 Subject: [PATCH] unmerged MTZ: export the events the merge would keep, not every event The export declared its rows FULL - LDTYPE=2 in the batch header, M=0 in M/ISYM - while filtering them on --min-partiality alone (0.02), where the 3D combine also applies --min-captured-fraction (0.7 on rotation) to the same summed event. AIMLESS reads the FULL declaration, reports "all runs have only fulls" and ignores FRACTIONCALC, so an event that caught a twentieth of its rocking curve entered the merge whole, with a small sigma, and was weighted heavily. There was no cut for the reader to make: the column that would let it make one is the one the reading program has been told to ignore. Dropping those events moves AIMLESS's Rmerge at 1.8 A from 1.353 to 0.694 and CC(1/2) from 0.985 to 0.993. FRACTIONCALC itself is unchanged, values above 1 included. Each part's partiality is the erf pair the predictor computed on that frame, from that frame's own refined lattice and mosaicity, so the parts of one event do not tile the rocking curve exactly - the offset steps by the wedge to within 12% of it, and the sums land in a peak at 1.000 whose 95th percentile is 1.09. It is an honest estimate of a captured fraction, and it is not the number rugnux scales on, so clamping it would only hide the spread. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 1 + image_analysis/WriteReflections.cpp | 32 ++++++++++++++++++++--------- rugnux/rugnux_cli.cpp | 2 +- 3 files changed, 24 insertions(+), 11 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 041ef7257..1a295feda 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -8,6 +8,7 @@ * `rugnux` places a detector swung out on a 2theta arm where the file says it stands. * `rugnux` writes the unmerged MTZ by default, with a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * `rugnux` writes a batch header in the unmerged MTZ for every image the observations span, not only for the images that produced one, so a scaling program reads one run per sweep. +* `rugnux` leaves an event out of the unmerged MTZ when less of its rocking curve was captured than `--min-captured-fraction`, as the merge does, since the exported rows are declared full. * `rugnux` handles symmetry better: the lattice, the point group, the setting and the systematic absences. * `rugnux` fits the direction of the goniometer axis and not its length, so the cell chosen by the first pass is the one its own refinement scored. * `rugnux` reports the detector geometry it measured - the direct beam, the tilt and the beam centre - and what a single sweep can and cannot determine. diff --git a/image_analysis/WriteReflections.cpp b/image_analysis/WriteReflections.cpp index e5a17a105..f908d59f2 100644 --- a/image_analysis/WriteReflections.cpp +++ b/image_analysis/WriteReflections.cpp @@ -602,15 +602,26 @@ float DetectorY(const Reflection &r) { return std::isfinite(r.observed_y) ? r.ob // 8-41% of events have parts summing to zero or less and 10-21% of the intensity-weighted centroids // fall outside the event's own frame range. Partialities are the rocking curve's own weights and are // positive by construction. -// min_partiality is the combine's own floor on the assembled full (--min-partiality, default 0.02, -// 0 = off). An event that caught a thousandth of its rocking curve is not a measurement of that -// reflection - rugnux does not merge one either - and writing it as a full hands the reading program -// a whole observation whose intensity is noise and whose sigma is small, so it is weighted heavily. -// On a crystal whose rocking curves span twenty frames those events are 7% of the file and cost -// POINTLESS the point group. Everything above the floor is written with its honest FRACTIONCALC, -// truncated edge-of-sweep events included, for the reader to cut where it wants. +// The floor is the SAME pair the 3D combine applies to the same event - --min-partiality (default +// 0.02) and --min-captured-fraction (default 0.7 on rotation) - so what is exported as a full is +// exactly what rugnux merges as a full. The captured-fraction floor is what makes the FULL +// declaration true: the batch header says LDTYPE=2 and M/ISYM carries M=0, AIMLESS reads that as +// "all runs have only fulls" and ignores FRACTIONCALC entirely, so an event that caught a twentieth +// of its rocking curve does not enter the merge at a twentieth of its intensity - it enters whole, +// with a small sigma, and is weighted heavily. There is no cut for the reader to make: the column +// that would let it make one is one the reading program has been told to ignore. Measured, dropping +// those events moves AIMLESS's Rmerge at 1.8 A from 1.353 to 0.694 and CC(1/2) from 0.985 to 0.993. +// +// FRACTIONCALC itself is written as summed, above 1 included. Each part's partiality is the erf pair +// the predictor computed on THAT frame, from that frame's own refined lattice and its own mosaicity +// estimate, so the parts of one event do not tile the rocking curve exactly - measured, the offset +// from one part to the next steps by the wedge to within 12% of it, and the sums land in a peak at +// 1.000 whose 95th percentile is 1.09. It is the honest estimate of a captured fraction and it is +// not the number rugnux scales on (the merge recomputes every partiality from a frame-order-smoothed +// mosaicity, RotationScaleMerge::SmoothMosaicity), so clamping it to 1 would hide the spread and buy +// a reading program nothing. std::vector SumRockingEvents(const std::vector &outcomes, - double min_partiality) { + double min_partiality, double min_captured_fraction) { constexpr float MAX_FRAME_GAP = 2.0f; // == RotationScaleMerge's: what makes one rocking event // The sort key travels with the part instead of being read back through the pointer, the way the @@ -664,7 +675,7 @@ std::vector SumRockingEvents(const std::vector & } Reflection full = *parts[i].r; i = j; - if (sum_p < min_partiality) + if (sum_p < min_partiality || sum_p < min_captured_fraction) continue; // The Lorentz-polarization factor is applied by the writer, which multiplies I by rlp, so @@ -794,7 +805,8 @@ void WriteUnmergedMtzReflections(const std::vector &outcomes }; if (scanning && sum_partials) { for (const auto &r : SumRockingEvents(outcomes, - experiment.GetScalingSettings().GetMinPartiality())) + experiment.GetScalingSettings().GetMinPartiality(), + experiment.GetScalingSettings().GetMinCapturedFraction())) add_row(r); } else { for (const auto &outcome : outcomes) diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 28516c5d5..df3031a6e 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -171,7 +171,7 @@ void print_usage() { std::cout << " -z, --reference-mtz Reference MTZ file" << std::endl; std::cout << " --reference-column