mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 75 Wiki Activity

rugnux: the unmerged MTZ carries the sensor efficiency in its own column, not inside LP

Browse Source 
LP is a column other programs read and un-apply. It is documented as the
Lorentz-polarization factor, and until the sensor efficiency was folded into the
same product that is what it held. Measured on our own unmerged file, LP spanned a
factor of 1.1343 across the detector where pure L/P spans nothing of the sort - the
excess is the efficiency, 13% end to end at 13 keV and 34% at 18 keV.

Both reference implementations keep it out. Recomputing pure L/P from a stored XDS
file's own geometry over its 124k reflections leaves RLP flat to 0.1% from 8.6 to
33.7 degrees, where a folded-in efficiency would have shown a 7% trend - and XDS has
the sensor numbers in hand. DIALS fills LP from lorentz and polarization alone and
writes QE as a separate column, even a column of ones.

Split them: LP is L/P again, QE is the efficiency, and the intensity is unchanged,
so a reader that un-applies LP recovers what it expects and one that wants raw
counts divides by LP and multiplies by QE. Only the unmerged file moves - every
other column is bit-identical and the file grows by exactly one float per reflection
plus one header record. The merged files are byte for byte what they were.

The process file gains an optional qe dataset beside the existing one rather than
changing what that one means, so a file written before the efficiency existed still
loads, and reads back as a correction of exactly one - which is what it was. It is
stored rather than recomputed on read because the writer has no geometry to
recompute it from, and because recomputing would have written a radial trend into
every stored file that never had one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
This commit is contained in:
leonarski_f
2026-09-05 14:29:45 +02:00
co-authored by Claude Opus 5
parent db4af06a2b
commit 6140de7249
20 changed files with 149 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/WriteReflections.cpp
+37 -23
View File
@@ -666,8 +666,8 @@ std::vector<Reflection> SumRockingEvents(const std::vector<IntegrationOutcome> &
++j;
double sum_p = 0.0, sum_I = 0.0, sum_var = 0.0, sum_var_bkg = 0.0;
double p_corr = 0.0, p_frame = 0.0, p_x = 0.0, p_y = 0.0, p_delta_phi = 0.0, p_zeta = 0.0,
p_bkg = 0.0;
double p_corr = 0.0, p_qe = 0.0, p_frame = 0.0, p_x = 0.0, p_y = 0.0, p_delta_phi = 0.0,
p_zeta = 0.0, p_bkg = 0.0;
for (size_t m = i; m < j; ++m) {
const Reflection &r = *parts[m].r;
const double p = r.partiality;
@@ -676,6 +676,7 @@ std::vector<Reflection> SumRockingEvents(const std::vector<IntegrationOutcome> &
sum_var += static_cast<double>(r.sigma) * r.sigma * r.prescaling_corr * r.prescaling_corr;
sum_var_bkg += static_cast<double>(r.var_bkg) * r.prescaling_corr * r.prescaling_corr;
p_corr += p * r.prescaling_corr;
p_qe += p * r.qe_corr;
p_frame += p * r.image_number;
p_x += p * DetectorX(r);
p_y += p * DetectorY(r);
@@ -688,16 +689,14 @@ std::vector<Reflection> SumRockingEvents(const std::vector<IntegrationOutcome> &
if (sum_p < min_partiality || sum_p < min_captured_fraction)
continue;
// The prescaling correction is applied by the writer, which multiplies I by prescaling_corr,
// so divide the event's own factor back out of the sums here. Every term in it - Lorentz,
// polarization, sensor efficiency - is a function of the reflection's position and rocking
// geometry, which move by almost nothing between the parts of one event (median 2e-4 on the
// Lorentz term), so the file's I/LP is still the raw count sum.
// The prescaling factor is applied by the writer, which multiplies I by prescaling_corr, so divide the
// event's own factor back out of the sums here. It is the same geometry for every part of
// one event to a median 2e-4, so the file's I / LP * QE is still the raw count sum.
full.prescaling_corr = static_cast<float>(p_corr / sum_p);
full.qe_corr = static_cast<float>(p_qe / sum_p);
full.I = static_cast<float>(sum_I / full.prescaling_corr);
full.sigma = static_cast<float>(std::sqrt(sum_var) / full.prescaling_corr);
full.var_bkg = static_cast<float>(sum_var_bkg
/ (static_cast<double>(full.prescaling_corr) * full.prescaling_corr));
full.var_bkg = static_cast<float>(sum_var_bkg / (static_cast<double>(full.prescaling_corr) * full.prescaling_corr));
full.partiality = static_cast<float>(sum_p);
full.image_number = static_cast<float>(p_frame / sum_p);
full.observed_x = static_cast<float>(p_x / sum_p);
@@ -728,9 +727,23 @@ void WriteUnmergedMtzReflections(const std::vector<IntegrationOutcome> &outcomes
ds.wavelength = experiment.GetWavelength_A();
// Every data column belongs to HKL_base and only the batches carry a dataset id, as in a
// POINTLESS file. Down to FLAG this is POINTLESS's own column set; the four after it are what
// rugnux measures beyond it - the offset of the reflection from the centre of its rocking curve,
// the Lorentz geometry of that curve, and the background that was subtracted.
// POINTLESS file. Down to FLAG this is POINTLESS's own column set, plus QE; the four after it
// are what rugnux measures beyond it - the offset of the reflection from the centre of its
// rocking curve, the Lorentz geometry of that curve, and the background that was subtracted.
//
// QE is not POINTLESS's, it is DIALS's: the detector's angle-dependent quantum efficiency, kept
// OUT of LP and given its own column because that is what both reference implementations do.
// LP here is Lorentz x polarization and nothing else, as XDS's RLP is (measured: XDS's column is
// flat to 0.1% across a detector over which the efficiency term spans 7%) and as DIALS's is
// (its LP comes from Corrections::lp(), lorentz/polarization, and it writes QE separately -
// a column of ones when it has no correction to report). Folding the sensor response into LP
// made this file assert something false about itself: a program that divided LP out to recover
// raw counts stripped the efficiency with it, and on an ordinary 13 keV geometry that term
// spans 13% end to end.
// Direction: QE is a DIVISOR, as in DIALS, and is normalised to 1 at normal incidence, so it
// rises above 1 toward the detector edge where a longer crossing makes the sensor more
// efficient. DIALS stores the un-normalised absorbed fraction instead; the two differ by the
// per-dataset constant QE(0), which is an overall scale.
mtz.add_column("M/ISYM", 'Y', 0, -1, false);
mtz.add_column("BATCH", 'B', 0, -1, false);
mtz.add_column("I", 'J', 0, -1, false);
@@ -740,6 +753,7 @@ void WriteUnmergedMtzReflections(const std::vector<IntegrationOutcome> &outcomes
mtz.add_column("YDET", 'R', 0, -1, false);
mtz.add_column("ROT", 'R', 0, -1, false);
mtz.add_column("LP", 'R', 0, -1, false);
mtz.add_column("QE", 'R', 0, -1, false);
mtz.add_column("FLAG", 'I', 0, -1, false);
mtz.add_column("DELPHI", 'R', 0, -1, false);
mtz.add_column("ZETA", 'R', 0, -1, false);
@@ -765,16 +779,13 @@ void WriteUnmergedMtzReflections(const std::vector<IntegrationOutcome> &outcomes
// them there, so the index the reflection was actually measured at is recoverable - that is the
// crystal-frame information careless scales on, and what makes the file unmerged rather than a
// merge waiting to happen.
// I and SIGI are the integrated intensity with the prescaling correction applied and nothing
// else, which is what IOBS means in every unmerged format, and LP records the factor that was
// applied, so the raw counts are I/LP. That factor is geometry and instrument response, not a
// scale, and a program reading this file has no way to recompute it. The partiality is NOT
// divided out - that is a scale, FRACTIONCALC carries it, and every program this file is for
// wants to handle it its own way.
// Note that LP therefore carries slightly more than its name says: the prescaling correction now
// also holds the sensor's angle-dependent efficiency, for which DIALS writes a separate QE column
// and AIMLESS calls LP "the Lorentz/polarization correction (already applied)". A consumer that
// divides LP out to recover raw counts removes the efficiency term with it.
// I and SIGI are the integrated intensity with every deterministic per-reflection correction
// applied and nothing else, which is what IOBS means in every unmerged format. Two columns
// record what was applied, so the raw count sum is recoverable exactly as I / LP * QE:
// LP is Lorentz x polarization and QE the sensor efficiency (a divisor, hence the multiply).
// Neither is a scale, and a program that reads this file has no way to recompute them. The
// partiality is NOT divided out - that IS a scale, FRACTIONCALC carries it, and every program
// this file is for wants to handle it its own way.
gemmi::UnmergedHklMover hkl_mover(mtz.spacegroup);
// The batch headers are written for every image the observations span, not only for the images
// that produced one. AIMLESS starts a new run wherever the phi series jumps, so an image that
@@ -814,7 +825,10 @@ void WriteUnmergedMtzReflections(const std::vector<IntegrationOutcome> &outcomes
mtz.data.push_back(DetectorX(r));
mtz.data.push_back(DetectorY(r));
mtz.data.push_back(phi_start_deg(r.image_number) + wedge_deg / 2.0f);
mtz.data.push_back(r.prescaling_corr);
// prescaling_corr is the whole prescaling product; qe_corr is its sensor-efficiency part. LP is what
// is left when that part is divided out, and QE is that part in the divide-by direction.
mtz.data.push_back(r.prescaling_corr / r.qe_corr);
mtz.data.push_back(1.0f / r.qe_corr);
mtz.data.push_back(0.0f); // FLAG: nothing here is a rejected observation
mtz.data.push_back(r.delta_phi_deg);
mtz.data.push_back(r.zeta);