From 6140de7249b0203635c902b2ee87a051b7e29eb0 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 5 Sep 2026 14:29:45 +0200 Subject: [PATCH] rugnux: the unmerged MTZ carries the sensor efficiency in its own column, not inside LP LP is a column other programs read and un-apply. It is documented as the Lorentz-polarization factor, and until the sensor efficiency was folded into the same product that is what it held. Measured on our own unmerged file, LP spanned a factor of 1.1343 across the detector where pure L/P spans nothing of the sort - the excess is the efficiency, 13% end to end at 13 keV and 34% at 18 keV. Both reference implementations keep it out. Recomputing pure L/P from a stored XDS file's own geometry over its 124k reflections leaves RLP flat to 0.1% from 8.6 to 33.7 degrees, where a folded-in efficiency would have shown a 7% trend - and XDS has the sensor numbers in hand. DIALS fills LP from lorentz and polarization alone and writes QE as a separate column, even a column of ones. Split them: LP is L/P again, QE is the efficiency, and the intensity is unchanged, so a reader that un-applies LP recovers what it expects and one that wants raw counts divides by LP and multiplies by QE. Only the unmerged file moves - every other column is bit-identical and the file grows by exactly one float per reflection plus one header record. The merged files are byte for byte what they were. The process file gains an optional qe dataset beside the existing one rather than changing what that one means, so a file written before the efficiency existed still loads, and reads back as a correction of exactly one - which is what it was. It is stored rather than recomputed on read because the writer has no geometry to recompute it from, and because recomputing would have written a radial trend into every stored file that never had one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- common/Reflection.h | 7 +++ docs/CBOR.md | 1 + docs/CHANGELOG.md | 1 + docs/HDF5.md | 1 + docs/RUGNUX_ADVANCED.md | 2 +- docs/RUGNUX_INTEGRATION.md | 17 ++++-- frame_serialize/CBORStream2Deserializer.cpp | 2 + frame_serialize/CBORStream2Serializer.cpp | 1 + image_analysis/WriteReflections.cpp | 60 ++++++++++++------- image_analysis/WriteReflections.h | 9 ++- .../bragg_prediction/BraggPrediction.cpp | 1 + .../bragg_prediction/BraggPredictionGPU.cu | 1 + .../bragg_prediction/BraggPredictionRot.cpp | 1 + .../bragg_prediction/BraggPredictionRotGPU.cu | 1 + reader/HDF5MetadataSource.cpp | 9 +++ rugnux/rugnux_cli.cpp | 2 +- tests/CBORTest.cpp | 7 ++- tests/JFJochReaderTest.cpp | 2 + tests/WriteReflectionsTest.cpp | 58 ++++++++++++++++++ writer/HDF5DataFilePluginReflection.cpp | 5 +- 20 files changed, 149 insertions(+), 39 deletions(-) diff --git a/common/Reflection.h b/common/Reflection.h index e3e86ec17..e1bb0c772 100644 --- a/common/Reflection.h +++ b/common/Reflection.h @@ -33,6 +33,13 @@ struct Reflection { // a scale: the fitted per-image scale and the partiality stay out of it and are divided in // separately below. (Named after DIALS's prescaling_correction, which holds the same product.) float prescaling_corr; + // The sensor-efficiency part of prescaling_corr on its own: QE(0)/QE(alpha), always <= 1, and 1 + // where the sensor is opaque or its material and thickness are unknown. The one term singled out + // of the product above, and only so the written files can say which part of it is + // Lorentz-polarization and which is detector response: LP means L/P everywhere else in the + // field, and prescaling_corr / qe_corr is that. Defaulted to 1 so a reflection read from a file + // written before this existed is a no-op rather than a zero. + float qe_corr = 1.0f; float partiality; // fraction of the reflection recorded in the sampled (rocking) slice float zeta; float image_scale_corr; // I_true = image_scale_corr * I; = prescaling_corr / (partiality * image_scale) diff --git a/docs/CBOR.md b/docs/CBOR.md index 8be543079..522f1542d 100644 --- a/docs/CBOR.md +++ b/docs/CBOR.md @@ -168,6 +168,7 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s | - image | float | image number (present for each spot) | | | | - rp | float | Distance to Ewald sphere \[Angstrom^-1\] | | | | - rlp | float | Prescaling correction: the product of every deterministic per-reflection correction (reciprocal Lorentz, reciprocal polarization, sensor efficiency). The key name is historical and kept for compatibility | | | +| - qe | float | The sensor-efficiency part of `rlp` on its own, QE(0)/QE(alpha); 1 where the sensor is opaque or unknown. Optional | | | | - partiality | float | Partiality of the reflection | | | | - phi | float | phi angle from XDS: difference from middle of current frame, not absolute \[deg\] | | | | - zeta | float | Lorentz zeta factor (reciprocal-space geometry term) | | | diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 1ec333f6b..80b284329 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -13,6 +13,7 @@ * `rugnux` corrects each reflection for the sensor's quantum efficiency at the angle the diffracted beam meets the detector. * `rugnux` applies the sensor quantum-efficiency correction to still images processed on a GPU, which were left uncorrected while the same data on a CPU-only machine was corrected. +* The unmerged MTZ keeps the sensor efficiency out of its `LP` column and records it in a new `QE` column, as XDS and DIALS do, so raw counts are `I / LP * QE`; the merged intensities are unchanged, and `_process.h5` gains a matching optional `qe` dataset. * `rugnux` takes the sensor attenuation length from tabulated coefficients rather than a wavelength-cubed approximation, which corrects the parallax term of the spot-width variance on CdTe sensors. * `rugnux --model` treats the model as a hypothesis: it decides the enantiomorph and the indexing only where its R-work beats that of the same model in random orientations, and a model the data reject is still scored, placed and mapped, but leaves the reflection files byte for byte what a run with no model writes. * `rugnux --model` places the model against the data as a rigid body before scoring it, writes sigma_A-weighted 2mFo-DFc and mFo-DFc maps in place of the unweighted 2Fo-Fc and Fo-Fc, and writes the model as it was placed - `_model.cif`, and `_model.pdb` where the PDB format can express the cell - in the cell and space group of the reflection files beside it. diff --git a/docs/HDF5.md b/docs/HDF5.md index 2b5c4c7ea..d3345709f 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -436,6 +436,7 @@ mostly onto the standard | `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) | | `observed_frame` | | standard | image number of the reflection | | `lp` | | standard | reciprocal of the prescaling correction (stored as `1/prescaling_corr`). The NXreflections name is `lp`, but the stored product also carries the sensor efficiency term | +| `qe` | | **extension** | the sensor-efficiency part of `lp` on its own, in the same reciprocal convention, so `lp / qe` is the pure Lorentz–polarization reciprocal. Absent in files written before it existed; the reader then takes the whole of `lp` for Lorentz–polarization, which is what those files mean | | `partiality` | | standard | recorded fraction of the reflection | | `delta_phi` | deg | **extension** | XDS Δφ: offset from the centre of the current frame | | `zeta` | | **extension** | Lorentz ζ factor (reciprocal-space geometry term) | diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 8649a450e..c1e30bdbd 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -335,7 +335,7 @@ Scaling and merging: | `--reference-column