Files
Jungfraujoch/image_analysis/bragg_prediction/BraggPredictionRot.cpp
T
leonarski_fandClaude Opus 5.5 0a515bb086 rc179: everything on rc178 after its merge into main (#88), except the CHANGELOG
The rc178 branch continued after #88 was squash-merged; this is its content from b9ada7914 (the tree
#88 squashed) to 094eee31a, as one change on main. The individual commits stay on the rc178 branch:

- still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes
- AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets
- DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked
- tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout
- Dense per-pixel maps through the panel table (panel layout stage 1)
- Panel id on spots and the panel table in the start message
- Viewer and preview show a panel detector as it lies on the detector plane
- Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster
- Panel layout stage 2: prediction and Bragg integration through the panel table
- XTC reader: frames handed up as stored, streams indexed in parallel
- XFEL reader: integer frames stay integer, one gain, placement by panel
- EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery"
- still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed
- float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count
- CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them
- Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table
- Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane
- still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good
- still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file
- still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition
- stills fixed cost: geometry refinement and pre-scan do the same work in less wall time
- stills geometry sample and trial spot finding run as many workers as the image pass
- Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame
- StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's
- MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in)
- Stills given a cell: the written cell is the mean over the frames that are merged
- Stills: the radial curvature correction decided per run on the same empty apertures
- StillsPartialityRefine: measure the run width only on shells with signal
- Stills pre-scan: report the paired z of the radial-correction comparison
- module placement: the five concurrent fits each keep every thread
- still battery: score R-free against the deposited data through the same dimple protocol
- still battery: the R-free target is the lower of the deposition's dimple and published values
- Stills: background clip follows the radial curve; signal disk from the lattice spacing
- Stills keep an overloaded observation as a lower bound instead of dropping it
- Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip
- battery: R-free against the deposition without refinement only, one protocol for rotation and stills
- still battery: EuXFEL sets run on a 2,000-image subset for now
- stills geometry refinement: fit the detector distance and the cell together
- stills geometry refinement: a distance given with --detector-distance is held in the joint fit
- Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count
- Metal integration test: the CPU comparison runs on a rotation experiment
- Stills: kept overloads go back to the drop where they are not concentrated at low resolution
- stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated
- battery results: XFEL still battery 20261022-1358 on rc178 0cf62452c - 12 sets, cold, R_free unrefined
- battery results: still summary gains the resolution of our merge and of the deposition
- Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value
- Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed
- XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock
- stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes
- rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid
- stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree
- CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get
- broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4)
- frontend: Mol* viewer of the placed model and its density on the Rugnux live page
- writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does
- rugnux live maps: one unit cell centred on the model; the viewer opens on all of it
- Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer
- Mol* viewer: open with the maps drawn within 10 A of the centre of view
- Mol* viewer: aaredb's structure-viewer setup, on a black stage
- rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons
- Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2)
- ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI)
- stills tilt decision: the modules stay where the run placed them
- still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it
- battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC
- Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons
- BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it
- ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization
- BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain
- ShadowFinder: measure the modulation robustly under a wide one-sided shadow
- ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation
- ShadowFinder: back to the predicted polarization factor, in the plane the background shows
- MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again
- battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables
- XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel
- stills geometry trial: the hypotheses a search asks are indexed side by side
- stills tilt decision: the fits start beside the pre-scan and the geometry trial
- XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache
- XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step
- GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block
- GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart
- docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts
- battery: doc_check.py cross-checks the manifests against the dataset pages
- stills module placement: the placement fit gets half the threads, each held-out fit an eighth
- still battery: the EuXFEL sets run on all their frames, as the published still runs did
- battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table
- macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-23 17:56:42 +02:00

289 lines
16 KiB
C++

// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <algorithm>
#include <cmath>
#include "../../common/JFJochMath.h"
#include "BraggPredictionRot.h"
#include "../SensorAbsorption.h"
#include "../bragg_integration/SystematicAbsence.h"
#include "RockingSlice.h"
#include "RotationColumnRange.h"
int BraggPredictionRot::Calc(const DiffractionExperiment &experiment, const CrystalLattice &lattice,
const BraggPredictionSettings &settings) {
const auto geom = experiment.GetDiffractionGeometry();
// The plane the panels span (without panels, the image): a quick reject, a reflection in a gap
// between panels is dropped when it is integrated.
const PlaneBox bounds = experiment.GetModules().PlaneBounds(experiment.GetXPixelsNum(), experiment.GetYPixelsNum());
const auto det_x0 = static_cast<float>(bounds.x0), det_x1 = static_cast<float>(bounds.x1);
const auto det_y0 = static_cast<float>(bounds.y0), det_y1 = static_cast<float>(bounds.y1);
const float one_over_dmax = 1.0f / settings.high_res_A;
const float one_over_dmax_sq = one_over_dmax * one_over_dmax;
float one_over_wavelength = 1.0f / geom.GetWavelength_A();
const Coord Astar = lattice.Astar();
const Coord Bstar = lattice.Bstar();
const Coord Cstar = lattice.Cstar();
const Coord S0 = geom.GetScatteringVector();
std::vector<float> rot = geom.GetDetectorMatrix().transpose().arr();
// Precompute detector geometry constants
float beam_x = geom.GetBeamX_pxl();
float beam_y = geom.GetBeamY_pxl();
float det_distance = geom.GetDetectorDistance_mm();
float pixel_size = geom.GetPixelSize_mm();
float F = det_distance / pixel_size;
const auto gon_opt = experiment.GetGoniometer();
if (!gon_opt.has_value())
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"BraggPredictionRotationCPU requires a goniometer axis");
const GoniometerAxis& gon = *gon_opt;
const Coord m2 = gon.GetAxis().Normalize();
const Coord m1 = (m2 % S0).Normalize();
const Coord m3 = (m1 % m2).Normalize();
const float m2_S0 = m2 * S0;
const float four_S0_sq = 4 * S0 * S0;
const float m3_S0 = m3 * S0;
int i = 0;
const Coord beam_dir = S0.Normalize();
const float mos_angle_rad = settings.mosaicity_deg * static_cast<float>(PI) / 180.f;
const float half_wedge_angle_rad = settings.wedge_deg * static_cast<float>(PI) / 180.f / 2.0f ;
// Energy bandwidth widens the rocking curve. Differentiating Bragg's law at fixed d gives
// dtheta = (dlambda/lambda) tan(theta_B), a spread in the same glancing angle the mosaic spread
// smears, so it adds to sigma_M in quadrature. It is NOT divided by zeta here: rotating the
// crystal by dphi changes theta by zeta*dphi, so the 1/zeta that turns an angular width into a
// rotation width is already the one c1 (and the epsilon3 cutoff) applies to sigma_M. The fitted
// sigma_M has this term deconvolved out (CalcMosaicityXDS), so it is not counted twice.
// sin(theta_B) = lambda/(2d) = lambda*|p0|/2. Zero bandwidth leaves every reflection untouched.
const float bandwidth_sigma = settings.bandwidth_sigma;
const float half_wavelength_A = geom.GetWavelength_A() / 2.0f;
// Angle-dependent sensor efficiency. Per-dataset constants (thickness, material, wavelength)
// collapse to two numbers here; the per-reflection part is one exponential below. Inert - and
// bit-identical to not applying it - wherever the sensor is opaque, which is every long
// wavelength, so it needs no flag and no threshold anyone has to choose.
const auto &det = experiment.GetDetectorSetup();
const auto sensor_qe = sensor_absorption::SensorQE::Build(
det.GetSensorMaterial(), det.GetSensorThickness_um(), geom.GetWavelength_A());
// The air in the sample-to-pixel flight path carries the same cos(alpha) dependence, with the
// opposite sign. See sensor_absorption::FlightPathAttenuation; off leaves d_over_L at 0, which is
// bit-identical to not applying it.
const auto air = sensor_absorption::FlightPathAttenuation::Build(
experiment.GetBraggIntegrationSettings().GetFlightPath(), geom.GetDetectorDistance_mm(),
geom.GetWavelength_A());
// The phi test after the atan2 below, taken before it: a solution with |phi| above phi_limit fails
// that test whatever its own bandwidth term (RotationPhiLimit). |phi| > phi_limit is
// cos(phi) < cos(phi_limit), which the (cos, sin) pair answers without the angle, so every solution
// rejected here is rejected by that test too. -1 = nothing is rejected here.
const double phi_limit = RotationPhiLimit(settings.min_zeta, mos_angle_rad, bandwidth_sigma, half_wavelength_A,
one_over_dmax, half_wedge_angle_rad, settings.mosaicity_multiplier);
const double cos_phi_limit = phi_limit > 0.0 ? std::cos(phi_limit) : -1.0;
// Which l of an (h, k) column can pass that test at all, so the walk does not visit the rest of
// the box. f(p0) = |p0|^2 + 2 S0.p0 is zero where p0 lies on the Ewald sphere, and rotating p0 by
// phi about the axis moves S0.p0 by at most |S0_perp| |p0_perp| |phi|. So a reflection whose
// solution lies within phi_limit has |f| <= 2 |S0_perp| |p0| phi_limit <= f_max, and along a column
// f is a quadratic in l, so the l that can pass are at most two intervals. Both are widened - f_max
// by 1%, each end by a whole index - so every l the tests below keep is still walked, in the same
// order and with the same arithmetic; the rest the phi test would have rejected anyway.
auto dot = [](const Coord &a, const Coord &b) {
return static_cast<double>(a.x) * b.x + static_cast<double>(a.y) * b.y + static_cast<double>(a.z) * b.z;
};
const double S0_perp = std::sqrt(std::max(0.0, dot(S0, S0) - static_cast<double>(m2_S0) * m2_S0));
const double f_max = 1.01 * 2.0 * S0_perp * one_over_dmax * phi_limit;
const double C_C = dot(Cstar, Cstar), S0_C = dot(S0, Cstar), inv_2C_C = 1.0 / (2.0 * C_C);
// p0 = A* h + B* k + C* l, evaluated as ((A* h) + (B* k)) + (C* l) exactly as before, with the
// terms that do not change in the inner loops taken out of them.
std::vector<Coord> Cstar_l(2 * settings.max_l + 1);
for (int l = -settings.max_l; l <= settings.max_l; l++)
Cstar_l[l + settings.max_l] = Cstar * l;
for (int h = -settings.max_h; h <= settings.max_h; h++) {
const Coord Astar_h = Astar * h;
for (int k = -settings.max_k; k <= settings.max_k; k++) {
const Coord AB = Astar_h + Bstar * k;
// The l of this column that can pass (above).
RotationColumn col{-settings.max_l, settings.max_l, settings.max_l + 1, settings.max_l + 1};
if (phi_limit > 0.0 && !enumerate_full_box
&& !RotationColumnRange(C_C, inv_2C_C, 2.0 * (dot(AB, Cstar) + S0_C),
dot(AB, AB) + 2.0 * dot(S0, AB), f_max, settings.max_l, col))
continue;
for (int l = col.l_lo; l <= col.l_hi; l++) {
if (l >= col.skip_lo && l <= col.skip_hi)
l = col.skip_hi + 1;
if (l > col.l_hi)
break;
if (systematic_absence(h, k, l, settings.centering))
continue;
const Coord &Cl = Cstar_l[l + settings.max_l];
const Coord p0(AB.x + Cl.x, AB.y + Cl.y, AB.z + Cl.z);
float p0_sq = p0 * p0;
if (p0_sq <= 0.0f || p0_sq > one_over_dmax_sq)
continue;
const float p0_m1 = p0 * m1;
const float p0_m2 = p0 * m2;
const float p0_m3 = p0 * m3;
const float rho_sq = p0_sq - (p0_m2 * p0_m2);
const float p_m3 = (- p0_sq / 2 - p0_m2 * m2_S0) / m3_S0;
const float p_m2 = p0_m2;
const float p_m1_opt[2] = {
std::sqrt(rho_sq - p_m3 * p_m3),
-std::sqrt(rho_sq - p_m3 * p_m3)
};
// No solution for Laue equations
if ((rho_sq < p_m3 * p_m3) || (p0_sq > four_S0_sq))
continue;
// Effective rocking width for this reflection: mosaicity broadened by the bandwidth
// term. sin(theta_B) <= 1 is guaranteed by the p0_sq test just above.
float mos_eff_rad = mos_angle_rad;
if (bandwidth_sigma > 0.0f) {
const float sin_theta = half_wavelength_A * std::sqrt(p0_sq);
const float dphi_bw = bandwidth_sigma * sin_theta / std::sqrt(1.0f - sin_theta * sin_theta);
mos_eff_rad = std::sqrt(mos_angle_rad * mos_angle_rad + dphi_bw * dphi_bw);
}
for (const auto& p_m1 : p_m1_opt) {
const float cosphi = (p_m1 * p0_m1 + p_m3 * p0_m3) / rho_sq;
const float sinphi = (p_m1 * p0_m3 - p_m3 * p0_m1) / rho_sq;
const double c = cosphi, s = sinphi;
if (c < cos_phi_limit * std::sqrt(c * c + s * s))
continue;
float phi = -1.0f * std::atan2(sinphi, cosphi);
// Most solutions are far from this frame. zeta >= min_zeta, so the rocking-curve test
// below (epsilon3) rejects every solution with min_zeta * (|phi| - half wedge) above the
// limit; testing that first, with a margin well above float rounding, skips the rest
// of the arithmetic for them and changes nothing about the ones that pass.
if (settings.min_zeta > 0.0f
&& settings.min_zeta * (std::fabs(static_cast<double>(phi)) - half_wedge_angle_rad)
> settings.mosaicity_multiplier * static_cast<double>(mos_eff_rad) + 1e-5)
continue;
Coord p = m1 * p_m1 + m2 * p_m2 + m3 * p_m3; // p0 vector "rotated" to diffracting condition
Coord S = S0 + p;
const Coord e1 = (S % S0).Normalize();
const float zeta_abs = std::fabs(m2 * e1);
if (zeta_abs < settings.min_zeta)
continue;
// Is any of this reflection's rocking curve inside THIS image's oscillation range?
// phi is the offset from the frame's mid-exposure angle to the exact diffracting
// condition, so the curve - a Gaussian of width mos_eff/zeta in phi - has to be
// measured against the frame's EDGE, not its centre. Testing |phi|*zeta alone
// asks whether the frame centre is within the curve, which is a different and
// stricter question: it rejects a reflection whose curve overlaps the exposure
// but whose exact condition falls outside it. Since consecutive frame centres are
// one wedge apart, the nearest centre can be half a wedge away, so a reflection is
// rejected on EVERY frame - lost entirely, not merely clipped - once
// mos_eff < zeta * wedge / (2 * multiplier). That is coarse slicing on a sharp
// crystal at high zeta, where the reflection is fully recorded and cleanest.
float epsilon3 = std::fabs(phi * zeta_abs) - half_wedge_angle_rad * zeta_abs;
if (epsilon3 > settings.mosaicity_multiplier * mos_eff_rad)
continue;
// Reciprocal Lorentz (Kabsch 2010): L^-1 = |m2 . (S x S0)| / (|S| |S0|) =
// |zeta * sin angle(S,S0)|. The original divided by the scalar product
// S.S0 = |S||S0|cos(2theta), adding a spurious 1/cos(2theta) (1.8x at 1 A) that
// corrupts the absolute/Wilson scale (it cancels within a resolution shell, so
// CC1/2 / CCref / R-meas are neutral).
const float lorentz_reciprocal = std::fabs(m2 * (S % S0)) / (S.Length() * S0.Length());
const float c1 = zeta_abs / (std::sqrt(2.0f) * mos_eff_rad);
const float partiality = RockingSlicePartiality(phi, half_wedge_angle_rad, c1);
// Place the partial where this frame recorded it: at the centre of the frame's slice of
// the rocking curve, which is the exact-condition position walked along its Debye ring by
// the rotation between the two (RockingSlice.h). Walking along the ring is turning S
// about the beam, by the rotation's component along the ring over the ring's radius.
const float c_slice = RockingSliceCentroid_rad(phi, half_wedge_angle_rad, c1, partiality);
const Coord S_par = beam_dir * (beam_dir * S), S_perp = S - S_par;
const Coord S_turn = beam_dir % S; // |S_turn| = |S_perp|
const float psi = c_slice * ((m2 % p) * S_turn) / (S_perp * S_perp);
S = S_par + S_perp * std::cos(psi) + S_turn * std::sin(psi);
// Inlined RecipToDetector: the full transposed detector matrix, tilt and discrete orientation
// Apply rotation matrix transpose
float S_rot_x = rot[0] * S.x + rot[1] * S.y + rot[2] * S.z;
float S_rot_y = rot[3] * S.x + rot[4] * S.y + rot[5] * S.z;
float S_rot_z = rot[6] * S.x + rot[7] * S.y + rot[8] * S.z;
if (S_rot_z <= 0)
continue;
float x = beam_x + F * S_rot_x / S_rot_z;
float y = beam_y + F * S_rot_y / S_rot_z;
if ((x < det_x0) || (x >= det_x1) || (y < det_y0) || (y >= det_y1))
continue;
float dist_ewald_sphere = std::fabs(S.Length() - one_over_wavelength);
// Sensor quantum efficiency at this reflection's own angle of incidence on the
// detector. The angle is taken against the DETECTOR NORMAL - S_rot is the
// diffracted direction in the detector's own frame, so its z component over its
// length is that cosine already, at no cost. Taking it here rather than from the
// resolution is what makes it right on a tilted detector, where the incidence
// angle stops being a function of resolution and the correction stops
// cancelling within a resolution shell.
const float cos_alpha = S_rot_z / S.Length();
const float qe_corr = sensor_qe.Factor(cos_alpha);
const float flight_corr = air.Factor(cos_alpha);
if (i == max_reflections)
GrowCapacity(2 * max_reflections);
float d = 1.0f / sqrtf(p0_sq);
reflections[i] = Reflection{
.h = h,
.k = k,
.l = l,
.delta_phi_deg = phi * 180.0f / static_cast<float>(PI),
.predicted_x = x,
.predicted_y = y,
.observed_x = NAN,
.observed_y = NAN,
.d = d,
.dist_ewald = dist_ewald_sphere,
.prescaling_corr = lorentz_reciprocal,
.qe_corr = qe_corr,
.flight_corr = flight_corr,
.partiality = partiality,
.zeta = zeta_abs,
.image_scale_corr = lorentz_reciprocal * qe_corr * flight_corr / partiality,
};
i++;
}
}
}
}
return TruncateToOutput(i);
}