Files
Jungfraujoch/image_analysis/bragg_prediction/BraggPrediction.cpp
T
leonarski_fandClaude Opus 5.5 0a515bb086 rc179: everything on rc178 after its merge into main (#88), except the CHANGELOG
The rc178 branch continued after #88 was squash-merged; this is its content from b9ada7914 (the tree
#88 squashed) to 094eee31a, as one change on main. The individual commits stay on the rc178 branch:

- still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes
- AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets
- DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked
- tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout
- Dense per-pixel maps through the panel table (panel layout stage 1)
- Panel id on spots and the panel table in the start message
- Viewer and preview show a panel detector as it lies on the detector plane
- Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster
- Panel layout stage 2: prediction and Bragg integration through the panel table
- XTC reader: frames handed up as stored, streams indexed in parallel
- XFEL reader: integer frames stay integer, one gain, placement by panel
- EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery"
- still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed
- float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count
- CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them
- Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table
- Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane
- still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good
- still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file
- still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition
- stills fixed cost: geometry refinement and pre-scan do the same work in less wall time
- stills geometry sample and trial spot finding run as many workers as the image pass
- Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame
- StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's
- MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in)
- Stills given a cell: the written cell is the mean over the frames that are merged
- Stills: the radial curvature correction decided per run on the same empty apertures
- StillsPartialityRefine: measure the run width only on shells with signal
- Stills pre-scan: report the paired z of the radial-correction comparison
- module placement: the five concurrent fits each keep every thread
- still battery: score R-free against the deposited data through the same dimple protocol
- still battery: the R-free target is the lower of the deposition's dimple and published values
- Stills: background clip follows the radial curve; signal disk from the lattice spacing
- Stills keep an overloaded observation as a lower bound instead of dropping it
- Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip
- battery: R-free against the deposition without refinement only, one protocol for rotation and stills
- still battery: EuXFEL sets run on a 2,000-image subset for now
- stills geometry refinement: fit the detector distance and the cell together
- stills geometry refinement: a distance given with --detector-distance is held in the joint fit
- Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count
- Metal integration test: the CPU comparison runs on a rotation experiment
- Stills: kept overloads go back to the drop where they are not concentrated at low resolution
- stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated
- battery results: XFEL still battery 20261022-1358 on rc178 0cf62452c - 12 sets, cold, R_free unrefined
- battery results: still summary gains the resolution of our merge and of the deposition
- Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value
- Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed
- XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock
- stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes
- rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid
- stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree
- CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get
- broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4)
- frontend: Mol* viewer of the placed model and its density on the Rugnux live page
- writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does
- rugnux live maps: one unit cell centred on the model; the viewer opens on all of it
- Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer
- Mol* viewer: open with the maps drawn within 10 A of the centre of view
- Mol* viewer: aaredb's structure-viewer setup, on a black stage
- rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons
- Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2)
- ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI)
- stills tilt decision: the modules stay where the run placed them
- still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it
- battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC
- Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons
- BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it
- ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization
- BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain
- ShadowFinder: measure the modulation robustly under a wide one-sided shadow
- ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation
- ShadowFinder: back to the predicted polarization factor, in the plane the background shows
- MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again
- battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables
- XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel
- stills geometry trial: the hypotheses a search asks are indexed side by side
- stills tilt decision: the fits start beside the pre-scan and the geometry trial
- XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache
- XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step
- GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block
- GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart
- docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts
- battery: doc_check.py cross-checks the manifests against the dataset pages
- stills module placement: the placement fit gets half the threads, each held-out fit an eighth
- still battery: the EuXFEL sets run on all their frames, as the published still runs did
- battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table
- macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-23 17:56:42 +02:00

238 lines
11 KiB
C++
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <algorithm>
#include "../../common/JFJochMath.h"
#include "../../common/Logger.h"
#include "BraggPrediction.h"
#include "../SensorAbsorption.h"
#include "../bragg_integration/SystematicAbsence.h"
void BraggPrediction::GrowCapacity(int count) {
reflections.resize(count);
max_reflections = count;
}
void BraggPrediction::OrderOutput(int count) {
order_keys.resize(count);
for (int i = 0; i < count; i++) {
const Reflection &r = reflections[i];
order_keys[i] = {r.h, r.k, r.l, r.delta_phi_deg, i};
}
std::sort(order_keys.begin(), order_keys.end(),
[](const OrderKey &a, const OrderKey &b) {
if (a.h != b.h) return a.h < b.h;
if (a.k != b.k) return a.k < b.k;
if (a.l != b.l) return a.l < b.l;
return a.delta_phi_deg < b.delta_phi_deg;
});
order_scratch.resize(count);
for (int i = 0; i < count; i++) order_scratch[i] = reflections[order_keys[i].index];
std::copy(order_scratch.begin(), order_scratch.end(), reflections.begin());
}
int BraggPrediction::TruncateToOutput(int count) {
if (count <= output_limit)
return count;
// Partiality first, then excitation error: exactly one of the two says anything on each path. On the
// rotation path dist_ewald is identically zero - BraggPredictionRot picks the rocking coordinate so
// that 2*S0.p + p.p = 0, i.e. |S| = 1/lambda exactly - so ranking by it there left an hkl-lexicographic
// prefix rather than the best-recorded reflections. On the still path every partiality is 1, so the
// ranking falls through to dist_ewald, which is what it has always been.
std::partial_sort(reflections.begin(), reflections.begin() + output_limit,
reflections.begin() + count,
[](const Reflection &a, const Reflection &b) {
if (a.partiality != b.partiality) return a.partiality > b.partiality;
if (a.dist_ewald != b.dist_ewald) return a.dist_ewald < b.dist_ewald;
if (a.h != b.h) return a.h < b.h;
if (a.k != b.k) return a.k < b.k;
return a.l < b.l;
});
return output_limit;
}
namespace {
// Number of bandwidth sigmas included in the (radially thickened) Ewald-shell
// acceptance window. 3σ captures essentially the whole pink-beam smear; matches
// the conservative end of the mosaicity cutoff used by callers.
constexpr float kBandwidthCutoffSigmas = 3.0f;
}
BraggPrediction::BraggPrediction(int max_reflections)
: max_reflections(max_reflections), reflections(max_reflections) {}
const std::vector<Reflection> &BraggPrediction::GetReflections() const {
return reflections;
}
int BraggPrediction::Calc(const DiffractionExperiment &experiment, const CrystalLattice &lattice,
const BraggPredictionSettings &settings) {
const auto geom = experiment.GetDiffractionGeometry();
// The plane the panels span (without panels, the image): a quick reject, a reflection in a gap
// between panels is dropped when it is integrated.
const PlaneBox bounds = experiment.GetModules().PlaneBounds(experiment.GetXPixelsNum(), experiment.GetYPixelsNum());
const auto det_x0 = static_cast<float>(bounds.x0), det_x1 = static_cast<float>(bounds.x1);
const auto det_y0 = static_cast<float>(bounds.y0), det_y1 = static_cast<float>(bounds.y1);
const float one_over_dmax = 1.0f / settings.high_res_A;
const float one_over_dmax_sq = one_over_dmax * one_over_dmax;
float one_over_wavelength = 1.0f / geom.GetWavelength_A();
const Coord Astar = lattice.Astar();
const Coord Bstar = lattice.Bstar();
const Coord Cstar = lattice.Cstar();
const Coord S0 = geom.GetScatteringVector();
std::vector<float> rot = geom.GetDetectorMatrix().transpose().arr();
// Precompute detector geometry constants
float beam_x = geom.GetBeamX_pxl();
float beam_y = geom.GetBeamY_pxl();
float det_distance = geom.GetDetectorDistance_mm();
float pixel_size = geom.GetPixelSize_mm();
float F = det_distance / pixel_size;
// Angle-dependent sensor efficiency; per-dataset constants collapsed to two numbers. Inert
// wherever the sensor is opaque, which is every long wavelength.
const auto &det_setup = experiment.GetDetectorSetup();
const auto sensor_qe = sensor_absorption::SensorQE::Build(
det_setup.GetSensorMaterial(), det_setup.GetSensorThickness_um(), geom.GetWavelength_A());
// The air in the sample-to-pixel flight path carries the same cos(alpha) dependence, with the
// opposite sign. See sensor_absorption::FlightPathAttenuation; off leaves d_over_L at 0, which is
// bit-identical to not applying it.
const auto air = sensor_absorption::FlightPathAttenuation::Build(
experiment.GetBraggIntegrationSettings().GetFlightPath(), det_distance,
geom.GetWavelength_A());
const float epsilon = 1e-5f;
const float s0_sq = S0 * S0;
const float rad_to_deg = 180.0f / static_cast<float>(PI);
int i = 0;
for (int h = -settings.max_h; h <= settings.max_h; h++) {
// Precompute A* h contribution
const float Ah_x = Astar.x * h;
const float Ah_y = Astar.y * h;
const float Ah_z = Astar.z * h;
for (int k = -settings.max_k; k <= settings.max_k; k++) {
// Accumulate B* k contribution
const float AhBk_x = Ah_x + Bstar.x * k;
const float AhBk_y = Ah_y + Bstar.y * k;
const float AhBk_z = Ah_z + Bstar.z * k;
for (int l = -settings.max_l; l <= settings.max_l; l++) {
if (systematic_absence(h, k, l, settings.centering))
continue;
float recip_x = AhBk_x + Cstar.x * l;
float recip_y = AhBk_y + Cstar.y * l;
float recip_z = AhBk_z + Cstar.z * l;
float recip_sq = recip_x * recip_x + recip_y * recip_y + recip_z * recip_z;
if (recip_sq > one_over_dmax_sq)
continue;
float S_x = recip_x + S0.x;
float S_y = recip_y + S0.y;
float S_z = recip_z + S0.z;
float S_len = sqrtf(S_x * S_x + S_y * S_y + S_z * S_z);
float dist_ewald_sphere = std::fabs(S_len - one_over_wavelength);
// Energy bandwidth thickens the Ewald shell radially: at the
// diffraction condition |S|-1/λ shifts by recip_z·(Δλ/λ), i.e.
// σ_bw = |recip_z|·bandwidth_sigma (= bλ/2d²). Broaden the acceptance
// window in quadrature so high-resolution shells (smeared most, ∝1/d²)
// are not clipped.
float radial_cutoff = settings.ewald_dist_cutoff;
if (settings.bandwidth_sigma > 0.0f) {
const float bw_tol = kBandwidthCutoffSigmas * settings.bandwidth_sigma * std::fabs(recip_z);
radial_cutoff = std::sqrt(radial_cutoff * radial_cutoff + bw_tol * bw_tol);
}
if (dist_ewald_sphere <= radial_cutoff ) {
const float s0_p0 = S0.x * recip_x + S0.y * recip_y + S0.z * recip_z;
const float val = s0_sq * recip_sq - s0_p0 * s0_p0;
float delta_phi_deg = NAN;
if (std::fabs(val) >= epsilon && s0_sq > epsilon) {
const float a_num = (s0_sq - 0.25f * recip_sq) * recip_sq;
if (a_num >= 0.0f) {
const float A = std::sqrt(a_num / val);
const float B = (A * s0_p0 + 0.5f * recip_sq) / s0_sq;
const float p_star_x = A * recip_x - B * S0.x;
const float p_star_y = A * recip_y - B * S0.y;
const float p_star_z = A * recip_z - B * S0.z;
const float p_star_sq = p_star_x * p_star_x + p_star_y * p_star_y + p_star_z * p_star_z;
const float denom = std::sqrt(p_star_sq * recip_sq);
if (denom >= epsilon) {
float c = (p_star_x * recip_x + p_star_y * recip_y + p_star_z * recip_z) / denom;
c = std::fmax(-1.0f, std::fmin(1.0f, c));
delta_phi_deg = std::acos(c) * rad_to_deg;
}
}
}
// Inlined RecipToDetector: the full transposed detector matrix, tilt and discrete orientation
// Apply rotation matrix transpose
float S_rot_x = rot[0] * S_x + rot[1] * S_y + rot[2] * S_z;
float S_rot_y = rot[3] * S_x + rot[4] * S_y + rot[5] * S_z;
float S_rot_z = rot[6] * S_x + rot[7] * S_y + rot[8] * S_z;
if (S_rot_z <= 0)
continue;
// Project to detector coordinates
// Assume detector is along x,y,z coordinates after rotation
float x = beam_x + F * S_rot_x / S_rot_z;
float y = beam_y + F * S_rot_y / S_rot_z;
if ((x < det_x0) || (x >= det_x1) || (y < det_y0) || (y >= det_y1))
continue;
// Sensor quantum efficiency at this reflection's angle of incidence on the
// detector - the diffracted direction against the detector normal, so the
// detector tilt is carried for free. See sensor_absorption::SensorQE.
const float cos_alpha =
S_rot_z / std::sqrt(S_x * S_x + S_y * S_y + S_z * S_z);
const float qe_corr = sensor_qe.Factor(cos_alpha);
const float flight_corr = air.Factor(cos_alpha);
if (i == max_reflections)
GrowCapacity(2 * max_reflections);
float d = 1.0f / sqrtf(recip_sq);
reflections[i] = Reflection{
.h = h,
.k = k,
.l = l,
.delta_phi_deg = delta_phi_deg,
.predicted_x = x,
.predicted_y = y,
.observed_x = NAN,
.observed_y = NAN,
.d = d,
.dist_ewald = dist_ewald_sphere,
.prescaling_corr = 1.0f,
.qe_corr = qe_corr,
.flight_corr = flight_corr,
.partiality = 1.0f,
.zeta = 1.0,
.image_scale_corr = qe_corr * flight_corr
};
++i;
}
}
}
}
return TruncateToOutput(i);
}