mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 87 Wiki Activity

rc179: everything on rc178 after its merge into main (#88), except the CHANGELOG

Browse Source 
The rc178 branch continued after #88 was squash-merged; this is its content from b9ada7914 (the tree
#88 squashed) to 094eee31a, as one change on main. The individual commits stay on the rc178 branch:

- still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes
- AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets
- DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked
- tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout
- Dense per-pixel maps through the panel table (panel layout stage 1)
- Panel id on spots and the panel table in the start message
- Viewer and preview show a panel detector as it lies on the detector plane
- Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster
- Panel layout stage 2: prediction and Bragg integration through the panel table
- XTC reader: frames handed up as stored, streams indexed in parallel
- XFEL reader: integer frames stay integer, one gain, placement by panel
- EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery"
- still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed
- float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count
- CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them
- Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table
- Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane
- still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good
- still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file
- still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition
- stills fixed cost: geometry refinement and pre-scan do the same work in less wall time
- stills geometry sample and trial spot finding run as many workers as the image pass
- Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame
- StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's
- MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in)
- Stills given a cell: the written cell is the mean over the frames that are merged
- Stills: the radial curvature correction decided per run on the same empty apertures
- StillsPartialityRefine: measure the run width only on shells with signal
- Stills pre-scan: report the paired z of the radial-correction comparison
- module placement: the five concurrent fits each keep every thread
- still battery: score R-free against the deposited data through the same dimple protocol
- still battery: the R-free target is the lower of the deposition's dimple and published values
- Stills: background clip follows the radial curve; signal disk from the lattice spacing
- Stills keep an overloaded observation as a lower bound instead of dropping it
- Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip
- battery: R-free against the deposition without refinement only, one protocol for rotation and stills
- still battery: EuXFEL sets run on a 2,000-image subset for now
- stills geometry refinement: fit the detector distance and the cell together
- stills geometry refinement: a distance given with --detector-distance is held in the joint fit
- Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count
- Metal integration test: the CPU comparison runs on a rotation experiment
- Stills: kept overloads go back to the drop where they are not concentrated at low resolution
- stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated
- battery results: XFEL still battery 20261022-1358 on rc178 0cf62452c - 12 sets, cold, R_free unrefined
- battery results: still summary gains the resolution of our merge and of the deposition
- Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value
- Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed
- XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock
- stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes
- rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid
- stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree
- CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get
- broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4)
- frontend: Mol* viewer of the placed model and its density on the Rugnux live page
- writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does
- rugnux live maps: one unit cell centred on the model; the viewer opens on all of it
- Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer
- Mol* viewer: open with the maps drawn within 10 A of the centre of view
- Mol* viewer: aaredb's structure-viewer setup, on a black stage
- rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons
- Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2)
- ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI)
- stills tilt decision: the modules stay where the run placed them
- still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it
- battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC
- Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons
- BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it
- ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization
- BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain
- ShadowFinder: measure the modulation robustly under a wide one-sided shadow
- ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation
- ShadowFinder: back to the predicted polarization factor, in the plane the background shows
- MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again
- battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables
- XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel
- stills geometry trial: the hypotheses a search asks are indexed side by side
- stills tilt decision: the fits start beside the pre-scan and the geometry trial
- XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache
- XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step
- GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block
- GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart
- docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts
- battery: doc_check.py cross-checks the manifests against the dataset pages
- stills module placement: the placement fit gets half the threads, each held-out fit an eighth
- still battery: the EuXFEL sets run on all their frames, as the published still runs did
- battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table
- macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-23 17:56:42 +02:00
1 parent afbe844b20
commit 0a515bb086
218 files changed
+55733 -1764
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

image_analysis/bragg_integration/BraggIntegrationEngineCPU.cpp
+113 -66
View File
@@ -7,6 +7,8 @@
#include <cmath>
#include <cstdint>
#include <limits>
#include <tuple>
#include <utility>
#include <vector>
#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86))
@@ -23,6 +25,11 @@ namespace {
// The engine reads pixels in the INT32_MIN(masked)/INT32_MAX(saturated) convention.
inline bool valid(int32_t v) { return v != INT32_MIN && v != INT32_MAX; }
// A direction on the plane in a panel's pixel axes (MakeBraggStencil). Unchanged without panels.
inline std::pair<double, double> to_panel_axes(const BraggPanel &pn, double ux, double uy) {
return {ux * pn.fx + uy * pn.fy, ux * pn.sx + uy * pn.sy};
}
// Ask for the cache lines from first to last ahead of their use. Only a hint: it changes no result.
inline void prefetch(const void *p) {
#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86))
@@ -78,27 +85,33 @@ BraggIntegrationEngineCPU::BraggIntegrationEngineCPU(const DiffractionExperiment
: BraggIntegrationEngine(experiment),
refl_mask(static_cast<int>(xpixel), static_cast<int>(ypixel), 0),
owner(static_cast<int>(xpixel), static_cast<int>(ypixel), BRAGG_OWNER_NONE),
static_mask(mask.GetPackedMask()) {}
static_mask(mask.GetPackedMask()) {
if (mask.GetMask().size() != npixel)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Pixel mask size doesn't match the image");
}
void BraggIntegrationEngineCPU::MarkNeighbours(const std::vector<Reflection> &predicted,
const std::vector<BraggPanel> &panels,
const std::vector<uint32_t> &order) {
const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
// --- Reflection mask: mark the r2 signal region of every predicted reflection, so a neighbour's
// signal is excluded from this reflection's r2..r3 background ring. MASK_FLUX where the
// neighbour puts flux on this frame (NEIGHBOUR_MASK_MIN_PARTIALITY) - only those are excluded
// - and MASK_TAIL where only the tail of a rocking curve does, which is counted for the density
// guard but read as background. The region is the INNER stencil ellipse, taken in each
// neighbour's OWN frame - an elongated neighbour whose streak is still marked as a disk would
// leak its tails into this reflection's ring. ---
// leak its tails into this reflection's ring. Only on the neighbour's own panel: beyond it the
// image holds pixels that are elsewhere on the detector. ---
refl_mask.Clear();
for (const uint32_t i : order) {
const auto &r = predicted[i];
const BraggPanel &pn = panels[i];
const uint8_t level = r.partiality >= NEIGHBOUR_MASK_MIN_PARTIALITY ? MASK_FLUX : MASK_TAIL;
const BraggStencil st = MakeBraggStencil(r.predicted_x, r.predicted_y, stencil);
const int x0 = std::max(0, static_cast<int>(std::floor(r.predicted_x - st.ex_in - 1.0f)));
const int x1 = std::min(W - 1, static_cast<int>(std::ceil(r.predicted_x + st.ex_in + 1.0f)));
const int y0 = std::max(0, static_cast<int>(std::floor(r.predicted_y - st.ey_in - 1.0f)));
const int y1 = std::min(H - 1, static_cast<int>(std::ceil(r.predicted_y + st.ey_in + 1.0f)));
const BraggStencil st = MakeBraggStencil(pn, stencil);
const int x0 = std::max(pn.x0, static_cast<int>(std::floor(r.predicted_x - st.ex_in - 1.0f)));
const int x1 = std::min(pn.x1, static_cast<int>(std::ceil(r.predicted_x + st.ex_in + 1.0f)));
const int y0 = std::max(pn.y0, static_cast<int>(std::floor(r.predicted_y - st.ey_in - 1.0f)));
const int y1 = std::min(pn.y1, static_cast<int>(std::ceil(r.predicted_y + st.ey_in + 1.0f)));
for (int y = y0; y <= y1; ++y)
for (int x = x0; x <= x1; ++x) {
const auto d = BraggStencilDistances(st, x - r.predicted_x, y - r.predicted_y);
@@ -115,10 +128,12 @@ void BraggIntegrationEngineCPU::CountNeighbourCrowding(const std::vector<Reflect
const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
counts.predicted += npredicted;
const std::vector<uint32_t> order = FrameOrder(predicted, npredicted);
MarkNeighbours(predicted, order);
const std::vector<BraggPanel> panels = Panels(predicted, npredicted);
MarkNeighbours(predicted, panels, order);
for (const uint32_t i : order) {
const auto &r = predicted[i];
const BraggStencil st = MakeBraggStencil(r.predicted_x, r.predicted_y, stencil);
const BraggPanel &pn = panels[i];
const BraggStencil st = MakeBraggStencil(pn, stencil);
const int x0 = std::max(0, static_cast<int>(std::floor(r.predicted_x - st.ex_out - 1.0f)));
const int x1 = std::min(W - 1, static_cast<int>(std::ceil(r.predicted_x + st.ex_out + 1.0f)));
const int y0 = std::max(0, static_cast<int>(std::floor(r.predicted_y - st.ey_out - 1.0f)));
@@ -131,7 +146,8 @@ void BraggIntegrationEngineCPU::CountNeighbourCrowding(const std::vector<Reflect
|| d.inner < r2_sq || d.outer >= r3_sq)
continue;
const size_t idx = static_cast<size_t>(y) * W + x;
if (idx / 32 < static_mask.size() && ((static_mask[idx / 32] >> (idx % 32)) & 1U))
if (!BraggPanelHolds(pn, x, y)
|| (idx / 32 < static_mask.size() && ((static_mask[idx / 32] >> (idx % 32)) & 1U)))
continue;
++n_ring;
if (refl_mask.Get(x, y) == 0)
@@ -147,7 +163,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
const std::vector<Reflection> &predicted,
size_t npredicted, int64_t image_number) {
if (npredicted == 0)
return Finalize(predicted, npredicted, std::vector<BraggFitResult>(npredicted), image_number);
return Finalize(predicted, npredicted, {}, std::vector<BraggFitResult>(npredicted), image_number);
counts.predicted += npredicted;
const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
@@ -155,7 +171,8 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
// Every loop below whose results do not depend on the order walks the reflections in this one.
const std::vector<uint32_t> order = FrameOrder(predicted, npredicted);
MarkNeighbours(predicted, order);
const std::vector<BraggPanel> panels = Panels(predicted, npredicted);
MarkNeighbours(predicted, panels, order);
owner.Clear();
// --- Signal-region ownership: a pixel inside two reflections' signal regions belongs to the
@@ -169,10 +186,11 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
const float claim_sq = claim * claim;
for (const uint32_t i : order) {
const auto &r = predicted[i];
const int x0 = std::max(0, static_cast<int>(std::floor(r.predicted_x - claim)));
const int x1 = std::min(W - 1, static_cast<int>(std::ceil(r.predicted_x + claim)));
const int y0 = std::max(0, static_cast<int>(std::floor(r.predicted_y - claim)));
const int y1 = std::min(H - 1, static_cast<int>(std::ceil(r.predicted_y + claim)));
const BraggPanel &pn = panels[i];
const int x0 = std::max(pn.x0, static_cast<int>(std::floor(r.predicted_x - claim)));
const int x1 = std::min(pn.x1, static_cast<int>(std::ceil(r.predicted_x + claim)));
const int y0 = std::max(pn.y0, static_cast<int>(std::floor(r.predicted_y - claim)));
const int y1 = std::min(pn.y1, static_cast<int>(std::ceil(r.predicted_y + claim)));
for (int y = y0; y <= y1; ++y)
for (int x = x0; x <= x1; ++x) {
const float dx = x - r.predicted_x, dy = y - r.predicted_y;
@@ -220,15 +238,16 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
// prefetch.)
if (k + 1 < npredicted) {
const Reflection &n = predicted[order[k + 1]];
const Box b = box_of(n, MakeBraggStencil(n.predicted_x, n.predicted_y, stencil));
const Box b = box_of(n, MakeBraggStencil(panels[order[k + 1]], stencil));
for (int y = b.y0; y <= b.y1 && b.x0 <= b.x1; ++y)
prefetch_row(img.address(static_cast<size_t>(y) * W + b.x0),
img.address(static_cast<size_t>(y) * W + b.x1));
}
const auto &r = predicted[i];
const BraggPanel &pn = panels[i];
// This reflection's stencil: the r1 signal disk, and the r2..r3 ring elongated radially by
// the analytic smear.
const BraggStencil st = MakeBraggStencil(r.predicted_x, r.predicted_y, stencil);
const BraggStencil st = MakeBraggStencil(pn, stencil);
const Box box = box_of(r, st);
const int x0 = box.x0, x1 = box.x1, y0 = box.y0, y1 = box.y1;
@@ -243,7 +262,9 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
for (int y = y0; y <= y1; ++y)
for (int x = x0; x <= x1; ++x) {
const auto d = BraggStencilDistances(st, x - r.predicted_x, y - r.predicted_y);
const int32_t px = img[y * W + x];
// Off the reflection's panel is a gap: unreadable, and not lost to the flux.
const bool on = BraggPanelHolds(pn, x, y);
const int32_t px = on ? img[y * W + x] : INT32_MIN;
if (do_clip && d.inner >= r2_sq && d.outer < r3_sq && valid(px) && refl_mask.Get(x, y) != MASK_FLUX)
clip_ring.push_back({px, d.rad});
if (BraggStencilInSignal(st, x - r.predicted_x, y - r.predicted_y, d.signal, r1_sq)) {
@@ -258,8 +279,10 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
}
++s.n_inner;
if (px == INT32_MAX
|| (px == INT32_MIN && !static_masked(static_cast<size_t>(y) * W + x)))
|| (px == INT32_MIN && on && !static_masked(static_cast<size_t>(y) * W + x)))
s.overloaded = 1;
if (px == INT32_MAX)
++s.n_saturated;
if (!valid(px)) continue;
s.I_sum += px;
s.I_sum_x += static_cast<int64_t>(x) * px;
@@ -287,10 +310,10 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
if (ClipsRing(s)) {
const int32_t thr = ClipThreshold(s);
for (const auto &[px, rad] : clip_ring)
if (px <= thr) {
if (static_cast<int64_t>(px) <= static_cast<int64_t>(thr) + RadialClipOffset(st.r0, rad)) {
clip.sum += px;
++clip.n;
if (bkg_radial) {
if (bkg_radial && radial_curve.empty()) {
// The radial curve is binned on the TRUE detector radius, so the
// offset here is the unshrunk radial projection.
const int b = std::clamp(static_cast<int>(std::lround(st.r0 + rad)), 0, a.n_rad - 1);
@@ -302,18 +325,19 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
FinishPassA(a, i, r, st, s, clip, bkg_vals);
}
return ProfilePasses(img, owner, predicted, npredicted, a, image_number);
return ProfilePasses(img, owner, predicted, npredicted, panels, a, image_number);
}
BraggIntegrationEngineCPU::PassA BraggIntegrationEngineCPU::StartPassA(size_t npredicted) const {
const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
PassA a;
a.rough.resize(npredicted);
// A pixel's radius is the reflection's radius plus the pixel's projection on the beam->reflection
// direction, so no per-pixel sqrt is needed.
a.n_rad = static_cast<int>(std::ceil(std::hypot(std::max<double>(beam_x, W - beam_x),
std::max<double>(beam_y, H - beam_y)))) + 2;
if (bkg_radial) {
// direction, so no per-pixel sqrt is needed. Out to the far corner of the plane the panels span.
const float bx0 = static_cast<float>(plane_bounds.x0), bx1 = static_cast<float>(plane_bounds.x1);
const float by0 = static_cast<float>(plane_bounds.y0), by1 = static_cast<float>(plane_bounds.y1);
a.n_rad = static_cast<int>(std::ceil(std::hypot(std::max<double>(beam_x - bx0, bx1 - beam_x),
std::max<double>(beam_y - by0, by1 - beam_y)))) + 2;
if (bkg_radial && radial_curve.empty()) {
a.rad_sum.assign(a.n_rad, 0);
a.rad_cnt.assign(a.n_rad, 0);
}
@@ -346,6 +370,7 @@ void BraggIntegrationEngineCPU::FinishPassA(PassA &a, size_t i, const Reflection
out.r_bin = std::clamp(static_cast<int>(std::lround(st.r0)), 0, a.n_rad - 1);
out.k_bin = BraggStencilKernelIndex(st, n_kern);
out.overloaded = s.overloaded != 0;
out.saturated = s.n_saturated;
if (overlap != OverlapMode::Off && s.n_own < s.n_disk) {
++counts.overlapped;
@@ -403,9 +428,13 @@ void BraggIntegrationEngineCPU::FinishPassA(PassA &a, size_t i, const Reflection
// error enters n_inner times over: var(I) = I_sum + n_inner^2 * bkg/n_bkg_used. Leaving
// the second term out understates sigma by sqrt(1 + n_inner/n_bkg) - 1.109x at the
// default r1=4/r2=6/r3=10 stencil, on every reflection of every dataset.
// bkg_var is in counted units, as the profile fit reads it; every variance formed from it is
// taken at the gain, so a CCD's sigma - and the strong flag and the guard read from it - is in
// photons rather than in ADU counted as photons.
const double g = pixel_gain;
out.bkg_var = out.bkg / n_bkg_used;
out.var_bkg = static_cast<double>(s.n_inner_valid) * out.bkg
+ static_cast<double>(s.n_inner_valid) * s.n_inner_valid * out.bkg_var;
out.var_bkg = g * (static_cast<double>(s.n_inner_valid) * out.bkg
+ static_cast<double>(s.n_inner_valid) * s.n_inner_valid * out.bkg_var);
out.I_sum = s.I_sum;
out.n_inner = static_cast<int>(s.n_inner_valid);
out.n_disk = s.n_disk;
@@ -413,7 +442,7 @@ void BraggIntegrationEngineCPU::FinishPassA(PassA &a, size_t i, const Reflection
const double var_bkg_term = static_cast<double>(s.n_inner_valid) * s.n_inner_valid * out.bkg_var;
out.sigma = 1.0;
if (s.I_sum > 0) {
out.sigma = std::max(out.sigma, std::sqrt(static_cast<double>(s.I_sum) + var_bkg_term));
out.sigma = std::max(out.sigma, std::sqrt(g * (static_cast<double>(s.I_sum) + var_bkg_term)));
// The centroid of the SIGNAL, not of the disk. Weighting by the raw counts weights by
// signal plus background, and the background is flat over a disk centred on the
// PREDICTION - so its own centroid is the prediction exactly, and it pulls the answer
@@ -465,7 +494,7 @@ bool BraggIntegrationEngineCPU::FitKept(const BraggFitOut &o) const {
template <class Sampler, class OwnerMap>
BraggFitOut BraggIntegrationEngineCPU::FitProfile(const Sampler &img, const OwnerMap &owner_map,
const BraggFitSetup &s, size_t i,
const BraggFitSetup &s, const BraggPanel &pn, size_t i,
const std::vector<float> &shell_profiles,
std::vector<float> &cells,
std::vector<FitPixel> &fit_pixels) const {
@@ -476,9 +505,15 @@ BraggFitOut BraggIntegrationEngineCPU::FitProfile(const Sampler &img, const Owne
BraggFitOut o{};
const float *P;
if (s.shell >= 0)
P = shell_profiles.data() + static_cast<size_t>(s.shell) * GG;
else {
if (s.shell >= 0) {
// The shell's learned profile is in the plane's axes; laid out here in the panel's.
const float *learned = shell_profiles.data() + static_cast<size_t>(s.shell) * GG;
cells.resize(GG);
for (int dy = -R; dy <= R; ++dy)
for (int dx = -R; dx <= R; ++dx)
cells[(dy + R) * G + (dx + R)] = learned[BraggProfileCell(pn, dx, dy, R)];
P = cells.data();
} else {
// The grid's image rows, asked for now (as in pass A): building the profile below takes long
// enough for them to arrive before the fit reads them.
const int gx0 = std::max(0, s.cx - Rf), gx1 = std::min(W - 1, s.cx + Rf);
@@ -506,7 +541,7 @@ BraggFitOut BraggIntegrationEngineCPU::FitProfile(const Sampler &img, const Owne
// pixels, what Reject cuts on. m_read / m_all is the fraction of the r1 disk the
// summation seed the runaway guard compares against actually summed; with nothing missing
// and nothing excluded it is 1 and the guard is untouched. ---
const BraggStencil st = MakeBraggStencil(s.px, s.py, stencil);
const BraggStencil st = MakeBraggStencil(pn, stencil);
fit_pixels.clear();
for (int dy = -Rf; dy <= Rf; ++dy)
for (int dx = -Rf; dx <= Rf; ++dx) {
@@ -523,7 +558,7 @@ BraggFitOut BraggIntegrationEngineCPU::FitProfile(const Sampler &img, const Owne
o.p_grid += Pp;
o.p_peak = std::max(o.p_peak, Pp);
if (in_disk) o.m_all += Pp;
const int32_t v = img[y * W + x];
const int32_t v = BraggPanelHolds(pn, x, y) ? img[y * W + x] : INT32_MIN;
if (!valid(v)) {
o.p_lost_peak = std::max(o.p_lost_peak, Pp);
continue;
@@ -557,7 +592,8 @@ BraggFitOut BraggIntegrationEngineCPU::FitProfile(const Sampler &img, const Owne
template <class Sampler, class OwnerMap>
std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &img, const OwnerMap &owner_map,
const std::vector<Reflection> &predicted,
size_t npredicted, PassA &a,
size_t npredicted,
const std::vector<BraggPanel> &panels, PassA &a,
int64_t image_number) {
const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
std::vector<BraggFitResult> results(npredicted);
@@ -568,7 +604,6 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
return !use_owner
|| BraggOwnedBy(owner_map.Get(x, y), static_cast<int>(i));
};
auto grid_idx = [this](int dx, int dy) { return (dy + R) * G + (dx + R); };
// --- Radial background curvature correction. The annulus mean is blind to the curvature of the
// radial background (a linear background cancels between the concentric disk and annulus), so
@@ -584,7 +619,11 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
double corr = 0.0;
for (int k = 0; k < k_len; ++k) {
const int b = std::clamp(rh.r_bin + k - k_off, 0, a.n_rad - 1);
const double v = a.rad_cnt[b] > 0 ? static_cast<double>(a.rad_sum[b]) / a.rad_cnt[b] : rh.bkg;
double v;
if (!radial_curve.empty())
v = radial_curve[b];
else
v = a.rad_cnt[b] > 0 ? static_cast<double>(a.rad_sum[b]) / a.rad_cnt[b] : rh.bkg;
corr += static_cast<double>(kern[k]) * v;
}
rh.bkg -= corr; // annulus mean -> mean over the signal disk
@@ -607,9 +646,9 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
if (overlap == OverlapMode::Reject && rh.n_own < overlap_min_peak * rh.n_disk) continue;
results[i] = {static_cast<float>(rh.I), static_cast<float>(rh.sigma), static_cast<float>(rh.bkg),
static_cast<float>(rh.obs_x), static_cast<float>(rh.obs_y),
static_cast<float>(rh.var_bkg), true, rh.has_obs, rh.overloaded};
static_cast<float>(rh.var_bkg), true, rh.has_obs, rh.overloaded, rh.saturated};
}
return Finalize(predicted, npredicted, results, image_number);
return Finalize(predicted, npredicted, panels, results, image_number);
}
auto shell_of = [&](float d) {
@@ -622,7 +661,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
// --- Learn the profile per shell (+ global) from the strong spots. ---
// Two things are learned. The empirical profile is the average grid in the DETECTOR frame, which is
// where it is applied. The width is a pair of second moments taken in each spot's OWN radial /
// where it is applied - in the plane's axes, which a turned panel's pixels are mapped onto. The width is a pair of second moments taken in each spot's OWN radial /
// tangential frame: a grid stacked in the detector frame is azimuthally averaged, so its <r^2> is
// sigma_r^2 + sigma_t^2 with no way back, and a radially smeared spot reads as a wide TANGENTIAL
// one. Rotating each contribution into the spot's frame keeps the two apart.
@@ -637,19 +676,22 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
for (size_t i = 0; i < npredicted; ++i) {
const auto &rh = rough[i];
if (!rh.ok || !rh.strong || rh.I <= 0.0) continue;
const double rx = predicted[i].predicted_x - beam_x, ry = predicted[i].predicted_y - beam_y;
const BraggPanel &pn = panels[i];
const double rx = pn.plane_x - beam_x, ry = pn.plane_y - beam_y;
const double Rpx = std::hypot(rx, ry);
const double ux = Rpx > 1e-6 ? rx / Rpx : 1.0, uy = Rpx > 1e-6 ? ry / Rpx : 0.0;
for (int dy = -R; dy <= R; ++dy)
const auto [ux, uy] = to_panel_axes(pn, Rpx > 1e-6 ? rx / Rpx : 1.0, Rpx > 1e-6 ? ry / Rpx : 0.0);
for (int dy = -R; dy <= R; ++dy)
for (int dx = -R; dx <= R; ++dx) {
const int x = rh.cx + dx, y = rh.cy + dy;
if (x < 0 || y < 0 || x >= W || y >= H) continue;
if (x < 0 || y < 0 || x >= W || y >= H || !BraggPanelHolds(pn, x, y)) continue;
const int32_t px = img[y * W + x];
if (!valid(px)) continue;
if (exclude && !clean(x, y, i)) continue;
const double v = (static_cast<double>(px) - rh.bkg) / rh.I;
shell_grid[rh.shell][grid_idx(dx, dy)] += v;
global_grid[grid_idx(dx, dy)] += v;
// The grid in the plane's axes (BraggProfileCell); the moments below are axis-free.
const int cell = BraggProfileCell(pn, dx, dy, R);
shell_grid[rh.shell][cell] += v;
global_grid[cell] += v;
if (dx * dx + dy * dy >= r1_sq) continue;
const double rad = dx * ux + dy * uy, tn = -dx * uy + dy * ux;
shell_mom[rh.shell].rad += v * rad * rad;
@@ -720,17 +762,19 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
fs = {};
if (!rh.ok) continue;
// An overloaded reflection is not measured by any fit - its brightest pixels are what is
// missing. It is passed on with its box sum and the flag, for the merge to drop its rocking
// event (Reflection::overloaded); the missing-peak cut below would otherwise drop just this
// frame and leave the event to be extrapolated from its flanks.
// missing. It is passed on with its box sum and the flag, for a rotation merge to drop its
// rocking event (Reflection::overloaded) and for a still to be kept as a lower bound
// (Finalize); the missing-peak cut below would otherwise drop just this frame and leave the
// event to be extrapolated from its flanks.
if (rh.overloaded) {
results[i] = {static_cast<float>(rh.I), static_cast<float>(rh.sigma), static_cast<float>(rh.bkg),
static_cast<float>(rh.obs_x), static_cast<float>(rh.obs_y),
static_cast<float>(rh.var_bkg), true, false, true};
static_cast<float>(rh.var_bkg), true, false, true, rh.saturated};
continue;
}
const int sh = rh.shell < 0 ? 0 : rh.shell;
const BraggStencil st = MakeBraggStencil(predicted[i].predicted_x, predicted[i].predicted_y, stencil);
const BraggPanel &pn = panels[i];
const BraggStencil st = MakeBraggStencil(pn, stencil);
fs.fit = 1;
fs.px = predicted[i].predicted_x;
fs.py = predicted[i].predicted_y;
@@ -740,7 +784,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
fs.shell = sh;
fs.footprint = st.fp_s2t > 0.0f;
if (!empirical) {
const double rx = predicted[i].predicted_x - beam_x, ry = predicted[i].predicted_y - beam_y;
const double rx = pn.plane_x - beam_x, ry = pn.plane_y - beam_y;
const double Rpx = std::hypot(rx, ry);
const double tan2t = Rpx / F_px;
double s2t = shell_sigma2[sh].tan;
@@ -754,7 +798,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
const double radial_extra = std::max(shell_sigma2[sh].rad - s2t,
sbw * sbw + c_radial * tan2t * tan2t);
if (Rpx > 1e-6 && radial_extra > 0.25) {
ux = rx / Rpx; uy = ry / Rpx;
std::tie(ux, uy) = to_panel_axes(pn, rx / Rpx, ry / Rpx);
s2r = s2t + radial_extra;
elong = true;
}
@@ -792,7 +836,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
std::vector<FitPixel> fit_pixels;
for (const uint32_t i : FrameOrder(predicted, npredicted))
if (setups[i].fit)
fits[i] = FitProfile(img, owner_map, setups[i], i, shell_profiles, cells_buf, fit_pixels);
fits[i] = FitProfile(img, owner_map, setups[i], panels[i], i, shell_profiles, cells_buf, fit_pixels);
}
for (size_t i = 0; i < npredicted; ++i) {
@@ -810,15 +854,17 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
// 1/den is the profile-fit variance with the background taken as exact. The fit is
// I = sum(P*(px-bkg)/v) / sum(P^2/v), so dI/dbkg = -wsum/den and the background estimate's
// own error adds (wsum/den)^2 * var(bkg) - the same term the box sum was missing.
double sigma = std::sqrt(1.0 / den + (wsum / den) * (wsum / den) * rh.bkg_var);
// Both variances at the gain (FinishPassA), so the guard below compares like with like.
const double g = pixel_gain;
double sigma = std::sqrt(g * (1.0 / den + (wsum / den) * (wsum / den) * rh.bkg_var));
// var_bkg is the NON-SIGNAL part of that variance, and 1/den is the fit's variance taken at the
// fitted intensity, so the signal part to remove is I itself - not max(0, I). Clamping it leaves
// a down-fluctuated reflection's own (deflated) variance standing as its background variance,
// which is 2-3x too small; the merge then weights exactly the down-fluctuated observations up.
// The whole point of a separate var_bkg is that it does not move with the observation's own
// fluctuation, and 1/den - I is what does not (1/den grows with I one for one).
double var_bkg = std::max(0.0, 1.0 / den - I
+ (wsum / den) * (wsum / den) * rh.bkg_var);
double var_bkg = g * std::max(0.0, 1.0 / den - I
+ (wsum / den) * (wsum / den) * rh.bkg_var);
// The seed is a sum over the disk the box sum actually read, so when pixels are missing from
// both - excluded to a neighbour, or unreadable - the fit's full-profile intensity has to be
// scaled down to that same disk before the two are comparable. Nothing dropped gives 1.
@@ -834,21 +880,22 @@ std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(const Sampler &
results[i] = {static_cast<float>(I), static_cast<float>(sigma),
static_cast<float>(rh.bkg),
static_cast<float>(rh.obs_x), static_cast<float>(rh.obs_y),
static_cast<float>(var_bkg), true, rh.has_obs, rh.overloaded};
static_cast<float>(var_bkg), true, rh.has_obs, rh.overloaded, rh.saturated};
}
return Finalize(predicted, npredicted, results, image_number);
return Finalize(predicted, npredicted, panels, results, image_number);
}
template std::vector<Reflection> BraggIntegrationEngineCPU::ProfilePasses(
const BraggIntegrationEngineCPU::BufferSampler &, const BraggIntegrationEngineCPU::FlatOwnerMap &,
const std::vector<Reflection> &, size_t, BraggIntegrationEngineCPU::PassA &, int64_t);
const std::vector<Reflection> &, size_t, const std::vector<BraggPanel> &,
BraggIntegrationEngineCPU::PassA &, int64_t);
std::vector<Reflection> BraggIntegrationEngineCPU::Run(const ImagePreprocessorBuffer &image,
const std::vector<Reflection> &predicted,
size_t npredicted, int64_t image_number) {
if (image.size() != npixel)
return Finalize(predicted, npredicted, std::vector<BraggFitResult>(npredicted), image_number);
return Finalize(predicted, npredicted, {}, std::vector<BraggFitResult>(npredicted), image_number);
return RunImpl(BufferSampler{image.data()}, predicted, npredicted, image_number);
}
@@ -856,7 +903,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::Run(const CompressedImage &im
const std::vector<Reflection> &predicted,
size_t npredicted, int64_t image_number) {
if (image.GetWidth() * image.GetHeight() != npixel)
return Finalize(predicted, npredicted, std::vector<BraggFitResult>(npredicted), image_number);
return Finalize(predicted, npredicted, {}, std::vector<BraggFitResult>(npredicted), image_number);
std::vector<uint8_t> scratch;
const auto *ptr = image.GetUncompressedPtr(scratch);