v1.0.0-rc.179 #90

Merged

leonarski_f merged 17 commits from rc179-clean into main 2026-10-24 14:52:30 +02:00

Conversation 0 Commits 17 Files Changed 394

+57778 -2178

leonarski_f commented 2026-10-24 14:52:20 +02:00

Owner

Copy link

Copy source

• Rugnux: Stills processing needs only the unit cell and space group beyond what the file records - detector distance, beam centre, tilt, panel placement, saturation level, gain and polarisation plane are found from the images - and is up to twice as fast.
• Rugnux: Stills can be indexed without a GPU, with a CPU implementation of the fast-feedback indexer (macOS, computers without a GPU).
• Rugnux: Rotation and stills data are integrated in the detector's own units with one gain model, improving data from CCD detectors.
• Web frontend: The Rugnux live page shows the placed model with its electron density and anomalous difference map in a Mol* viewer.

* Rugnux: Stills processing needs only the unit cell and space group beyond what the file records - detector distance, beam centre, tilt, panel placement, saturation level, gain and polarisation plane are found from the images - and is up to twice as fast. * Rugnux: Stills can be indexed without a GPU, with a CPU implementation of the fast-feedback indexer (macOS, computers without a GPU). * Rugnux: Rotation and stills data are integrated in the detector's own units with one gain model, improving data from CCD detectors. * Web frontend: The Rugnux live page shows the placed model with its electron density and anomalous difference map in a Mol* viewer.

leonarski_f added 17 commits 2026-10-24 14:52:20 +02:00

rc179: everything on rc178 after its merge into main (#88), except the CHANGELOG ... 0a515bb086

The rc178 branch continued after #88 was squash-merged; this is its content from b9ada7914 (the tree
#88 squashed) to 094eee31a, as one change on main. The individual commits stay on the rc178 branch:

- still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes
- AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets
- DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked
- tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout
- Dense per-pixel maps through the panel table (panel layout stage 1)
- Panel id on spots and the panel table in the start message
- Viewer and preview show a panel detector as it lies on the detector plane
- Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster
- Panel layout stage 2: prediction and Bragg integration through the panel table
- XTC reader: frames handed up as stored, streams indexed in parallel
- XFEL reader: integer frames stay integer, one gain, placement by panel
- EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery"
- still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed
- float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count
- CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them
- Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table
- Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane
- still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good
- still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file
- still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition
- stills fixed cost: geometry refinement and pre-scan do the same work in less wall time
- stills geometry sample and trial spot finding run as many workers as the image pass
- Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame
- StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's
- MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in)
- Stills given a cell: the written cell is the mean over the frames that are merged
- Stills: the radial curvature correction decided per run on the same empty apertures
- StillsPartialityRefine: measure the run width only on shells with signal
- Stills pre-scan: report the paired z of the radial-correction comparison
- module placement: the five concurrent fits each keep every thread
- still battery: score R-free against the deposited data through the same dimple protocol
- still battery: the R-free target is the lower of the deposition's dimple and published values
- Stills: background clip follows the radial curve; signal disk from the lattice spacing
- Stills keep an overloaded observation as a lower bound instead of dropping it
- Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip
- battery: R-free against the deposition without refinement only, one protocol for rotation and stills
- still battery: EuXFEL sets run on a 2,000-image subset for now
- stills geometry refinement: fit the detector distance and the cell together
- stills geometry refinement: a distance given with --detector-distance is held in the joint fit
- Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count
- Metal integration test: the CPU comparison runs on a rotation experiment
- Stills: kept overloads go back to the drop where they are not concentrated at low resolution
- stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated
- battery results: XFEL still battery 20261022-1358 on rc178 0cf62452c - 12 sets, cold, R_free unrefined
- battery results: still summary gains the resolution of our merge and of the deposition
- Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value
- Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed
- XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock
- stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes
- rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid
- stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree
- CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get
- broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4)
- frontend: Mol* viewer of the placed model and its density on the Rugnux live page
- writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does
- rugnux live maps: one unit cell centred on the model; the viewer opens on all of it
- Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer
- Mol* viewer: open with the maps drawn within 10 A of the centre of view
- Mol* viewer: aaredb's structure-viewer setup, on a black stage
- rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons
- Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2)
- ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI)
- stills tilt decision: the modules stay where the run placed them
- still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it
- battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC
- Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons
- BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it
- ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization
- BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain
- ShadowFinder: measure the modulation robustly under a wide one-sided shadow
- ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation
- ShadowFinder: back to the predicted polarization factor, in the plane the background shows
- MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again
- battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables
- XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel
- stills geometry trial: the hypotheses a search asks are indexed side by side
- stills tilt decision: the fits start beside the pre-scan and the geometry trial
- XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache
- XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step
- GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block
- GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart
- docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts
- battery: doc_check.py cross-checks the manifests against the dataset pages
- stills module placement: the placement fit gets half the threads, each held-out fit an eighth
- still battery: the EuXFEL sets run on all their frames, as the published still runs did
- battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table
- macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu

FFBIDXIndexerCPU: the fast-feedback indexer on the CPU, used where there is no GPU ... ea383f9e83

A port of the fast-feedback indexer's CUDA kernels to plain C++/Eigen: the same half-sphere and
rotation samples, the same scores and the same 32 output cells for Refine(). The loops are
reorganised so the inner one runs over samples and vectorises (GCC/Clang, no intrinsics), and the
cell search separates the rotation about the candidate vector out of the per-spot terms. The
vector refinement reproduces the library's warp sums - partial sums its shuffle reduction leaves in
the other lanes carry into the next round - and its float arithmetic (FMA contraction, the
Fibonacci points in float), because the candidates searched depend on it: on the library's 27 test
spot lists and 4 synthetic stills the CPU and GPU best cells agree to ~1e-6 relative, the top-32
sets in 30 of 31 cases completely.

FFBIDX now resolves to the GPU library where a GPU is visible and to the CPU port otherwise (no
GPU, --no-gpu, or a build without CUDA); Auto takes FFBIDX for stills whenever a cell is known,
also without a GPU, instead of FFTW. With a GPU nothing changes (a stills set and a rotation set
reproduce the baseline exactly). Without a GPU the stills indexer pool gets one indexer per worker
instead of eight, since the CPU indexer holds no card memory.

Single core, synthetic still, loaded 5950X: ~8 ms per call at 30 spots, ~15 ms at 80, ~35 ms at
250; 16 threads ~1300/700/330 calls/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 9a8eb8669d1be6c3ec16db6786280affd6830ee5)

docs: the fast-feedback indexer and its CPU implementation ... 0c12178371

CPU_DATA_ANALYSIS_INDEXING.md gains a section on the fast-feedback indexer (scores, the three
stages, the GPU library and the CPU port); the pages that said ffbidx needs a GPU now say it runs on
the CPU without one. ACKNOWLEDGEMENT.md credits the port to the library's CUDA kernels, and
THIRD_PARTY_NOTICES.md lists FFBIDXIndexerCPU.cpp as adapted from the library (BSD-3-Clause, the
licence text already in licenses/).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit eeb87e6020e1c65c1f6895e1cd4b102700166a99)

Polarization plane decided once, in the pre-scan, and followed by the integration too ... c74678b06f

The shadow finder already compares the background's cos 2phi harmonic with the profile a polarized
source gives it, and applies the stated degree in the plane the background shows where it shows
one clearly (correlation > 0.5, at least 0.3 of a fully polarized source's modulation). The
integration's Lorentz-polarization factor still assumed the plane along lab x, so on a run whose
background is suppressed along the image's y axis the two disagreed, and only a negative
--polarization made them agree.

ShadowFinder now reports what it applied (GetAppliedPolarization), the pre-scan writes it into the
experiment (and keeps it in the beam-stop memo with the mask), and everything built after the
pre-scan - the Bragg integration, the azimuthal mapping - reads the same plane. The degree is never
fitted; only the plane can turn, and only where the shadow finder already turned it. Where the plane
is not turned the experiment is not touched, so every other run is bit-identical: six rotation sets
give the same p.hkl md5 as rc178, and the stills sets that do not turn give the same R_free, R_split,
indexed fraction and d_min. On the one stills run whose plane turns, R_free and the resolution cut
are the ones a stated -0.998 gave before, now from a bare invocation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit f50bfe15f7dccf588bd8bdd6114a32e3ac9b36ed)

still battery: no --polarization on the two forms of the turned-plane LCLS run ... 15793a6bc2

The plane is now taken from the background for the integration as well as the shadow mask, and the
degree (0.99 by default against the 0.998 passed) moves neither the mask nor R_free beyond 0.0001.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit ebf0f24351cf45c5e3d7aefeae1e5df83ff0fd55)

One gain mechanism: frames analysed in their own units, the gain applied where a photon count is assumed ... 711f902aa7

The gain of an input in detector units (SMV, marCCD, XTC, the XFEL formats) is measured once, before
anything reads the frames (RugnuxMeasureGain: photon-peak spacing and background variance per unit
mean), or given with --gain, and handed to the integration and the spot finding by one function
(ApplyRugnuxPixelGain). The rotation pre-scan no longer measures it, and the stills reader no longer
converts frames of a detector whose photons are not resolved: SACLA MPCCD, Rayonix and CCD frames are
integrated in ADU at the gain, rotation and stills alike.

Where a photon count was assumed, it is now taken at the gain (bit-identical at gain 1):
- integration: the background clip's one-photon floor, the sigma floor (one photon), the profile fit's
  background floor (0.01 photon), and a still's overload kept as a lower bound, which was left in ADU
  while every other reflection left the integrator in photons - the cause of the earlier loss of the
  engine route on clipped CCD stills (merged low resolution inflated by the gain, the rocking-width
  fit's bandwidth collapsing); CPU and CUDA, Metal through the shared CPU code;
- the adaptive spot threshold (Poisson arm and its one-photon floor), CPU and GPU;
- the pre-scan spot-width measurement (strong-spot SNR, the ring clip) behind the stills bandwidth and
  the rotation signal radius;
- the hot/dead pixel tests, CPU and GPU; the empty-aperture ring clip.
BackgroundPixelGain is gone: one pixel_gain.

A detector whose background resolves single photons (photon peaks: JUNGFRAU in keV, AGIPD) is a photon
counter and its frames are still counted in whole photons as read: analysed in keV at the gain instead,
a photon-counting still set lost 0.008 R-free (fewer spots, a noisier beam-centre trial) - the rounding
is the detector's own discrimination. --gain sets the value, the background still says which kind.

Validation, against rc178: photon-counting and g<=1 rotation p.hkl md5-identical (lyso, thau, cytc,
myob, insu, a small-molecule set, 5reo, 6cee, 6i3j, 2xfw, 6s1u, 8v4j); CCD g>1 at or within noise of rc178 (5uth
R_meas 0.113 -> 0.108). Stills R-free (0-cycle REFMAC): MPCCD sets 0.184 -> 0.185 and 0.199 -> 0.196,
Rayonix identical, photon-resolving sets back on their rc178 numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit da8dbfc8147354b63e375f2d9521dffb80938b6d)

CI: the Linux portable job runs the [portable] tests too, as the Windows and macOS jobs do ... 385e2a2838

So the CPU fast-feedback indexer's portable case (and the other cross-platform checks) runs on Linux,
macOS and Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu

v1.0.0-rc.179: VERSION and update_version.sh ... a9215fe652

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu

ShadowFinder: the deficit and the reflection guard in photons on a detector read in ADU ... e866262ed6

The shadow is a ratio and does not depend on the units, but two tests in the shadow finder are made
in counts: the Poisson significance of a pixel's deficit against its ring, and the maximum a pixel
must reach (25) to have recorded a reflection that is never masked. On a frame analysed in ADU at a
gain g > 1 the first was overstated by sqrt(g) and the second fired on a few photons - on a stop
that lets a little through, the whole stop was given back as "reflections". Both are now taken in
photons: the counts behind the deficit divided by the gain, the reflection count multiplied by it.
CPU and CUDA alike (ShadowMaskSetup carries the gain). g = 1 (every photon-counting input) is
bit-identical by construction.

Tests: ShadowFinder_ADetectorReadInADUGivesThePhotonMask (the photon scene times 9 at gain 9 gives
the photon mask; at gain 1 the stop is given back), and a device-vs-host section on the same scene.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 148177e3e54c9ac3a129c6ff3cd4fa24e6d34b07)

Integration: the radial background correction carries the corrected background into var_bkg (D14) ... 6940718cf5

The correction moved bkg and I but left the non-signal variance carried to the merge at the ring's
level. var_bkg = g (n bkg + n^2 bkg_var): its first term is the Poisson share of the background under
the signal disk, which is the corrected background, so it now moves with it (CPU, CUDA; Metal runs
the CPU correction). The second term - the noise of the ring estimate - and sigma, which is taken
from the counts in the disk, do not depend on the curve and are unchanged. Not touched: the strong
flag and the centroid are still taken at the uncorrected pedestal.

Only reflections the correction is applied to change; tests: the corrected empty aperture's var_bkg
moves by n_inner times the background change (CPU), and GPU var_bkg matches the CPU with a curve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 6d85d594a4199e86d67c779b5301e88e48a3ad49)

Metal: a frame with a handed-over radial curve in use is integrated by the CPU engine ... 3467c21919

With a curve from the whole frame (RadialBackgroundFromRings) the ring's high-side clip follows it
(RadialClipOffset) on the CPU and CUDA; the Metal clip kernel did not, so a Mac run of such a frame
differed from both. It now goes to the CPU engine, which gives that result exactly - the precedent is
the trimmed mean. Frames without a handed-over curve are unchanged.

Not compiled here (no macOS); BraggIntegrationEngineMetal_HandedOverRadialCurveStaysOnCPU checks it on
the Mac CI job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 36be3b8f13dbde2616fe7add3acc3d073ef082af)

Metal integration engine: the clip follows a handed-over radial curve and saturated pixels are counted, as on the CPU ... d0b4de2bd4

The two pixel-level differences left between the Metal kernels and the CPU engine are closed:
- clip_sums follows the radial curve handed over by RadialBackgroundFromRings (RadialClipOffset, written
  out line for line as radial_clip_offset; the curve in buffer 11, BraggPassAFrame::n_curve), so the
  CPU fallback for such frames (3467c2191) is removed;
- box_sums counts the saturated signal-disk pixels (BraggBoxSums::n_saturated), so a still's overload
  kept as a lower bound counts them at the saturation limit, as on the CPU and CUDA.
Everything else - the gain (clip floor, sigma floor, fit floor, Finalize), the radial correction and D14 -
is the CPU engine's own host code on the Metal path already.

The structs the host hands the kernels carry static_asserts on their size in the shared headers, so the
host and the Metal compiler check the same numbers.

The [Metal] bit-identity test now covers the handed-over curve under every integrator, stills keeping
overloads (overloads_kept compared and required non-zero), gains 4 and 1.7, and a footprint still with a
curve; `clipped` is compared too. A [portable] CPU-only case checks everywhere that these cases reach
their modes. Not compiled here (no macOS): the Mac CI job is the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 285d40c99b6356a52d224e610bc7b28d69973a06)

CHANGELOG: rc.179 ... 134fc56f70

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu

Radial background: one mechanism for stills and rotation, decided per run on empty apertures (D5) ... f199f23c54

Wherever the correction is set for the whole run its curve is the frame's peak-excluded ring
background (the adaptive spot finder's, whole frame) and the ring's clip follows it - as stills have
done since integ-pal - now rotation too. One empty-aperture measurement (BackgroundAperture) decides
it for both workflows:

- The apertures' corrected difference is now taken as the integrator applies the correction: the
  clip follows the curve (each ring pixel's threshold raised by the curve's rise from the centre, in
  whole counts) as well as the kernel correction. Before, the tally clipped flat, so it measured a
  correction the engine never applies.
- Rotation measures the stencils the spot width can give it (r1 = 4, 5, 6 with the ring-area rule,
  RotationStencil) and decides on the one it gets, and separately on the pre-adaptive stencil the
  geometry pre-pass and the starvation guard return to.
- Stills: on wherever the corrected ring predicts the empty disk better (unchanged rule).
  Rotation: the frame's curve on every image only where it takes >= 10% of the apertures' mean
  square error away (ROTATION_RADIAL_MIN_GAIN); otherwise the existing per-image ice gate with the
  reflections' curve is left exactly as it was. Why a margin on rotation only: a still has no other
  correction, a sweep has one. Measured over 18 rotation sets (forced on every image): photon-counting
  sweeps gain 0.1-5% on the apertures (z 3-4) and lose ISa/anomalous signal when corrected
  (insulin, lysozyme, cytochrome c; 9yzk R-free +0.009 at 1.2%), as the earlier stills-rule trial
  found; the home-source CCD and smooth-ice runs gain 13-31%.

Result (CPU A/B vs 87684012f, 18 rotation sets): 12 .hkl byte-identical (every photon-counting
control without ice), 5uth R-free 0.210 -> 0.201 / ISa 10.4 -> 10.8, the ice set neutral; stills:
lcls242_s turns the correction on (the clip-following tally), R-free 0.2399 -> 0.2406.
No integration kernel changed; the tally is host code. Tests: BackgroundAperture_CorrectedClipFollowsTheCurve,
RadialHelps margins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit dd3f3f2da39f5338e7dda1874bbdd535229bce98)

Radial background: with the frame's curve, the correction is taken over the pixels read, at their 2 theta ... fac2a17428

With the frame's curve in use (D5: stills, and rotation runs whose empty apertures ask for it) the
correction was a precomputed annulus-minus-disk kernel over radial offset, applied to the curve at the
reflection's distance from the PONI. The empty apertures that decide it compute the same quantity over
the pixels they actually read, at each pixel's own 2 theta. The two differ exactly next to the beam stop:
the ring's beam-side pixels are masked by the shadow, but the kernel still averaged the curve over them
(the curve there holds the shadow's edge, a cliff of several hundred counts), and on a tilted detector
the PONI is a few pixels from where the beam lands, where the curve is steepest. On a binned-Rayonix
still run with the beam stop off the beam this sent low-resolution reflections next to the stop to large
negative intensities (median -854 -> -4171 at 42-50 px), destabilised the per-image scales and collapsed
the lowest shell's CC1/2 (0.906 -> 0.448, "not usable" warning).

Now, where the frame's curve is in use:
- the correction is the curve's mean over the ring pixels the background was taken from (after the
  clip) minus its mean over the readable disk pixels (BraggCurveCorrection), summed in pass A while the
  box is read anyway - in fixed point (1/256 count), so CPU and GPU sums are exact and order-independent;
- the curve is read at the 2 theta of the reflection's position (CurveRadius, from the geometry), the
  radius it is tabulated against, and the clip offset follows it there;
- BackgroundAperture takes the curve's mean over the ring pixels the clip kept, so the decision
  measures exactly what is applied.
The kernel stays for the curve accumulated from the reflections' rings (rotation's per-image ice gate),
unchanged. Metal unchanged (it hands such frames to the CPU engine).

Validation. Rotation, CPU A/B vs 87684012f over 18 sets: .hkl byte-identical on the 12 sets where D5 is
off (and on the 3 ice-gated D14 sets identical to D5); the three D5-on sets improve: a home-source CCD
set R-free 0.210 -> 0.194 (D5 0.201), ISa 10.4 -> 10.8, anomalous 3.70 -> 3.94; the ice set
0.315 -> 0.312, ISa 22.3 -> 22.7 (D5 lost ISa there); the third neutral. Stills (0-cycle REFMAC, 9 sets):
R-free within +0.0021/-0.0039 of 87684012f on every set, verdicts unchanged except one draw -> win,
no "not usable" warning on any set (two base runs had one). The lowest shell's CC1/2 on the run above
stays low (0.44): it is set by three to five reflections at 30-41 A in the penumbra of the stop, which
integrate as garbage with every variant (the uncorrected run included); with the merge's low-resolution
limit at 30 A it reads 0.79.

Tests: BraggIntegrationEngineCPU_RadialCurveOverThePixelsRead (shadow edge on a tilted detector; fails
with the kernel or with the PONI radius), the GPU parity test now with a masked ring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit d225b546c613ab2266c63ff2add02624b04d5e25)

battery results: rc179 fac2a1 - full still battery (cold, quiet) and full rotation battery with REFMAC, replacing the rc178 ones ... fd510ac56c

Stills: 7 wins, 4 draws, 1 loss of 12 with no distance, gain or polarisation given; the twelve runs take
1,168 s. Rotation: 255 sets, no verdict change against rc178 6f6f69; REFMAC R-free identical on 178 of 201
sets, the rest within +0.003 or better (largest gain -0.034); total time +5%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu

Metal integration engine: with the frame's curve, the correction over the pixels read, at their 2 theta, as on the CPU ... 774357818f

Mirrors fac2a1742 in the Metal engine, which since d0b4de2bd runs frames with a handed-over curve
itself and so differed from the CPU engine there:
- box_sums sums the curve over the readable disk pixels and over the ring pixels behind the
  background, clip_sums over the pixels the clip keeps, in whole 1/BRAGG_CURVE_FIXED counts
  (curve_fixed = CurveFixed); integer sums, so independent of visit order;
- the curve is read at each reflection's CurveRadius (computed on the host, buffer curve_r), and the
  clip offset follows it there instead of at the distance from the PONI.
The sums travel in the shared pass-A structs (BraggBoxSums::curve_disk/curve_ring, 96 -> 112 bytes;
BraggClipSums::curve, 16 -> 24 bytes), which the CPU engine now fills instead of three locals, and
the correction (BraggCurveCorrection, double) moved from RunImpl into FinishPassA, which the Metal
engine already calls with the kernels' sums - so the GPU does no double arithmetic and the CPU's
result is unchanged.

Tests: [Metal] ShadowEdgeBitIdenticalToCPU (the beam-stop shadow on a tilted detector of
BraggIntegrationEngineCPU_RadialCurveOverThePixelsRead, 48 reflections, box sum and Gaussian fit);
[portable] ShadowEdgeReachesItsModes (CPU: beam off the PONI, rings next to the shadow lose pixels,
the correction cuts the background error from ~3 counts to below 0.6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 7af57c66da67fcb8d56f6f90278e57fa11075ff9)

leonarski_f merged commit b00c668e3a into main 2026-10-24 14:52:30 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) mx-gitea perl_d (perl_d) takaba_k (takaba_k) wang_m (wang_m)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#90