Jungfraujoch/docs/CPU_DATA_ANALYSIS_IMAGE.md

Data analysis: from images to spots (§0–§3)

Part of the CPU/GPU data-analysis reference; the section numbers are continuous across its four parts.

:local:
:depth: 2

0. Getting the image onto the GPU: device-side bitshuffle+LZ4 decoding

Images arrive bitshuffle+LZ4 compressed (HDF5 filter 32008), and everything from §1 onwards runs on the GPU when one is present. Instead of decompressing on the host and uploading the image, the compressed chunk is uploaded — a few MB rather than tens of MB — and decoded on the device. The approach follows Jon Wright (ESRF); the kernels are Jungfraujoch's own.

One kernel does the work: one CUDA block owns one bitshuffle block, from the compressed payload through to finished pixels.

1. LZ4 into shared memory, one warp per bitshuffle block. Blocks are independent, so the parallelism is across them; within the warp every lane runs the same sequence parser over the same bytes, and the literal and match copies are split across the 32 lanes so the stores coalesce. An overlapping match is treated as a pattern of period offset sourced from bytes that already precede the write position, which keeps it parallel rather than a serial byte loop; offset == 1 (a run of one repeated byte, the common case in sparse detector data) and power-of-two offsets avoid the modulo altogether. Because the lanes cooperate on the copies, each one is followed by __syncwarp() — a later match can read bytes another lane wrote, and since Volta that ordering is not implicit.
2. The bitshuffle inverse fused with preprocessing. The whole CUDA block then reads that shared buffer back: one thread owns one group of 8 elements across every byte-plane, so once it has transposed its 8 bytes out of each plane it holds 8 complete elements — and it applies the pixel mask, the error marker and the saturation cap and emits 8 finished int32 pixels directly. Nothing of the block reaches device memory but the pixels — neither the bitshuffled bytes nor the decompressed image is ever materialised. For 8-bit images there is a single plane and the assembly degenerates to a copy.

Decoding into shared memory is worth more than the bandwidth it saves: an LZ4 match reads back bytes written a few sequences earlier, so every copy step is a dependent round trip — tens of cycles in shared memory against hundreds in device memory. It is paid for in residency, because the buffer holds a whole bitshuffle block, and a block larger than 16 kB (larger than either writer this pipeline reads produces) falls back to a pair of kernels instead, the first writing the shuffled image to device memory and the second un-transposing and preprocessing out of it.

The block offsets inside the container can only be discovered by reading the block lengths in order, so that scan stays on the host.

Only BSHUF_LZ4 is decoded on the device. For the zstd variants (BSHUF_ZSTD, BSHUF_ZSTD_RLE, BSHUF_ZSTD_RLE_HUFF), and for uncompressed or float images, BSLZ4DecoderGPU::Supports() returns false and the pipeline decompresses on the host and uploads as before.

The container arrives off the network or off disk and is not trusted. Everything checkable on the host — declared sizes, the block scan, a block size that is not a multiple of 8 elements, a block count the chunk could not hold, trailing bytes — is rejected before any work is queued; the kernel additionally flags a block that did not decode to exactly its declared length, which becomes an exception once the caller has synchronised. That last check matters because the decode buffers are reused frame to frame: a block that stopped early would leave the previous image's most significant byte-plane in place, which reads not as a missing corner but as real pixels several powers of two too bright.

1. Geometry, reciprocal-space mapping, and basic quantities

1.1 Coordinate conventions

For a pixel coordinate (x,y) (in pixels), Jungfraujoch converts to a laboratory direction vector via:

1. shift by the beam-centre pixel (x_\mathrm{beam}, y_\mathrm{beam}) — the PONI, pyFAI's point of normal incidence, which coincides with the direct-beam impact point only for an untilted detector (see Detector geometry),
2. scale by pixel size p (mm),
3. set detector distance D (mm),
4. apply detector orientation rotation R_\mathrm{det} (PyFAI-like parameterization).

The unnormalized detector coordinate (mm) is: $ \mathbf{r}\mathrm{det}(x,y) = \begin{pmatrix} (x-x\mathrm{beam})p\ (y-y_\mathrm{beam})p\ D \end{pmatrix}. $

The lab-frame vector is: $ \mathbf{r}\mathrm{lab} = R\mathrm{det},\mathbf{r}_\mathrm{det}. $

By this construction (x_\mathrm{beam}, y_\mathrm{beam}) maps to (0,0,D) before the rotation — the point where the detector normal through the sample meets the detector — which is what makes it the PONI rather than the direct beam; the two differ by D\tan(\mathrm{tilt}) on a tilted detector. The laboratory frame is fixed the same way everywhere in the system: +z along the beam propagation, +x along increasing pixel column (the fast axis) and +y along increasing pixel row (the slow axis) — a right-handed triple that coincides with XDS's laboratory frame, which is what makes the geometry echo of Rugnux directly comparable. The absolute hand of an indexing — and with it the Bijvoet hands of §14.6–§14.7 — follows from this convention.

Let the incident wavevector magnitude be k = 1/\lambda in Å^{-1}, and define: $ \mathbf{S}_0 = (0,0,k). $

The reciprocal-space scattering vector associated with pixel (x,y) is: $ \mathbf{s}(x,y) = k,\frac{\mathbf{r}\mathrm{lab}}{\lVert \mathbf{r}\mathrm{lab}\rVert} - \mathbf{S}_0. $

This \mathbf{s} is the fundamental quantity used for spot finding (resolution filters), indexing, and refinement.

1.2 Two-theta, azimuth, resolution and q

The scattering angle 2\theta is computed from \mathbf{r}_\mathrm{lab} via: $ 2\theta = \mathrm{atan2}!\left(\sqrt{x_\mathrm{lab}^2 + y_\mathrm{lab}^2},; z_\mathrm{lab}\right), $

evaluated as a two-argument arctangent, so the mapping stays correct where a strongly tilted detector's far corner reaches past 2\theta = 90°.

Resolution (Å) at a pixel is: $ d = \frac{\lambda}{2\sin\theta}. $

The magnitude q = 2\pi/d is used for radial binning and ice-ring handling.

1.3 Distance from the Ewald sphere

For a reciprocal lattice point \mathbf{p} (Å^{-1}), define: $ \Delta_\mathrm{Ewald}(\mathbf{p}) = \lVert \mathbf{p} + \mathbf{S}_0\rVert - k. $ Jungfraujoch uses |\Delta_\mathrm{Ewald}| as an operational proxy for excitation error. This appears in:

• still prediction (accept if |\Delta_\mathrm{Ewald}|\le \Delta_\mathrm{cut}),
• profile radius estimation (see §11.1),
• still partiality option in scaling/merging (§10.2).

1.4 The beam centre

A run starts from the beam centre in the file, or from --beam-x/--beam-y where they are given. Header centres are often typed rather than measured, and on rotation data a wrong one is not repaired by anything downstream: the error is fixed in the laboratory frame, so accumulating a sweep smears every reciprocal-lattice point around a circle, and a displacement \delta p on the detector multiplies the FFT amplitude at an axis of length a by J_0(2\pi\,\delta p\,a/(D\lambda)). Past the first zero the true axis is gone and its harmonic wins. Rugnux therefore measures the centre itself on every run and treats the measurement as a second hypothesis to be tested against the file's, not as a replacement for it. The steps, in the order they run:

Step	Runs	Effect on the centre
Header check	every run, before a frame is read	none; reported
Background measurement (--beam-center-check)	every run with the beam-stop pre-scan	none at this point; reported
Second first pass at the measured centre	rotation	adopts the measured centre in the cases listed below
Both centres judged on the merge	rotation, two-pass	adopts the measured centre where its first pass merges better
Post-refinement (§7.5)	rotation, two-pass	refines the centre from the integrated reflections, within 15 px

On most runs the measured centre is reported, the second first pass finds the same lattice at both centres, and the run continues at the file's centre until post-refinement moves it.

What the file says. Before any frame is read, a centre within one pixel of the geometric centre of the detector, or a whole number of pixels in both coordinates, is reported as a value written rather than measured, and a centre that lands on a masked pixel is a warning — a real beam does not sit on a dead pixel or in a module gap. These lines change nothing.

Measured on every run (--beam-center-check, on by default). The isotropy of the scattered background places the point the beam lands on; the leverage is the curvature of the solvent ring. The fit runs over the mean image the beam-stop pre-scan already builds (§1.5), with the opaque part of the shadow masked, so it costs no frames of its own. For the same reason --detect-beam-stop=off leaves nothing to measure it on, and the check then does nothing.

The fit starts with a whole-detector capture. The centrosymmetry score of the projection about a candidate centre \mathbf{c}, \sum_\mathbf{x} I(\mathbf{x})\,I(2\mathbf{c}-\mathbf{x}), is the self-convolution (I*I)(2\mathbf{c}), so one FFT pair scores every candidate centre on the detector at half-pixel spacing: the cost is O(N\log N) and independent of how far the header is from the truth. Three surfaces are scored — the 2D point inversion, which measures the same isotropic background the walk below fits, and a 1D line mirror per detector axis, which along the spindle is exact Friedel physics (a reflection's mate half a turn later lands mirrored in the line through the beam across the spindle). The beam-stop shadow is blanked out of the scored image, a one-sided opaque obstruction being a centrosymmetry defect in its own right. What comes back is a shortlist and a margin per surface, not a centre: the surface can be locally flat over tens of pixels, and the peak-to-runner-up margin says so. A local walk is then seeded at the capture and refines it; where it declines there it is started again from the file's centre, and only where neither start gives it something to fit does the capture stand alone. The transforms run on the GPU where one is present and on FFTW otherwise, with everything that decides anything shared between the two.

The run log then carries:

• Beam centre capture: the strongest point-symmetry centre, the two line-mirror coordinates and the three margins (under about 3 % the surface is flat).
• Beam centre check: the file puts the direct beam at (x,y), the scattered background puts it at (x,y) +- σ px - a difference of d px, against the t px this geometry asks the centre to be right to. The comparison is against the direct beam, not the PONI: beam_x_pxl is the point of normal incidence, and the two part by D\tan(\mathrm{rot})/p as soon as the detector is tilted, while the background is centrosymmetric about where the beam lands. t = 0.183\,D\lambda/(p\,a_\mathrm{max}) is the displacement at which the J_0 factor above has fallen to 0.70 for the longest axis of the cell given with -C (200 Å where none is given); it runs from well under a pixel to several. It is reported, not used as a gate: a centre change far smaller than t can still decide between a cell and its harmonic.
• One of three follow-ups: the difference is under three times the fit's σ (the file's centre is as good as this measurement can tell); more than t of it lies across the spindle, where it can cost peaks or an axis; or it is a real difference, mostly along the spindle, where a wrong centre does not announce itself.

None of these lines moves the centre. What consumes the measurement is the second first pass below (rotation runs) and the bound on the post-refinement.

The second first pass (rotation). After the first pass — and after the rotation-axis sign rescue below — the first pass is indexed again at the measured centre, however small the difference. A trial centre gets its own azimuthal mapping, spot engines and spot cache: raw centroids do not move with the centre, but which spots are in the list does (the resolution limit, the ice-ring flag, the beam-stop mask and the strongest-N ranking are all radial), and the starting centre's spots are parked and restored so a trial that adopts nothing leaves the run as it was.

The indexed frame count is not used to choose between two centres that both index: acceptance is a fractional-Miller test, so a cell twice as long must place every spot twice as accurately to score the same, and a halved axis can index more frames than the true cell. "Indexes" below means more than half of the validation frames; "same lattice" means the same Bravais class with primitive volumes within 2 %. The outcomes:

1. The file's centre indexes and the measured one does not. The file's centre stands.
2. Both index the same lattice. The file's centre is kept, and the log says the difference does not decide this crystal's cell. A centre off by about one reflection spacing still indexes the same cell, with part of the sweep given indices one out, so where the two centres are further apart than both t and three σ of the fit, the measured centre is kept as a hypothesis for the merge (next paragraph).
3. Both index the same cell in a different Bravais class. Whether the centre could have decided the class is arithmetic: a shift of n pixels of size p moves an axis of length a by n\,p\,a/(D\lambda) relative, and the lattice search holds an axis equality to 3 %. Where the shift is worth that much on the pair of reduced-cell axes the equality compares, both centres are carried to the merge; otherwise the file's centre and its class are kept.
4. Both index, on different lattices. Both cells are reported in a warning. Where the primitive volumes differ by an integer factor of 2 to 4 — an axis harmonic, which a centre error along the spindle produces — the choice is made on the pooled validation spots: the measured centre is adopted where its lattice, larger or smaller, beats chance and puts a share of the spots on itself over its own wrong-spindle null that exceeds the file's by more than the binomial noise of the two (z = 3.29). Otherwise the file's centre is kept, with a warning.
5. The file's centre does not index. Nothing defends it, so:
   • where the measured centre indexes, it is adopted;
   • where it does not, the rotation-axis sign is flipped at the measured centre, since each of the two errors can hide the other; where that indexes, both are adopted;
   • where it still does not, the first-pass depth ladder of §6 (how much of each frame the pass reads) is run at both centres, so the centre moves only when it is the centre that pays. The measured centre is adopted where its lean rung indexes and beats the file's; where both index the same lattice once less of each frame is read, both centres are carried to the merge; where the file's centre wins or ties, the depth was the problem and the file's centre stays;
   • where nothing indexes at either centre, the pooled spots decide as in case 4; failing that, the file's centre is put back, so a run that fails for some other reason fails at the geometry it was given.

Both centres judged on the merge (rotation, two-pass). Where case 3 or 5 carried both centres, the whole first pass is run again at the measured centre — with its own short-axis pass and post-refinement — and the two arms are compared on their search merges (P1, whole range, before the correction surfaces). Where case 2 held a hypothesis, this is done only if the file's arm merges inconsistently: \mathrm{CC}_{1/2} below 0.9 in the lowest-resolution shell of its search merge, where every reflection is strong and a consistent merge reads close to 1. The measured centre wins where:

• both arms ended on the same lattice, and it merges more reflections at $I/\sigma \ge 2$, or its lowest-shell \mathrm{CC}_{1/2} is higher by more than 0.05;
• the arms ended on different lattices, and its search-merge \mathrm{CC}_{1/2} is higher by more than 0.05.

Otherwise, or where the measured arm does not complete, the run keeps the file's centre. The log lines are Beam centre check: running the first pass again at the measured centre … followed by … the run adopts the measured centre and the lattice it finds or the measured centre does not win. Over the validation battery about one rotation run in ten runs this second arm, and about one in ten ends on the measured centre by one of the routes above.

After a first pass that failed. A rotation sweep leaves one further thing the input often cannot settle, decided on the same count the rest of the first pass uses — the right answer indexes and the wrong one does not.

• The rotation-axis sign. A miniCBF header names the axis but gives no direction, and an NXmx vector is only meaningful together with the detector mounting. So after a poor first pass the opposite sign is tried and whichever indexes more validation frames is kept, the flipped one only where its lattice also beats chance on the pooled spots (§4.1), so that one frame against none on a sparse pattern cannot flip it. The decision holds for the rest of the run, and it flips the axis, not the angles, because prediction reads the axis too. It runs before the second first pass above and before the long-axis rescue, since with the sign wrong every candidate lattice is wrong.

Post-refinement. On a two-pass rotation run, whatever centre the first pass ends on is refined together with the distance, cell, orientation and axis by the post-refinement of §7.5, and the second pass is integrated there. The background measurement does not move that fit; it only widens where the fit may go: the refined beam is committed only within 15 px of the nearer of the pass-1 centre and the measured one (the measured one counting where its σ is within \max(1\ \mathrm{px}, t)), so a header that is far out can still be corrected. Where the refined pass is judged worse, the run goes back to the pass-1 geometry.

Stills. There is no second first pass and no post-refinement on stills; the background measurement is reported only. The centre moves only through the stills geometry refinement, --refine-geometry, which bundle-adjusts the beam centre, distance and cell over strongly indexed frames and is on by default where a reference cell is given.

What ends up in the output. The report's BEAM_CENTRE (the PONI) and DIRECT_BEAM are the geometry the result was integrated at, and so is JFJOCH_DATASET_SETTINGS; POSTREFINE_BEAM_CENTRE gives the post-refinement's move as before -> after. Which of the steps above changed the centre, and why, is in the run log only.

Options.

• --beam-x/--beam-y set the starting centre. The background check still runs and can still adopt the measured centre in the cases above; add --beam-center-check=off to hold a typed centre until post-refinement.
• --beam-center-check=off skips the background measurement and the second first pass, and removes the measured centre from the post-refinement bound.
• --detect-beam-stop=off removes the projection the background measurement is read from.
• --rotation-no-postrefine removes the post-refinement, and with it the merge-judged arms.

1.5 Finding the beam stop

The beam stop and its holder arm shadow part of the detector. A reflection behind them is attenuated but otherwise ordinary — it integrates low, with a plausible \sigma, and no outlier test catches it — so the shadow is found and masked instead. --detect-beam-stop[=N|off] (on by default, N=60 frames) projects those frames to a per-pixel mean and maximum, and writes the result into the pixel mask as bit 9, from where it excludes those pixels from every later stage.

The shadow is a place where the background is missing, so it is found by comparing each pixel's background against the background at the same radius. The mean is pooled over a small box first — a single pixel of a sparse background carries too few counts to tell a shadow from a Poisson hole, and the stop is much wider than the box — and the comparison is against the median of the pixel's own radius ring, taken over the pixels not already excluded and iterated a few times so the shadow stays out of the baseline it is measured against. A pixel is shadow when the ratio falls below 0.50 and the deficit is significant against its own Poisson scatter, \sqrt{2\,(E-N+N\ln(N/E))} over the pooled counts. A ratio says nothing when the background behind it is a handful of photons, and it is that significance that makes the comparison scale-free rather than tuned to one exposure: rebuilt from six frames of a low-background sweep, the same ratio without it masks three quarters of the detector. The index of dispersion does not do this job — it is 1.0 inside the shadow and 1.0 outside it, a shadowed pixel being Poisson at a low rate and a lit one Poisson at a high rate — only the rate relative to the ring separates them.

Nothing assumes the stop and the beam are concentric, because only the per-ring comparison is used; a ring lying wholly inside the stop has no unshadowed pixel for its median to find, which is exactly the case where the comparison must fail. Such a ring is judged against what this detector's background typically is — the median over the rings the walk is willing to judge — and is shadow in its entirety when it falls far below that. Comparing it instead against the largest background further out reads an ordinary background inside a strong ring as blocked, and the walk then runs out to that ring and returns a filled disk of good detector with diffraction rings plainly visible inside it.

The rings are drawn about the centre the data measure, not the one the file claims. Displacing the centre costs nothing for tens of pixels and a great deal beyond: at 150 px out the comparison describes the background's own radial fall-off rather than the hardware, and returns a tenth of the detector as shadow with nothing blocking it — and header centres are wrong by that much. The centre is therefore fitted from the same per-pixel mean the mask is computed from, before the mask is read, and named to the finder; it costs no frame of its own. That first fit is itself biased by the shadow it has not yet masked (on a sweep with a third of the detector behind a pin it landed 5 px out and called itself 0.40 px), and it does not have to be unbiased: the ring comparison does not notice tens of pixels, and the centre the run reports and consumes is the second fit of §1.4, the one that runs after the mask is loaded with the shadow out of the way. The order is fit, mask, fit, and only the second answer leaves. Two rounds are enough, measured rather than assumed — on clean sweeps the two fits agree to 0.03 px, so a third would draw the same rings.

A ring is only flat once the polarization is divided out. The comparison assumes the background is flat around a ring with nothing in the beam, and it is not: a polarized source suppresses the background in its own plane by the azimuthal factor of §2.2, a factor of three at 2\theta=55° and four at 70° — several times the dip the test is looking for, so on a short-distance geometry the two in-plane lobes of every outer ring read as shadow (measured: 5 % and 11 % of two detectors, with nothing visible under either mask). The mean projection is therefore divided by that factor before the ring comparison, and the Poisson deficit multiplies it back in so the significance is still the significance of the counts that were recorded. The factor is the geometry's own, evaluated about the centre just fitted rather than the one in the file: polarization is the only correction a ring carries that varies along it — solid angle, detector response and air absorption are functions of 2\theta and the ring median absorbs them — and the geometry is what knows where the polarization plane lies once the detector is tilted, the stored image quarter-turned or the detector rotated in its own plane.

What survives is then shaped into a region, and what makes it specific is size, not connection to the beam. A shadow is cast by something physical and is correspondingly large, while the background wanders a pixel or two at a time, so the connected components holding at least 2000 core pixels are kept and the rest dropped; measured on clean sweeps spanning 0.05 to 9.5 counts/px/frame of background, every one returns exactly one such component — the beam stop — and the largest spurious candidate anywhere is 74 pixels. Requiring the region to touch the direct beam instead is written for the stop and its holder arm; hardware standing in the beam further out — a pin, a loop — casts a shadow that begins some way out in radius, with lit detector between it and the stop and no bridge across the gap, and on one sweep that discarded 950 k correctly-found pixels and sent them into integration as measured-and-near-zero. The kept components are bridged across the module gaps the holder arm crosses, grown outward through the partially shadowed penumbra — much wider for a pin than for a stop edge — closed, and with the interior of the disk filled; the rings found lying wholly inside the stop join the region after the size filter rather than being asked to be large themselves.

Last, any pixel that ever recorded a real reflection — a maximum over the frames well above background, in a cluster, so that a single-frame zinger does not count — is given back, because a beam stop cannot have blocked a reflection that was measured.

1.6 Defective pixels

A pixel that reads high frame after frame, whatever the crystal does, is integrated into whichever reflection's box it falls in; under rotation that is one pixel collecting a different reflection on every frame that reaches it, and a merged intensity hundreds or thousands of times its shell mean, carried by one or two observations the merge's outlier test cannot judge. The file's mask misses such pixels on many detectors, so on rotation data from a counting sensor Rugnux measures them on the pre-scan's own sample of frames (HotPixelFinder, rugnux/HotPixels.h) and masks them as bit 10. The frames are read a second time for it, once the beam-stop projection has measured where the scattered background puts the beam: the rings have to be drawn about the true beam, and a file's centre can be far enough out that a ring crosses the background's radial fall-off.

On each frame a pixel is lit when it exceeds b + 3.3\,\max(\sqrt{b}, 1.4826\,\mathrm{MAD}) + 2, with b the larger of the median of its 2 px iso-2\theta ring and of that ring's sixteenth in azimuth, so ice and powder rings, the polarization dip and partial shadows set their own level. It is persistent when it is lit on at least \max(k_1, k_B) of the sampled frames: one reflection stays on a pixel for (\Delta\phi + w)/|\zeta| of rotation (w = 5°, a generous rocking width), which covers fewer than k_1 = 1 + \lceil (\Delta\phi + w)/(|\zeta|\,\delta) \rceil frames sampled \delta apart; and k_B is the binomial bound, from the lit rate of the ring's other pixels, that fewer than 0.01 pixels of the whole detector reach by chance. A persistent pixel is masked if it reads on average at least ten times its ring and its mean excess is above the Poisson bound; a weaker one cannot make an outlier. This holds for a patch of such pixels as for a lone one: a patch lit through the whole sweep is stationary in the lab, so it is no reflection of the rotating crystal, and a reflection crossing it would read the patch. Pixels holding the detector's error value on most frames (already invalid on every frame, but not in the static mask) are masked with them. A CCD (no sensor depth: read with an offset, not Poisson) is left alone. One log line says how many pixels were masked and why.

---

2. Azimuthal integration (radial profiles)

Azimuthal integration produces a radial profile I(q) or I(d) by histogramming pixels into radial bins. Pixels are not split across bins; each pixel contributes wholly to a single bin. By default the profile is purely radial (a single azimuthal bin), but the azimuth can optionally be split into up to 512 \phi sectors (azim_bins, --azim-phi-bins), giving a 2D q\times\phi profile that exposes azimuthal anisotropy such as detector shadowing or sample texture.

2.1 Histogram estimator

Let bin index b(x,y) be precomputed from q(x,y) (or equivalently from d(x,y)) and, when \phi sectors are enabled, the azimuth \phi(x,y) — so b = b_q + b_\phi B_q. For each bin b:

• accumulate corrected intensity and its square: $ S_b = \sum_{(x,y):,b(x,y)=b} I(x,y),C(x,y),\qquad S^{(2)}_b = \sum I(x,y)^2,C(x,y)^2, $
• and count: $ N_b = #{(x,y):,b(x,y)=b \text{ and pixel is valid}}. $

The profile reports both the mean \bar{I}_b = S_b / N_b (when N_b>0) and a per-bin sample standard deviation \sigma_b = \sqrt{(S^{(2)}_b - S_b^2/N_b)/(N_b-1)} (a spread/error estimate for each radial point). Invalid pixels (masked, saturated, detector error codes) are excluded.

2.2 Corrections applied

Two standard corrections are available:

(i) Solid angle / geometric correction. A flat pixel's solid angle falls off with the incidence angle \alpha between the scattered ray and the detector normal. With the in-plane detector offsets u=(x-x_\mathrm{beam})p and v=(y-y_\mathrm{beam})p — measured from the PONI (§1.1), which is what the tilt-invariance below rests on — and detector distance D, $ \cos\alpha = \frac{D}{\sqrt{u^2+v^2+D^2}},\qquad C_\Omega = \cos^3\alpha, $ applied — like the polarization term below — as a divisor (intensities are scaled by 1/\cos^3\alpha), so pixels at oblique incidence, which subtend a smaller solid angle, are boosted. Because \alpha is evaluated in the detector's own frame it is invariant under detector tilt (\mathrm{rot1}/\mathrm{rot2}/\mathrm{rot3}), matching PyFAI's solidAngleArray and MAX IV azint. It reduces to the commonly quoted \cos^3(2\theta) form only for an untilted detector, where the incidence angle coincides with the scattering angle.

(ii) Polarization correction. With polarization coefficient P (beamline dependent) and azimuth \phi: $ C_\mathrm{pol}(2\theta,\phi) = \frac{1}{2}\left(1+\cos^2(2\theta) - P\cos(2\phi)\left(1-\cos^2(2\theta)\right)\right), $ applied as a divisor to intensities (i.e. scale by 1/C_\mathrm{pol}) when enabled. This is the factor of Kahn et al. (1982); \phi is the azimuth in the lab frame, so the correction follows detector tilt and a swung-out 2\theta arm without further work.

The polarization plane is taken to contain the lab x axis — a horizontally polarized source. A vertically polarized one is expressed by a negative P: flipping the sign of the \cos 2\phi term is exactly a 90^\circ rotation of the plane. The plane is not autodetected and is not read from the file — no format Rugnux reads declares one — so a vertical beamline must say so with --polarization. P is the polarization degree (XDS's FRACTION_OF_POLARIZATION is (1+P)/2). The default P = 0.99, an undulator value, is applied to every Rugnux run; it is not fitted, because a single dataset does not determine it. Bending-magnet and wiggler beamlines are typically lower (0.8–0.95), and a known value for one should be passed with --polarization.

2.3 Background estimate for profiles

A background estimate is derived from the profile as its mean intensity over a fixed low-to-mid Q window (default 2\pi/5 to 2\pi/3 Å^{-1}). This background is used for monitoring and diagnostics; it is not the same as the local Bragg-spot background used in summation integration (§9.2).

---

3. Spot finding (strong pixels → Bragg spots)

Spot finding is a two-stage process:

1. Strong-pixel selection using intensity and/or local signal-to-noise criteria.
2. Connected-component labeling (CCL) to group strong pixels into candidate spots, followed by spot-level filtering and feature extraction.

3.1 Strong-pixel detection by local statistics

For each pixel i with value v_i, consider a square window (nominally 31\times 31 pixels) around it. Let the window contain n valid pixels (excluding masked/bad/saturated), and define: $ \Sigma = \sum v,\qquad \Sigma_2 = \sum v^2. $

To avoid biasing the local statistics by the test pixel itself, Jungfraujoch evaluates the pixel against the window with the pixel removed: $ \Sigma' = \Sigma - v_i,\quad \Sigma_2' = \Sigma_2 - v_i^2,\quad n' = n-1. $

A variance-like quantity proportional to n'^2 is formed: $ V = n'\Sigma_2' - (\Sigma')^2, $ and the deviation-from-mean quantity: $ \Delta = v_i n' - \Sigma'. $

A pixel is considered strong if:

• it is above a photon/count threshold, and
• its window contains enough valid neighbours (more than 100), so the local statistics are meaningful, and
• \Delta>0, and
• the squared deviation exceeds a scaled variance: $ \Delta^2 > V\cdot T^2, $ where T is the configured signal-to-noise threshold.

This is equivalent to a local z-score criterion but implemented in integer arithmetic to be robust and fast.

The test is applied in two passes over the image. The first is as described above. The second repeats it with every pixel found strong by the first excluded from the local background — it is treated exactly like a saturated pixel, so it contributes to no window it falls into and stays strong itself. This matters for any spot wide enough to reach into its own background box: on a single pass such a spot inflates the mean and variance it is then tested against, and its outer pixels fail the criterion. Excluding the core recovers them, so the spot is reported with its true extent rather than its brightest few pixels. Both the CPU and GPU implementations run these two passes and return the same spot list for the same frame.

Special cases:

• saturated pixels can be forced to “strong” (useful for detecting overloaded Bragg spots),
• invalid pixels are never strong.

3.2 Adaptive (self-calibrating) detection

The local-statistics test above needs a fixed photon/count threshold whose correct value depends on the background level, which varies between datasets. The adaptive mode (--adaptive-spots; the default in rugnux and in the viewer for both stills and rotation data, --no-adaptive-spots reverts) derives that threshold from each image's own noise, per resolution ring, so no per-dataset value is needed. It admits more spots than the fixed threshold, including genuine reflections that belong to no indexed lattice; these are down-weighted rather than filtered in the per-image geometry fit (§7.4).

Pixels are binned into the same resolution rings as the azimuthal integrator (§2). For each ring a robust background is estimated in three passes: one plain pass over all valid pixels, then two $\sigma$-clipping passes that keep only pixels within \pm 3\sigma of the current ring mean (removing the Bragg peaks from the background estimate). This yields a per-ring background mean \mu_b and scatter \sigma_b.

The ring's detection threshold is the larger of two arms, $ t_b = \max!\big(;\mu_b + z,\sqrt{\sigma_b^2 + \sigma_\mathrm{read}^2};,;; k_\mathrm{Poisson}(\mu_b, p);\big), $ where k_\mathrm{Poisson}(\mu_b,p) is the smallest count whose Poisson$(\mu_b)$ upper tail is \le p. The Poisson arm is correct where the background is countable (a bright low-resolution ring gets a high threshold); the Gaussian arm — floored by a detector-level excess-noise constant \sigma_\mathrm{read} — takes over on near-empty high-resolution rings, where the Poisson arm degenerates to "one photon is significant" and would flood. The operating point p = E/N is set from a single portable knob E, the expected number of false pixels tolerated per frame (--spot-false-pixels, default 100), with N the number of valid pixels. Because p and every \mu_b,\sigma_b come from the image itself, the same E lands a sensible photon threshold on strong and weak datasets alike, with no per-dataset tuning. Rings too sparse to characterise (fewer than ~40 pixels) fall back to a whole-frame background. A pixel is strong when v_i \ge t_b for its ring (saturated pixels are still forced strong); the strong pixels then feed the same CCL stage (§3.4). The signal-to-noise and photon-count criteria of §3.1 are not used in this mode.

Because detection reads the pixel's ring, a pixel that falls outside the azimuthal-integration q range has no ring and can never be strong: the integration range bounds what adaptive detection can see. Both upper limits are therefore optional and default to the detector itself — the azimuthal integration runs to the highest q any pixel of the detector reaches (--azim-max-q unset), and spot finding is not clipped in resolution (--spot-high-resolution unset), for rotation data as well as stills. Setting either one narrows detection accordingly — appropriate for weak, high-background data, where the spots admitted at the detector edge are dominated by noise.

Fused GPU engine. The per-ring reduction the adaptive threshold needs is the same reduction the azimuthal integrator performs. On the GPU path the two are fused into a single image pass (AdaptiveSpotFinderGPU): one reduction accumulates the corrected per-ring sums for the azimuthal profile (§2) and the raw per-ring statistics for the threshold, after which a light kernel flags the strong pixels. One GPU pass therefore replaces both the separate azimuthal-integration pass and the host-side adaptive spot-finding pass, at a small fraction of the CPU finder's cost per frame and producing the same spot list and azimuthal profile. It is enabled by default in the offline rugnux path, the interactive viewer and the online receiver.

Online. spot_finding_settings in the REST API carries adaptive_threshold and false_pixels_per_frame, so the mode is reachable from the broker and from the web frontend as well as from rugnux and the viewer. It defaults to off online, unlike rugnux and the viewer, because the broker serves both workflows and only one of them can run it: spots are found in software only on the DECTRIS/SIMPLON path, while the JUNGFRAU and EIGER workflows find them on the FPGA at its own fixed threshold. Setting adaptive_threshold on those is refused with an error rather than accepted and ignored, so a detection setting that had no effect cannot be mistaken for one that did.

3.3 Resolution and ice-ring handling

Spot finding can be restricted to a resolution range [d_\mathrm{high}, d_\mathrm{low}] by masking pixels outside the range. Optionally, spots in identified ice-ring regions can be tagged so that subsequent indexing/refinement may include or exclude them (see §4 and §6).

A single per-image ice-ring score is derived from a radial profile: for each hexagonal-ice powder ring (see Where the ring positions come from below), the profile intensity at the ring is divided by a smooth background estimated from the whole profile — a running median of the non-ice bins, interpolated under each ring — and the strongest ring's ratio is reported (1 = no ice, >1 = ice above background). A whole-profile background is used rather than a couple of adjacent shoulder bins so the estimate is robust to the radial binning: at a coarse Q-spacing a local shoulder can be only ~1 bin and would double-count the ring's own edge (offline processing defaults to a fine 0.01 Å⁻¹ spacing — in q = 2\pi/d, like every q in this document (§1.2) — --azim-q-spacing, so the rings are well resolved). The reported quantity is the ice magnitude rather than a significance: with many photons any real ice ring is statistically significant, so significance does not discriminate.

The profile the score is read off is the peak-excluded one, not the plain azimuthal integration: where adaptive spot finding runs (§3.2 — the offline and viewer default), the score uses the sigma-clipped per-resolution-ring background that finder already computes for its threshold. This matters more than it sounds. A plain azimuthal profile is a per-ring mean, so a few strong low-resolution reflections landing in a ring's bin raise it exactly as ice would; measured over a rotation battery, that alone ranked ice-free crystals above crystals that really are iced. An ice ring is azimuthally smooth and survives the sigma clip, while Bragg peaks do not, so on the clipped profile ice-free crystals sit near 1 and crystals with confirmed ice above 2. Only where no adaptive finder ran (the FPGA workflow) does the score fall back to the plain profile.

The radial profile sees ice only as a smooth powder ring. Ice in large crystallites diffracts as discrete spots, leaves the profile flat, and is invisible to the score above, so a second channel is read from the spot list itself: the spots found on the ice bands are counted against the spots found in the ice-free flanks [w,2w) either side of each band, rescaled to the bands' own q width (a flank landing on another ring is dropped with its width). The indicator is the ratio pooled over the run — per image the control is a handful of spots and the ratio means nothing — and it is taken before the spot-count filter, which orders ice spots last and would discard them first. The two channels barely overlap: smooth ice reads high on the profile and ~1 on the spots, textured ice the reverse, and a clean crystal ~1 on both. Both counts are stored per image (spot_count_ice_rings, spot_count_ice_control; HDF5 /entry/MX/peakCountIceRingRes, /entry/MX/peakCountIceRingControl).

Both channels are used offline as a gate on ice handling: unless the run reaches --ice-min-score (default 1.5) on the profile or --ice-min-spot-ratio (default 2.0) on the spots — 0 disables a channel — ice-ring flagging and the exclusion from the scale fit (§10.10) are skipped. The fixed bands cover 16–26 % of the unique reflections at typical resolutions whether or not the crystal has ice, and more than that on a detector reaching past 1.5 Å, so handling ice on a clean crystal is a pure loss.

Where the ring positions come from. The eleven bands from 3.895 to 1.522 Å are the measured positions of Moreau, Atakisi & Thorne (Acta Cryst D77, 2021, 540–554). That list ends at 1.522 Å by its own scope — the paper states that hexagonal ice "has 11 diffraction rings between 4 and 1.5 Å resolution", and its subject was detecting ice in deposited data rather than masking it — not because ice stops there. On a detector that reaches further, the rings it does not list are the ones left in the data.

The eight bands below it are calculated, since past that paper there is nothing measured to copy. Enumerating hkl from the ice Ih cell is not enough: ice Ih is P6_3/mmc with oxygen on 4f, and most of what enumeration emits is extinguished by the oxygen sublattice rather than by the space group — which is why (004) at 1.830 Å and (104) at 1.657 Å are missing from the measured list even though they lie inside its range and its reflection conditions allow them (for even l the 4f structure factor carries a factor \cos 2\pi l z at every hk, and z \approx 1/16 kills l=4 — (104) along with (004)). Structure factors are computed instead — oxygen only, the hydrogens being half-occupancy disordered and weak to X-rays — and the lines kept are those reaching 3 % of the strongest. That rule reproduces the measured eleven exactly, and every line it drops inside their range computes to zero, which is what makes it trustworthy below 1.522 Å. The cell is that of Röttger et al. (Acta Cryst B50, 1994, 644–648). The list stops at 1.170 Å because below it the calculated real lines fall to 2–3 % while the extinct ones rise to about 1 %, and an oxygen-only calculation cannot separate them any further.

One consequence is worth stating: the profile score is the strongest ring's ratio, a maximum over the bands, so a longer list can only raise it. The gate at 1.5 is therefore read against a list of this length, and lengthening it again would need the gate re-checked.

Rings this sample actually shows. Hexagonal ice is the only phase whose rings can be named in advance, and it is not the only thing that powders: a shower of microcrystals around the crystal, or a salt out of the cryoprotectant, leaves the same textured rings at $d$-spacings no fixed list carries, and their spots are otherwise handed to the indexer as if they were this crystal's. The rings are therefore also measured from the run's own pre-scan spots — read off as what stands above the smooth fall-off of spot density with q — on every run, and reported whether or not anything acted on them (POWDER_* in the results report: the ring count, the fraction of the pre-scan's spots they hold, and their $d$-spacings). The two lists are not the same list: on one such crystal 16 of the 24 measured rings were ice and the rest were not. A measured ring is set aside from indexing only where a first pass that found no lattice needs it set aside (§6); the fixed ice bands above are what the ice score, the scale fit and the resolution cut read.

A further optional safeguard removes isolated high-resolution “spur” spots by detecting large gaps in 1/d (or q) space and discarding spots beyond the gap. This is intended for macromolecular diffraction where edge-of-detector backgrounds can be extremely low.

3.4 Connected-component labeling (CCL)

Strong pixels are grouped into connected components (adjacent strong pixels) using a CCL algorithm. Each component yields a candidate spot with:

• centroid (x,y) (often intensity-weighted),
• pixel count (spot size),
• integrated spot intensity proxy (sum of pixel values),
• resolution d at the centroid (or mean over pixels),
• and quality flags (e.g. ice-ring classification).

Spot-level filters include minimum/maximum pixel count and resolution limits.

The upper bound is on how large a round spot may be, not on how bright one may be. Under the self-calibrating threshold (§3.2) a component's area above the contour grows as \sigma^2\ln(A/T) — without bound in the peak amplitude A — so an upper bound on pixel count alone becomes an intensity ceiling: measured on a strongly diffracting crystal, footprints run 3 px at 30–100 counts to 50 px above 10 000, four times the slope the fixed local-box test gives, and the old bound of 50 discarded the brightest reflections on every image. The bound is therefore 200 (the same as CrystFEL peakfinder8's --max-pix-count; XDS has no such parameter at all and guards on shape instead, with SPOT_MAXIMUM-CENTROID), and a component above 50 pixels must in addition fill at least a fifth of the square its bounding box fits inside. A Bragg reflection is round and fills about half of that square however bright it is; an ice arc, a cosmic-ray track or a lit detector row fills a fifth or less, which is what an upper bound was ever protecting against. Below 50 pixels no shape is asked for, so everything accepted before still is. The shape test is inert on every dataset it has been measured on — it exists to bound the shape of what the larger size bound now admits, on data carrying arcs or tracks, not because the crystals measured here needed it. The test is integer arithmetic, so the host and the GPU extractor agree by construction.

The host implementation (StrongPixelSet::sparseccl) is the SparseCCL of the ACTS/traccc project: it runs over the strong pixels sorted row-major, uses a sliding window over the previous line and a union-find whose root is each component's lowest index. On the GPU the same labelling runs on the device (SpotExtractorGPU): the packed strong-pixel bitmask is compacted into that same sorted list without leaving the card, each pixel finds its at most four earlier 8-neighbours by binary search, and a lock-free union-find with path halving labels them. Only the finished spot list — a few hundred entries — comes back to the host, instead of the whole bitmask (2.26 MB per frame at 18 MP). The two implementations produce the same components, in the same order, with the same pixel counts and intensities; tests/SpotExtractorGPUParityTest.cpp holds them to it. The device version is also insensitive to frame content: the host sliding window becomes quadratic when many pixels light up in one detector line — a hot module, or a diffraction ring where it runs tangent to a row — which costs tens to hundreds of milliseconds on such a frame, while the device version stays under a millisecond.

3.5 Adaptive per-image minimum spot size

The minimum-pixels-per-spot filter (§3.4) trades sensitivity against noise: a small value keeps faint one- or two-pixel spots — real signal on strong data, but detector noise on high-background frames — while a larger value keeps only well-formed spots. The best value is dataset-dependent, so for serial-stills indexing it can be chosen per image rather than fixed. The frame is indexed three times, at min-pix 3, 2 and 1, and the setting that maximises

\frac{n_\mathrm{indexed}^2}{n_\mathrm{total}} \quad\text{(indexed-spot count weighted by indexed fraction)}

is kept; the frame is then integrated once at that min-pix. The fraction factor discounts the extra spots a smaller min-pix admits unless the lattice actually explains them, so strong frames keep their real weak spots (extending resolution) while noise-flooded frames stay strict. Because min-pix filters the connected components after detection, strong-pixel detection AND the connected-component labelling both run once per frame, and the three attempts only repeat the spot-level filter; the azimuthal profile is the one that single detection pass computed. The winning attempt's spot list is kept rather than re-extracted, so the frame that is integrated is exactly the frame that was scored. This is a stills-only, indexing-path option — rotation indexing builds one global lattice from all frames and keeps a fixed min-pix. In rugnux it is the default; giving an explicit --min-pix-per-spot pins a fixed value instead.

3.6 Predicting the resolution the merged data will reach

A per-image resolution estimate is read off the finished spot list. It predicts how far the merged data will reach, not how far the furthest spot on this image lies. Each non-ice spot is weighted by \sqrt{I} — the intensity is a summed photon count, so \sqrt{I} is its Poisson significance — the 1/d^2 is found beyond which a fraction f=0.30 of that weight lies, and the estimate is that resolution taken 2.25\times further in 1/d. It is deliberately not limited to what the detector records: the quantile sits in the middle of the fall-off, well inside the recorded range, so it goes on measuring the crystal where the detector stops before the diffraction does, and on such a run it reads finer than the detector corner. The dataset value is the median over images.

Both constants carry a mechanism. A quantile from the middle of the distribution measures the shape of the fall-off, which is the crystal's own \exp(-B/2d^{2}), where the extreme end of it measures where detection stops — a threshold that moves with the exposure and with how many reflections the unit cell puts on a frame. And merging averages many observations of each reflection, so intensities go on being measurable a fixed factor in 1/d past the point at which one image's spot finder still detects them; that factor is the 2.25. Both are calibrated on rotation data against the resolution at which per-shell CC1/2 falls through 0.30, and the estimate is good to about 0.2 Å there. It is a prediction and not a measurement of what a run achieved: nothing downstream is cut on it, and it is reported alone (Rugnux SPOT_RESOLUTION_ESTIMATE, and per image in the stream, the plots and HDF5).