The spot finder flagged strong pixels on the device and then labelled them on the
host, so every frame sent the packed bitmask back - 2.26 MB on a large detector -
and the host walked all of it to recover a few hundred pixels. Do the labelling on
the device instead: compact the bitmask into a flat-index-sorted list, find each
pixel's backward neighbours by binary search, union them lock-free with path
halving, then label, accumulate and filter in one kernel. Only the spot list comes
back, and only one stream synchronisation per frame.
The gain in the ordinary case is modest - about a quarter off per-image spot
finding - because the host algorithm is genuinely fast on a normal frame. What
justifies it is the frame that is not ordinary. The host labels a sorted sparse
list through a window spanning two detector lines, so its cost is quadratic in how
many strong pixels share a line. A lit band of detector rows - a hot module, a
panel edge - costs 33 ms at two rows and 377 ms at fifteen, all of it under the
pixel cap that was supposed to bound this, and none of it maskable when the cause
is a diffraction ring rather than a defect: a ring runs tangent to a row at its
top and bottom, which is exactly the shape that hurts. The device version is flat
at 0.05 to 0.64 ms across every geometry tried, so an online run no longer stalls
a quarter of a second on an ice ring. Rejecting an over-cap frame is now free too,
since the count is known before any pixel is written.
Also label once and filter three times. The per-image minimum-pixel search runs the
extraction at three settings, but that setting only decides which components are
kept - it does not change the components - so the search itself need not be
repeated. This helps the host path as much as the device one.
The resolution mask moves to the device as a bit mask, uploaded when the limits
change rather than per frame, since the compaction needs it there.
Parity is asserted permanently rather than argued: five cases covering realistic
frames, occupancy from a hundred pixels to past the cap, the pathological
geometries including rings, the resolution mask, and a hundred-repeat determinism
check - requiring the same partition, the same spot order, and identical counts.
The centroid is a float sum and therefore order-dependent, so the device walks each
component from its root in ascending order and fuses its multiply-add the way the
host's does; note that whether the host fuses at all depends on the architecture
flags, so exact centroid equality is asserted where the compiler fuses and a
two-ulp bound otherwise. Making those accumulators integer would remove that
dependence entirely and is worth doing separately.
Regression set: all 37 crystals identical to the last printed digit. Unit suite
passes with the new cases.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were opt-in adaptive-spot refinements that did not help. Soft per-spot
weighting was index-rate neutral across the battery (re-ranking only bites when
spots exceed the max-spot cap, which weak serial data does not reach). The
local-SNR gate was neutral on index rate and degraded merged CC1/2 on flooded
XFEL data. Drops the flags, ApplyWeights/FilterByLocalSNR, the per-spot weight
field, and the by-weight FilterSpotsByCount branch (now strongest-first only).
--adaptive-spots itself is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add --soft-weight (implies --adaptive-spots): give every detected spot a
continuous quality weight in (0,1] and keep the highest-weight spots rather than
the brightest, so a deliberately loose detector self-cleans -- bright ice / salt
/ jet blobs and single-pixel noise no longer evict faint clean Bragg spots from
the max-spots cut.
The weight is a product of dimensionless gates (AdaptiveSpotFinderCPU::ApplyWeights,
computed against the per-ring background the adaptive finder already builds): a
logistic ramp in the spot's SNR and a soft size band (rises from one pixel,
plateaus, falls for oversized ice/salt/streak blobs). It carries on
DiffractionSpot -> SpotToSave and is consumed by FilterSpotsByCount, which ranks
by {non-ice, weight, intensity} when requested and by intensity otherwise, so the
classic and FPGA paths are unchanged.
Honest result: on the serial-stills battery this is index-rate-NEUTRAL. The
weighted ranking only changes the outcome when the spot count exceeds the
max-spots cap and the weight disagrees with intensity in a way that affects
indexing; the adaptive detectors already produce clean spot lists and the weak
sets sit under the cap, so re-ranking is a wash there (and a wash, not a
regression, on the one set that floods). Its intended benefit -- robustness to
ice/jet-contaminated frames and to a loosened detector -- is not exercised by
this battery; kept opt-in as the substrate for that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis.
If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Improve logic on switching on/off spot finding
* jfjoch_broker: Increase maximum spot count for FFBIDX to 65536
* jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD)
* jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis
* jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection
* jfjoch_writer: Save more information about each reflection
Reviewed-on: #32
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.11 instead).
* jfjoch_broker: Experimental rotation (3D) indexing
* jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values
Reviewed-on: #18
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5