abf156a3aa62af0b73baa72f32fdcda486c4dc18
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leonarski_fandClaude Opus 5
|
16bf3408f0 |
Address code-review findings; make detection limits detector-driven
One changeset, developed together in response to a review of this branch, so the files carry several of the changes at once. Full test suite passes (733 cases). Spot finding - Split ImageSpotFinder into Detect() (flag strong pixels - the expensive per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with Run() = both. The per-image min-pix escalation now detects ONCE and repeats only the cheap extraction, instead of re-running the whole finder four times per frame as it did on the default path. It also keeps the winning attempt's spot list rather than re-extracting it, so the frame that is integrated is exactly the frame that was scored - which a GPU re-extract could not guarantee (float atomic ordering). - spot_finding_time_s no longer swallows indexing time, and indexing_time_s now sums every escalation call instead of reporting only the last. Detection limits follow the detector - The azimuthal-integration upper q and the spot-finding high-resolution limit are now std::optional, in the C++ structs AND in the OpenAPI schema, and resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_ recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a pixel outside that q range could never be strong - the integration range silently bounded what detection could see, regardless of the requested resolution limit. Regenerated the C++ and TypeScript clients; the viewer and the web frontend each gained a "to detector edge" switch. Detection defaults are now per workflow (measured, not assumed) - Stills: adaptive detection, min-pix chosen per image, no resolution clipping. - Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit. On a 33-crystal rotation battery, adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge), and the detector-edge limit cost indexing on a strong rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and --no-adaptive-spots is new. Indexer seed escalation - Stop escalating once a seed's lattice explains >= 90% of the seed spots. Previously any frame with >= 80 spots always paid three indexer calls, online broker included. Merge-consistency filter - --min-image-cc gated on a per-image CC computed BEFORE the stills partiality post-refinement and never refreshed; the refiner now recomputes it, so the reported CC describes the data that are actually merged. - Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the merge, the error model and MergeStats can no longer disagree about which images are in (the --scale path merged unfiltered while its statistics were filtered). Per-image B-factor refinement (-B) removed - Measured on four serial-stills datasets: it is a no-op where the per-image fit is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on 14-25% of images). It had also been silently DISCARDED since the partiality post-refinement landed - reported but not applied. Rather than fix and keep a knob with no demonstrated benefit, the flag and the whole image_scale_b_factor chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs. ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS. (The Wilson per-image b_factor is a different quantity and stays.) Stills partiality width now fits both of its components - sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d* with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2 on d*^2. The angular-only width is fitted over a d*^2-dense population, so it was pinned by the high-resolution edge and collapsed at low d*: median partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55% of them under the merge's partiality floor, and the survivors divided by those values - which inflated the merged low-resolution intensity scale 3.6x (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x, no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining R-free are all blind to that ramp, which is why it survived earlier validation; the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged). Removed dead code from add-then-remove churn - Prediction-time "still partiality" (unreachable: no setter), the phantom IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the constant it always was), StillsPartialityRefine's caller-less Settings constructor and its reference to a long-gone env var, ProcessImage's unread bool return, an unused include, and a dead viewer overlay hook. Also - Viewer: the magnifier compared a QImage with itself, so its scene rect was set once ever and it could not pan into a larger dataset; the hover tail timer could fire after leaveEvent and resurrect the resolution readout outside the image. - update_version.sh regenerated the frontend lock file BEFORE bumping the version (every release shipped an off-by-one lock), and did git rm/git add on a path that has not existed since the client moved to src/client - with no set -e, both failed silently. - fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a redirect, not a test, so dkms add never ran. - Unit tests for the adaptive-threshold host functions, which had none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 4.8
|
9fdeed282a |
Add fused GPU adaptive spot finder (azint + spot finding in one pass)
AdaptiveSpotFinderGPU does the per-resolution-ring reduction once on the GPU and drives both products from it: the azimuthal-integration profile (corrected space) and the self-calibrating adaptive spot-detection threshold (raw counts). This replaces the separate GPU azint pass and the host-side adaptive spot finder that runs on the GPU path today. On a ~4.5 MP detector it does both jobs in ~1 ms/frame versus ~40 ms for the CPU adaptive finder (~42x), with an identical spot list and azimuthal profile. The per-ring threshold math (Poisson tail + read-floored Gaussian, operating point from the false-pixels-per-frame knob) is factored into AdaptiveThreshold.h so the CPU and GPU finders share one source of truth and cannot drift. Wired opt-in via a MXAnalysisWithoutFPGA constructor flag, default on for the rugnux offline path and the interactive viewer, off for the online receiver (so the broker path is unchanged). When on, Analyze() skips the separate azint pass and lifts the profile from the fused engine. The viewer gains an "Adaptive threshold" checkbox that greys out the signal/noise and photon-count sliders (the adaptive finder uses neither). Dedicated tests exercise both products (spot-finding parity vs the CPU finder, azimuthal profile vs a standalone GPU azint) plus a speed benchmark. Validated end-to-end on lysozyme serial stills: fused == CPU-adaptive index rate and merge stats. Docs: new section 3.2 in docs/CPU_DATA_ANALYSIS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 4.8
|
20bbcb1cd3 |
Remove --soft-weight and --local-snr spot-finder options
Both were opt-in adaptive-spot refinements that did not help. Soft per-spot weighting was index-rate neutral across the battery (re-ranking only bites when spots exceed the max-spot cap, which weak serial data does not reach). The local-SNR gate was neutral on index rate and degraded merged CC1/2 on flooded XFEL data. Drops the flags, ApplyWeights/FilterByLocalSNR, the per-spot weight field, and the by-weight FilterSpotsByCount branch (now strongest-first only). --adaptive-spots itself is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 4.8
|
ecf79af018 |
Remove threshold-free persistence spot-detection variant
Drops --persistence-spots and AdaptiveSpotFinderCPU::RunPersistence (the 0-D
topological-persistence detector added in
|
||
|
leonarski_fandClaude Opus 4.8
|
ca7cbe206a |
Add opt-in local-SNR spot gate and acceptance-fraction knob (serial stills)
Two opt-in tools for weak serial-stills tuning; both default-off, so the default pipeline is bit-identical (verified: a serial-stills reference run reproduces HEAD's 7.85% indexing rate exactly). --local-snr <sigma> (AdaptiveSpotFinderCPU::FilterByLocalSNR): after the loose per-ring adaptive threshold builds connected-component spots, drop any spot that does not stand this many sigmas above its OWN LOCAL background (robust median/MAD of a square annulus), not just the azimuthal ring mean. On structured-background (XFEL) frames the ring mean underestimates the local diffuse level in some sectors, so the ring threshold floods; a real Bragg peak still stands many local sigmas proud. Validated on XFEL stills to separate real peaks from flood at the pixel level (real median local-SNR ~70 vs flood ~2.6; SNR>=5 keeps ~99.8% of real peaks, ~14% of flood). GPU-portable (a per-spot local reduction). NOTE: on the current serial-stills battery it is index-rate/CC1/2 neutral -- the flood that survives as CC clusters overlaps weak-real spots, and only lattice-fit separates those -- but it is the correct tool for genuinely floody data (ice/jet/loosened detector) and the right substrate for the online FPGA path. --min-indexed-fraction <f>: exposes the previously hardcoded 0.20 minimum indexed-spot fraction (AnalyzeIndexing) as a per-run setting. Lowering it admits weaker/sparser crystals; on flooded XFEL data the extra lattices are spurious (pair with --min-image-cc to gate them), on clean synchrotron data there are no marginal frames so it is a no-op -- useful as a gating-experiment primitive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 4.8
|
7c5bedfd74 |
Add soft per-spot quality weighting for adaptive spot detection
Add --soft-weight (implies --adaptive-spots): give every detected spot a
continuous quality weight in (0,1] and keep the highest-weight spots rather than
the brightest, so a deliberately loose detector self-cleans -- bright ice / salt
/ jet blobs and single-pixel noise no longer evict faint clean Bragg spots from
the max-spots cut.
The weight is a product of dimensionless gates (AdaptiveSpotFinderCPU::ApplyWeights,
computed against the per-ring background the adaptive finder already builds): a
logistic ramp in the spot's SNR and a soft size band (rises from one pixel,
plateaus, falls for oversized ice/salt/streak blobs). It carries on
DiffractionSpot -> SpotToSave and is consumed by FilterSpotsByCount, which ranks
by {non-ice, weight, intensity} when requested and by intensity otherwise, so the
classic and FPGA paths are unchanged.
Honest result: on the serial-stills battery this is index-rate-NEUTRAL. The
weighted ranking only changes the outcome when the spot count exceeds the
max-spots cap and the weight disagrees with intensity in a way that affects
indexing; the adaptive detectors already produce clean spot lists and the weak
sets sit under the cap, so re-ranking is a wash there (and a wash, not a
regression, on the one set that floods). Its intended benefit -- robustness to
ice/jet-contaminated frames and to a loosened detector -- is not exercised by
this battery; kept opt-in as the substrate for that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
leonarski_fandClaude Opus 4.8
|
6de03bc443 |
Add threshold-free persistence variant of adaptive spot detection
Add --persistence-spots, a second parameter-free detector alongside --adaptive-spots. Instead of a hard per-ring threshold it builds the noise-normalised image z = (I - ring_mean) / sqrt(ring_sigma^2 + read^2) (same per-ring background as the hard variant) and scores every intensity maximum by its 0-D topological persistence: sweeping the height from high to low, each maximum is born and, when its basin meets a taller one at a saddle, dies with persistence = birth - saddle, in sigma. A lone noise spike merges into the background almost immediately (persistence ~1 sigma); a real peak stands many sigma proud. Emitting maxima whose persistence clears the same z(E) significance bar needs no photon threshold and no min-pix, and it deblends touching peaks (each keeps its own maximum). Implemented with the same union-find idiom as the connected-component labeller. On serial stills this auto-adapts with no per-dataset tuning like --adaptive-spots, finding fewer but cleaner (deblended) spots; the hard-threshold variant remains more sensitive on the very weakest data. Both share the per-ring background and read-noise floor. comp_of is allocated lazily so the default and hard-adaptive paths pay nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 4.8
|
9a8c946555 |
Add self-calibrating adaptive spot detection for offline stills
The offline CPU spot finder marks a pixel strong when it clears a fixed photon
count AND a local-window SNR. The fixed photon floor forces per-dataset tuning:
its sweet spot tracks the background level (weak sets want a low threshold,
strong or high-background sets a high one) and the usable window is narrow, so
users hand-tune --spot-threshold/--spot-sigma per dataset.
Add an opt-in --adaptive-spots mode (AdaptiveSpotFinderCPU) that replaces the
fixed floor with a per-resolution-ring threshold derived from each image's own
noise. Per ring it computes a peak-excluded background mean and sigma (one plain
pass + two sigma-clip passes over the assembled photon image, binned by the
azimuthal-integration ring index) and sets
thr = max( PoissonTail(mean, p), mean + z * sqrt(sigma^2 + read^2) )
with p = false_pixels_per_frame / n_pixels the single portable knob (default
100) and z = Phi^-1(1 - p). The Poisson arm is the correct significance where
the background is countable (it carries the sqrt(mean) shot noise, so a bright
low-resolution ring gets a high threshold); the read-noise-floored Gaussian arm
keeps the threshold physical where the background vanishes (empty high-resolution
rings), without which those rings flood. read is a detector-level constant, not
a per-dataset knob. Both arms are needed: Poisson alone floods near-zero
background, Gaussian alone drops the shot-noise term and under-thresholds bright
rings.
One --adaptive-spots setting then adapts across a wide range of serial datasets
with no per-dataset threshold, matching or beating hand-tuned thresholds and the
peakfinder8/xgandalf reference on both weak large-cell and strong serial data,
with equal merged R-free.
The finder runs on the CPU (offline/viewer path) and reads the host image, which
the GPU pipeline already keeps in sync, so it works in either build. The default
(non-adaptive) path and the online/FPGA path are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|