Image buffer: the per-image CBOR metadata headroom had been re-derived from the
online reflection cap alone, which cut it from 4 MiB to 2.55 MB while the measured
worst case - reflections plus the capped spot list plus the three azimuthal arrays -
is 2.9 MB, so the receiver dropped the frames with the most to say. Restore it and
give it a name that both the code and its guard test read: written down twice, the
two had drifted and the test kept passing against the value the code had left.
Spot finding: an unset low_resolution_limit means no limit at that end, as an unset
high_resolution_limit already did. An optional rather than a zero sentinel, because
zero is not a natural "no limit" here - every pixel lies above it, so the plain
comparison masked the whole image instead of none of it, and nothing validated the
zero. The API field is no longer required; a zero is folded into the unset case at
the boundary, where older clients still send it, so one spelling reaches the
analysis code. The FPGA takes its fixed-point ceiling instead, since ap_ufixed<16,9>
wraps above 512 A and would have masked everything.
image_preprocessing: check the CUDA calls on the fused decode path - the one new GPU
file with none, and the path fed by bytes we did not produce. An unchecked
synchronise returned the host-written sentinel as if it were a measurement, so the
decode looked successful and the fallback to the host decoder never fired.
rugnux: --stride no longer writes one past the end of the per-image arrays, whose
count floored where the worker loop ceils, and the written process file links the
images actually processed rather than the first N - each frame's picture now sits
next to its own analysis.
Powder calibration: the face-centred calibrants no longer list their systematically
absent rings, so the distance fit starts from a reflection that exists rather than
an extinct one; the triclinic calibrant covers both signs of h and k instead of a
single octant, which is only valid for a diagonal metric. The test asserted the old
behaviour - one ring formula for every cubic standard - and is rewritten.
CBOR: skip an unknown tagged value in the end block, as the other four blocks
already do. One advance lands on the tagged item rather than past it, so an older
reader fed a newer end message threw and never finalized its file.
Viewer: a settings value the setter rejects no longer escapes as an uncaught throw
from a worker slot, and the field offers only what the setter accepts.
Space-group search: judge stage B on the same "present" cut stage A already computes.
Merged sigma is floored so no reflection reads above ISa, so on a low-ISa merge the
fixed cut left both stage B tests unsatisfiable - every screw axis passed unchallenged
and the centering rescue switched itself off on exactly the weak data it exists for.
Where the fixed cut is the smaller of the two they are equal and this is inert: over
the 37-crystal rotation battery every crystal reports the identical space group and
identical merge statistics, so it is a no-op there and the low-ISa case it targets
remains unmeasured.
rugnux: --polarization reaches --mode azint, which parsed the flag and then dropped
it; that mode also applies the same polarization default as every other mode.
Acknowledge the ACTS/traccc project, whose sparse connected-component labelling both
spot extractors take their algorithm from, with its citation and its license.
The rc.161 change list is brought back to one line per entry, and the user-visible
changes that were missing from it added.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One changeset, developed together in response to a review of this branch, so the
files carry several of the changes at once. Full test suite passes (733 cases).
Spot finding
- Split ImageSpotFinder into Detect() (flag strong pixels - the expensive
per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with
Run() = both. The per-image min-pix escalation now detects ONCE and repeats
only the cheap extraction, instead of re-running the whole finder four times
per frame as it did on the default path. It also keeps the winning attempt's
spot list rather than re-extracting it, so the frame that is integrated is
exactly the frame that was scored - which a GPU re-extract could not guarantee
(float atomic ordering).
- spot_finding_time_s no longer swallows indexing time, and indexing_time_s now
sums every escalation call instead of reporting only the last.
Detection limits follow the detector
- The azimuthal-integration upper q and the spot-finding high-resolution limit
are now std::optional, in the C++ structs AND in the OpenAPI schema, and
resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_
recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a
pixel outside that q range could never be strong - the integration range
silently bounded what detection could see, regardless of the requested
resolution limit. Regenerated the C++ and TypeScript clients; the viewer and
the web frontend each gained a "to detector edge" switch.
Detection defaults are now per workflow (measured, not assumed)
- Stills: adaptive detection, min-pix chosen per image, no resolution clipping.
- Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit.
On a 33-crystal rotation battery, adaptive detection helped four hard crystals
but deterministically broke three (a lost space group, a halved indexing rate,
a collapsed merge), and the detector-edge limit cost indexing on a strong
rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and
--no-adaptive-spots is new.
Indexer seed escalation
- Stop escalating once a seed's lattice explains >= 90% of the seed spots.
Previously any frame with >= 80 spots always paid three indexer calls, online
broker included.
Merge-consistency filter
- --min-image-cc gated on a per-image CC computed BEFORE the stills partiality
post-refinement and never refreshed; the refiner now recomputes it, so the
reported CC describes the data that are actually merged.
- Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the
merge, the error model and MergeStats can no longer disagree about which
images are in (the --scale path merged unfiltered while its statistics were
filtered).
Per-image B-factor refinement (-B) removed
- Measured on four serial-stills datasets: it is a no-op where the per-image fit
is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas
+23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on
14-25% of images). It had also been silently DISCARDED since the partiality
post-refinement landed - reported but not applied. Rather than fix and keep a
knob with no demonstrated benefit, the flag and the whole image_scale_b_factor
chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and
read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs.
ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS.
(The Wilson per-image b_factor is a different quantity and stays.)
Stills partiality width now fits both of its components
- sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d*
with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2
on d*^2. The angular-only width is fitted over a d*^2-dense population, so it
was pinned by the high-resolution edge and collapsed at low d*: median
partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55%
of them under the merge's partiality floor, and the survivors divided by those
values - which inflated the merged low-resolution intensity scale 3.6x
(~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x,
no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and
R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining
R-free are all blind to that ramp, which is why it survived earlier validation;
the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged).
Removed dead code from add-then-remove churn
- Prediction-time "still partiality" (unreachable: no setter), the phantom
IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the
constant it always was), StillsPartialityRefine's caller-less Settings
constructor and its reference to a long-gone env var, ProcessImage's unread
bool return, an unused include, and a dead viewer overlay hook.
Also
- Viewer: the magnifier compared a QImage with itself, so its scene rect was set
once ever and it could not pan into a larger dataset; the hover tail timer
could fire after leaveEvent and resurrect the resolution readout outside the
image.
- update_version.sh regenerated the frontend lock file BEFORE bumping the
version (every release shipped an off-by-one lock), and did git rm/git add on
a path that has not existed since the client moved to src/client - with no
set -e, both failed silently.
- fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a
redirect, not a test, so dkms add never ran.
- Unit tests for the adaptive-threshold host functions, which had none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For stills indexing the minimum-pixels-per-spot filter is now chosen per image
instead of being fixed: the frame is indexed at min-pix 3/2/1 and the setting that
maximises indexed-spot count weighted by indexed fraction (n_indexed^2 / n_total)
is kept, then integrated once at that min-pix. The fraction factor keeps a smaller
min-pix's extra spots only when the lattice actually explains them, so strong frames
retain their real weak spots (extending resolution) while noise-flooded frames stay
strict.
The mode is selected by the presence of --min-pix-per-spot, now optional
(SpotFindingSettings::min_pix_per_spot is std::optional<int64_t>): omit it for the
adaptive per-image path, give a value to force a fixed min-pix. It applies only to
the stills indexing path -- rotation indexing builds one global lattice and keeps a
fixed min-pix, and the online receiver and the FPGA host path always carry a concrete
value, so neither changes. IndexAndRefine::ProcessImage now returns whether the frame
indexed, to drive the per-image selection.
Exposed in the jfjoch_viewer spot-finding settings (adaptive-threshold and
adaptive-min-pix checkboxes, each greying out the control it overrides); the broker
uses neither.
Validated on the full rotation regression battery (no regression) and the whole
serial-stills target battery at full image count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
* jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin.
* jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins.
Reviewed-on: #60
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times
* frontend: Fix selection of FFTW in indexing settings
Reviewed-on: #51
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Default EIGER readout time is 20 microseconds
* jfjoch_broker: Multiple improvements regarding performance
* jfjoch_broker: Image buffer allows to track frames in preparation and sending
* jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer
* jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets
* jfjoch_writer: Fixes regarding properly closing files in long data collections
* jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code
Reviewed-on: #34
leonarski_fandClaude Opus 5