One changeset, developed together in response to a review of this branch, so the
files carry several of the changes at once. Full test suite passes (733 cases).
Spot finding
- Split ImageSpotFinder into Detect() (flag strong pixels - the expensive
per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with
Run() = both. The per-image min-pix escalation now detects ONCE and repeats
only the cheap extraction, instead of re-running the whole finder four times
per frame as it did on the default path. It also keeps the winning attempt's
spot list rather than re-extracting it, so the frame that is integrated is
exactly the frame that was scored - which a GPU re-extract could not guarantee
(float atomic ordering).
- spot_finding_time_s no longer swallows indexing time, and indexing_time_s now
sums every escalation call instead of reporting only the last.
Detection limits follow the detector
- The azimuthal-integration upper q and the spot-finding high-resolution limit
are now std::optional, in the C++ structs AND in the OpenAPI schema, and
resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_
recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a
pixel outside that q range could never be strong - the integration range
silently bounded what detection could see, regardless of the requested
resolution limit. Regenerated the C++ and TypeScript clients; the viewer and
the web frontend each gained a "to detector edge" switch.
Detection defaults are now per workflow (measured, not assumed)
- Stills: adaptive detection, min-pix chosen per image, no resolution clipping.
- Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit.
On a 33-crystal rotation battery, adaptive detection helped four hard crystals
but deterministically broke three (a lost space group, a halved indexing rate,
a collapsed merge), and the detector-edge limit cost indexing on a strong
rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and
--no-adaptive-spots is new.
Indexer seed escalation
- Stop escalating once a seed's lattice explains >= 90% of the seed spots.
Previously any frame with >= 80 spots always paid three indexer calls, online
broker included.
Merge-consistency filter
- --min-image-cc gated on a per-image CC computed BEFORE the stills partiality
post-refinement and never refreshed; the refiner now recomputes it, so the
reported CC describes the data that are actually merged.
- Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the
merge, the error model and MergeStats can no longer disagree about which
images are in (the --scale path merged unfiltered while its statistics were
filtered).
Per-image B-factor refinement (-B) removed
- Measured on four serial-stills datasets: it is a no-op where the per-image fit
is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas
+23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on
14-25% of images). It had also been silently DISCARDED since the partiality
post-refinement landed - reported but not applied. Rather than fix and keep a
knob with no demonstrated benefit, the flag and the whole image_scale_b_factor
chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and
read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs.
ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS.
(The Wilson per-image b_factor is a different quantity and stays.)
Stills partiality width now fits both of its components
- sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d*
with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2
on d*^2. The angular-only width is fitted over a d*^2-dense population, so it
was pinned by the high-resolution edge and collapsed at low d*: median
partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55%
of them under the merge's partiality floor, and the survivors divided by those
values - which inflated the merged low-resolution intensity scale 3.6x
(~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x,
no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and
R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining
R-free are all blind to that ramp, which is why it survived earlier validation;
the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged).
Removed dead code from add-then-remove churn
- Prediction-time "still partiality" (unreachable: no setter), the phantom
IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the
constant it always was), StillsPartialityRefine's caller-less Settings
constructor and its reference to a long-gone env var, ProcessImage's unread
bool return, an unused include, and a dead viewer overlay hook.
Also
- Viewer: the magnifier compared a QImage with itself, so its scene rect was set
once ever and it could not pan into a larger dataset; the hover tail timer
could fire after leaveEvent and resurrect the resolution readout outside the
image.
- update_version.sh regenerated the frontend lock file BEFORE bumping the
version (every release shipped an off-by-one lock), and did git rm/git add on
a path that has not existed since the client moved to src/client - with no
set -e, both failed silently.
- fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a
redirect, not a test, so dkms add never ran.
- Unit tests for the adaptive-threshold host functions, which had none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation.
* jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window.
* rugnux: Accept an absolute `-o` output prefix in offline processing.
* Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed).
* Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files)
* jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction
* jfjoch_broker: Image pusher statistics are accessible via the REST interface
* jfjoch_writer: Supports TCP image socket and for these auto-forking option
Reviewed-on: #36
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.96 instead).
* jfjoch_broker: For DECTRIS detectors add dark data collection during initialization for bad pixel mask
* jfjoch_broker: Refactor of calibration logic for more clear code (likely to introduce problems)
* jfjoch_viewer: Add option to handle user pixel mask (experimental)
* jfjoch_viewer: More options for ROI
* jfjoch_viewer: Add window to display calibration
Reviewed-on: #2
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5