IndexAndRefine::integration_outcome is a whole-run vector (resize(GetImageNum()))
holding up to ~0.8 MB of reflections per indexed image. It is consumed only by
the offline scaling/merge pass in rugnux (ScaleAllImages / GetIntegrationOutcome).
In the viewer's interactive/live analysis it is written but never read - dead
storage that, with auto-reanalyze on during live follow, grew ~0.8 MB per frame
for the whole run.
Add a retain_outcomes constructor flag (default true). When false, skip the
whole-run resize and the per-image store; the current image's reflections are
still returned via the outgoing message. The viewer's live IndexAndRefine passes
false. rugnux (including GUI processing jobs, which build their own Rugnux ->
IndexAndRefine) and the online receiver keep the default true, so scaling/merge
still has the accumulated data.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The viewer could grow to ~100 GB RAM when live-following an HTTP broker. The
rc.153 images_in_flight backpressure only throttled imageLoaded; the heavy
per-frame payload rides datasetLoaded, fanned out over ~10 queued cross-thread
connections with no cap. In HTTPSyncDataset follow mode (entered when an operator
clicks an image while following live) RefreshDatasetOnly_i emits a fresh full
dataset every autoload tick with no imageLoaded, so the gate never engaged and
the queued events - each pinning a full JFJochReaderDataset (full-detector
PixelMask + per-image plots) - accumulated without bound.
Backpressure datasetLoaded the same way as imageLoaded: a datasets_in_flight
counter (cap 2), all emits routed through EmitDatasetLoaded_i, and
AutoLoadTimerExpired gated on it (covers HTTPSyncDataset). The window routes the
worker's datasetLoaded through a single OnDatasetReady sink that fans out
synchronously via datasetReady and acks with datasetConsumed. Under load stale
datasets are dropped; the next tick sends the latest.
Share the pixel mask instead of deep-copying it: JFJochReaderDataset::pixel_mask
is now shared_ptr<const PixelMask>, so per-frame dataset copies share the ~72 MB
mask. UpdateUserMask does copy-on-write; JFJochHttpReader caches the mask by
arm_date so a live refresh reuses one shared mask per acquisition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D)
* jfjoch_viewer: Some graphical improvements.
* jfjoch_viewer: Simplify und unify data analysis settings.
* jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition.
Reviewed-on: #65
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_viewer: Add reciprocal space viewer
* jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset
* jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress)
Reviewed-on: #57
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times
* frontend: Fix selection of FFTW in indexing settings
Reviewed-on: #51
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Better track time for each operation in the processing stack
* jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress)
* jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress)
* jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0)
Reviewed-on: #46
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release.
* jfjoch_broker: Add thresholding to prefer shorter vectors after FFT
* jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress)
* jfjoch_viewer: Display file opening errors
* jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available
Reviewed-on: #29
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Report changes in the image buffer, so viewer doesn't reload constantly
* jfjoch_viewer: Improve performance of loading images
* jfjoch_viewer: Auto-throttle image loading in HTTP-sync / movie modes
* jfjoch_viewer: Auto-foreground calculated with histogram
* jfjoch_viewer: Fix rare segmentation fault
Reviewed-on: #28
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.111 instead).
* jfjoch_broker: Add binary export of data analysis plots over OpenAPI
* jfjoch_broker: Minor fixes to HTTP error handling
* jfjoch_viewer: Prefer binary plots over JSON plots
* jfjoch_viewer: Change foreground with F button + wheel
* jfjoch_viewer: Change way how angles are displayed
* jfjoch_viewer: Display resolution of the mouse cursor in top left corner
Reviewed-on: #26
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.111 instead).
* jfjoch_viewer: Fix issue when HTTP sync silently disconnected when it was enabled when the broker was starting measurement.
* jfjoch_broker: Add protections on time of geometry optimization and reduce rotation recalculations
Reviewed-on: #25
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.11 instead).
* jfjoch_broker: Experimental rotation (3D) indexing
* jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values
Reviewed-on: #18
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_viewer: Add keyboard shortcuts and option to copy image to clipboard
* jfjoch_broker: Fix bit-width and exposure time for PSI EIGER detectors
Reviewed-on: #15
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_viewer: Auto load is better handling change of states
* jfjoch_viewer: Fix DBus registration
* jfjoch_viewer: Handle charts better with vertical lines on hover and status bar update
* jfjoch_viewer: Calculate ROI in a more efficient way
Reviewed-on: #6
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.96 instead).
* jfjoch_broker: For DECTRIS detectors add dark data collection during initialization for bad pixel mask
* jfjoch_broker: Refactor of calibration logic for more clear code (likely to introduce problems)
* jfjoch_viewer: Add option to handle user pixel mask (experimental)
* jfjoch_viewer: More options for ROI
* jfjoch_viewer: Add window to display calibration
Reviewed-on: #2
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* Fixes in CI pipeline
* jfjoch_broker: Remove PNG preview, no dependency on libpng
* jfjoch_writer: Fix UTC timestamp being generated wrong (mix between milli- and microseconds)
* jfjoch_viewer: Show data collection time in dataset tooltip
* jfjoch_viewer: Allow to choose the calibrant (presets for LaB6 and silver behenate)
* jfjoch_viewer: Auto foreground value
* Use external libjpeg-turbo and libtiff: simpler build stack, these are built and linked statically in automated Docker builds
* Remove OpenBLAS dependency
Reviewed-on: #1
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Fable 5