A single per-image ice_ring_score - the strongest hexagonal-ice ring band/shoulder
intensity ratio from the azimuthal profile (1 = no ice) - computed in the CPU and
FPGA analysis paths and the offline azint worker, then plumbed through every layer:
DataMessage/EndMessage, CBOR (frame_serialize), HDF5 (/entry/MX/iceRingScore),
ScanResult, receiver plots (PlotType::IceRingScore), the OpenAPI spec (plot_type +
scan_result schema, with regenerated broker/gen and frontend client) and
OpenAPIConvert, the reader + Qt viewer, and the React frontend plot. Documented in
docs/CBOR.md, docs/HDF5.md and docs/CPU_DATA_ANALYSIS.md, with the general
"add a per-image quantity" recipe added to CLAUDE.md.
Verified in HDF5: lysoC (weak ice) mean 1.23, EP_cs_01-17 (heavy ice) mean 1.67 /
max 2.23. This is a monitoring quantity - it does not gate scaling (which already
excludes all ice rings) or merging (handled by the CC1/2 ring mask).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When per-image CBOR metadata comes within 32 bytes of the buffer slot
size, slot_size - (metadata_size + 32) wrapped around (size_t), passing a
huge output_size to CompressImage. That defeated the buffer-too-small
guard and let the compressor write the full image past the end of the
slot, corrupting adjacent memory; AppendImage then threw a plain
JFJochException that aborted the whole collection after the fact.
Detect metadata_size + 32 >= slot_size explicitly and throw
CompressionBufferTooSmallException, so the existing catch drops just this
frame gracefully - the case the change was meant to handle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Compress() and FrameTransformation::CompressImage() returned int64_t with a
negative value meaning "did not fit". That is a footgun: the negative result
silently converts to a huge size_t if a caller forgets to check it. Return
size_t and instead throw a named CompressionBufferTooSmallException (deriving
from JFJochException, Compression category) when the output would not fit the
destination buffer.
The receiver catches it explicitly and drops just that frame, as before; the
offline/GetCompressedImage path uses a worst-case buffer so it never throws.
Add a test that a too-small destination throws and a worst-case buffer does not.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JFJochBitShuffleCompressor::Compress now takes a dest_size and returns a
negative value when the compressed output would not fit, instead of writing
past the destination buffer. The check is lazy: before each block it verifies
the remaining space still covers that block's worst case (mirrored by the new
MaxCompressedBlockSize helper, consistent with MaxCompressedSize so a dest
sized to MaxCompressedSize never fails). On overflow the dest content is
undefined - no rescue.
The receiver uses this to compress directly into the writer buffer slot and
drop just the oversized frame instead of pre-reserving the full worst-case
image size next to the per-image CBOR metadata.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A serial-crystallography run on a detector with a large converted geometry
(JF17T16, modules tiled vertically + horizontally) aborted with
"Array out of bounds (Not enough memory to save image)". An indexed still on
such a detector predicts/integrates close to the kMaxReflections (10000) cap;
at ~170 B per serialized Reflection that is ~1.7 MB of per-image CBOR metadata,
which overflowed the fixed 1 MB the buffer slot reserved on top of the image.
The serialization guard then threw and cancelled the whole run.
- Raise the per-image metadata headroom from 1 MB to 4 MB
(GetImageBufferLocationSize). The worst case - 10000 reflections + 2000 spots
(API max) + 65534 azimuthal bins - serializes to 2.78 MB, leaving margin while
staying negligible next to the multi-MB image slot.
- When metadata still does not fit, drop just that frame (log metadata/image/slot
sizes + recycle the slot) instead of aborting, in both the FPGA and Lite
receivers.
- Add a regression test asserting the worst-case metadata fits the headroom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
* jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin.
* jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins.
Reviewed-on: #60
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Improve performance of preview JPEG image generator at receiver startup (saving about 150 ms on measurement start for 16M)
Reviewed-on: #54
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times
* frontend: Fix selection of FFTW in indexing settings
Reviewed-on: #51
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection
* jfjoch_broker: Minor bug fixes for rare conditions
Reviewed-on: #50
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Better track time for each operation in the processing stack
* jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress)
* jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress)
* jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0)
Reviewed-on: #46
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS)
* jfjoch_viewer: Fix calibration (pedestal) images being open flipped
* jfjoch_process: Add space group detection (EXPERIMENTAL)
Reviewed-on: #39
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Default EIGER readout time is 20 microseconds
* jfjoch_broker: Multiple improvements regarding performance
* jfjoch_broker: Image buffer allows to track frames in preparation and sending
* jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer
* jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets
* jfjoch_writer: Fixes regarding properly closing files in long data collections
* jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code
Reviewed-on: #34
This is an UNSTABLE release and not recommended for production use (please use rc.11 instead).
* jfjoch_broker: Experimental rotation (3D) indexing
* jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values
Reviewed-on: #18
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Fable 5