New CompressionAlgorithm that emits a standard Zstandard frame: zero/0xFF runs
become RLE_Blocks (like BSHUF_ZSTD_RLE) and literal regions become
Compressed_Blocks with per-block adaptive Huffman literals and no sequences
(Number_of_Sequences=0). Short runs are absorbed into the literal stream;
incompressible literals fall back to Raw_Blocks so the worst case stays within
ZSTD_compressBound.
The Huffman tree + bitstream are produced by zstd's own HUF_compress{1,4}X_repeat
(the same calls ZSTD_compressLiterals uses); only the frame/block/literals-section
framing is hand-written, with comments citing zstd_compression_format.md so it can
be checked clause by clause. Output decodes with stock ZSTD_decompress, so no
reader changes are needed (decode routes like BSHUF_ZSTD).
On sparse diffraction this gives ~12% smaller files than bitshuffle/LZ4 at about
the same end-to-end speed, sitting between LZ4 and full ZSTD; for maximum ratio
use BSHUF_ZSTD. Robust on any input: tests round-trip pure zeros, Poisson(10),
Mersenne-Twister noise (checked against the size bound), an extreme-sparsity mask,
and a real lyso image through stock ZSTD_decompress.
API: exposed as "bszstd_rlehuf"; regenerate the Python/TS clients (update_version.sh)
to surface the new value there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compress() and FrameTransformation::CompressImage() returned int64_t with a
negative value meaning "did not fit". That is a footgun: the negative result
silently converts to a huge size_t if a caller forgets to check it. Return
size_t and instead throw a named CompressionBufferTooSmallException (deriving
from JFJochException, Compression category) when the output would not fit the
destination buffer.
The receiver catches it explicitly and drops just that frame, as before; the
offline/GetCompressedImage path uses a worst-case buffer so it never throws.
Add a test that a too-small destination throws and a worst-case buffer does not.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the fixed-element DefaultBlockSize with a byte target divided by
elem_size to get the block element count, so the per-block working set (and
thus cache behaviour) stays constant across pixel bit depths instead of halving
from 8- to 16- to 32-bit. The target is per-algorithm, following the measured
sweet spots on sparse data: LZ4 wants a small, cache-resident block for
throughput (16 kB), ZSTD/RLE want a large block for ratio (128 kB). The gap is
widest on extreme-sparsity inputs such as the uint32 pixel_mask, where
large-block ZSTD reaches 100-1800x vs ~160x for LZ4.
The block size is read back per-dataset from the bitshuffle stream header
(block_size = header_bytes / elem_size) and the HDF5 filter params, so the
decompressor and external readers (XDS/Neggia/Durin/CrystFEL) need no change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On sparse lyso frames the larger block improves compression ratio across all
bshuf algorithms (16-bit data): ZSTD 8.58 -> 9.30, LZ4 7.38 -> 7.58, RLE
6.82 -> 6.90. 16384 captures most of the gain available from even larger
blocks (ZSTD tops out ~9.55 at 65536) while staying close to the cache sweet
spot: the cheap codecs (LZ4, RLE) peak in throughput once a block's working
set fits L1d (~4096 elem here), so very large blocks trade real throughput for
diminishing ratio - and that penalty is worse on the Xeon Gold/Platinum
production hosts (smaller private L2, shared-L3 contention under many parallel
compression threads).
The block size is stored per-dataset in the bitshuffle HDF5 filter params, so
existing readers (XDS/Neggia/Durin/CrystFEL) stay compatible.
Move the per-block bitshuffle scratch off the inline member array onto a
lazily-sized heap vector, like tmp_space, so the block size no longer bloats
every stack-allocated compressor (incl. the transient ones in
CBORStream2Serializer).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JFJochBitShuffleCompressor::Compress now takes a dest_size and returns a
negative value when the compressed output would not fit, instead of writing
past the destination buffer. The check is lazy: before each block it verifies
the remaining space still covers that block's worst case (mirrored by the new
MaxCompressedBlockSize helper, consistent with MaxCompressedSize so a dest
sized to MaxCompressedSize never fails). On overflow the dest content is
undefined - no rescue.
The receiver uses this to compress directly into the writer buffer slot and
drop just the oversized frame instead of pre-reserving the full worst-case
image size next to the per-image CBOR metadata.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)
Reviewed-on: #43
leonarski_fandClaude Opus 4.8