New CompressionAlgorithm that emits a standard Zstandard frame: zero/0xFF runs
become RLE_Blocks (like BSHUF_ZSTD_RLE) and literal regions become
Compressed_Blocks with per-block adaptive Huffman literals and no sequences
(Number_of_Sequences=0). Short runs are absorbed into the literal stream;
incompressible literals fall back to Raw_Blocks so the worst case stays within
ZSTD_compressBound.
The Huffman tree + bitstream are produced by zstd's own HUF_compress{1,4}X_repeat
(the same calls ZSTD_compressLiterals uses); only the frame/block/literals-section
framing is hand-written, with comments citing zstd_compression_format.md so it can
be checked clause by clause. Output decodes with stock ZSTD_decompress, so no
reader changes are needed (decode routes like BSHUF_ZSTD).
On sparse diffraction this gives ~12% smaller files than bitshuffle/LZ4 at about
the same end-to-end speed, sitting between LZ4 and full ZSTD; for maximum ratio
use BSHUF_ZSTD. Robust on any input: tests round-trip pure zeros, Poisson(10),
Mersenne-Twister noise (checked against the size bound), an extreme-sparsity mask,
and a real lyso image through stock ZSTD_decompress.
API: exposed as "bszstd_rlehuf"; regenerate the Python/TS clients (update_version.sh)
to surface the new value there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compress() and FrameTransformation::CompressImage() returned int64_t with a
negative value meaning "did not fit". That is a footgun: the negative result
silently converts to a huge size_t if a caller forgets to check it. Return
size_t and instead throw a named CompressionBufferTooSmallException (deriving
from JFJochException, Compression category) when the output would not fit the
destination buffer.
The receiver catches it explicitly and drops just that frame, as before; the
offline/GetCompressedImage path uses a worst-case buffer so it never throws.
Add a test that a too-small destination throws and a worst-case buffer does not.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JFJochBitShuffleCompressor::Compress now takes a dest_size and returns a
negative value when the compressed output would not fit, instead of writing
past the destination buffer. The check is lazy: before each block it verifies
the remaining space still covers that block's worst case (mirrored by the new
MaxCompressedBlockSize helper, consistent with MaxCompressedSize so a dest
sized to MaxCompressedSize never fails). On overflow the dest content is
undefined - no rescue.
The receiver uses this to compress directly into the writer buffer slot and
drop just the oversized frame instead of pre-reserving the full worst-case
image size next to the per-image CBOR metadata.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
leonarski_fandClaude Opus 4.8