mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests 1 Actions 3 Packages Projects Releases 63 Wiki Activity

Compressor: scale block size to a per-algorithm byte target

Browse Source 
Replace the fixed-element DefaultBlockSize with a byte target divided by
elem_size to get the block element count, so the per-block working set (and
thus cache behaviour) stays constant across pixel bit depths instead of halving
from 8- to 16- to 32-bit. The target is per-algorithm, following the measured
sweet spots on sparse data: LZ4 wants a small, cache-resident block for
throughput (16 kB), ZSTD/RLE want a large block for ratio (128 kB). The gap is
widest on extreme-sparsity inputs such as the uint32 pixel_mask, where
large-block ZSTD reaches 100-1800x vs ~160x for LZ4.

The block size is read back per-dataset from the bitshuffle stream header
(block_size = header_bytes / elem_size) and the HDF5 filter params, so the
decompressor and external readers (XDS/Neggia/Durin/CrystFEL) need no change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
leonarski_f
2026-06-27 07:56:08 +02:00
co-authored by Claude Opus 4.8
parent aadba5b343
commit 74584c23ac
5 changed files with 51 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/FrameTransformationTest.cpp
+14 -7
View File
@@ -97,7 +97,8 @@ TEST_CASE("FrameTransformation_Raw_NoCompression_bshuf_lz4" ,"") {
auto image = transformation.GetCompressedImage();
REQUIRE(read_be64(image.GetCompressed()) == experiment.GetPixelsNum() * experiment.GetByteDepthImage());
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::DefaultBlockSize *
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::BlockSize(experiment.GetCompressionAlgorithm(),
experiment.GetByteDepthImage()) *
experiment.GetByteDepthImage());
std::vector<int16_t> output;
@@ -238,7 +239,8 @@ TEST_CASE("FrameTransformation_Converted_bshuf_lz4" ,"") {
auto image = transformation.GetCompressedImage();
REQUIRE(read_be64(image.GetCompressed()) == experiment.GetPixelsNum() * experiment.GetByteDepthImage());
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::DefaultBlockSize *
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::BlockSize(experiment.GetCompressionAlgorithm(),
experiment.GetByteDepthImage()) *
experiment.GetByteDepthImage());
std::vector<uint16_t> output;
@@ -297,7 +299,8 @@ TEST_CASE("FrameTransformation_Converted_bshuf_zstd" ,"") {
auto image = transformation.GetCompressedImage();
REQUIRE(read_be64(image.GetCompressed()) == experiment.GetPixelsNum() * experiment.GetByteDepthImage());
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::DefaultBlockSize *
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::BlockSize(experiment.GetCompressionAlgorithm(),
experiment.GetByteDepthImage()) *
experiment.GetByteDepthImage());
std::vector<uint16_t> output;
@@ -356,7 +359,8 @@ TEST_CASE("FrameTransformation_Converted_bshuf_zstd_rle" ,"") {
auto image = transformation.GetCompressedImage();
REQUIRE(read_be64(image.GetCompressed()) == experiment.GetPixelsNum() * experiment.GetByteDepthImage());
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::DefaultBlockSize *
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::BlockSize(experiment.GetCompressionAlgorithm(),
experiment.GetByteDepthImage()) *
experiment.GetByteDepthImage());
std::vector<uint16_t> output;
@@ -418,7 +422,8 @@ TEST_CASE("FrameTransformation_Converted_bshuf_zstd_32bit" ,"") {
auto image = transformation.GetCompressedImage();
REQUIRE(read_be64(image.GetCompressed()) == experiment.GetPixelsNum() * experiment.GetByteDepthImage());
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::DefaultBlockSize *
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::BlockSize(experiment.GetCompressionAlgorithm(),
experiment.GetByteDepthImage()) *
experiment.GetByteDepthImage());
std::vector<uint32_t> output;
@@ -481,7 +486,8 @@ TEST_CASE("FrameTransformation_Converted_bshuf_zstd_8bit" ,"") {
auto image = transformation.GetCompressedImage();
REQUIRE(read_be64(image.GetCompressed()) == experiment.GetPixelsNum() * experiment.GetByteDepthImage());
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::DefaultBlockSize *
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::BlockSize(experiment.GetCompressionAlgorithm(),
experiment.GetByteDepthImage()) *
experiment.GetByteDepthImage());
std::vector<int8_t> output;
@@ -548,7 +554,8 @@ TEST_CASE("FrameTransformation_Converted_bshuf_zstd_unsigned_16bit" ,"") {
auto image = transformation.GetCompressedImage();
REQUIRE(read_be64(image.GetCompressed()) == experiment.GetPixelsNum() * experiment.GetByteDepthImage());
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::DefaultBlockSize *
REQUIRE(read_be32(image.GetCompressed() + 8) == JFJochBitShuffleCompressor::BlockSize(experiment.GetCompressionAlgorithm(),
experiment.GetByteDepthImage()) *
experiment.GetByteDepthImage());
std::vector<uint16_t> output;