Compressor: scale block size to a per-algorithm byte target
Replace the fixed-element DefaultBlockSize with a byte target divided by elem_size to get the block element count, so the per-block working set (and thus cache behaviour) stays constant across pixel bit depths instead of halving from 8- to 16- to 32-bit. The target is per-algorithm, following the measured sweet spots on sparse data: LZ4 wants a small, cache-resident block for throughput (16 kB), ZSTD/RLE want a large block for ratio (128 kB). The gap is widest on extreme-sparsity inputs such as the uint32 pixel_mask, where large-block ZSTD reaches 100-1800x vs ~160x for LZ4. The block size is read back per-dataset from the bitshuffle stream header (block_size = header_bytes / elem_size) and the HDF5 filter params, so the decompressor and external readers (XDS/Neggia/Durin/CrystFEL) need no change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -140,7 +140,9 @@ void HDF5DataFile::CreateFile(const DataMessage& msg, std::shared_ptr<HDF5File>
|
||||
xpixel = msg.image.GetWidth();
|
||||
ypixel = msg.image.GetHeight();
|
||||
|
||||
dcpl.SetCompression(msg.image.GetCompressionAlgorithm(), JFJochBitShuffleCompressor::DefaultBlockSize);
|
||||
dcpl.SetCompression(msg.image.GetCompressionAlgorithm(),
|
||||
JFJochBitShuffleCompressor::BlockSize(msg.image.GetCompressionAlgorithm(),
|
||||
msg.image.GetByteDepth()));
|
||||
dcpl.SetChunking( {1, ypixel, xpixel});
|
||||
|
||||
H5Pset_fill_time(dcpl.GetID(), H5D_FILL_TIME_NEVER);
|
||||
|
||||
Reference in New Issue
Block a user