v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00
parent 538f3504d3
commit 4dc2534dbf
287 changed files with 9146 additions and 2340 deletions
@@ -5,6 +5,7 @@
 #include <type_traits>

 #include "BSLZ4DecoderGPU.h"
+#include "BSLZ4DecodeWarp.h"
 #include "../../common/JFJochException.h"
 #include "../../compression/JFJochDecompress.h"   // BSHUF_BLOCKED_MULT and the container layout

@@ -14,20 +15,8 @@ namespace {
            throw JFJochException(JFJochExceptionCategory::GPUCUDAError, cudaGetErrorString(val));
    }

-    // One WARP per LZ4 block. Every lane runs the same sequence parser over the same bytes - a
-    // broadcast read, so no divergence - and the literal and match copies are split across the 32
-    // lanes so the stores coalesce. One thread per block instead has each thread streaming its own
-    // 8 kB region, which coalesces not at all and measured 13x slower.
-    //
-    // Because the lanes cooperate on the copies, a match can source bytes that OTHER lanes wrote in
-    // an earlier sequence. Since Volta that needs an explicit __syncwarp() - implicit reconvergence
-    // is not part of the programming model - so there is one after every copy loop. The full mask is
-    // correct: the early return and every break test warp-uniform values, so lanes never diverge
-    // permanently.
-    //
-    // Bounds: every read is clamped against iend and every write against oend, so a malformed or
-    // corrupt payload cannot walk off either buffer. It can still stop early, which leaves the block
-    // short; lane 0 flags that at the end and the host turns it into an exception.
+    // One CUDA block per 256/32 = 8 LZ4 blocks; the parser itself is lz4_decode_block_warp, shared
+    // with the fused decode+preprocess kernel so the two cannot decode a chunk differently.
    __global__ void lz4_decode_blocks(const uint8_t *__restrict__ src,
                                      const BSLZ4BlockDesc *__restrict__ desc,
                                      uint8_t *__restrict__ dst,
@@ -37,105 +26,11 @@ namespace {
        const int b = (blockIdx.x * blockDim.x + threadIdx.x) >> 5;
        if (b >= nblocks) return;

-        const uint8_t *ip = src + desc[b].in_off;
-        const uint8_t *const iend = ip + desc[b].in_len;
+        const uint8_t *const ip = src + desc[b].in_off;
        uint8_t *const obase = dst + desc[b].out_off;
-        uint8_t *op = obase;
-        uint8_t *const oend = obase + desc[b].nelem * elem_size;
-        bool malformed = false;
-
-        while (ip < iend) {
-            const uint32_t token = *ip++;
-            uint32_t litlen = token >> 4;
-            if (litlen == 15) {
-                // read_variable_length(&ip, iend - RUN_MASK, initial_check=1) in the reference: the
-                // chain may not start within, nor run into, the last RUN_MASK (15) input bytes. A
-                // valid stream never does - the literals it counts have to follow it - so a chain
-                // that reaches there is corruption, and this is the only place it shows up.
-                if ((size_t)(iend - ip) <= 15) { malformed = true; break; }
-                uint32_t s;
-                do {
-                    s = *ip++;
-                    litlen += s;
-                    if ((size_t)(iend - ip) < 15) { malformed = true; break; }
-                } while (s == 255);
-                if (malformed) break;
-            }
-            if (litlen) {
-                // Clamped by the INPUT as well as the output: a corrupt litlen must not read past the
-                // end of this block's payload or write past the end of the block. Clamping keeps the
-                // kernel in bounds; needing to clamp at all means the stream is not decodable, which
-                // is what the reference reports as an error, so record it.
-                if (litlen > (uint32_t)(oend - op) || litlen > (uint32_t)(iend - ip))
-                    malformed = true;
-                const uint32_t n = min(min(litlen, (uint32_t)(oend - op)), (uint32_t)(iend - ip));
-                for (uint32_t i = lane; i < n; i += 32) op[i] = ip[i];
-                __syncwarp();
-                op += n; ip += litlen;
-            }
-
-            // LZ4's parsing restrictions: an encoder may not leave a match within MFLIMIT (12) bytes
-            // of the end of the block, nor fewer than 2+1+LASTLITERALS (8) input bytes after a
-            // literal run that is not the last one. So once either limit is reached this can ONLY be
-            // the final sequence, and the final sequence must consume the payload exactly. The
-            // reference applies this whether or not the run was empty, which is why the test sits
-            // outside the copy - a zero-length literal run near the end is just as illegal.
-            if ((size_t)(oend - op) < 12 || (size_t)(iend - ip) < 8) {
-                malformed = (ip != iend) || (op != oend);
-                break;   // necessarily EOF
-            }
-            if (iend - ip < 2) break;   // last sequence carries literals only
-
-            const uint32_t offset = (uint32_t)ip[0] | ((uint32_t)ip[1] << 8);
-            ip += 2;
-            uint32_t matchlen = token & 0x0F;
-            if (matchlen == 15) {
-                // read_variable_length(&ip, iend - LASTLITERALS + 1, initial_check=0): bounded by the
-                // last 4 input bytes rather than 15, and with no check before the first read.
-                uint32_t s;
-                do {
-                    s = *ip++;
-                    matchlen += s;
-                    if ((size_t)(iend - ip) < 4) { malformed = true; break; }
-                } while (s == 255);
-                if (malformed) break;
-            }
-            matchlen += 4;   // minmatch
-
-            if (offset == 0 || offset > (uint32_t)(op - obase)) { malformed = true; break; }
-            const uint8_t *mp = op - offset;
-            // A match may reach the end of the block but never past it - the reference treats an
-            // overrun as an error rather than truncating, and so must this.
-            if (matchlen > (uint32_t)(oend - op))
-                malformed = true;
-            const uint32_t n = min(matchlen, (uint32_t)(oend - op));
-            if (offset >= matchlen) {
-                for (uint32_t i = lane; i < n; i += 32) op[i] = mp[i];
-            } else {
-                // An overlapping match is a pattern of period `offset`. mp[0..offset-1] all lie
-                // before op and are already final, so each output byte can be sourced from them
-                // independently - which keeps this parallel rather than a serial byte loop. Long
-                // zero runs in sparse detector data arrive here with offset == 1, and a runtime
-                // modulo is an emulated division, so the two cheap cases are peeled off first.
-                if (offset == 1) {
-                    const uint8_t v = mp[0];
-                    for (uint32_t i = lane; i < n; i += 32) op[i] = v;
-                } else if ((offset & (offset - 1)) == 0) {
-                    const uint32_t m = offset - 1;
-                    for (uint32_t i = lane; i < n; i += 32) op[i] = mp[i & m];
-                } else {
-                    for (uint32_t i = lane; i < n; i += 32) op[i] = mp[i % offset];
-                }
-            }
-            __syncwarp();
-            op += n;
-        }
-
-        // A block must decode to exactly its declared length AND consume exactly its payload. Both
-        // are conditions LZ4_decompress_safe reports to the host path, and both are needed: a corrupt
-        // stream can land on the right output length while leaving input over, or run its input out
-        // early. Either way the bytes are not the ones that were compressed.
-        if (lane == 0 && (malformed || op != oend || ip != iend))
+        const bool ok = lz4_decode_block_warp(ip, ip + desc[b].in_len,
+                                              obase, obase + desc[b].nelem * elem_size, lane);
+        if (lane == 0 && !ok)
            atomicExch(status, 1u);
    }

@@ -223,7 +118,6 @@ bool BSLZ4DecoderGPU::Supports(const CompressedImage &image) {
 BSLZ4DecoderGPU::BSLZ4DecoderGPU(size_t in_max_uncompressed_bytes, std::shared_ptr<CudaStream> in_stream)
    : stream(std::move(in_stream)),
      max_uncompressed_bytes(in_max_uncompressed_bytes) {
-    gpu_shuffled = CudaDevicePtr<uint8_t>(max_uncompressed_bytes);
    gpu_status = CudaDevicePtr<uint32_t>(1);
    host_status = CudaHostPtr<uint32_t>(1);
    // The compressed buffer and the descriptors are grown to fit the first image instead of being
@@ -231,6 +125,20 @@ BSLZ4DecoderGPU::BSLZ4DecoderGPU(size_t in_max_uncompressed_bytes, std::shared_p
    // tens; sizing this from the UNCOMPRESSED size cost ~73 MB per worker to hold ~4 MB.
 }

+// gpu_shuffled holds a whole uncompressed frame - 72 MB at 18 Mpx, per worker - and only the
+// DecodeShuffled() route ever writes it. That route is taken when a bitshuffle block is too large for
+// the fused kernel, which neither writer this pipeline reads produces, so on a real frame the buffer
+// is never touched. So allocate it the first time it is actually asked for, at the size the caller
+// declared, rather than in the constructor.
+void BSLZ4DecoderGPU::EnsureUncompressedCapacity(size_t bytes) {
+    if (gpu_shuffled.get() && bytes <= max_uncompressed_bytes)
+        return;
+    const size_t want = std::max(bytes, max_uncompressed_bytes);
+    cuda_err(cudaStreamSynchronize(*stream));
+    gpu_shuffled = CudaDevicePtr<uint8_t>(want);
+    max_uncompressed_bytes = want;
+}
+
 void BSLZ4DecoderGPU::EnsureCompressedCapacity(size_t bytes) {
    if (bytes <= compressed_capacity)
        return;
@@ -251,7 +159,17 @@ void BSLZ4DecoderGPU::EnsureBlockCapacity(size_t nblocks) {
    max_blocks = want;
 }

-BSLZ4ShuffledImage BSLZ4DecoderGPU::DecodeShuffled(const CompressedImage &image) {
+uint32_t BSLZ4DecoderGPU::BlockBytes(const CompressedImage &image) {
+    if (image.GetCompressedSize() < 12)
+        return 0;   // not a chunk at all; the caller's decode reports it properly
+    return be32(image.GetCompressed() + 8);
+}
+
+// Everything up to and including getting the chunk onto the device: the container scan, which is the
+// only part that has to happen on the host, the capacity checks, and the uploads. What decodes the
+// blocks is left to the caller of this - either the LZ4 kernel below or the fused kernel in the
+// preprocessor - because that is the whole difference between the two routes.
+BSLZ4ShuffledImage BSLZ4DecoderGPU::PrepareChunk(const CompressedImage &image) {
    const uint8_t *src = image.GetCompressed();
    const size_t clen = image.GetCompressedSize();
    const size_t elem_size = elem_size_of(image.GetMode());
@@ -259,8 +177,6 @@ BSLZ4ShuffledImage BSLZ4DecoderGPU::DecodeShuffled(const CompressedImage &image)

    if (clen < 12)
        throw JFJochException(JFJochExceptionCategory::Compression, "bslz4 chunk shorter than its header");
-    if (total_bytes > max_uncompressed_bytes)
-        throw JFJochException(JFJochExceptionCategory::Compression, "bslz4 image larger than the decoder was sized for");
    if (be64(src) != total_bytes)
        throw JFJochException(JFJochExceptionCategory::Compression, "bslz4 header size does not match the image");

@@ -325,34 +241,56 @@ BSLZ4ShuffledImage BSLZ4DecoderGPU::DecodeShuffled(const CompressedImage &image)
    cuda_err(cudaMemcpyAsync(gpu_compressed.get(), src, clen, cudaMemcpyHostToDevice, *stream));

    const int nb = static_cast<int>(nblk);
-    if (nb > 0) {
+    if (nb > 0)
        cuda_err(cudaMemcpyAsync(gpu_desc.get(), host_desc.get(), nblk * sizeof(BSLZ4BlockDesc),
                                 cudaMemcpyHostToDevice, *stream));
-        lz4_decode_blocks<<<(nb * 32 + 255) / 256, 256, 0, *stream>>>(
-                gpu_compressed.get(), gpu_desc.get(), gpu_shuffled.get(), gpu_status.get(),
-                nb, static_cast<uint32_t>(elem_size));
-        cuda_err(cudaGetLastError());
-    }
-    cuda_err(cudaMemcpyAsync(host_status.get(), gpu_status.get(), sizeof(uint32_t),
-                             cudaMemcpyDeviceToHost, *stream));
-    // Stop the clock here rather than after the un-transpose: getting the chunk onto the device and
-    // LZ4-decoding it is the part that replaced the host decompression, and it is the same work on
-    // both the raw-bytes path and the fused one, where the un-transpose is inseparable from
-    // preprocessing and is reported with it.
-    cuda_err(cudaEventRecord(decode_stop, *stream));
-    decode_timed = true;

    BSLZ4ShuffledImage ret;
-    ret.shuffled = gpu_shuffled.get();
+    ret.compressed = gpu_compressed.get();
    ret.desc = gpu_desc.get();
+    ret.status = gpu_status.get();
    ret.nblocks = nb;
    ret.elem_size = static_cast<uint32_t>(elem_size);
+    ret.block_bytes = block_bytes;
    ret.tail_elems = static_cast<uint32_t>(leftover_bytes / elem_size);
    ret.tail_elem0 = static_cast<uint32_t>(out_off / elem_size);
    ret.tail_src = leftover_bytes > 0 ? gpu_compressed.get() + off : nullptr;
    return ret;
 }

+BSLZ4ShuffledImage BSLZ4DecoderGPU::DecodeShuffled(const CompressedImage &image) {
+    EnsureUncompressedCapacity(image.GetUncompressedSize());
+    BSLZ4ShuffledImage ret = PrepareChunk(image);
+    if (ret.nblocks > 0) {
+        lz4_decode_blocks<<<(ret.nblocks * 32 + 255) / 256, 256, 0, *stream>>>(
+                ret.compressed, ret.desc, gpu_shuffled.get(), ret.status,
+                ret.nblocks, ret.elem_size);
+        cuda_err(cudaGetLastError());
+    }
+    ret.shuffled = gpu_shuffled.get();
+    QueueDecodeStatus();
+    // Stop the clock here rather than after the un-transpose: getting the chunk onto the device and
+    // LZ4-decoding it is the part that replaced the host decompression.
+    cuda_err(cudaEventRecord(decode_stop, *stream));
+    decode_timed = true;
+    return ret;
+}
+
+BSLZ4ShuffledImage BSLZ4DecoderGPU::UploadCompressed(const CompressedImage &image) {
+    BSLZ4ShuffledImage ret = PrepareChunk(image);
+    // All this route decodes on its own is the PCIe upload, so that is what the clock brackets. The
+    // LZ4 pass happens inside the caller's kernel, inseparably from the un-transpose and the
+    // preprocessing, and is reported with them.
+    cuda_err(cudaEventRecord(decode_stop, *stream));
+    decode_timed = true;
+    return ret;
+}
+
+void BSLZ4DecoderGPU::QueueDecodeStatus() {
+    cuda_err(cudaMemcpyAsync(host_status.get(), gpu_status.get(), sizeof(uint32_t),
+                             cudaMemcpyDeviceToHost, *stream));
+}
+
 void BSLZ4DecoderGPU::Decode(const CompressedImage &image, uint8_t *gpu_out) {
    const BSLZ4ShuffledImage s = DecodeShuffled(image);