v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

tests/BraggIntegrationEngineGPUTest.cpp

@@ -305,6 +305,61 @@ TEST_CASE("BraggIntegrationEngineGPU_MatchesCPU") {
     }
 }
 
+// The mask and the owner map are cleared by the run that marked them rather than at the start of the
+// next one, so a reused engine has to give the same answer as a fresh one. A first frame whose spots
+// are somewhere else entirely is what would show a leftover mark: a stale mask pixel is read as a
+// neighbour's signal and dropped from the background ring, a stale owner steals a pixel outright.
+TEST_CASE("BraggIntegrationEngineGPU_ReusedEngineMatchesFresh") {
+    if (get_gpu_count() == 0) {
+        WARN("No CUDA GPU present. Skipping BraggIntegrationEngineGPU_ReusedEngineMatchesFresh");
+        return;
+    }
+
+    for (OverlapMode ovl : {OverlapMode::Off, OverlapMode::Exclude}) {
+        const DiffractionExperiment experiment =
+                MakeExperiment(IntegratorMode::ProfileGaussian, std::nullopt, 4.0f, false, DetJF(2),
+                               0.0f, 0.0f, 0.0f, 0.0f, ovl);
+        const size_t width = experiment.GetXPixelsNum();
+        const size_t height = experiment.GetYPixelsNum();
+        const size_t npixel = experiment.GetPixelsNum();
+
+        // Two frames whose spot grids do not line up, so the first frame's marks fall on the second
+        // frame's background rings rather than back onto its own disks.
+        const Scene first = BuildScene(width, height, 47);
+        const Scene second = BuildScene(width, height, 60);
+        REQUIRE(first.predicted.size() > 60);
+        REQUIRE(second.predicted.size() > 60);
+
+        auto integrate = [&](BraggIntegrationEngineGPU &engine, const Scene &scene,
+                             const std::shared_ptr<CudaStream> &stream) {
+            ImagePreprocessorBufferGPU img(npixel);
+            for (size_t i = 0; i < npixel; ++i) img[i] = scene.image[i];
+            REQUIRE(cudaMemcpyAsync(img.getGPUBuffer(), img.getBuffer().data(),
+                                    npixel * sizeof(int32_t), cudaMemcpyHostToDevice, *stream) == cudaSuccess);
+            return engine.Run(img, scene.predicted, scene.predicted.size(), 7);
+        };
+
+        auto stream_fresh = std::make_shared<CudaStream>();
+        BraggIntegrationEngineGPU fresh(experiment, stream_fresh);
+        const auto out_fresh = integrate(fresh, second, stream_fresh);
+
+        auto stream_reused = std::make_shared<CudaStream>();
+        BraggIntegrationEngineGPU reused(experiment, stream_reused);
+        integrate(reused, first, stream_reused);
+        const auto out_reused = integrate(reused, second, stream_reused);
+
+        INFO("overlap mode " << static_cast<int>(ovl));
+        REQUIRE(out_reused.size() == out_fresh.size());
+        for (size_t i = 0; i < out_fresh.size(); ++i) {
+            INFO("reflection " << i);
+            CHECK(out_reused[i].h == out_fresh[i].h);
+            CHECK(out_reused[i].I == out_fresh[i].I);
+            CHECK(out_reused[i].sigma == out_fresh[i].sigma);
+            CHECK(out_reused[i].bkg == out_fresh[i].bkg);
+        }
+    }
+}
+
 // Hidden ([.]) benchmark: the raison d'etre of the GPU port is < 2 ms/frame (vs ~142 ms on the CPU
 // for ProfileIntegrate2D). Run explicitly with:  ./jfjoch_test "[bragg_bench]"
 TEST_CASE("BraggIntegrationEngineGPU_Benchmark", "[.][bragg_bench]") {