v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

tests/DiffractionExperimentTest.cpp

@@ -1077,4 +1077,105 @@ TEST_CASE("DiffractionExperiment_PedestalRun","[DiffractionExperiment]") {
     x.Mode(DetectorMode::PedestalG2);
     REQUIRE(x.IsPedestalRun());
     REQUIRE(!x.IsApplyPixelMask());
-}
+}
+TEST_CASE("DiffractionExperiment_ImagesPerFile_Defaults", "[DiffractionExperiment]") {
+    // images_per_file is optional in the settings, because at setup time the acquisition it should
+    // suit is not necessarily known. GetImagesPerFile() is the single place that resolves it, and
+    // everything downstream (receiver, pusher, puller, writer) requires the answer to be a fixed
+    // non-zero number - so every branch below must produce one.
+    const GoniometerAxis rotation("omega", 0.0f, 0.1f, {-1, 0, 0}, {});
+    const GoniometerAxis stationary("omega", 0.0f, 0.0f, {-1, 0, 0}, {});
+
+    SECTION("taken literally when given") {
+        DiffractionExperiment x(DetJF4M());
+        x.ImagesPerTrigger(5000).NumTriggers(1).Goniometer(rotation).ImagesPerFile(250);
+        CHECK(x.GetImagesPerFile() == 250);
+    }
+
+    SECTION("stills default to 1000") {
+        DiffractionExperiment x(DetJF4M());
+        x.ImagesPerTrigger(50000).NumTriggers(1);
+        CHECK(x.GetImagesPerFile() == DEFAULT_IMAGES_PER_FILE);
+    }
+
+    SECTION("a rotation sweep goes into one file") {
+        DiffractionExperiment x(DetJF4M());
+        x.ImagesPerTrigger(3600).NumTriggers(1).Goniometer(rotation);
+        CHECK(x.GetImagesPerFile() == 3600);
+    }
+
+    SECTION("a long rotation sweep is split again") {
+        DiffractionExperiment x(DetJF4M());
+        x.ImagesPerTrigger(ROTATION_SINGLE_FILE_IMAGE_LIMIT + 1).NumTriggers(1).Goniometer(rotation);
+        CHECK(x.GetImagesPerFile() == DEFAULT_IMAGES_PER_FILE);
+    }
+
+    SECTION("a goniometer that does not turn is not a rotation sweep") {
+        DiffractionExperiment x(DetJF4M());
+        x.ImagesPerTrigger(3600).NumTriggers(1).Goniometer(stationary);
+        CHECK(x.GetImagesPerFile() == DEFAULT_IMAGES_PER_FILE);
+    }
+
+    SECTION("a grid scan splits on whole fast-axis rows") {
+        DiffractionExperiment x(DetJF4M());
+        x.ImagesPerTrigger(10000).NumTriggers(1)
+         .GridScan(GridScanSettings(30, 10.0f, 10.0f, false, false));
+        const int64_t images_per_file = x.GetImagesPerFile();
+        CHECK(images_per_file % 30 == 0);
+        CHECK(images_per_file >= DEFAULT_IMAGES_PER_FILE);
+        CHECK(images_per_file < DEFAULT_IMAGES_PER_FILE + 30);
+    }
+
+    SECTION("a single self-contained file holds the whole run") {
+        DiffractionExperiment x(DetJF4M());
+        x.ImagesPerTrigger(5000).NumTriggers(1)
+         .SetFileWriterFormat(FileWriterFormat::NXmxIntegrated).ImagesPerFile(250);
+        CHECK(x.GetImagesPerFile() == 5000);
+    }
+}
+
+// The saturation limit is exclusive inside Jungfraujoch and inclusive in the file, so the two sides
+// must convert. They stopped agreeing once, and the value then lost a count on every
+// write-read-write cycle; these pin the pairing and what each stored type declares.
+TEST_CASE("DiffractionExperiment_SaturationAndErrorMarkers", "[DiffractionExperiment]") {
+    SECTION("the conversions are inverses") {
+        for (int64_t limit : {256L, 65536L, 32768L, 2147483648L}) {
+            CHECK(SaturationLimitFromValue(SaturationValueFromLimit(limit)) == limit);
+            CHECK(SaturationValueFromLimit(limit) == limit - 1);
+        }
+    }
+
+    struct Case { int64_t bits; bool is_signed; int64_t saturation; int64_t underload; int64_t error; };
+    // saturation_value is the highest value that is still a count; error is the marker actually
+    // stored in the pixels. Unsigned images reserve the top code, signed ones the bottom.
+    const std::vector<Case> cases = {
+            {16, false, UINT16_MAX - 1,  0,             UINT16_MAX},
+            {32, false, UINT32_MAX - 1,  0,             UINT32_MAX},
+            {16, true,  INT16_MAX - 1,   INT16_MIN + 1, INT16_MIN},
+            {32, true,  INT32_MAX - 1,   INT32_MIN + 1, INT32_MIN},
+    };
+
+    for (const auto &c: cases) {
+        DiffractionExperiment x(DetJF(1));
+        x.BitDepthImage(c.bits).PixelSigned(c.is_signed);
+        StartMessage message;
+        x.FillMessage(message);
+
+        CAPTURE(c.bits, c.is_signed);
+        CHECK(message.saturation_value == c.saturation);
+        CHECK(message.saturation_value == SaturationValueFromLimit(x.GetSaturationLimit()));
+        REQUIRE(message.underload_value.has_value());
+        CHECK(message.underload_value.value() == c.underload);
+        // The marker the pixels really carry - GetUnderflow() was written here, and it is -1 for an
+        // unsigned image, which matches no pixel such an image can hold.
+        REQUIRE(message.error_value.has_value());
+        CHECK(message.error_value.value() == c.error);
+        CHECK(message.error_value.value() == x.GetImageFillValue());
+
+        // DIALS remaps the top two codes whenever bit_depth_readout is present, without looking at
+        // the pixel type. For an unsigned image they fall below underload_value and are masked, as
+        // intended; for a signed one they land inside the trusted range and a saturated pixel would
+        // be integrated as a count of -2. Unsigned 32-bit cannot be read at all without the field.
+        CHECK(message.bit_depth_readout.has_value() == !c.is_signed);
+    }
+}