v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

tests/DiffractionGeometryTest.cpp

@@ -547,4 +547,87 @@ TEST_CASE("DiffractionGeometry_PONI_matrix_consistency") {
     CHECK(S0.x == Catch::Approx(0.0f));
     CHECK(S0.y == Catch::Approx(0.0f));
     CHECK(S0.z == Catch::Approx(1.0f));
-}
+}
+// Cross-check of a TILTED detector against two independent implementations, pyFAI and DIALS/dxtbx.
+//
+// Every other geometry test here is either self-consistent (round trips) or exercises one angle at a
+// time. This one pins all three PONI angles at once, non-zero and of mixed sign, against reference
+// positions computed outside Jungfraujoch. That matters because the errors this guards against are
+// second order: a wrong composition order or a swapped axis is invisible unless two angles are
+// non-zero simultaneously, and a wrong pivot is invisible to anything that only checks directions.
+//
+// HOW TO REGENERATE THE NUMBERS
+//
+// pyFAI (`pip install pyFAI`), which is an independent implementation of the PONI convention:
+//
+//     from pyFAI.geometry import Geometry
+//     from pyFAI.detectors import Detector
+//     px = 75e-6
+//     det = Detector(pixel1=px, pixel2=px, max_shape=(2164, 2030))
+//     g = Geometry(dist=0.150, poni1=1275*px + px/2, poni2=1000*px + px/2,
+//                  rot1=0.05, rot2=+0.03, rot3=0.02,      # (+rot1, -rot2, +rot3); see WritePoniFile
+//                  detector=det, wavelength=1e-10)
+//     t3, t1, t2 = g.calc_pos_zyx(d1=[y], d2=[x])         # metres, pyFAI's own axes
+//     lab_mm = (t2*1e3, t1*1e3, t3*1e3)                   # pyFAI (t1,t2,t3) -> our (x,y,z)
+//
+// The half pixel in poni1/poni2 is the origin convention (docs/DETECTOR_GEOMETRY.md): our beam
+// centre is pixel-centred, pyFAI measures from the sensor edge.
+//
+// DIALS: write a master, patch this geometry into it, and read the panel back.
+//
+//     source /opt/dials-v3-27-0/dials_env.sh
+//     build/tools/jfjoch_hdf5_test <input.h5> -n1 -S -o g          # writes g_master.h5
+//     # with h5py, set /entry/instrument/detector/{beam_center_x,beam_center_y,distance},
+//     # transformations/{rot1,rot2,rot3}, and recompute transformations/translation - both its
+//     # magnitude and its @vector - as {bx*px, by*px, distance} normalised, since the writer
+//     # derives it from the beam centre and distance.
+//     p = ExperimentListFactory.from_filenames(['g_master.h5'])[0].detector[0]
+//     lab = p.get_origin() + x*px_mm*p.get_fast_axis() + y*px_mm*p.get_slow_axis()
+//
+// Use get_origin()/get_fast_axis()/get_slow_axis() as above, NOT get_pixel_lab_coord(): that applies
+// a parallax correction from the sensor thickness and material which DiffractionGeometry does not
+// model, and it costs ~0.1 mm at the detector edge - enough to look like a geometry error.
+//
+// DIALS reports in the imgCIF frame, which is ours turned 180 degrees about x (diag(1,-1,-1)) - a
+// proper rotation, not a mirror. The test applies that mapping, so it pins the frame relation too.
+TEST_CASE("DiffractionGeometry_Tilted_vs_PyFAI_and_DIALS", "[DiffractionGeometry]") {
+    DiffractionGeometry geom;
+    geom.BeamX_pxl(1000.0f).BeamY_pxl(1275.0f).DetectorDistance_mm(150.0f)
+        .PixelSize_mm(0.075f).Wavelength_A(1.0f)
+        .PoniRot1_rad(0.05f).PoniRot2_rad(-0.03f).PoniRot3_rad(0.02f);
+
+    struct Reference {
+        int x, y;
+        float pyfai[3];   // our frame:   x, y, z   [mm]
+        float dials[3];   // imgCIF frame: x, y, z  [mm]
+    };
+
+    const std::vector<Reference> reference = {
+        {    0,    0, {-69.399541334f, -98.819975327f, 150.623559791f},
+                      {-69.399544982f,  98.819979800f, -150.623559832f}},
+        { 2029, 2163, { 85.802269836f,  60.488193357f, 147.887421982f},
+                      { 85.802273560f, -60.488196450f, -147.887421893f}},
+        { 1000, 1275, {  7.405508016f,  -4.642730847f, 149.745128473f},
+                      {  7.405508016f,   4.642730847f, -149.745128473f}},
+        {  300, 1800, {-44.232874953f,  35.676608756f, 153.548927011f},
+                      {-44.232877406f, -35.676610671f, -153.548927191f}},
+        { 1700,  400, { 58.519162201f, -71.195011374f, 145.153948055f},
+                      { 58.519164629f,  71.195014535f, -145.153947836f}},
+    };
+
+    // 2 um, i.e. 1/37 of a pixel. The references agree with each other to ~4e-6 mm; the margin is
+    // set by float32 rounding in DiffractionGeometry and in the HDF5 file the DIALS values came from.
+    const double margin = 2e-3;
+
+    for (const auto &r: reference) {
+        const Coord lab = geom.LabCoord(static_cast<float>(r.x), static_cast<float>(r.y));
+
+        CHECK(lab.x == Catch::Approx(r.pyfai[0]).margin(margin));
+        CHECK(lab.y == Catch::Approx(r.pyfai[1]).margin(margin));
+        CHECK(lab.z == Catch::Approx(r.pyfai[2]).margin(margin));
+
+        CHECK(lab.x == Catch::Approx( r.dials[0]).margin(margin));
+        CHECK(lab.y == Catch::Approx(-r.dials[1]).margin(margin));
+        CHECK(lab.z == Catch::Approx(-r.dials[2]).margin(margin));
+    }
+}