v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

reader/HDF5MetadataSource.cpp

@@ -604,21 +604,76 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
         metadata.SourceName(master_file->GetString("/entry/source/name"));
         dataset->experiment.ImportInstrumentMetadata(metadata);
 
+        // The rotation axis is whatever the file calls it. The name is free-form throughout the API,
+        // the CBOR stream and the writer, so looking only for "omega" - as this did - read a sweep
+        // recorded as "phi" back as stills, silently. Prefer an axis that actually turns; fall back
+        // to a stationary one, which still says where the head was.
         if (master_file->Exists("/entry/sample/transformations")) {
-            if (master_file->Exists("/entry/sample/transformations/omega")) {
-                auto omega = ReadAxis(master_file.get(), "omega");
-                dataset->experiment.Goniometer(omega);
-            } else if (master_file->Exists("/entry/sample/grid_scan")) {
-                GridScanSettings grid(
-                    master_file->GetInt("/entry/sample/grid_scan/n_fast"),
-                    master_file->GetFloat("/entry/sample/grid_scan/step_x") * 1e6f,
-                    master_file->GetFloat("/entry/sample/grid_scan/step_y") * 1e6f,
-                    master_file->GetOptBool("/entry/sample/grid_scan/snake_scan").value_or(false),
-                    master_file->GetOptBool("/entry/sample/grid_scan/vertical_scan").value_or(false)
-                );
-                grid.ImageNum(number_of_images);
-                dataset->experiment.GridScan(grid);
+            // A Smargon chi/phi is tagged with equipment_component - it is a head position, not the
+            // spindle. Recognised by that tag and not by name: phi is an ordinary spindle name in MX,
+            // so a file from anywhere else must not have its rotation axis read back as a head
+            // position, nor its spindle mistaken for one here.
+            auto is_smargon_axis = [this](const std::string &name) {
+                const std::string dname = "/entry/sample/transformations/" + name;
+                if (!master_file->Exists(dname))
+                    return false;
+                HDF5DataSet axis(*master_file, dname);
+                return axis.AttrExists("equipment_component")
+                       && (axis.ReadAttrStr("equipment_component") == "smargon");
+            };
+
+            std::optional<GoniometerAxis> stationary;
+            for (const auto &name: master_file->FindLeafs("/entry/sample/transformations")) {
+                if (is_smargon_axis(name))
+                    continue;
+                auto axis = ReadAxis(master_file.get(), name);
+                if (!axis.has_value())
+                    continue;
+                if (axis->IsScanning()) {
+                    dataset->experiment.Goniometer(axis);
+                    stationary.reset();
+                    break;
+                }
+                if (!stationary.has_value())
+                    stationary = axis;
             }
+            if (stationary.has_value())
+                dataset->experiment.Goniometer(stationary);
+
+            // chi and phi are ordinary stationary axes in the file; the settings still keep them in
+            // their own Smargon field, so put them back there. Without this a re-opened file lost
+            // the head position entirely - nothing in reader/ read it.
+            std::optional<GoniometerAxis> chi, phi;
+            if (is_smargon_axis("chi"))
+                chi = ReadAxis(master_file.get(), "chi");
+            if (is_smargon_axis("phi"))
+                phi = ReadAxis(master_file.get(), "phi");
+            if (chi.has_value() || phi.has_value()) {
+                SmargonPosition smargon;
+                if (chi.has_value()) {
+                    smargon.chi_deg = chi->GetStart_deg();
+                    smargon.chi_axis = chi->GetAxis();
+                }
+                if (phi.has_value()) {
+                    smargon.phi_deg = phi->GetStart_deg();
+                    smargon.phi_axis = phi->GetAxis();
+                }
+                dataset->experiment.Smargon(smargon);
+            }
+        }
+
+        // Independent of the axis: a grid scan can be taken at a given head position, so the two are
+        // not alternatives.
+        if (master_file->Exists("/entry/sample/grid_scan")) {
+            GridScanSettings grid(
+                master_file->GetInt("/entry/sample/grid_scan/n_fast"),
+                master_file->GetFloat("/entry/sample/grid_scan/step_x") * 1e6f,
+                master_file->GetFloat("/entry/sample/grid_scan/step_y") * 1e6f,
+                master_file->GetOptBool("/entry/sample/grid_scan/snake_scan").value_or(false),
+                master_file->GetOptBool("/entry/sample/grid_scan/vertical_scan").value_or(false)
+            );
+            grid.ImageNum(number_of_images);
+            dataset->experiment.GridScan(grid);
         }
 
         auto tmp = master_file->ReadOptVector<float>("/entry/sample/unit_cell");
@@ -682,13 +737,19 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
 
         DetectorSetup detector = DetDECTRIS(image_size_x, image_size_y, detector_name, {});
         detector.PixelSize_um(master_file->GetFloat("/entry/instrument/detector/x_pixel_size") * 1e6);
+        // Whether the stored image is mirrored in Y. A file written before this was recorded is
+        // mirrored - that is the only thing Jungfraujoch has ever produced - so absence means true.
+        detector.MirrorY(master_file
+                         ->GetOptBool("/entry/instrument/detector/detectorSpecific/mirror_y")
+                         .value_or(true));
         // Sensor thickness/material drive the parallax/absorption model, so take them from the file
         // rather than the DetectorSetup default (NXmx stores thickness in metres).
         if (master_file->Exists("/entry/instrument/detector/sensor_thickness"))
             detector.SensorThickness_um(master_file->GetFloat("/entry/instrument/detector/sensor_thickness") * 1e6);
         if (master_file->Exists("/entry/instrument/detector/sensor_material"))
             detector.SensorMaterial(master_file->GetString("/entry/instrument/detector/sensor_material"));
-        detector.SaturationLimit(master_file->GetInt("/entry/instrument/detector/saturation_value"));
+        detector.SaturationLimit(SaturationLimitFromValue(
+                master_file->GetInt("/entry/instrument/detector/saturation_value")));
         // The reader hands every image out as signed int32 whatever the file stored (see PixelSigned
         // below), so that is the container depth the rest of the code has to see. DetectorSetup defaults
         // DECTRIS to 16 bits and GetByteDepthImage() prefers the detector's value over the image
@@ -1060,16 +1121,25 @@ std::optional<GoniometerAxis> HDF5MetadataSource::ReadAxis(HDF5Object *file, con
     std::vector<double> angle;
     dataset.ReadVector(angle);
 
-    if (angle.size() < 2)
+    if (angle.empty())
+        return {};
+
+    // Not everything in the group is an axis. The writer's own AXISNAME_end and the two rotation
+    // width scalars carry only units, and a file from anywhere else may hold whatever it likes.
+    // Missing attribute means "not a transformation", so skip it rather than throwing: the search
+    // for the goniometer walks every leaf and only stops early on an axis that turns, so a master
+    // whose axis was stationary reached omega_end and could not be opened at all.
+    if (!dataset.AttrExists("transformation_type")
+            || (dataset.ReadAttrStr("transformation_type") != "rotation"))
         return {};
 
     std::vector<double> end = file->ReadOptVector<double>(dname + "_end");
 
+    // A single value, or every value the same, is a stationary axis: it says where the head was
+    // rather than that anything turned. Increment 0 is the honest description of that, and
+    // GoniometerAxis::IsScanning is what separates it from a sweep.
     double start = angle[0];
-    double incr = angle[1] - angle[0];
-
-    if (dataset.ReadAttrStr("transformation_type") != "rotation")
-        return {};
+    double incr = (angle.size() < 2) ? 0.0 : angle[1] - angle[0];
 
     std::vector<double> axis_vec = dataset.ReadAttrVec("vector");
     if (axis_vec.size() != 3)