v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

common/DiffractionExperiment.cpp

@@ -659,8 +659,11 @@ void DiffractionExperiment::FillMessage(StartMessage &message) const {
     message.incident_energy = GetIncidentEnergy_keV() * 1e3f;
     message.image_size_x = GetXPixelsNum();
     message.image_size_y = GetYPixelsNum();
-    message.saturation_value = GetSaturationLimit() - 1;
-    message.error_value = GetUnderflow();
+    message.mirror_y = IsDetectorMirroredY();
+    message.saturation_value = SaturationValueFromLimit(GetSaturationLimit());
+    // The marker actually stored in the pixels: UINTx_MAX unsigned, INTx_MIN signed. GetUnderflow()
+    // was written here, which is -1 for an unsigned image and so matches no pixel it can contain.
+    message.error_value = GetImageFillValue();
     message.frame_time = std::chrono::duration<float>(GetImageTime()).count();
     message.count_time = std::chrono::duration<float>(GetImageCountTime()).count();
     message.number_of_images = GetImageNum();
@@ -669,7 +672,23 @@ void DiffractionExperiment::FillMessage(StartMessage &message) const {
     message.sensor_material = detector.GetSensorMaterial();
     message.sensor_thickness = detector.GetSensorThickness_um() * 1e-6f;
     message.bit_depth_image = GetByteDepthImage() * 8;
-    message.bit_depth_readout = GetBitDepthReadout();
+    // Both fields carry the depth of the STORED image, deliberately. NXmx has no field for it -
+    // only bit_depth_readout, defined as the depth of the detector electronics - so every NXmx
+    // reader takes bit_depth_readout as the width of the stored pixel (DIALS uses it to decide
+    // which values are masking markers, and cannot read a 32-bit image without it). The electronic
+    // readout depth is a constant of the detector and of no use to a data consumer, so reporting it
+    // here would only mislead: it differs from the image depth exactly when summation widens the
+    // image, which is when getting it wrong does damage.
+    // Only for unsigned images. DIALS remaps 2^bit_depth_readout-1 and -2 to -1 and -2 whenever the
+    // field is present, without looking at the pixel type: for an unsigned image those land below
+    // underload_value and are masked, which is exactly what we want, but for a signed one they land
+    // INSIDE the trusted range and a clipped pixel is integrated as a count of -2. Signed images
+    // read correctly without it; unsigned 32-bit cannot be read at all without it.
+    if (!IsPixelSigned())
+        message.bit_depth_readout = message.bit_depth_image;
+    // Lowest valid value, so a reader can tell the error marker from data. Unsigned images use 0;
+    // signed ones reserve INTx_MIN as the marker (GetUnderflow), so the lowest valid is one above.
+    message.underload_value = IsPixelSigned() ? GetUnderflow() + 1 : 0;
     message.indexing_algorithm = GetIndexingAlgorithm();
     message.images_per_trigger = dataset.GetImageNumPerTrigger();
 
@@ -904,6 +923,11 @@ DiffractionExperiment &DiffractionExperiment::Goniometer(const std::optional<Gon
     return *this;
 }
 
+DiffractionExperiment &DiffractionExperiment::Smargon(const std::optional<SmargonPosition> &input) {
+    dataset.Smargon(input);
+    return *this;
+}
+
 std::optional<GoniometerAxis> DiffractionExperiment::GetGoniometer() const {
     return dataset.GetGoniometer();
 }
@@ -1103,19 +1127,43 @@ std::vector<uint16_t> DiffractionExperiment::ExportROIMap() const {
     return roi_mask.GetROIMap(GetDiffractionGeometry(), GetXPixelsNumConv(), GetYPixelsNumConv());
 }
 
-DiffractionExperiment &DiffractionExperiment::ImagesPerFile(int64_t input) {
+DiffractionExperiment &DiffractionExperiment::ImagesPerFile(const std::optional<int64_t> &input) {
     dataset.ImagesPerFile(input);
     return *this;
 }
 
+// Images per file is optional in the settings, because at setup time the acquisition it should suit
+// is not necessarily known yet. Everything downstream of here - receiver, pusher, puller, writer -
+// requires a fixed, non-zero number, so this is the one place that resolves it and it must always
+// return one.
 int64_t DiffractionExperiment::GetImagesPerFile() const {
-    auto tmp = dataset.GetImagesPerFile();
+    // A single self-contained file holds the whole run whatever the setting says.
+    if (file_writer.GetFileFormat() == FileWriterFormat::NXmxIntegrated)
+        return std::max<int64_t>(GetImageNum(), 1);
 
-    if (tmp == 0
-        || file_writer.GetFileFormat() == FileWriterFormat::NXmxIntegrated)
-        return GetImageNum();
-    else
-        return tmp;
+    if (const auto requested = dataset.GetImagesPerFile())
+        return requested.value();   // taken literally
+
+    const int64_t image_num = GetImageNum();
+
+    // Rotation: keep the sweep in one data file. A legacy master splits over several files is read
+    // by some programs as only its first one, and a rotation dataset is small enough that the file
+    // is not a problem - past the limit the writer parallelism is worth more, one file going to one
+    // writer (see ZMQStream2Pusher::SendImage).
+    const auto goniometer = GetGoniometer();
+    if (goniometer.has_value() && goniometer->IsScanning()
+        && (image_num > 0) && (image_num <= ROTATION_SINGLE_FILE_IMAGE_LIMIT))
+        return image_num;
+
+    // Grid scan: split on whole rows of the fast axis, so a file is a meaningful piece of the grid
+    // rather than an arbitrary cut across it.
+    if (const auto grid_scan = GetGridScan()) {
+        const int64_t n_fast = grid_scan->GetNFast();
+        if (n_fast > 0)
+            return ((DEFAULT_IMAGES_PER_FILE + n_fast - 1) / n_fast) * n_fast;
+    }
+
+    return DEFAULT_IMAGES_PER_FILE;   // stills, serial
 }
 
 int64_t DiffractionExperiment::GetImageBufferLocationSize() const {
@@ -1411,6 +1459,69 @@ bool DiffractionExperiment::IsDetectorModuleSync() const {
     return detector.IsModuleSync();
 }
 
+bool DiffractionExperiment::IsDetectorMirroredY() const {
+    return detector.IsMirrorY();
+}
+
+// Base first, sample last - the order things are physically mounted in, which is also the order an
+// NXmx depends_on chain composes in. The grid stage is a base stage (an Aerotech xyz at SLS) that
+// the spindle sits on; the spindle carries the head; the head carries the sample.
+//
+// An axis that moves carries one value per image; one that does not carries a single value. Nothing
+// is derived on the way out to NXmx - that is the point of holding the positions rather than a start
+// and an increment, so that measured positions can be carried here later without changing anything
+// downstream.
+std::vector<DetectorTransformation> DiffractionExperiment::BuildTransformationChain(int64_t image_num) const {
+    std::vector<DetectorTransformation> chain;
+    std::string parent;   // empty = mounted on the base
+
+    const auto add = [&chain, &parent](DetectorTransformation axis) {
+        axis.DependsOn(parent);
+        parent = axis.GetName();
+        chain.push_back(std::move(axis));
+    };
+
+    const auto to_float = [](const std::vector<double> &input) {
+        return std::vector<float>(input.begin(), input.end());
+    };
+
+    if (const auto grid_scan = GetGridScan(); grid_scan.has_value() && (image_num > 0)) {
+        add(DetectorTransformation("grid_scan_x", TransformationType::Translation, {1, 0, 0})
+                    .Values(to_float(grid_scan->GetXContainer_m(image_num))));
+        add(DetectorTransformation("grid_scan_y", TransformationType::Translation, {0, 1, 0})
+                    .Values(to_float(grid_scan->GetYContainer_m(image_num))));
+    }
+
+    if (const auto goniometer = GetGoniometer()) {
+        DetectorTransformation axis(goniometer->GetName(), TransformationType::Rotation,
+                                    goniometer->GetAxis());
+        if (image_num > 0)
+            axis.Values(to_float(goniometer->GetAngleContainer(image_num)));
+        else
+            axis.Value(goniometer->GetStart_deg());
+        add(std::move(axis));
+    } else if (GetGridScan().has_value()) {
+        // A grid scan still sits on a spindle that simply does not turn. NXmx cannot say "there is no
+        // rotation", and a sample chain of translations alone is not something readers accept.
+        add(DetectorTransformation("omega", TransformationType::Rotation, {-1, 0, 0}).Value(0.0f));
+    }
+
+    // Smargon chi and phi are ordinary axes that happen not to move. They still carry one entry per
+    // image: a reader takes the number of images from the innermost axis of the sample chain when no
+    // axis varies, and chi/phi are innermost whenever they are present.
+    if (const auto smargon = dataset.GetSmargonPosition()) {
+        const auto n = static_cast<size_t>(std::max<int64_t>(image_num, 1));
+        add(DetectorTransformation("chi", TransformationType::Rotation, smargon->chi_axis)
+                    .EquipmentComponent("smargon")
+                    .Values(std::vector<float>(n, smargon->chi_deg)));
+        add(DetectorTransformation("phi", TransformationType::Rotation, smargon->phi_axis)
+                    .EquipmentComponent("smargon")
+                    .Values(std::vector<float>(n, smargon->phi_deg)));
+    }
+
+    return chain;
+}
+
 int64_t DiffractionExperiment::GetEigerBitDepth() const {
     auto tmp = detector_settings.GetEigerBitDepth();
     if (tmp.has_value())
@@ -1729,7 +1840,10 @@ int64_t DiffractionExperiment::GetDarkMaskNumberOfFrames() const {
 }
 
 bool DiffractionExperiment::IsRotationIndexing() const {
-    return GetGoniometer().has_value() && indexing.GetRotationIndexing();
+    // A stationary axis is not rotation data, however it got here. Without this, a constant axis
+    // reaches RotationIndexerCounter, whose stride stays zero, and indexing then never runs at all.
+    const auto goniometer = GetGoniometer();
+    return goniometer.has_value() && goniometer->IsScanning() && indexing.GetRotationIndexing();
 }
 
 DiffractionExperiment &DiffractionExperiment::RunNumber(uint64_t input) {