v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

common/PixelMask.cpp

@@ -3,13 +3,16 @@
 
 #include "PixelMask.h"
 #include "RawToConvertedGeometry.h"
+#include "TableChecksum.h"
 #include "JFJochException.h"
 #include "JFJochCompressor.h"
 
 PixelMask::PixelMask() = default;
 
 PixelMask::PixelMask(size_t width, size_t height)
-        : mask(width*height, 0) {}
+        : mask(width*height, 0) {
+    UpdateBinaryMask();
+}
 
 PixelMask::PixelMask(const DiffractionExperiment &experiment)
         : PixelMask(experiment.GetXPixelsNumConv(),
@@ -17,7 +20,9 @@ PixelMask::PixelMask(const DiffractionExperiment &experiment)
     CalcEdgePixels(experiment);
 }
 
-PixelMask::PixelMask(const std::vector<uint32_t> &in_mask) : mask(in_mask) {}
+PixelMask::PixelMask(const std::vector<uint32_t> &in_mask) : mask(in_mask) {
+    UpdateBinaryMask();
+}
 
 uint32_t PixelMask::LoadMask(const std::vector<uint32_t> &input_mask, uint8_t bit) {
     uint32_t ret = 0;
@@ -35,7 +40,7 @@ uint32_t PixelMask::LoadMask(const std::vector<uint32_t> &input_mask, uint8_t bi
     return ret;
 }
 
-void PixelMask::UpdateRawMask(const DiffractionExperiment &experiment) {
+void PixelMask::UpdateDerived(const DiffractionExperiment &experiment) {
     switch (experiment.GetDetectorType()) {
         case DetectorType::JUNGFRAU:
         case DetectorType::EIGER:
@@ -46,6 +51,14 @@ void PixelMask::UpdateRawMask(const DiffractionExperiment &experiment) {
             raw_mask.clear();
             break;
     }
+    UpdateBinaryMask();
+}
+
+void PixelMask::UpdateBinaryMask() {
+    binary_mask.resize(mask.size());
+    for (size_t i = 0; i < mask.size(); i++)
+        binary_mask[i] = (mask[i] != 0);
+    binary_mask_checksum = TableChecksum(binary_mask.data(), binary_mask.size());
 }
 
 void PixelMask::CalcEdgePixels_i(const DiffractionExperiment &experiment) {
@@ -96,7 +109,7 @@ void PixelMask::CalcEdgePixels_i(const DiffractionExperiment &experiment) {
 
 void PixelMask::CalcEdgePixels(const DiffractionExperiment &experiment)  {
     CalcEdgePixels_i(experiment);
-    UpdateRawMask(experiment);
+    UpdateDerived(experiment);
 }
 
 const std::vector<uint32_t> &PixelMask::GetMaskRaw() const {
@@ -110,6 +123,14 @@ const std::vector<uint32_t> &PixelMask::GetMask() const {
     return mask;
 }
 
+const std::vector<uint8_t> &PixelMask::GetBinaryMask() const {
+    return binary_mask;
+}
+
+uint64_t PixelMask::GetBinaryMaskChecksum() const {
+    return binary_mask_checksum;
+}
+
 const std::vector<uint32_t> &PixelMask::GetMask(const DiffractionExperiment& experiment) const {
     if (experiment.IsGeometryTransformed())
         return GetMask();
@@ -180,7 +201,7 @@ void PixelMask::LoadDetectorBadPixelMask(const DiffractionExperiment &experiment
 
     CalcEdgePixels_i(experiment);
 
-    UpdateRawMask(experiment);
+    UpdateDerived(experiment);
 }
 
 PixelMaskStatistics PixelMask::GetStatistics() const {
@@ -207,12 +228,12 @@ PixelMaskStatistics PixelMask::GetStatistics() const {
 void PixelMask::LoadUserMask(const DiffractionExperiment& experiment, const std::vector<uint32_t> &in_mask) {
     if (in_mask.size() == mask.size()) {
         LoadMask(in_mask, UserMaskedPixelBit);
-        UpdateRawMask(experiment);
+        UpdateDerived(experiment);
     } else if (in_mask.size() == experiment.GetModulesNum() * RAW_MODULE_SIZE) {
         std::vector<uint32_t> tmp(experiment.GetPixelsNumConv(), 0);
         RawToConvertedGeometry(experiment, tmp.data(), in_mask. data());
         LoadMask(tmp, UserMaskedPixelBit);
-        UpdateRawMask(experiment);
+        UpdateDerived(experiment);
     } else
         throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
                               "Size of input user mask invalid");
@@ -223,13 +244,13 @@ void PixelMask::LoadBeamStopMask(const DiffractionExperiment& experiment, const
         throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
                               "Size of input beam stop mask invalid");
     LoadMask(in_mask, BeamStopPixelBit);
-    UpdateRawMask(experiment);
+    UpdateDerived(experiment);
 }
 
 void PixelMask::ClearBeamStopMask(const DiffractionExperiment& experiment) {
     for (auto &i: mask)
         i &= ~(1u << BeamStopPixelBit);
-    UpdateRawMask(experiment);
+    UpdateDerived(experiment);
 }
 
 void PixelMask::LoadUserMask(const DiffractionExperiment& experiment, const CompressedImage& image) {
@@ -307,6 +328,7 @@ void PixelMask::LoadDECTRISBadPixelMask(const std::vector<uint32_t> &input_mask)
         }
     }
     raw_mask = {}; // For DECTRIS - there is no raw mask
+    UpdateBinaryMask();
 }
 
 void PixelMask::LoadDarkBadPixelMask(const DiffractionExperiment& experiment, const std::vector<uint32_t> &input_mask) {
@@ -325,5 +347,5 @@ void PixelMask::LoadDarkBadPixelMask(const DiffractionExperiment& experiment, co
             mask[i] &= ~(1 << NoisyPixelBit);
         }
     }
-    UpdateRawMask(experiment);
+    UpdateDerived(experiment);
 }