v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/MXAnalysisWithoutFPGA.cpp

@@ -43,15 +43,12 @@ MXAnalysisWithoutFPGA::MXAnalysisWithoutFPGA(const DiffractionExperiment &in_exp
       indexer(in_indexer),
       prediction(CreateBraggPrediction(experiment.IsRotationIndexing())),
       mask(in_mask),
-      mask_resolution(experiment.GetPixelsNum(), false),
       mask_high_res(-1),
       mask_low_res(-1) {
 #ifdef JFJOCH_USE_CUDA
     if (get_gpu_count() == 0) {
 #endif
         preprocessor_buffer = std::make_unique<ImagePreprocessorBuffer>(experiment.GetPixelsNum());
-        spotFinder = std::make_unique<ImageSpotFinderCPU>(experiment.GetXPixelsNum(), experiment.GetYPixelsNum());
-        azint = std::make_unique<AzIntEngineCPU>(integration);
         preprocessor = std::make_unique<ImagePreprocessorCPU>(in_experiment, in_mask);
         bragg_engine = std::make_unique<BraggIntegrationEngineCPU>(in_experiment);
         if (experiment.ROI().size() >= 1)
@@ -59,15 +56,17 @@ MXAnalysisWithoutFPGA::MXAnalysisWithoutFPGA(const DiffractionExperiment &in_exp
 #ifdef JFJOCH_USE_CUDA
     } else {
         stream = std::make_shared<CudaStream>();
-        preprocessor_buffer = std::make_unique<ImagePreprocessorBufferGPU>(experiment.GetPixelsNum());
+        // The host copy of the preprocessed image is only read when a CPU engine wants it, which is
+        // the same condition that drives copy_image_to_host below. Skipping it also skips page-locking
+        // 4 bytes per pixel per worker.
+        preprocessor_buffer = std::make_unique<ImagePreprocessorBufferGPU>(
+                experiment.GetPixelsNum(), /*host_mirror=*/!enable_fused_adaptive_gpu);
         // The preprocessed image only has to come back to the host if a CPU engine reads it. Every
         // engine built below runs on the GPU, except the CPU adaptive finder that is kept when the fused
         // GPU engine is off - so that is the one case that needs the copy. Every caller currently passes
         // enable_fused_adaptive_gpu = true, so on the GPU path the copy is off in practice.
         preprocessor = std::make_unique<ImagePreprocessorGPU>(in_experiment, in_mask, stream,
                                                              /*copy_image_to_host=*/!enable_fused_adaptive_gpu);
-        spotFinder = std::make_unique<ImageSpotFinderGPU>(experiment.GetXPixelsNum(), experiment.GetYPixelsNum(), stream);
-        azint = std::make_unique<AzIntEngineGPU>(integration, stream);
         bragg_engine = std::make_unique<BraggIntegrationEngineGPU>(in_experiment, stream);
         if (experiment.ROI().size() >= 1)
             roi = std::make_unique<ROIIntegrationGPU>(experiment, stream);
@@ -140,7 +139,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
 
     if (!fused) {
         const auto azint_start_time = std::chrono::steady_clock::now();
-        azint->Run(*preprocessor_buffer, profile);
+        AzInt().Run(*preprocessor_buffer, profile);
         const auto azint_end_time = std::chrono::steady_clock::now();
         output.azint_time_s = std::chrono::duration<float>(azint_end_time - azint_start_time).count();
     }
@@ -156,7 +155,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
 
         ImageSpotFinder &finder = spot_finding_settings.adaptive_threshold
                                       ? static_cast<ImageSpotFinder &>(*adaptiveSpotFinder)
-                                      : *spotFinder;
+                                      : FixedThresholdFinder();
         const auto integrate_fn = [this](const std::vector<Reflection> &predicted, size_t npredicted,
                                          int64_t image_number) {
             return bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number);
@@ -289,6 +288,36 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
             integration.Settings(), spot_finding_settings.ice_ring_width_Q_recipA);
 }
 
+ImageSpotFinder &MXAnalysisWithoutFPGA::FixedThresholdFinder() {
+    if (!spotFinder) {
+#ifdef JFJOCH_USE_CUDA
+        if (stream)
+            spotFinder = std::make_unique<ImageSpotFinderGPU>(experiment.GetXPixelsNum(),
+                                                              experiment.GetYPixelsNum(), stream);
+        else
+#endif
+            spotFinder = std::make_unique<ImageSpotFinderCPU>(experiment.GetXPixelsNum(),
+                                                              experiment.GetYPixelsNum());
+        // It missed every mask update that happened before it existed, so it takes the current one
+        // now. Without this it would find spots outside the resolution limits.
+        if (mask_resolution)
+            spotFinder->SetResolutionMaskBits(*mask_resolution);
+    }
+    return *spotFinder;
+}
+
+AzIntEngine &MXAnalysisWithoutFPGA::AzInt() {
+    if (!azint) {
+#ifdef JFJOCH_USE_CUDA
+        if (stream)
+            azint = std::make_unique<AzIntEngineGPU>(integration, stream);
+        else
+#endif
+            azint = std::make_unique<AzIntEngineCPU>(integration);
+    }
+    return *azint;
+}
+
 void MXAnalysisWithoutFPGA::RebuildROI() {
     if (experiment.ROI().empty()) {
         roi.reset();
@@ -331,17 +360,14 @@ void MXAnalysisWithoutFPGA::RunROIOnly(DataMessage &output) {
 void MXAnalysisWithoutFPGA::UpdateMaskResolution(const SpotFindingSettings &settings) {
     mask_low_res = settings.low_resolution_limit;
     mask_high_res = settings.high_resolution_limit;
-    // An unset limit masks nothing at that end. At the high-resolution end 0 does that on its own - no
-    // pixel has d < 0, and the detector's own edge is where the pixels stop anyway; at the low-resolution
-    // end every pixel lies above any finite stand-in, so it takes an infinite one.
-    const float high_res = mask_high_res.value_or(0.0f);
-    const float low_res = mask_low_res.value_or(INFINITY);
-    auto const &resolution_map = integration.Resolution();
-    for (int i = 0; i < mask_resolution.size(); i++)
-        mask_resolution[i] = (resolution_map[i] > low_res) || (resolution_map[i] < high_res);
+    // The mask is a pure function of the resolution map and the two limits, so the mapping builds it -
+    // once for all the workers, which otherwise each walked every pixel of the detector to arrive at
+    // the same bits.
+    mask_resolution = integration.ResolutionMaskBits(mask_high_res, mask_low_res);
 
     // The finders keep their own copy (the GPU ones a bit-packed device copy), so the mask is handed
     // over here - when the limits change - rather than with every image.
-    spotFinder->SetResolutionMask(mask_resolution);
-    adaptiveSpotFinder->SetResolutionMask(mask_resolution);
+    if (spotFinder)
+        spotFinder->SetResolutionMaskBits(*mask_resolution);
+    adaptiveSpotFinder->SetResolutionMaskBits(*mask_resolution);
 }