v1.0.0.rc-161 (#71)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.

**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.

**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.

**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**

Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

reader/HDF5MetadataSource.cpp

@@ -141,6 +141,7 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
     auto h = file.ReadOptVector<int32_t>(image_group_name + "/h");
     auto k = file.ReadOptVector<int32_t>(image_group_name + "/k");
     auto l = file.ReadOptVector<int32_t>(image_group_name + "/l");
+    auto image_number = file.ReadOptVector<float>(image_group_name + "/observed_frame");
     auto predicted_x = file.ReadOptVector<float>(image_group_name + "/predicted_x");
     auto predicted_y = file.ReadOptVector<float>(image_group_name + "/predicted_y");
     auto obs_x = file.ReadOptVector<float>(image_group_name + "/observed_x");
@@ -149,6 +150,8 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
     auto int_sum = file.ReadOptVector<float>(image_group_name + "/int_sum");
     auto int_err = file.ReadOptVector<float>(image_group_name + "/int_err");
     auto bkg = file.ReadOptVector<float>(image_group_name + "/background_mean");
+    // Written since the merge stopped back-deriving it; older _process.h5 do not carry it.
+    auto var_bkg = file.ReadOptVector<float>(image_group_name + "/background_variance");
     auto lp = file.ReadOptVector<float>(image_group_name + "/lp");
     auto partiality = file.ReadOptVector<float>(image_group_name + "/partiality");
     auto phi = file.ReadOptVector<float>(image_group_name + "/delta_phi");
@@ -157,7 +160,8 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
 
     if (h.size() != l.size() || h.size() != k.size() || h.size() != d.size()
         || h.size() != predicted_x.size() || h.size() != predicted_y.size()
-        || h.size() != int_sum.size() || h.size() != int_err.size() || h.size() != bkg.size())
+        || h.size() != int_sum.size() || h.size() != int_err.size() || h.size() != bkg.size()
+        || h.size() != image_number.size())
         throw JFJochException(JFJochExceptionCategory::HDF5, "Wrong size of reflections dataset");
 
     for (size_t i = 0; i < h.size(); i++) {
@@ -175,6 +179,15 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
         if (zeta.size() > i)
             zeta_val = zeta[i];
 
+        // A file written before this dataset existed has to have the non-signal variance reconstructed,
+        // not zeroed. The combine takes var_bkg as the authoritative non-signal term, so a zero would
+        // leave it the signal alone and weight a weak reflection by ~1/I instead of ~1/sigma^2 - orders
+        // of magnitude too high, and worst exactly where the reflection is weakest. The integrator's own
+        // identity sigma^2 = I + var_bkg inverts to recover what the file does not store.
+        float var_bkg_val = std::max(0.0f, int_err.at(i) * int_err.at(i) - int_sum.at(i));
+        if (var_bkg.size() > i)
+            var_bkg_val = var_bkg[i];
+
         float image_scale_corr_val = 1.0f; // Default is 1.0, if we don't know any better
         if (image_scale_corr.size() > i)
             image_scale_corr_val = image_scale_corr[i];
@@ -191,6 +204,7 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
             .h = h.at(i),
             .k = k.at(i),
             .l = l.at(i),
+            .image_number = image_number.at(i),
             .delta_phi_deg = delta_phi_val,
             .predicted_x = predicted_x.at(i),
             .predicted_y = predicted_y.at(i),
@@ -199,6 +213,7 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
             .d = d.at(i),
             .I = int_sum.at(i),
             .bkg = bkg.at(i),
+            .var_bkg = var_bkg_val,
             .sigma = int_err.at(i),
             .rlp = lp_val,
             .partiality = partiality_val,
@@ -357,6 +372,7 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
                 dataset->spot_count_low_res = master_file->ReadOptVector<float>("/entry/MX/peakCountLowRes");
                 dataset->spot_count_indexed = master_file->ReadOptVector<float>("/entry/MX/peakCountIndexed");
                 dataset->spot_count_ice_rings = master_file->ReadOptVector<float>("/entry/MX/peakCountIceRingRes");
+                dataset->spot_count_ice_control = master_file->ReadOptVector<float>("/entry/MX/peakCountIceRingControl");
 
                 dataset->indexing_result = master_file->ReadOptVector<float>("/entry/MX/imageIndexed");
                 dataset->bkg_estimate = master_file->ReadOptVector<float>("/entry/MX/bkgEstimate");
@@ -372,8 +388,15 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
                 dataset->b_factor = master_file->ReadOptVector<float>("/entry/MX/bFactor");
                 dataset->image_scale_factor = master_file->ReadOptVector<float>("/entry/MX/imageScaleFactor");
                 dataset->image_scale_cc = master_file->ReadOptVector<float>("/entry/MX/imageScaleCC");
-                dataset->image_scale_b = master_file->ReadOptVector<float>("/entry/MX/imageScaleBFactor");
                 dataset->integrated_reflections = master_file->ReadOptVector<float>("/entry/MX/integratedReflections");
+                dataset->sweep_quality = master_file->ReadOptVector<uint8_t>("/entry/MX/sweepQuality");
+                if (master_file->Exists("/entry/MX/sweepQualityReasons")) {
+                    const auto dim = master_file->GetDimension("/entry/MX/sweepQualityReasons");
+                    for (size_t i = 0; i < (dim.empty() ? 0 : dim[0]); i++)
+                        dataset->sweep_quality_reasons.push_back(
+                            master_file->ReadElement<std::string>("/entry/MX/sweepQualityReasons", i)
+                                    .value_or(""));
+                }
             }
             if (master_file->Exists("/entry/image"))
                 dataset->max_value = master_file->ReadOptVector<int64_t>("/entry/image/max_value");
@@ -466,6 +489,9 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
                                        data_file, "/entry/MX/nPeaks",
                                        number_of_images, fimages);
 
+                        ReadVector(dataset->spot_count_ice_control,
+                                   data_file, "/entry/MX/peakCountIceRingControl",
+                                   number_of_images, fimages);
                         ReadVector(dataset->spot_count_ice_rings,
                                    data_file, "/entry/MX/peakCountIceRingRes",
                                    number_of_images, fimages);
@@ -541,8 +567,9 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
         auto ring_current_A = master_file->GetOptFloat("/entry/source/current");
         if (ring_current_A) dataset->experiment.RingCurrent_mA(ring_current_A.value() * 1000.0);
 
-        dataset->experiment.DetectIceRings(
-            master_file->GetOptBool("/entry/instrument/detector/detectorSpecific/detect_ice_rings").value_or(false));
+        dataset->file_detect_ice_rings =
+            master_file->GetOptBool("/entry/instrument/detector/detectorSpecific/detect_ice_rings");
+        dataset->experiment.DetectIceRings(dataset->file_detect_ice_rings.value_or(false));
         dataset->experiment.PoniRot1_rad(
             master_file->GetOptFloat("/entry/instrument/detector/transformations/rot1").value_or(0.0));
         dataset->experiment.PoniRot2_rad(
@@ -662,6 +689,17 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
         if (master_file->Exists("/entry/instrument/detector/sensor_material"))
             detector.SensorMaterial(master_file->GetString("/entry/instrument/detector/sensor_material"));
         detector.SaturationLimit(master_file->GetInt("/entry/instrument/detector/saturation_value"));
+        // The reader hands every image out as signed int32 whatever the file stored (see PixelSigned
+        // below), so that is the container depth the rest of the code has to see. DetectorSetup defaults
+        // DECTRIS to 16 bits and GetByteDepthImage() prefers the detector's value over the image
+        // format's, so leaving it at the default computed the overflow as a 16-bit one and called every
+        // count above 32767 saturated - the integration accept gate then dropped the WHOLE reflection,
+        // silently removing the strongest reflections of a strong crystal (measured on a lysozyme set:
+        // max accepted pixel 32738 against a declared saturation of 108833). Taking bit_depth_image from
+        // the file instead does not work either: it describes an UNSIGNED container, so pairing it with
+        // signed pixels halves the range (a 16-bit file capped at 32767, an 8-bit one at 127). The real
+        // cap is the file's own saturation_value, set just above.
+        detector.BitDepthImage(32);
         detector.MinFrameTime(std::chrono::microseconds(0));
         detector.MinCountTime(std::chrono::microseconds(0));
         detector.ReadOutTime(std::chrono::nanoseconds(0));
@@ -778,6 +816,7 @@ static void ReadSpotsFromFiles(HDF5Object &master_file,
                                hsize_t source_image,
                                int64_t image_number,
                                const DiffractionGeometry &geom,
+                               float plot_d_min_A,
                                DataMessage &message) {
     auto spot_count_opt = ReadElementMasterFirst<uint32_t>(master_file,
                                                            source_file,
@@ -889,7 +928,7 @@ static void ReadSpotsFromFiles(HDF5Object &master_file,
     message.spot_count_indexed = ReadElementMasterFirst<uint32_t>(
         master_file, source_file, "/entry/MX/peakCountIndexed", master_image, source_image);
 
-    GenerateSpotPlot(message, 1.5);
+    GenerateSpotPlot(message, message.spots, plot_d_min_A);
 }
 
 void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_image,
@@ -907,7 +946,8 @@ void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_im
     const auto source_image = static_cast<hsize_t>(image_id);
 
     ReadSpotsFromFiles(*master_file, *source_file, master_image, source_image,
-                       requested_image, dataset->experiment.GetDiffractionGeometry(), message);
+                       requested_image, dataset->experiment.GetDiffractionGeometry(),
+                       dataset->experiment.GetDetectorMaxResolution_A(), message);
 
     if (!dataset->az_int_bin_to_q.empty()) {
         if (dataset->azimuthal_bins == 0) {
@@ -951,8 +991,6 @@ void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_im
         message.mosaicity_deg = dataset->mosaicity_deg[image_number];
     if (dataset->b_factor.size() > image_number)
         message.b_factor = dataset->b_factor[image_number];
-    if (dataset->image_scale_b.size() > image_number)
-        message.image_scale_b_factor = dataset->image_scale_b[image_number];
     if (dataset->image_scale_factor.size() > image_number)
         message.image_scale_factor = dataset->image_scale_factor[image_number];
     if (dataset->image_scale_cc.size() > image_number)
@@ -1186,6 +1224,7 @@ std::vector<SpotToSave> HDF5MetadataSource::ReadSpots(int64_t requested_image) c
                        image, meta_image_id,
                        requested_image,
                        cached_geom,
+                       dataset_ ? dataset_->experiment.GetDetectorMaxResolution_A() : 0.0f,
                        tmp_message);
 
     return tmp_message.spots;