v1.0.0-rc.164 (#74)

* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
* rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
* rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
* rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
* rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
* rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
* rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present.
* rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file.
* rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
* Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere.
* Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
* A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
* A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires.
* `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
* The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged.
* The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count.
* `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible.
* `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so.
* The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents.

Reviewed-on: #74
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

reader/HDF5MetadataSource.cpp

@@ -132,9 +132,22 @@ std::string dataset_name(const std::string &path) {
     return file;
 }
 
+// Per-image reflections and lattices are written in the setting the images were INDEXED in; the
+// unit cell, the run lattice and the space group beside them are in the setting the merge settled
+// on, which the space-group search can re-seat to. /entry/MX/reindexMatrix is the integral change of
+// basis between the two, so applying it here is what makes the file read as one consistent dataset.
+// No matrix means the two settings are the same one.
+CrystalLattice ApplyReindex(const CrystalLattice &latt, const std::optional<std::array<int32_t, 9>> &m) {
+    if (!m)
+        return latt;
+    const auto &v = *m;
+    return latt.Multiply(gemmi::Mat33(v[0], v[1], v[2], v[3], v[4], v[5], v[6], v[7], v[8]));
+}
+
 bool ReadReflectionsFromGroup(HDF5Object &file,
                               const std::string &image_group_name,
-                              std::vector<Reflection> &reflections) {
+                              std::vector<Reflection> &reflections,
+                              const std::optional<std::array<int32_t, 9>> &reindex) {
     if (!file.Exists("/entry/reflections") || !file.Exists(image_group_name))
         return false;
 
@@ -165,6 +178,15 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
         throw JFJochException(JFJochExceptionCategory::HDF5, "Wrong size of reflections dataset");
 
     for (size_t i = 0; i < h.size(); i++) {
+        int32_t hh = h.at(i), kk = k.at(i), ll = l.at(i);
+        if (reindex) {
+            const auto &m = *reindex;
+            const int32_t h0 = hh, k0 = kk, l0 = ll;
+            hh = m[0] * h0 + m[1] * k0 + m[2] * l0;
+            kk = m[3] * h0 + m[4] * k0 + m[5] * l0;
+            ll = m[6] * h0 + m[7] * k0 + m[8] * l0;
+        }
+
         float lp_val = 0.0;
         if (lp.size() > i && lp[i] != 0.0f)
             lp_val = 1.0f / lp[i];
@@ -201,9 +223,9 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
         }
 
         Reflection r{
-            .h = h.at(i),
-            .k = k.at(i),
-            .l = l.at(i),
+            .h = hh,
+            .k = kk,
+            .l = ll,
             .image_number = image_number.at(i),
             .delta_phi_deg = delta_phi_val,
             .predicted_x = predicted_x.at(i),
@@ -687,6 +709,13 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
                 .gamma = tmp[5]
             });
         dataset->experiment.SpaceGroupNumber(master_file->GetOptInt("/entry/sample/space_group_number"));
+        // The setting the cell and space group just read are in, relative to the setting the per-image
+        // reflections and lattices were written in. Absent on every file written before the offline
+        // analysis started recording it, and on every run that never re-seated its lattice - both mean
+        // the identity, and both are read as such.
+        if (const auto m = master_file->ReadOptVector<int32_t>("/entry/MX/reindexMatrix"); m.size() == 9)
+            dataset->reindex_matrix = std::array<int32_t, 9>{m[0], m[1], m[2], m[3], m[4],
+                                                             m[5], m[6], m[7], m[8]};
         dataset->experiment.SampleName(master_file->GetString("/entry/sample/name"));
 
 
@@ -1070,7 +1099,7 @@ void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_im
         );
 
         if (tmp.size() == 9)
-            message.indexing_lattice = CrystalLattice(tmp);
+            message.indexing_lattice = ApplyReindex(CrystalLattice(tmp), dataset->reindex_matrix);
 
         std::optional<std::string> lattice;
         if (master_file->Exists("/entry/MX/bravaisLattice"))
@@ -1102,8 +1131,10 @@ void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_im
     const std::string master_reflection_group_name = fmt::format("/entry/reflections/image_{:06d}", image_number);
     const std::string source_reflection_group_name = fmt::format("/entry/reflections/image_{:06d}", image_id);
 
-    if (!ReadReflectionsFromGroup(*master_file, master_reflection_group_name, message.reflections))
-        ReadReflectionsFromGroup(*source_file, source_reflection_group_name, message.reflections);
+    if (!ReadReflectionsFromGroup(*master_file, master_reflection_group_name, message.reflections,
+                                  dataset->reindex_matrix))
+        ReadReflectionsFromGroup(*source_file, source_reflection_group_name, message.reflections,
+                                 dataset->reindex_matrix);
     if (!message.reflections.empty()) {
         CalcISigma(message);
         CalcWilsonBFactor(message, !message.b_factor.has_value());
@@ -1214,6 +1245,10 @@ std::vector<IntegrationOutcome> HDF5MetadataSource::ReadReflections(size_t start
     // reflections lazily from the source data files instead.
     const bool master_reflections_authoritative = master_file->Exists("/entry/reflections");
 
+    // Everything below comes out in the setting of the dataset's unit cell and space group, not in the
+    // setting it was written in (see ApplyReindex).
+    const auto &reindex = dataset_->reindex_matrix;
+
     for (size_t img = start_image; img <= end_image_val; img++) {
         IntegrationOutcome outcome;
 
@@ -1235,7 +1270,7 @@ std::vector<IntegrationOutcome> HDF5MetadataSource::ReadReflections(size_t start
         }
 
         // ── reflections ──────────────────────────────────────────────────────
-        ReadReflectionsFromGroup(*meta_file, refl_group, outcome.reflections);
+        ReadReflectionsFromGroup(*meta_file, refl_group, outcome.reflections, reindex);
 
         // ── per-image mosaicity ───────────────────────────────────────────────
         if (meta_file->Exists("/entry/MX/mosaicity")) {
@@ -1252,7 +1287,7 @@ std::vector<IntegrationOutcome> HDF5MetadataSource::ReadReflections(size_t start
                 auto lattice_vec = meta_file->ReadOptVector<float>(
                     "/entry/MX/latticeIndexed", {meta_image_id, 0}, {1, 9});
                 if (lattice_vec.size() == 9)
-                    outcome.latt = CrystalLattice(lattice_vec);
+                    outcome.latt = ApplyReindex(CrystalLattice(lattice_vec), reindex);
             } catch (...) {
             }
         }