v1.0.0-rc.164 (#74)

* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
* rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
* rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
* rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
* rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
* rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
* rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present.
* rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file.
* rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
* Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere.
* Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
* A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
* A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires.
* `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
* The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged.
* The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count.
* `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible.
* `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so.
* The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents.

Reviewed-on: #74
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

broker/JFJochStateMachine.cpp

@@ -64,7 +64,7 @@ bool JFJochStateMachine::ImportPedestalG1G2(const JFJochReceiverOutput &receiver
     return true;
 }
 
-void JFJochStateMachine::CalibrateJUNGFRAU(std::unique_lock<std::mutex> &ul) {
+bool JFJochStateMachine::CalibrateJUNGFRAU(std::unique_lock<std::mutex> &ul) {
     if (!gain_calibration.empty()) {
         if (gain_calibration.size() != experiment.GetModulesNum())
             throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
@@ -73,14 +73,20 @@ void JFJochStateMachine::CalibrateJUNGFRAU(std::unique_lock<std::mutex> &ul) {
             calibration->GainCalibration(i) = gain_calibration[i];
     }
 
-    TakePedestalInternalG0(ul);
+    // Abandon the sequence on the first failure. Collecting G1 on top of a G0 that was never
+    // measured only produces a calibration that looks complete.
+    if (!TakePedestalInternalG0(ul))
+        return false;
     if (!experiment.IsFixedGainG1()) {
         for (int i = 0; i < experiment.GetStorageCellNumber(); i++) {
-            TakePedestalInternalG1(ul, i);
-            TakePedestalInternalG2(ul, i);
+            if (!TakePedestalInternalG1(ul, i))
+                return false;
+            if (!TakePedestalInternalG2(ul, i))
+                return false;
         }
     }
     pixel_mask.LoadDetectorBadPixelMask(experiment, calibration.get());
+    return true;
 }
 
 void JFJochStateMachine::CalibrateDetector(std::unique_lock<std::mutex> ul) {
@@ -90,46 +96,56 @@ void JFJochStateMachine::CalibrateDetector(std::unique_lock<std::mutex> ul) {
     UpdatePixelMaskStatistics(pixel_mask.GetStatistics());
 
     logger.Info("Calibration sequence started");
+    bool calibrated;
     try {
         if (experiment.GetDetectorType() == DetectorType::EIGER) {
             // PSI EIGER - only reset calibration
             calibration.reset();
+            calibrated = true;
         } else if (experiment.GetDetectorType() == DetectorType::DECTRIS) {
             // DECTRIS - take dark data for mask
             calibration.reset();
-            TakeDarkMaskInternal(ul);
+            calibrated = TakeDarkMaskInternal(ul);
         } else {
             // PSI JUNGFRAU - take pedestal
             calibration = std::make_unique<JFCalibration>(experiment);
-            CalibrateJUNGFRAU(ul);
+            calibrated = CalibrateJUNGFRAU(ul);
         }
         // Update pixel mask statistics
         UpdatePixelMaskStatistics(pixel_mask.GetStatistics());
-        // configure detector for standard operation
-        services.ConfigureDetector(experiment);
+        // configure detector for standard operation - only worth doing if there is a calibration to
+        // operate with, and a cancelled sequence has left the detector mid-abort anyway
+        if (calibrated)
+            services.ConfigureDetector(experiment);
     } catch (const std::exception &e) {
         logger.Error("Calibration sequence error {}", e.what());
-        SetState(JFJochState::Error, e.what(), BrokerStatus::MessageSeverity::Error);
+        // The calibration is in an undefined state, so the detector has to be initialized again.
+        SetState(JFJochState::Inactive, e.what(), BrokerStatus::MessageSeverity::Error);
+        c.notify_all(); // ul unlocks on the way out
         throw;
     }
-    SetState(JFJochState::Idle, "Calibration sequence done", BrokerStatus::MessageSeverity::Success);
-    logger.Info("Calibration sequence done");
+    // The steps above report a cancellation or a failure through SetState and return false; that
+    // must not be overwritten with success here.
+    if (calibrated) {
+        SetState(JFJochState::Idle, "Calibration sequence done", BrokerStatus::MessageSeverity::Success);
+        logger.Info("Calibration sequence done");
+    }
     ul.unlock(); // Notify all outside of mutex
     c.notify_all();
 }
 
-void JFJochStateMachine::TakeDarkMaskInternal(std::unique_lock<std::mutex> &ul) {
+bool JFJochStateMachine::TakeDarkMaskInternal(std::unique_lock<std::mutex> &ul) {
     if (cancel_sequence) {
         SetState(JFJochState::Inactive,
                  "Mask sequence cancelled",
-                 BrokerStatus::MessageSeverity::Warning);
-        return;
+                 BrokerStatus::MessageSeverity::Error);
+        return false;
     }
 
     services.LoadDetectorPixelMask(pixel_mask);
 
     if (experiment.GetDarkMaskNumberOfFrames() == 0)
-        return;
+        return true;
 
     DiffractionExperiment local_experiment(experiment);
     local_experiment.Mode(DetectorMode::DarkMask);
@@ -145,14 +161,16 @@ void JFJochStateMachine::TakeDarkMaskInternal(std::unique_lock<std::mutex> &ul)
     auto mask_output = services.Stop();
     ul.lock();
 
-    if (mask_output.receiver_output.dark_mask_result.size() == local_experiment.GetPixelsNum()) {
-        pixel_mask.LoadDarkBadPixelMask(local_experiment, mask_output.receiver_output.dark_mask_result);
-        SetState(JFJochState::Idle);
-    } else
-        SetState(JFJochState::Error, "Mask not collected properly", BrokerStatus::MessageSeverity::Error);
+    if (mask_output.receiver_output.dark_mask_result.size() != local_experiment.GetPixelsNum()) {
+        SetState(JFJochState::Inactive, "Mask not collected properly", BrokerStatus::MessageSeverity::Error);
+        return false;
+    }
+    pixel_mask.LoadDarkBadPixelMask(local_experiment, mask_output.receiver_output.dark_mask_result);
+    SetState(JFJochState::Idle);
+    return true;
 }
 
-void JFJochStateMachine::TakePedestalInternalG0(std::unique_lock<std::mutex> &ul) {
+bool JFJochStateMachine::TakePedestalInternalG0(std::unique_lock<std::mutex> &ul) {
     DiffractionExperiment local_experiment(experiment);
     std::string message;
     if (local_experiment.IsFixedGainG1()) {
@@ -171,12 +189,12 @@ void JFJochStateMachine::TakePedestalInternalG0(std::unique_lock<std::mutex> &ul
     if (cancel_sequence) {
         SetState(JFJochState::Inactive,
                  "Pedestal sequence cancelled",
-                 BrokerStatus::MessageSeverity::Warning);
-        return;
+                 BrokerStatus::MessageSeverity::Error);
+        return false;
     }
 
     if (local_experiment.GetPedestalG0Frames() == 0)
-        return;
+        return true;
 
     SetState(JFJochState::Calibration, message, BrokerStatus::MessageSeverity::Info);
     services.ConfigureDetector(local_experiment);
@@ -190,15 +208,17 @@ void JFJochStateMachine::TakePedestalInternalG0(std::unique_lock<std::mutex> &ul
     auto pedestal_output = services.Stop();
     ul.lock();
 
-    if (ImportPedestalG0(pedestal_output.receiver_output))
-        SetState(JFJochState::Idle);
-    else
-        SetState(JFJochState::Error,
+    if (!ImportPedestalG0(pedestal_output.receiver_output)) {
+        SetState(JFJochState::Inactive,
                  "Pedestal not collected properly",
                  BrokerStatus::MessageSeverity::Error);
+        return false;
+    }
+    SetState(JFJochState::Idle);
+    return true;
 }
 
-void JFJochStateMachine::TakePedestalInternalG1(std::unique_lock<std::mutex> &ul, int32_t storage_cell) {
+bool JFJochStateMachine::TakePedestalInternalG1(std::unique_lock<std::mutex> &ul, int32_t storage_cell) {
     DiffractionExperiment local_experiment(experiment);
     local_experiment.Mode(DetectorMode::PedestalG1);
 
@@ -211,12 +231,12 @@ void JFJochStateMachine::TakePedestalInternalG1(std::unique_lock<std::mutex> &ul
     if (cancel_sequence) {
         SetState(JFJochState::Inactive,
                  "Pedestal sequence cancelled",
-                 BrokerStatus::MessageSeverity::Warning);
-        return;
+                 BrokerStatus::MessageSeverity::Error);
+        return false;
     }
 
     if (local_experiment.GetPedestalG1Frames() == 0)
-        return;
+        return true;
 
 
     SetState(JFJochState::Calibration,
@@ -233,13 +253,16 @@ void JFJochStateMachine::TakePedestalInternalG1(std::unique_lock<std::mutex> &ul
     auto pedestal_output = services.Stop();
     ul.lock();
 
-    if (!ImportPedestalG1G2(pedestal_output.receiver_output, 1, storage_cell))
-        SetState(JFJochState::Error,
+    if (!ImportPedestalG1G2(pedestal_output.receiver_output, 1, storage_cell)) {
+        SetState(JFJochState::Inactive,
                  "Pedestal not collected properly",
                  BrokerStatus::MessageSeverity::Error);
+        return false;
+    }
+    return true;
 }
 
-void JFJochStateMachine::TakePedestalInternalG2(std::unique_lock<std::mutex> &ul, int32_t storage_cell) {
+bool JFJochStateMachine::TakePedestalInternalG2(std::unique_lock<std::mutex> &ul, int32_t storage_cell) {
     DiffractionExperiment local_experiment(experiment);
     local_experiment.Mode(DetectorMode::PedestalG2);
 
@@ -251,12 +274,12 @@ void JFJochStateMachine::TakePedestalInternalG2(std::unique_lock<std::mutex> &ul
     if (cancel_sequence) {
         SetState(JFJochState::Inactive,
                  "Pedestal sequence cancelled",
-                 BrokerStatus::MessageSeverity::Warning);
-        return;
+                 BrokerStatus::MessageSeverity::Error);
+        return false;
     }
 
     if (local_experiment.GetPedestalG2Frames() == 0)
-        return;
+        return true;
 
 
     SetState(JFJochState::Calibration,
@@ -273,10 +296,13 @@ void JFJochStateMachine::TakePedestalInternalG2(std::unique_lock<std::mutex> &ul
     auto pedestal_output = services.Stop();
     ul.lock();
 
-    if (!ImportPedestalG1G2(pedestal_output.receiver_output, 2, storage_cell))
-        SetState(JFJochState::Error,
+    if (!ImportPedestalG1G2(pedestal_output.receiver_output, 2, storage_cell)) {
+        SetState(JFJochState::Inactive,
                  "Pedestal not collected properly",
                  BrokerStatus::MessageSeverity::Error);
+        return false;
+    }
+    return true;
 }
 
 void JFJochStateMachine::Initialize() {
@@ -289,6 +315,7 @@ void JFJochStateMachine::Initialize() {
         throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Detector information not provided");
 
     ResetError(); // Clear error, we don't care what was it
+    start_exception = nullptr; // Re-initialising discards a pending start failure
 
     logger.Info("Initialize");
     SetState(JFJochState::Busy, "Configuring indexing threads", BrokerStatus::MessageSeverity::Info);
@@ -311,6 +338,7 @@ void JFJochStateMachine::Pedestal() {
     if (state != JFJochState::Idle)
         throw WrongDAQStateException("Must be idle to take pedestal");
 
+    start_exception = nullptr; // A new operation supersedes a pending start failure
     SetState(JFJochState::Busy, "Updating calibration", BrokerStatus::MessageSeverity::Info);
 
     measurement = std::async(std::launch::async, &JFJochStateMachine::CalibrateDetector, this, std::move(ul));
@@ -334,6 +362,9 @@ void JFJochStateMachine::InitializeThread(std::unique_lock<std::mutex> ul) {
     } catch (const std::exception &e) {
         logger.Error("Initialize error {}", e.what());
         SetState(JFJochState::Error, e.what(), BrokerStatus::MessageSeverity::Error);
+        // Wake anyone in WaitTillNotBusy/WaitTillMeasurementDone - the state has left Busy, and
+        // without this they sleep out their whole timeout before noticing.
+        c.notify_all(); // ul unlocks on the way out
         throw;
     }
     CalibrateDetector(std::move(ul));
@@ -352,6 +383,10 @@ void JFJochStateMachine::Start(const DatasetSettings &settings, bool async) {
     if (measurement.valid())
         measurement.get(); // In case measurement was running - clear thread
 
+    // Clear before ImportDatasetSettings, which can throw: a rejected /start must not leave the
+    // previous run's failure behind for the next wait call to report.
+    start_exception = nullptr;
+
     experiment.ImportDatasetSettings(settings);
 
     cancel_sequence = false;
@@ -362,24 +397,28 @@ void JFJochStateMachine::Start(const DatasetSettings &settings, bool async) {
 
     experiment.IncrementRunNumber();
 
-    start_exception = nullptr;
     SetState(JFJochState::Busy, "Preparing measurement", BrokerStatus::MessageSeverity::Info);
     measurement = std::async(std::launch::async, &JFJochStateMachine::MeasurementThread, this);
     if (!async) {
         c.wait(ul, [&]() { return state != JFJochState::Busy; });
         // A synchronous start propagates the failure to the caller. The state has already been set
         // by MeasurementThread (Idle for an ordinary failure, Error for a critical detector fault).
-        if (start_exception) {
-            auto e = start_exception;
-            start_exception = nullptr;
-            std::rethrow_exception(e);
-        }
+        // start_exception is left in place - the next Start() or Initialize() clears it - so that a
+        // wait call made afterwards reports the same failure instead of an apparent timeout.
+        if (start_exception)
+            std::rethrow_exception(start_exception);
     }
 }
 
 BrokerStatus JFJochStateMachine::WaitTillNotBusy(std::chrono::milliseconds timeout) {
     std::unique_lock ul(m);
     c.wait_for(ul, timeout, [&]() { return state != JFJochState::Busy; });
+    // An asynchronous start reports its failure here, since /start itself returned before the
+    // measurement thread ran. Without this the state is plain Idle and the caller cannot tell a
+    // failed start from a timeout. rethrow_exception does not consume the exception_ptr, so
+    // repeated calls all report the same failure.
+    if (start_exception)
+        std::rethrow_exception(start_exception);
     return GetStatus();
 }
 
@@ -606,6 +645,7 @@ void JFJochStateMachine::LoadDetectorSettings(const DetectorSettings &settings)
             break;
         case JFJochState::Idle:
             if (ImportDetectorSettings(settings)) {
+                start_exception = nullptr; // A new operation supersedes a pending start failure
                 SetState(JFJochState::Busy, "Loading settings", BrokerStatus::MessageSeverity::Info);
                 measurement = std::async(std::launch::async, &JFJochStateMachine::CalibrateDetector, this, std::move(ul));
             } else {
@@ -801,6 +841,11 @@ BrokerStatus JFJochStateMachine::WaitTillMeasurementDone() {
 
     c.wait(ul, [&] { return !IsRunning(); });
 
+    // A start that failed asynchronously never reached Measuring, so the state is Idle and would
+    // otherwise be reported as a successfully finished collection.
+    if (start_exception)
+        std::rethrow_exception(start_exception);
+
     return GetStatus();
 }
 
@@ -809,6 +854,9 @@ BrokerStatus JFJochStateMachine::WaitTillMeasurementDone(std::chrono::millisecon
 
     c.wait_for(ul, timeout, [&] { return !IsRunning(); });
 
+    if (start_exception)
+        std::rethrow_exception(start_exception);
+
     return GetStatus();
 }
 
@@ -1158,6 +1206,7 @@ void JFJochStateMachine::SetDarkMaskSettings(const DarkMaskSettings &settings) {
     }
     if ((experiment.GetDetectorType() == DetectorType::DECTRIS) && (state == JFJochState::Idle)) {
         // Need to redo the calibration
+        start_exception = nullptr; // A new operation supersedes a pending start failure
         SetState(JFJochState::Busy, "Loading settings", BrokerStatus::MessageSeverity::Info);
         measurement = std::async(std::launch::async, &JFJochStateMachine::CalibrateDetector, this, std::move(ul));
     }