v1.0.0.rc-161 (#71)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.

**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.

**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.

**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**

Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/indexing/IndexerThreadPool.cpp

@@ -14,10 +14,28 @@
 #include "FFTIndexerCPU.h"
 #endif
 
-IndexerThread::IndexerThread(const IndexingSettings &settings, int threadid) {
+// The indexer for one RESOLVED algorithm, or nullptr if this build/host cannot serve it.
+static std::unique_ptr<Indexer> MakeIndexer(IndexingAlgorithmEnum algorithm, const IndexingSettings &settings) {
+#ifdef JFJOCH_USE_CUDA
+    if (get_gpu_count() > 0) {
+        if (algorithm == IndexingAlgorithmEnum::FFT)
+            return std::make_unique<FFTIndexerGPU>(settings);
+        if (algorithm == IndexingAlgorithmEnum::FFBIDX)
+            return std::make_unique<FFBIDXIndexer>();
+    }
+#endif
+#ifdef JFJOCH_USE_FFTW
+    if (algorithm == IndexingAlgorithmEnum::FFTW)
+        return std::make_unique<FFTIndexerCPU>(settings);
+#endif
+    return nullptr;
+}
+
+IndexerThread::IndexerThread(const IndexingSettings &settings, int threadid, IndexerConstruction construction)
+    : settings_(settings), construction_(construction) {
     std::unique_lock<std::mutex> lock(m);
     state = TaskState::STARTING;
-    worker_thread = std::thread(&IndexerThread::Worker, this, std::cref(settings), threadid);
+    worker_thread = std::thread(&IndexerThread::Worker, this, threadid);
     c_running.wait(lock, [this] { return state != TaskState::STARTING; });
     if (state == TaskState::ERROR) {
         worker_thread.join();
@@ -26,7 +44,7 @@ IndexerThread::IndexerThread(const IndexingSettings &settings, int threadid) {
     }
 }
 
-void IndexerThread::Worker(const IndexingSettings &settings, int threadid) {
+void IndexerThread::Worker(int threadid) {
     try {
         pin_gpu();
     } catch (const std::exception &e) {
@@ -37,58 +55,42 @@ void IndexerThread::Worker(const IndexingSettings &settings, int threadid) {
 
     std::unique_ptr<Indexer> fft_indexer, ffbidx_indexer, fftw_indexer;
 
-#ifdef JFJOCH_USE_CUDA
-    try {
-        if (get_gpu_count() > 0) {
-            if (settings.GetAlgorithm() == IndexingAlgorithmEnum::Auto
-                || settings.GetAlgorithm() == IndexingAlgorithmEnum::FFT)
-                fft_indexer = std::make_unique<FFTIndexerGPU>(settings);
+    // Preconstruct: build every indexer the requested algorithm could resolve to before the pool
+    // reports ready, so no cuFFT planning happens once frames are flowing, and a failure is fatal
+    // for the pool instead of being met frame by frame. OnFirstUse skips this and builds in the
+    // dispatch below.
+    if (construction_ == IndexerConstruction::Preconstruct) {
+        try {
+            const auto requested = settings_.GetAlgorithm();
 
-            if (settings.GetAlgorithm() == IndexingAlgorithmEnum::Auto
-                || settings.GetAlgorithm() == IndexingAlgorithmEnum::FFBIDX)
-                ffbidx_indexer = std::make_unique<FFBIDXIndexer>();
+            if (requested == IndexingAlgorithmEnum::Auto || requested == IndexingAlgorithmEnum::FFT)
+                fft_indexer = MakeIndexer(IndexingAlgorithmEnum::FFT, settings_);
+
+            if (requested == IndexingAlgorithmEnum::Auto || requested == IndexingAlgorithmEnum::FFBIDX)
+                ffbidx_indexer = MakeIndexer(IndexingAlgorithmEnum::FFBIDX, settings_);
+
+            if ((requested == IndexingAlgorithmEnum::Auto && get_gpu_count() == 0)
+                || requested == IndexingAlgorithmEnum::FFTW)
+                fftw_indexer = MakeIndexer(IndexingAlgorithmEnum::FFTW, settings_);
+        } catch (const std::exception &e) {
+            spdlog::error("Failed to initialize indexer: {}", e.what());
+            {
+                std::unique_lock<std::mutex> lock(m);
+                state = TaskState::ERROR;
+            }
+            c_running.notify_all();
+            return;
+        } catch (...) {
+            spdlog::error("Failed to initialize indexer");
+            {
+                std::unique_lock<std::mutex> lock(m);
+                state = TaskState::ERROR;
+            }
+            c_running.notify_all();
+            return;
         }
-    } catch (const std::exception &e) {
-        spdlog::error("Failed to initialize GPU indexer: {}", e.what());
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
-    } catch (...) {
-        spdlog::error("Failed to initialize GPU indexer");
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
     }
-#endif
-#ifdef JFJOCH_USE_FFTW
-    try {
-        if ((settings.GetAlgorithm() == IndexingAlgorithmEnum::Auto && (get_gpu_count() == 0))
-            || settings.GetAlgorithm() == IndexingAlgorithmEnum::FFTW)
-            fftw_indexer = std::make_unique<FFTIndexerCPU>(settings);
-    } catch (const std::exception &e) {
-        spdlog::error("Failed to initialize FFTW indexer: {}", e.what());
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
-    } catch (...) {
-        spdlog::error("Failed to initialize FFTW indexer");
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
-    }
-#endif
+
     {
         std::unique_lock<std::mutex> lock(m);
         state = TaskState::IDLE;
@@ -110,28 +112,33 @@ void IndexerThread::Worker(const IndexingSettings &settings, int threadid) {
             std::unique_ptr<IndexerResult> tmp_result;
             try {
                 auto algorithm = input->experiment.GetIndexingAlgorithm();
-                Indexer *indexer = nullptr;
+                std::unique_ptr<Indexer> *slot = nullptr;
 
-                if (algorithm == IndexingAlgorithmEnum::FFT && fft_indexer) {
-                    indexer = fft_indexer.get();
-                } else if (algorithm == IndexingAlgorithmEnum::FFBIDX && ffbidx_indexer) {
-                    indexer = ffbidx_indexer.get();
-                } else if (algorithm == IndexingAlgorithmEnum::FFTW && fftw_indexer) {
-                    indexer = fftw_indexer.get();
-                } else {
+                switch (algorithm) {
+                    case IndexingAlgorithmEnum::FFT:    slot = &fft_indexer;    break;
+                    case IndexingAlgorithmEnum::FFBIDX: slot = &ffbidx_indexer; break;
+                    case IndexingAlgorithmEnum::FFTW:   slot = &fftw_indexer;   break;
+                    default: break;
+                }
+
+                // A preconstructing worker already holds it; an OnFirstUse worker builds it here,
+                // on the first frame that resolves to this algorithm.
+                if (slot && !*slot)
+                    *slot = MakeIndexer(algorithm, settings_);
+
+                if (!slot || !*slot) {
                     // Algorithm is already resolved here (never Auto/None - see
-                    // IndexerThreadPool::Run). Reaching this means the resolved algorithm
-                    // has no matching indexer in this worker (e.g. a GPU algorithm on a
-                    // host without a GPU) - fail loudly instead of silently not indexing.
+                    // IndexerThreadPool::Run, which also checked this host can serve it). Reaching
+                    // this means the resolved algorithm has no matching indexer in this build -
+                    // fail loudly instead of silently not indexing.
                     throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
                                           "Internal error: no indexer available for the resolved "
                                           "indexing algorithm");
                 }
 
-                if (indexer) {
-                    indexer->Setup(input->experiment);
-                    tmp_result = std::make_unique<IndexerResult>(indexer->Run(input->recip));
-                }
+                Indexer &indexer = **slot;
+                indexer.Setup(input->experiment);
+                tmp_result = std::make_unique<IndexerResult>(indexer.Run(input->recip));
             } catch (std::exception &e) {
                 tmp_result = nullptr;
                 spdlog::error("Indexer thread {} failed: {}", threadid, e.what());
@@ -182,13 +189,13 @@ IndexerThread::~IndexerThread() {
     Finalize();
 }
 
-IndexerThreadPool::IndexerThreadPool(const IndexingSettings &settings)
+IndexerThreadPool::IndexerThreadPool(const IndexingSettings &settings, IndexerConstruction construction)
     : worker_busy(settings.GetIndexingThreads(), 0),
       worker_free_count(settings.GetIndexingThreads()),
       viable_cell_min_spots(settings.GetViableCellMinSpots()),
       blocking(settings.GetBlockingBehavior()) {
     for (size_t i = 0; i < settings.GetIndexingThreads(); ++i)
-        tasks.emplace_back(std::make_unique<IndexerThread>(std::cref(settings), i));
+        tasks.emplace_back(std::make_unique<IndexerThread>(std::cref(settings), i, construction));
 }
 
 int IndexerThreadPool::GetFreeWorker() {