indexing: select predicted reflections by partiality, build indexers where it pays

When more reflections are predicted for a frame than the output can hold, the
surplus was dropped by keeping those closest to the Ewald sphere. On the rotation
path that quantity is identically zero by construction - the rocking coordinate is
chosen so the scattering vector lands exactly on the sphere - so the comparison
fell through to h, k and l and the survivors were whichever came first in
lexicographic order. Measured on a large cell: every value within one float ulp of
zero, and the kept set had a MEAN PARTIALITY BELOW that of the full set, i.e. worse
than choosing at random. Rank by partiality instead, which the predictor already
computes and which is what the header always claimed was being kept. On the one
regression crystal large enough to cross the cap this lifts completeness from 84.8%
to 90.2% on the same observations; multiplicity and R_meas move the way they must
when the same measurements cover more of reciprocal space.

The online path asked for a cap of ten thousand but the truncation was hardcoded to
the offline limit, so the broker predicted and integrated up to six times what it
could transport and discarded the rest after paying for it. Honour the caller's
limit, which also makes the post-integration re-truncation dead code.

Indexer pool construction becomes a policy. The online service needs every indexer
resident before data arrives, because a cuFFT plan built on the first frame is
planning time inside the measurement; spending memory to be ready is the intended
trade there and stays the default. Offline there is no such deadline, and a stills
run with a known cell was holding a fully allocated FFT indexer per worker that the
algorithm resolution can never dispatch - 2.8 GB where 0.4 GB is needed. rugnux and
the viewer opt into building on first use; the broker, the receiver and the tests
are untouched. This also removes a dangling reference that was latent: the worker
held the settings by reference although the pool is routinely constructed from a
temporary, which only survived because eager construction finished inside the
constructor call.

Finally, refuse a first-pass lattice that indexes fewer than a sixth of the
validation frames. It fires on nothing in the regression set - the weakest real
crystal sits at 22 of 60, more than twice the floor - so it is a backstop, but the
failure it prevents is one the set does contain: a dataset with no crystal at all
adopts a lattice from its powder rings, integrates every image against it, and dies
much later inside the merge complaining about resolution. It now stops in the first
pass and says what to try.

Regression set: 36 of 37 crystals byte-identical, the exception being the
completeness gain above; 34 of 37 space groups, no failures. Full unit suite passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

image_analysis/indexing/IndexerThreadPool.cpp

@@ -14,10 +14,28 @@
 #include "FFTIndexerCPU.h"
 #endif
 
-IndexerThread::IndexerThread(const IndexingSettings &settings, int threadid) {
+// The indexer for one RESOLVED algorithm, or nullptr if this build/host cannot serve it.
+static std::unique_ptr<Indexer> MakeIndexer(IndexingAlgorithmEnum algorithm, const IndexingSettings &settings) {
+#ifdef JFJOCH_USE_CUDA
+    if (get_gpu_count() > 0) {
+        if (algorithm == IndexingAlgorithmEnum::FFT)
+            return std::make_unique<FFTIndexerGPU>(settings);
+        if (algorithm == IndexingAlgorithmEnum::FFBIDX)
+            return std::make_unique<FFBIDXIndexer>();
+    }
+#endif
+#ifdef JFJOCH_USE_FFTW
+    if (algorithm == IndexingAlgorithmEnum::FFTW)
+        return std::make_unique<FFTIndexerCPU>(settings);
+#endif
+    return nullptr;
+}
+
+IndexerThread::IndexerThread(const IndexingSettings &settings, int threadid, IndexerConstruction construction)
+    : settings_(settings), construction_(construction) {
     std::unique_lock<std::mutex> lock(m);
     state = TaskState::STARTING;
-    worker_thread = std::thread(&IndexerThread::Worker, this, std::cref(settings), threadid);
+    worker_thread = std::thread(&IndexerThread::Worker, this, threadid);
     c_running.wait(lock, [this] { return state != TaskState::STARTING; });
     if (state == TaskState::ERROR) {
         worker_thread.join();
@@ -26,7 +44,7 @@ IndexerThread::IndexerThread(const IndexingSettings &settings, int threadid) {
     }
 }
 
-void IndexerThread::Worker(const IndexingSettings &settings, int threadid) {
+void IndexerThread::Worker(int threadid) {
     try {
         pin_gpu();
     } catch (const std::exception &e) {
@@ -37,58 +55,42 @@ void IndexerThread::Worker(const IndexingSettings &settings, int threadid) {
 
     std::unique_ptr<Indexer> fft_indexer, ffbidx_indexer, fftw_indexer;
 
-#ifdef JFJOCH_USE_CUDA
-    try {
-        if (get_gpu_count() > 0) {
-            if (settings.GetAlgorithm() == IndexingAlgorithmEnum::Auto
-                || settings.GetAlgorithm() == IndexingAlgorithmEnum::FFT)
-                fft_indexer = std::make_unique<FFTIndexerGPU>(settings);
+    // Preconstruct: build every indexer the requested algorithm could resolve to before the pool
+    // reports ready, so no cuFFT planning happens once frames are flowing, and a failure is fatal
+    // for the pool instead of being met frame by frame. OnFirstUse skips this and builds in the
+    // dispatch below.
+    if (construction_ == IndexerConstruction::Preconstruct) {
+        try {
+            const auto requested = settings_.GetAlgorithm();
 
-            if (settings.GetAlgorithm() == IndexingAlgorithmEnum::Auto
-                || settings.GetAlgorithm() == IndexingAlgorithmEnum::FFBIDX)
-                ffbidx_indexer = std::make_unique<FFBIDXIndexer>();
+            if (requested == IndexingAlgorithmEnum::Auto || requested == IndexingAlgorithmEnum::FFT)
+                fft_indexer = MakeIndexer(IndexingAlgorithmEnum::FFT, settings_);
+
+            if (requested == IndexingAlgorithmEnum::Auto || requested == IndexingAlgorithmEnum::FFBIDX)
+                ffbidx_indexer = MakeIndexer(IndexingAlgorithmEnum::FFBIDX, settings_);
+
+            if ((requested == IndexingAlgorithmEnum::Auto && get_gpu_count() == 0)
+                || requested == IndexingAlgorithmEnum::FFTW)
+                fftw_indexer = MakeIndexer(IndexingAlgorithmEnum::FFTW, settings_);
+        } catch (const std::exception &e) {
+            spdlog::error("Failed to initialize indexer: {}", e.what());
+            {
+                std::unique_lock<std::mutex> lock(m);
+                state = TaskState::ERROR;
+            }
+            c_running.notify_all();
+            return;
+        } catch (...) {
+            spdlog::error("Failed to initialize indexer");
+            {
+                std::unique_lock<std::mutex> lock(m);
+                state = TaskState::ERROR;
+            }
+            c_running.notify_all();
+            return;
         }
-    } catch (const std::exception &e) {
-        spdlog::error("Failed to initialize GPU indexer: {}", e.what());
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
-    } catch (...) {
-        spdlog::error("Failed to initialize GPU indexer");
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
     }
-#endif
-#ifdef JFJOCH_USE_FFTW
-    try {
-        if ((settings.GetAlgorithm() == IndexingAlgorithmEnum::Auto && (get_gpu_count() == 0))
-            || settings.GetAlgorithm() == IndexingAlgorithmEnum::FFTW)
-            fftw_indexer = std::make_unique<FFTIndexerCPU>(settings);
-    } catch (const std::exception &e) {
-        spdlog::error("Failed to initialize FFTW indexer: {}", e.what());
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
-    } catch (...) {
-        spdlog::error("Failed to initialize FFTW indexer");
-        {
-            std::unique_lock<std::mutex> lock(m);
-            state = TaskState::ERROR;
-        }
-        c_running.notify_all();
-        return;
-    }
-#endif
+
     {
         std::unique_lock<std::mutex> lock(m);
         state = TaskState::IDLE;
@@ -110,28 +112,33 @@ void IndexerThread::Worker(const IndexingSettings &settings, int threadid) {
             std::unique_ptr<IndexerResult> tmp_result;
             try {
                 auto algorithm = input->experiment.GetIndexingAlgorithm();
-                Indexer *indexer = nullptr;
+                std::unique_ptr<Indexer> *slot = nullptr;
 
-                if (algorithm == IndexingAlgorithmEnum::FFT && fft_indexer) {
-                    indexer = fft_indexer.get();
-                } else if (algorithm == IndexingAlgorithmEnum::FFBIDX && ffbidx_indexer) {
-                    indexer = ffbidx_indexer.get();
-                } else if (algorithm == IndexingAlgorithmEnum::FFTW && fftw_indexer) {
-                    indexer = fftw_indexer.get();
-                } else {
+                switch (algorithm) {
+                    case IndexingAlgorithmEnum::FFT:    slot = &fft_indexer;    break;
+                    case IndexingAlgorithmEnum::FFBIDX: slot = &ffbidx_indexer; break;
+                    case IndexingAlgorithmEnum::FFTW:   slot = &fftw_indexer;   break;
+                    default: break;
+                }
+
+                // A preconstructing worker already holds it; an OnFirstUse worker builds it here,
+                // on the first frame that resolves to this algorithm.
+                if (slot && !*slot)
+                    *slot = MakeIndexer(algorithm, settings_);
+
+                if (!slot || !*slot) {
                     // Algorithm is already resolved here (never Auto/None - see
-                    // IndexerThreadPool::Run). Reaching this means the resolved algorithm
-                    // has no matching indexer in this worker (e.g. a GPU algorithm on a
-                    // host without a GPU) - fail loudly instead of silently not indexing.
+                    // IndexerThreadPool::Run, which also checked this host can serve it). Reaching
+                    // this means the resolved algorithm has no matching indexer in this build -
+                    // fail loudly instead of silently not indexing.
                     throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
                                           "Internal error: no indexer available for the resolved "
                                           "indexing algorithm");
                 }
 
-                if (indexer) {
-                    indexer->Setup(input->experiment);
-                    tmp_result = std::make_unique<IndexerResult>(indexer->Run(input->recip));
-                }
+                Indexer &indexer = **slot;
+                indexer.Setup(input->experiment);
+                tmp_result = std::make_unique<IndexerResult>(indexer.Run(input->recip));
             } catch (std::exception &e) {
                 tmp_result = nullptr;
                 spdlog::error("Indexer thread {} failed: {}", threadid, e.what());
@@ -182,13 +189,13 @@ IndexerThread::~IndexerThread() {
     Finalize();
 }
 
-IndexerThreadPool::IndexerThreadPool(const IndexingSettings &settings)
+IndexerThreadPool::IndexerThreadPool(const IndexingSettings &settings, IndexerConstruction construction)
     : worker_busy(settings.GetIndexingThreads(), 0),
       worker_free_count(settings.GetIndexingThreads()),
       viable_cell_min_spots(settings.GetViableCellMinSpots()),
       blocking(settings.GetBlockingBehavior()) {
     for (size_t i = 0; i < settings.GetIndexingThreads(); ++i)
-        tasks.emplace_back(std::make_unique<IndexerThread>(std::cref(settings), i));
+        tasks.emplace_back(std::make_unique<IndexerThread>(std::cref(settings), i, construction));
 }
 
 int IndexerThreadPool::GetFreeWorker() {