Stills -C: a volume-matched distance hypothesis and the de novo warning from the trial sample

Without the separate de novo geometry fit, a cell given fitted from a header distance 3% short on weak frames
settled 3% long, and the spot-count distance search, flat to 1-2% about the right distance, picked a peak 1.5%
off from which the refit could not recover (merge CC1/2 0.02). The stills geometry trial now also indexes its
96-frame sample de novo at the distance it decided; the distance at which the cell given has the volume of the
frames' de novo cell is a hypothesis of its own, taken where it puts >= 0.9x as many sampled spots on the
lattice and lies more than half the 1% fine step away. The same de novo cell, as a known cell, gives the
warning where it puts materially more spots on the lattice than the cell given (-C is still always used).
Cost: two trials on the 96-frame sample (about a second); nothing in live, which runs no geometry trial.
StillsTrial carries the consensus cell of a hypothesis indexed without a cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 456ead9c802b2747437f74dcd06eed146cf260de)

docs/CPU_DATA_ANALYSIS_IMAGE.md

@@ -302,7 +302,12 @@ closely and a finer step about the best decides against the file's distance, whi
 puts at least 1/0.9 as many spots on the lattice; a local grid of centres follows at the distance decided.
 The range is in millimetres, not in multiples of the file's value, so a stated camera length that is
 nominal or wrong does not limit where the search looks, and an input that states no distance (an XTC) is
-searched too. A distance given with `--detector-distance` is held: it is neither searched nor refined.
+searched too. The spot count is flat to a percent or two about the right distance, so with `-C` the same sample is also
+indexed de novo at the distance decided: a frame's de novo cell takes up the distance's error in its scale, and
+the distance at which the cell given has the volume of the frames' cell is taken where it puts at least 0.9 times
+as many sampled spots on the lattice and lies more than half the 1 % fine step from the distance decided. The same
+de novo cell, as a known cell, is the frames' evidence on the cell given (§7.7 of the indexing page). A distance
+given with `--detector-distance` is held: it is neither searched nor refined.
 Without a cell nothing is searched in distance: on its own a wrong distance indexes as a wrong cell, not
 as fewer indexed spots. The stills geometry refinement, `--refine-geometry`, bundle-adjusts the
 centre and the distance together with the cell over strongly indexed frames (§7.7); it is on by default for

docs/CPU_DATA_ANALYSIS_INDEXING.md

@@ -429,11 +429,11 @@ cell in any setting of the same lattice, and are described on its axes.
 With `-C` the run always uses the cell given; the frames' own evidence is reported, never acted on. Two
 warnings say where it disagrees with them, each with both numbers. Where the geometry refinement moves the cell's
 shape (a length beyond the common scale, which trades with the distance) more than 3 % or an angle more than 3°
-from what was given. And where, at the centre and distance the stills geometry trial decides (§1.4), the trial's
-96-frame sample indexed de novo puts more than 1/0.9 times as many spots on the lattice as the cell given: the cell
-given may then be another crystal form than most of the frames, or a shape they do not have - the same lattice in a
-shape it does not have still indexes the frames, at a distance of its own, and merges far worse than the run
-without `-C`.
+from what was given. And where, at the centre and distance the stills geometry trial decides (§1.4), the cell
+the trial's 96-frame sample indexes to de novo puts more than 1/0.9 times as many of its spots on the lattice as
+the cell given: the cell given may then be another crystal form than most of the frames, or a shape they do not
+have - the same lattice in a shape it does not have still indexes the frames, at a distance of its own, and
+merges far worse than the run without `-C`.
 
 **One crystal form.** A sample can hold two crystal forms, and a known-cell indexer fits both to the
 given cell's orientation search; their refined cells then fall into two volume populations, and

rugnux/Rugnux.h

@@ -1768,6 +1768,7 @@ class Rugnux {
         int indexed = 0;           // frames that indexed
         int64_t spots = 0;         // spots found on the frames that indexed
         int64_t on_lattice = 0;    // ... of which on the lattice
+        std::optional<UnitCell> cell;   // a hypothesis without a cell: the cell most of its frames indexed to
     };
     std::vector<DataMessage> FindStillsSampleSpots(const std::vector<int> &ordinals, int start_image);
     // A geometry hypothesis and the frames it is indexed on.

rugnux/RugnuxStills.cpp

@@ -934,6 +934,9 @@ std::vector<Rugnux::StillsTrial> Rugnux::IndexStillsSample(const std::vector<Sti
         f.get();
     for (const auto &indexer : indexers)
         indexer->RethrowIndexerError();
+    for (size_t h = 0; h < hypotheses.size(); h++)
+        if (!hypotheses[h].x.GetUnitCell())
+            total[h].cell = indexers[h]->GetConsensusUnitCell();
     return total;
 }
 
@@ -965,6 +968,8 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) {
     constexpr float STEP = 1.025f, MIN_DISTANCE_MM = 30.0f, MAX_DISTANCE_MM = 3000.0f;
     // The indexing tolerance of the scan, in reciprocal-lattice units (the run's default is 0.1).
     constexpr float SCAN_TOLERANCE = 0.2f;
+    // The step of the fine search about the scan's best (fine, below).
+    constexpr float FINE_STEP = 0.01f;
     // How many of the scan's local maxima are looked at more closely.
     constexpr size_t SCAN_PEAKS = 3;
     // A sample found at a distance this far from the one decided is found again there (see below).
@@ -1123,10 +1128,11 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) {
         int64_t best = score_here;
         const auto fine = [&](float about, float cx, float cy) {
             std::vector<TrialKey> keys;
-            for (const float f : {0.98f, 0.99f, 1.0f, 1.01f, 1.02f})
+            const float steps[] = {1 - 2 * FINE_STEP, 1 - FINE_STEP, 1, 1 + FINE_STEP, 1 + 2 * FINE_STEP};
+            for (const float f : steps)
                 keys.emplace_back(cx, cy, about * f, &frames);
             try_all(keys);
-            for (const float f : {0.98f, 0.99f, 1.0f, 1.01f, 1.02f}) {
+            for (const float f : steps) {
                 const int64_t score = trial(cx, cy, about * f, frames);
                 if (score > best) {
                     best = score;
@@ -1234,6 +1240,49 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) {
                         distance, best, best_distance);
             distance = best_distance;
         }
+        // With -C, the same sample indexed de novo, at the centre and distance decided. The count of spots on the
+        // lattice is flat to a percent or two about the right distance (a run 1.5% off scored as well as 0.5%
+        // off), and a cell given that is fitted from a start that far off can stay there; the frames' own cell
+        // places the distance more finely. Indexed de novo, a frame's cell takes up the distance's error in its
+        // scale, so the distance at which the cell given has the volume of the frames' cell (the cube root of
+        // the volume ratio) is a hypothesis of its own, taken where it puts as many of the sample's spots on the
+        // lattice as the distance decided, or nearly (FILE_KEPT_ABOVE), and lies further from it than half the fine
+        // step can tell (a distance the geometry refinement has fitted is better than that). The cell given is used whatever the
+        // frames say; where their de novo cell, as a known cell, puts materially more of the sample's spots on
+        // the lattice, the log warns with both counts - the cell given may be another crystal form than most of
+        // the frames, or a shape they do not have (the same lattice in a wrong shape still indexes the frames, at
+        // a distance of its own, and merges far worse than the run without -C). Two trials on the sample.
+        if (stills_user_cell_) {
+            DiffractionExperiment e = experiment_;
+            place(e, x, y, distance);
+            e.SetUnitCell(std::nullopt);
+            const std::optional<UnitCell> frames_cell = IndexStillsSample({{e, &frames}}, pool).at(0).cell;
+            if (frames_cell) {
+                e.SetUnitCell(*frames_cell);
+                const int64_t frames_count = IndexStillsSample({{e, &frames}}, pool).at(0).on_lattice;
+                const float by_volume = distance * static_cast<float>(std::cbrt(
+                        CrystalLattice(*stills_user_cell_).CalcVolume() / CrystalLattice(*frames_cell).CalcVolume()));
+                try_all({{x, y, distance, &frames}, {x, y, by_volume, &frames}});
+                const int64_t here = trial(x, y, distance, frames), there = trial(x, y, by_volume, frames);
+                logger.Info("Stills detector distance: the frames index de novo to {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} "
+                            "{:.2f} at {:.2f} mm, which the cell given matches in volume at {:.2f} mm: {} sampled spots "
+                            "on the lattice there, {} at {:.2f} mm", frames_cell->a, frames_cell->b, frames_cell->c,
+                            frames_cell->alpha, frames_cell->beta, frames_cell->gamma, distance, by_volume, there,
+                            here, distance);
+                if (there >= FILE_KEPT_ABOVE * here && std::abs(by_volume / distance - 1.0f) > FINE_STEP / 2)
+                    distance = by_volume;
+                const int64_t given = std::max(here, there);
+                if (given < FILE_KEPT_ABOVE * frames_count)
+                    logger.Warning("The cell given (-C) {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} puts {} of a sample's spots "
+                                   "on the lattice, the cell the frames index to de novo ({:.2f} {:.2f} {:.2f} {:.2f} {:.2f} "
+                                   "{:.2f}) {}: the run uses the cell given, but it may be another crystal form than most "
+                                   "of the frames, or a shape they do not have - compare with a run without -C",
+                                   stills_user_cell_->a, stills_user_cell_->b, stills_user_cell_->c,
+                                   stills_user_cell_->alpha, stills_user_cell_->beta, stills_user_cell_->gamma, given,
+                                   frames_cell->a, frames_cell->b, frames_cell->c, frames_cell->alpha,
+                                   frames_cell->beta, frames_cell->gamma, frames_count);
+            }
+        }
         // A centre decided at a distance this far off had little indexing to go on; decided again here.
         if (distance != file_distance)
             decide_centre(distance);
@@ -1275,29 +1324,6 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) {
         }
     }
 
-    // With -C: the same sample indexed de novo at the geometry decided, against the cell given there. The run uses
-    // the cell given whatever this says; where the frames index materially more of their spots de novo (below
-    // FILE_KEPT_ABOVE of the de novo count, the margin a file's value is kept by), the cell given may be another
-    // crystal form than most of the frames, or a shape they do not have, and the log says so with both counts. A
-    // cell of the wrong shape still indexes the frames, at a distance of its own, and merges far worse than the run
-    // without -C. Ninety-six frames, a few seconds.
-    if (stills_user_cell_) {
-        DiffractionExperiment e = experiment_;
-        place(e, x, y, distance);
-        e.SetUnitCell(std::nullopt);
-        const int64_t given = trial(x, y, distance, frames);
-        const StillsTrial de_novo = IndexStillsSample({{e, &frames}}, pool).at(0);
-        logger.Info("Stills cell check at {:.2f} mm: the cell given puts {} sampled spots on the lattice, the frames "
-                    "indexed de novo {}", distance, given, de_novo.on_lattice);
-        if (given < FILE_KEPT_ABOVE * de_novo.on_lattice)
-            logger.Warning("The cell given (-C) {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} puts {} of a sample's spots on the "
-                           "lattice at {:.1f} mm, where the frames indexed de novo put {}: the run uses the cell given, "
-                           "but it may be another crystal form than most of the frames, or a shape they do not have - "
-                           "compare with a run without -C",
-                           stills_user_cell_->a, stills_user_cell_->b, stills_user_cell_->c, stills_user_cell_->alpha,
-                           stills_user_cell_->beta, stills_user_cell_->gamma, given, distance, de_novo.on_lattice);
-    }
-
     logger.Info("Stills beam centre: the run uses the direct beam at ({:.2f},{:.2f}){}", x, y,
                 x == file_x && y == file_y ? ", the file's" : "");
     place(experiment_, x, y, distance);