Merge broker-rugnux-live into rc176: rugnux in jfjoch_broker as a live thread during a DECTRIS rotation collection (off by default), the rugnux library without file I/O (results in ProcessResult::outputs, the CLI writes them through JFJochRugnuxOutput), END carrying the merged table, map coefficients and placed model

Branch reviewed at e16db07ff (review: owner's decisions accepted - _maps.mtz instead of .ccp4 in the
viewer, -o "" computes everything incl. MR/validation, Ctrl+C/error paths may leave fewer files, the
writer links gemmi/zlib) and merged at 55184eb09 with its review fixes (END size guard, frame drop as
in Lite, outputs released before re-runs, pool priority, docs, frontend gating).

Conflicts: rugnux/Rugnux.h (kept rc176's host_memory_budget_bytes, output_prefix removed as on the
branch), rugnux/RugnuxPipeline.h (MtzTable future + rc176's GPUMerge pointer), MolecularReplacement.h/.cpp
(rc176's GPU cost model, packed transform and split functions with the branch's RugnuxModel signature;
search_model with rc176's other_budgets, result.model_name). tests/RugnuxLargeTest.cpp: rc176's
host-budget test writes its files through WriteRugnuxOutputs now that the library writes none.
Generated code (broker/gen, frontend client, Python client docs, Redoc) regenerated from the merged
jfjoch_api.yaml with update_version.sh (version unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

image_analysis/structure_refinement/ModelValidation.h

@@ -11,18 +11,29 @@
 #include <string>
 #include <vector>
 
+#include "gemmi/grid.hpp"
 #include "gemmi/symmetry.hpp"
 
 #include "../../common/Reflection.h"   // MergedReflection
 #include "../../common/UnitCell.h"
+#include "../../common/MtzTable.h"
+#include "RugnuxModel.h"
 
 class Logger;
 namespace gemmi { struct Structure; }
 
+// The maps of a validation: their coefficients, as <prefix>_maps.mtz holds them, and the maps themselves,
+// as the .ccp4 files hold them, where the caller asked for them (ModelValidationSchedule::keep_map_grids).
+// The anomalous map is there only where the merge kept the Bijvoet split.
+struct ModelMaps {
+    MtzTable coefficients;
+    std::optional<gemmi::Grid<float>> map_2fofc, map_fofc, map_anom;
+};
+
 // Result of validating merged intensities against an atomic model.
 struct ModelValidationResult {
     bool ok = false;
-    std::string model_path;        // the file that was asked for, so the report can name it
+    std::string model_name;        // RugnuxModel::name, so the report can name the model that was asked for
     // Why it did not run, when ok is false: the file could not be read, had no atoms, no cell or no
     // usable space group. A caller that asked for a model must be able to say so rather than finish
     // quietly with no R-free and no maps, which reads exactly like a run that was never asked.
@@ -57,7 +68,11 @@ struct ModelValidationResult {
     double radial_misfit = NAN;
     int shell_scale_bins = 0;
     double mean_atom_density_sigma = 0.0;  // mean 2mFo-DFc value at atom centres, in sigma
-    std::string maps_prefix;               // where the .ccp4 / _maps.mtz were written
+    // The maps, where this validation made them; the caller takes them over (null after that).
+    std::shared_ptr<ModelMaps> maps;
+    // Where the caller puts the maps beside the reflection files: "" for the answer's, "_mr" for the
+    // model's choice. Set by the caller; the report names <prefix><maps_suffix>.
+    std::string maps_suffix;
 
     // The rigid-body placement of the model against these data, where it was committed. The step is
     // taken only when it lowers R-free, so an unmoved model means the data said no, not that nothing
@@ -208,11 +223,10 @@ struct ModelValidationResult {
     std::shared_ptr<gemmi::Structure> placed_model;
 };
 
-// Given merged intensities and an atomic model (PDB or mmCIF, gzipped or not - the format is taken
-// from the file's content), scale the model structure factors (with an
+// Given merged intensities and an atomic model, scale the model structure factors (with an
 // optimized flat bulk-solvent contribution and an overall anisotropic B) to the WORKING observed
-// amplitudes, then report R-work / R-free and write 2Fo-Fc and Fo-Fc electron-density maps
-// (CCP4) plus an MTZ of map coefficients next to output_prefix. Nothing the reported R-free is
+// amplitudes, then report R-work / R-free and make 2Fo-Fc and Fo-Fc electron-density maps and
+// their coefficients (ModelValidationResult::maps). Nothing the reported R-free is
 // computed against was fitted on the free reflections: the scale, like the rigid-body placement
 // below, sees the working set only and is then applied to all of them.
 //
@@ -242,8 +256,8 @@ struct ModelValidationResult {
 // twin_law, where given, is a twin law of the data's group (a Miller-index operator in the data's
 // basis): the model is then also scored against the data as twinned by it (twin_target and below).
 //
-// model, where given, is the model itself - one already placed in the data's cell, by molecular
-// replacement - and model_path only names it; no other setting is then probed for it.
+// placed, where given, is the model itself - one already placed in the data's cell, by molecular
+// replacement - and `model` only names it; no other setting is then probed for it.
 //
 // report_shell_d_min is the merge statistics' own shell bounds, coarse to fine, and it is what
 // cc_model_shells is binned on. Sharing the grid is the point: a reader has to be able to put a
@@ -266,10 +280,10 @@ struct ModelFrameForecast {
 struct ModelValidationSchedule {
     // Called once, before the null, with the forecast above.
     std::function<void(const ModelFrameForecast &)> on_forecast;
-    // Where valid, waited on before the first file is written: false returns without writing any.
-    std::shared_future<bool> write_gate;
+    // Where valid, waited on before the maps are made: false returns without making any.
+    std::shared_future<bool> maps_gate;
     // Asked once the verdict, the setting and the indexing are decided, before any map is made: true
-    // where another validation will write this one's maps and map coefficients, which are then not made.
+    // where another validation's maps and map coefficients are the ones kept, so this one's are not made.
     std::function<bool(const ModelValidationResult &)> maps_superseded;
     // The null that says whether the model fits. False where the verdict is another validation's
     // (KeepModelVerdict) or was reached elsewhere (a molecular-replacement placement, scored for its maps).
@@ -280,6 +294,8 @@ struct ModelValidationSchedule {
     // GPU memory another validation running beside this one holds there. It limits how many engines
     // this one takes and never what runs on the GPU, so no number depends on whether the other runs.
     size_t gpu_beside_bytes = 0;
+    // Keep the maps themselves (ModelMaps), not only their coefficients: rugnux writes them as .ccp4.
+    bool keep_map_grids = false;
 };
 
 // What a model validation needs of its card, part by part: the bytes of one engine of each, how many
@@ -344,8 +360,7 @@ std::vector<size_t> OtherCardsTotalMemory(int device);
 
 ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &merged,
                                            const UnitCell &cell,
-                                           const std::string &model_path,
-                                           const std::string &output_prefix,
+                                           const RugnuxModel &model,
                                            Logger &logger,
                                            const gemmi::SpaceGroup *data_space_group = nullptr,
                                            bool probe_indexing_ambiguity = true,
@@ -354,7 +369,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
                                            const std::vector<float> &report_shell_d_min = {},
                                            const ModelValidationSchedule &schedule = {},
                                            const gemmi::Op *twin_law = nullptr,
-                                           const gemmi::Structure *model = nullptr);
+                                           const gemmi::Structure *placed = nullptr);
 
 // Reindex `merged` into the frame ValidateAgainstModel reported, so the reflection files that are
 // written describe the same indexing as the R-factors and the maps. Returns the space group they are
@@ -393,7 +408,7 @@ ModelValidationResult ReadModelAnomalousMap(const std::vector<MergedReflection>
 // serial-stills run: there the ambiguity has to be broken per image, at integration time, long before
 // there is a merge to fit a model to. `cell` and `space_group` override the model's own, where
 // the run already knows them. Empty on any failure (which is logged).
-std::vector<MergedReflection> ModelReferenceIntensities(const std::string &model_path,
+std::vector<MergedReflection> ModelReferenceIntensities(const RugnuxModel &model,
                                                         const std::optional<UnitCell> &cell,
                                                         const gemmi::SpaceGroup *space_group,
                                                         double d_min,