rugnux: module placement file through the OpenAPI model (--module-placement, <prefix>_module_placement.json)

rugnux's own JSON parser and writer for the placement file are gone: the file is the dataset_settings
fragment {"module_placement": [...]} and is read and written through the generated model and the
broker's conversion (JFJochAPIConvert), so /start and rugnux accept exactly the same thing. The unreleased
--panels becomes --module-placement and <prefix>_panels.json becomes <prefix>_module_placement.json; the
released --module-geometry (.h5/.geom) keeps its meaning. Keys are now written in the model's
(alphabetical) order.

The file is written for every stills run on a detector of several modules unless --panel-refine=off,
also where every module was held: the placement as given is a placement too. <prefix>_modules.h5 is still
written only where modules moved. The log names a module by its box corner, as the file does, not by its
centre. Results unchanged (a stills set on an 8-module detector gives the identical report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

docs/RUGNUX_CLI.md

@@ -361,8 +361,8 @@ Defaults are taken from the input file; override them to reprocess with a correc
 | `--saturation <num>` | Pixel value from which a pixel is overloaded, in the units the frames are read in. Replaces the file's limit and the one stills read off the frames |
 | `--crystfel-geometry <file>` | Place the panels of an HDF5/CXI input by a CrystFEL geometry file instead of the file's own metadata: module geometry, camera length, mask arrays and bad regions (optional; [Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)). Photon energy, camera length and - where the geometry names no mask array - the masks otherwise come from the data files (Cheetah CXI, SACLA) or the command line; gain and saturation are measured. Tilted panels, `dimN` layouts, per-panel data and detector shifts are refused |
 | `--module-geometry <file>` | Where the detector's modules are: an NXmx file with one NXdetector_module per module (what a run writes as `<prefix>_modules.h5`) or a CrystFEL geometry file whose panels are laid on the image. Overrides the input's own |
-| `--panels <file.json>` | Place the modules as a module placement file says (`<prefix>_panels.json` of an earlier run; [Module placement file](RUGNUX_FORMATS.md#module-placement-file-prefix_panelsjson)). Entries name modules by their box in the image; one that matches no module, or a module left out, is an error |
-| `--panel-refine[=off\|diagnose]` | Stills on a detector of several modules: by default the geometry bundle adjustment places each module as a rigid block where that predicts held-out spots better - the whole placement must beat the given one, and each module keeps its move only where its own held-out spots are predicted better, the others staying where they were given. `--panel-refine` keeps every move the spots support, without these tests; `=off` keeps the modules as given (a CrystFEL `.geom` included). A placement used is written to `<prefix>_modules.h5` and `<prefix>_panels.json`. `=diagnose` also runs a set of tests on the placement and writes them to the log |
+| `--module-placement <file.json>` | Place the modules as a module placement file says (`<prefix>_module_placement.json` of an earlier run, the `module_placement` fragment of the broker's dataset settings; [Module placement file](RUGNUX_FORMATS.md#module-placement-file-prefix_module_placementjson)). Entries name modules by their box in the image (`box_x`, `box_y`, `box_width`, `box_height`); one that matches no module, one named twice, or a module left out, is an error |
+| `--panel-refine[=off\|diagnose]` | Stills on a detector of several modules: by default the geometry bundle adjustment places each module as a rigid block where that predicts held-out spots better - the whole placement must beat the given one, and each module keeps its move only where its own held-out spots are predicted better, the others staying where they were given. `--panel-refine` keeps every move the spots support, without these tests; `=off` keeps the modules as given (a CrystFEL `.geom` included). A placement used is written to `<prefix>_modules.h5`; on stills `<prefix>_module_placement.json` is written unless `=off`, also where every module was held. `=diagnose` also runs a set of tests on the placement and writes them to the log |
 | `--mask-good [<file>:]<dataset>[@<bits>]` / `--mask-bad [<file>:]<dataset>[@<bits>]` | HDF5/CXI stills: a mask array in the frames' stored layout, in `<file>` or in each data file; usable only where non-zero (or with all `<bits>`) / unusable where non-zero (or with any `<bits>`). Repeatable ([Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)) |
 | `--rot1 <num>` | PONI detector rotation 1 (rad). On stills, `--rot1` or `--rot2` holds the tilt (not refined); without them stills refine the tilt where held-out frames agree |
 | `--rot2 <num>` | PONI detector rotation 2 (rad) |

docs/RUGNUX_FORMATS.md

@@ -156,38 +156,48 @@ turn; `--panel-refine=off` keeps them where they were given, a CrystFEL `.geom`'
 where that predicts held-out spots better, module by module; modules move only relative to one
 another, the beam centre, distance and crystal orientations carrying what all of them share. The
 placement is used from there on, and written to `<prefix>_modules.h5` (read back with
-`--module-geometry`) and `<prefix>_panels.json` (read back with `--panels`, and taken by the broker's
-`/start`).
+`--module-geometry`); on stills of a detector of several modules `<prefix>_module_placement.json` is
+written whenever placement is not turned off - also where every module was held, as given being a
+placement too - read back with `--module-placement` and taken by the broker's `/start`.
 
-#### Module placement file (`<prefix>_panels.json`)
+#### Module placement file (`<prefix>_module_placement.json`)
 
 ```json
 {
-  "module_geometry": [
-    {"x0": 0, "y0": 0, "width": 1030, "height": 514,
-     "ox": 0.31, "oy": -0.42, "fx": 1.0, "fy": 0.0002, "sx": -0.0002, "sy": 1.0},
+  "module_placement": [
+    {"box_height": 514, "box_width": 1030, "box_x": 0, "box_y": 0,
+     "plane_x": 0.31, "plane_y": -0.42,
+     "x_step_x": 1.0, "x_step_y": 0.0002, "y_step_x": -0.0002, "y_step_y": 1.0},
     ...
   ]
 }
 ```
 
+The file is a fragment of the broker's `dataset_settings` (`jfjoch_api.yaml`, `detector_module_placement`),
+read and written through the same OpenAPI model the broker uses; the keys come in alphabetical order.
+
 One entry per module, in pixels of the image as it is stored (for a JUNGFRAU: as the broker assembles
 it, after `mirror_y`; a pixel's centre at its index):
 
-* `x0`, `y0`, `width`, `height` - the module's box in the image (NXmx `data_origin`, `data_size`). They
-  name the module: entries match modules by their box, not by their place in the list, so the order of
-  a broker's `custom_geometry`, a master's `module_NN` groups or a reader's panels does not matter.
-* `ox`, `oy`, `fx`, `fy`, `sx`, `sy` - where the box lies on the detector plane: image pixel (x, y) of the
-  module is at (ox, oy) + (x - x0) (fx, fy) + (y - y0) (sx, sy). As assembled, `ox = x0`, `oy = y0`,
-  `(fx, fy) = (1, 0)`, `(sx, sy) = (0, 1)`; the refinement's fraction of a pixel and small turn are kept.
+* `box_x`, `box_y`, `box_width`, `box_height` - the module's box in the image (NXmx `data_origin`,
+  `data_size`): `box_x`, `box_y` are its lowest image x and y, whatever way its readout runs, so not the
+  readout corner `x0`, `y0` of a broker's `custom_geometry`. They name the module: entries match modules
+  by their box, not by their place in the list, so the order of a broker's `custom_geometry`, a master's
+  `module_NN` groups or a reader's panels does not matter. rugnux's log names modules the same way.
+* `plane_x`, `plane_y`, `x_step_x`, `x_step_y`, `y_step_x`, `y_step_y` - where the box lies on the
+  detector plane: image pixel (x, y) of the module is at
+  (plane_x, plane_y) + (x - box_x) (x_step_x, x_step_y) + (y - box_y) (y_step_x, y_step_y).
+  As assembled, `plane_x = box_x`, `plane_y = box_y`, `(x_step_x, x_step_y) = (1, 0)`,
+  `(y_step_x, y_step_y) = (0, 1)`; the refinement's fraction of a pixel and small turn are kept.
 
-A placement is valid only for the image layout it was measured on. `--panels <file>` and `/start` refuse
+A placement is valid only for the image layout it was measured on. `--module-placement <file>` and `/start` refuse
 it, naming the module, where an entry matches no module of the image or a module has no entry - another
 detector size, other gaps, a binned image or a raw (unassembled) JUNGFRAU stream. The beam centre and
 distance are not in the file: they belong to each dataset (`--beam-x`, `--detector-distance`, or the
 `/start` settings), and the placement only moves the modules relative to one another.
 
-With the broker, the file's `module_geometry` array goes into the `/start` dataset settings as it stands.
+With the broker, the file's `module_placement` is merged into the `/start` dataset settings as it stands; `/start`
+answers `400` where it does not name exactly the detector's modules.
 The image is still assembled as `custom_geometry` lays it out, at whole pixels; the placement goes into the
 start message and the master's `module_NN` groups, and the broker's own analysis uses it.
 

docs/RUGNUX_TUTORIAL.md

@@ -324,9 +324,10 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke
   [The results report](RUGNUX_REPORT.md#the-results-report).
 - `<prefix>_log.txt` — the full timestamped log of the run, at debug level, written by every run with
   an output prefix; send it with the report when asking about a run.
-- `<prefix>_modules.h5`, `<prefix>_panels.json` — serial stills on a detector of several modules, where
-  the geometry refinement placed the modules: where it placed each one, readable back with
-  `--module-geometry` / `--panels` (the JSON also by the broker's `/start`; see
+- `<prefix>_modules.h5`, `<prefix>_module_placement.json` — serial stills on a detector of several
+  modules: where the geometry refinement placed each one (the `.h5` only where it moved some; the JSON
+  whenever placement is not off), readable back with `--module-geometry` / `--module-placement` (the
+  JSON also by the broker's `/start`, merged into its dataset settings; see
   [Module geometry](RUGNUX_FORMATS.md#module-geometry)).
 - `<prefix>_plot.txt` — **the per-image table**, one row per processed image, whitespace-separated,
   with a single `#` legend line over its own columns so gnuplot plots it as it stands

rugnux/CMakeLists.txt

@@ -49,7 +49,7 @@ TARGET_LINK_LIBRARIES(JFJochRugnuxOutput JFJochRugnux JFJochWriter JFJochModelWr
 # rugnux is the single offline analysis CLI; --mode picks what it does: mx (the full pipeline, the
 # default), azint, scale (re-scale/merge stored reflections) or calibration (powder-ring geometry).
 ADD_EXECUTABLE(rugnux rugnux_cli.cpp RugnuxConsole.cpp RugnuxConsole.h)
-TARGET_LINK_LIBRARIES(rugnux JFJochRugnuxOutput JFJochRugnux JFJochReader JFJochImageAnalysis JFJochWriter)
+TARGET_LINK_LIBRARIES(rugnux JFJochRugnuxOutput JFJochRugnux JFJochReader JFJochImageAnalysis JFJochWriter JFJochAPIConvert)
 INSTALL(TARGETS rugnux RUNTIME COMPONENT rugnux)
 # macOS: sign the installed binary with the Developer ID (hardened runtime, secure timestamp) so the
 # .tgz can be notarized - see JFJOCH_MACOS_SIGNING_IDENTITY in the top-level CMakeLists.txt. The

rugnux/RugnuxStills.cpp

@@ -515,8 +515,8 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_
                 start[m].dx = scan[m].dx;
                 start[m].dy = scan[m].dy;
                 if (std::hypot(scan[m].dx, scan[m].dy) >= 1.0)
-                    logger.Info("Module {} at ({:.0f}, {:.0f}): {} spots on the lattices as placed, {} shifted by "
-                                "({:+.1f}, {:+.1f}) px", m, start[m].x0, start[m].y0, scan[m].on_lattice_at_zero,
+                    logger.Info("Module {} at box ({}, {}): {} spots on the lattices as placed, {} shifted by "
+                                "({:+.1f}, {:+.1f}) px", m, given.Get()[m].x0, given.Get()[m].y0, scan[m].on_lattice_at_zero,
                                 scan[m].on_lattice_at_best, scan[m].dx, scan[m].dy);
             }
         scan_mark.Stop();
@@ -596,8 +596,8 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_
                 if (moved_m && !(moved_t.module[m].Rms() < held_t.module[m].Rms())) {
                     hold[m] = true;
                     held_back++;
-                    logger.Info("Module {} at ({:.0f}, {:.0f}): held as given - held-out residual {:.3f} px placed, "
-                                "{:.3f} px as given ({} spots)", m, centres[m].x0, centres[m].y0,
+                    logger.Info("Module {} at box ({}, {}): held as given - held-out residual {:.3f} px placed, "
+                                "{:.3f} px as given ({} spots)", m, given.Get()[m].x0, given.Get()[m].y0,
                                 moved_t.module[m].Rms(), held_t.module[m].Rms(), moved_t.module[m].n);
                 }
             }
@@ -613,8 +613,8 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_
         if (apply) {
             for (size_t m = 0; m < rm.modules.size(); m++) {
                 const auto &ms = rm.modules[m];
-                logger.Info("Module {} at ({:.0f}, {:.0f}): shift ({:+.2f}, {:+.2f}) px, turn {:+.3f} deg, {} spots",
-                            m, ms.x0, ms.y0, ms.dx, ms.dy, ms.rot_rad * 180.0 / PI, ms.spots);
+                logger.Info("Module {} at box ({}, {}): shift ({:+.2f}, {:+.2f}) px, turn {:+.3f} deg, {} spots",
+                            m, given.Get()[m].x0, given.Get()[m].y0, ms.dx, ms.dy, ms.rot_rad * 180.0 / PI, ms.spots);
             }
             // The move is a step from where the modules were given; the experiment now has them there,
             // so every spot found and every prediction integrated from here on uses it. The image is

rugnux/rugnux_cli.cpp

@@ -12,7 +12,7 @@
 #include <filesystem>
 #include <fstream>
 #include <iomanip>
-#include <nlohmann/json.hpp>
+#include "../broker/ModulePlacementConvert.h"
 #include <future>
 #include <optional>
 #include <getopt.h>
@@ -117,39 +117,20 @@ enum class RugnuxSweepsMode { Joint, Separate };
 // buys nothing and runs into the 65534-bin limit on q x azimuth, so 32 is the default.
 constexpr int CALIBRATION_AZIM_BINS_DEFAULT = 32;
 
-// A module placement file (docs/RUGNUX_FORMATS.md, Module placement file): {"module_geometry": [module, ...]}, each module its box
-// in the image (x0, y0, width, height) and where that box lies on the detector plane (ox, oy, fx, fy, sx, sy:
-// DetectorModule). The same array is what the broker's /start takes as module_geometry.
-std::vector<DetectorModule> ReadModulePlacementJson(const std::string &path) {
+// A module placement file (docs/RUGNUX_FORMATS.md, Module placement file): the dataset_settings fragment
+// {"module_placement": [...]} of the broker's /start, read and written through the OpenAPI model.
+std::vector<DetectorModule> ReadModulePlacementFile(const std::string &path) {
     std::ifstream in(path);
     if (!in)
         throw std::runtime_error("cannot open the file");
-    const auto j = nlohmann::json::parse(in);
-    std::vector<DetectorModule> out;
-    for (const auto &e: j.at("module_geometry")) {
-        DetectorModule m;
-        m.x0 = e.at("x0");
-        m.y0 = e.at("y0");
-        m.width = e.at("width");
-        m.height = e.at("height");
-        m.ox = e.at("ox");
-        m.oy = e.at("oy");
-        m.fx = e.at("fx");
-        m.fy = e.at("fy");
-        m.sx = e.at("sx");
-        m.sy = e.at("sy");
-        out.push_back(m);
-    }
-    return out;
+    std::stringstream text;
+    text << in.rdbuf();
+    return ParseModulePlacement(text.str());
 }
 
-void WriteModulePlacementJson(const std::string &path, const DetectorModules &modules) {
-    nlohmann::ordered_json list = nlohmann::ordered_json::array();
-    for (const auto &m: modules.Get())
-        list.push_back({{"x0", m.x0}, {"y0", m.y0}, {"width", m.width}, {"height", m.height},
-                        {"ox", m.ox}, {"oy", m.oy}, {"fx", m.fx}, {"fy", m.fy}, {"sx", m.sx}, {"sy", m.sy}});
+void WriteModulePlacementFile(const std::string &path, const DetectorModules &modules) {
     std::ofstream out(path);
-    out << nlohmann::ordered_json{{"module_geometry", list}}.dump(2) << "\n";
+    out << ModulePlacementJson(modules);
     if (!out)
         throw std::runtime_error("write failed");
 }
@@ -323,7 +304,7 @@ enum {
     OPT_MASK_BAD,
     OPT_REFINE_MODULES,
     OPT_MODULE_GEOMETRY,
-    OPT_PANELS,
+    OPT_MODULE_PLACEMENT,
     OPT_ROT1,
     OPT_ROT2,
     OPT_ROT3,
@@ -420,7 +401,7 @@ static option long_options[] = {
     {"refine-geometry", optional_argument, nullptr, OPT_REFINE_GEOMETRY},
     {"panel-refine", optional_argument, nullptr, OPT_REFINE_MODULES},
     {"module-geometry", required_argument, nullptr, OPT_MODULE_GEOMETRY},
-    {"panels", required_argument, nullptr, OPT_PANELS},
+    {"module-placement", required_argument, nullptr, OPT_MODULE_PLACEMENT},
     {"detect-beam-stop", optional_argument, nullptr, OPT_DETECT_BEAM_STOP},
     {"beam-center-check", optional_argument, nullptr, OPT_BEAM_CENTER_CHECK},
 
@@ -982,7 +963,7 @@ struct RugnuxCli {
     bool force_modules = false;            // --panel-refine: every placeable module moved, no held-out tests
     bool module_diagnostics = false;       // --panel-refine=diagnose
     std::string module_geometry_file;      // --module-geometry <file>
-    std::string panels_file;               // --panels <file.json>
+    std::string module_placement_file;     // --module-placement <file.json>
 
     std::optional<float> bandwidth_fwhm; // relative FWHM of dlambda/lambda
 
@@ -1191,8 +1172,8 @@ int RugnuxCli::Parse(int argc, char **argv) {
         case OPT_MODULE_GEOMETRY:
             module_geometry_file = optarg;
             break;
-        case OPT_PANELS:
-            panels_file = optarg;
+        case OPT_MODULE_PLACEMENT:
+            module_placement_file = optarg;
             break;
         case OPT_FORCE_ROTATION_LATTICE: {
             if (rotation_indexing) {
@@ -2981,12 +2962,13 @@ int RugnuxInputRun::PrepareRugnuxInput() {
     }
     // Where a module placement file puts them (docs/RUGNUX_FORMATS.md, Module placement file): the modules are named by their boxes
     // in the image, so it has to be the same detector, assembled the same way.
-    if (!panels_file.empty()) {
+    if (!module_placement_file.empty()) {
         try {
-            experiment.Modules(ModulesPlaced(experiment.GetModules(), ReadModulePlacementJson(panels_file)));
-            logger.Info("--panels: {} modules placed from {}", experiment.GetModules().size(), panels_file);
+            experiment.Modules(ModulesPlaced(experiment.GetModules(), ReadModulePlacementFile(module_placement_file)));
+            logger.Info("--module-placement: {} modules placed from {}", experiment.GetModules().size(),
+                        module_placement_file);
         } catch (const std::exception &e) {
-            logger.Error("Cannot use --panels {}: {}", panels_file, e.what());
+            logger.Error("Cannot use --module-placement {}: {}", module_placement_file, e.what());
             throw RugnuxInputFailed{EXIT_FAILURE};
         }
     }
@@ -3763,11 +3745,15 @@ int RugnuxInputRun::RunPreparedInput() {
         } catch (const std::exception &e) {
             logger.Warning("Cannot write {}_modules.h5: {}", output_prefix, e.what());
         }
-        // The same, for --panels and the broker's /start (module_geometry).
+    }
+    // The same, for --module-placement and the broker's /start: on stills of a detector of several modules
+    // whenever placement was not turned off, also where every module was held - as given is a placement too.
+    if (!output_prefix.empty() && refine_modules && result.modules.size() >= 2
+        && (force_still || !RugnuxRotationData(experiment))) {
         try {
-            WriteModulePlacementJson(output_prefix + "_panels.json", result.modules);
+            WriteModulePlacementFile(output_prefix + "_module_placement.json", result.modules);
         } catch (const std::exception &e) {
-            logger.Warning("Cannot write {}_panels.json: {}", output_prefix, e.what());
+            logger.Warning("Cannot write {}_module_placement.json: {}", output_prefix, e.what());
         }
     }