From bcee3fb5b7a385c0435ea8e329066eea9415c8cb Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 3 Nov 2026 10:42:01 +0100 Subject: [PATCH] rugnux: module placement file through the OpenAPI model (--module-placement, _module_placement.json) rugnux's own JSON parser and writer for the placement file are gone: the file is the dataset_settings fragment {"module_placement": [...]} and is read and written through the generated model and the broker's conversion (JFJochAPIConvert), so /start and rugnux accept exactly the same thing. The unreleased --panels becomes --module-placement and _panels.json becomes _module_placement.json; the released --module-geometry (.h5/.geom) keeps its meaning. Keys are now written in the model's (alphabetical) order. The file is written for every stills run on a detector of several modules unless --panel-refine=off, also where every module was held: the placement as given is a placement too. _modules.h5 is still written only where modules moved. The log names a module by its box corner, as the file does, not by its centre. Results unchanged (a stills set on an 8-module detector gives the identical report). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4 --- docs/RUGNUX_CLI.md | 4 +-- docs/RUGNUX_FORMATS.md | 38 +++++++++++++++--------- docs/RUGNUX_TUTORIAL.md | 7 +++-- rugnux/CMakeLists.txt | 2 +- rugnux/RugnuxStills.cpp | 12 ++++---- rugnux/rugnux_cli.cpp | 66 ++++++++++++++++------------------------- 6 files changed, 63 insertions(+), 66 deletions(-) diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index bba548482..19fc46a51 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -361,8 +361,8 @@ Defaults are taken from the input file; override them to reprocess with a correc | `--saturation ` | Pixel value from which a pixel is overloaded, in the units the frames are read in. Replaces the file's limit and the one stills read off the frames | | `--crystfel-geometry ` | Place the panels of an HDF5/CXI input by a CrystFEL geometry file instead of the file's own metadata: module geometry, camera length, mask arrays and bad regions (optional; [Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)). Photon energy, camera length and - where the geometry names no mask array - the masks otherwise come from the data files (Cheetah CXI, SACLA) or the command line; gain and saturation are measured. Tilted panels, `dimN` layouts, per-panel data and detector shifts are refused | | `--module-geometry ` | Where the detector's modules are: an NXmx file with one NXdetector_module per module (what a run writes as `_modules.h5`) or a CrystFEL geometry file whose panels are laid on the image. Overrides the input's own | -| `--panels ` | Place the modules as a module placement file says (`_panels.json` of an earlier run; [Module placement file](RUGNUX_FORMATS.md#module-placement-file-prefix_panelsjson)). Entries name modules by their box in the image; one that matches no module, or a module left out, is an error | -| `--panel-refine[=off\|diagnose]` | Stills on a detector of several modules: by default the geometry bundle adjustment places each module as a rigid block where that predicts held-out spots better - the whole placement must beat the given one, and each module keeps its move only where its own held-out spots are predicted better, the others staying where they were given. `--panel-refine` keeps every move the spots support, without these tests; `=off` keeps the modules as given (a CrystFEL `.geom` included). A placement used is written to `_modules.h5` and `_panels.json`. `=diagnose` also runs a set of tests on the placement and writes them to the log | +| `--module-placement ` | Place the modules as a module placement file says (`_module_placement.json` of an earlier run, the `module_placement` fragment of the broker's dataset settings; [Module placement file](RUGNUX_FORMATS.md#module-placement-file-prefix_module_placementjson)). Entries name modules by their box in the image (`box_x`, `box_y`, `box_width`, `box_height`); one that matches no module, one named twice, or a module left out, is an error | +| `--panel-refine[=off\|diagnose]` | Stills on a detector of several modules: by default the geometry bundle adjustment places each module as a rigid block where that predicts held-out spots better - the whole placement must beat the given one, and each module keeps its move only where its own held-out spots are predicted better, the others staying where they were given. `--panel-refine` keeps every move the spots support, without these tests; `=off` keeps the modules as given (a CrystFEL `.geom` included). A placement used is written to `_modules.h5`; on stills `_module_placement.json` is written unless `=off`, also where every module was held. `=diagnose` also runs a set of tests on the placement and writes them to the log | | `--mask-good [:][@]` / `--mask-bad [:][@]` | HDF5/CXI stills: a mask array in the frames' stored layout, in `` or in each data file; usable only where non-zero (or with all ``) / unusable where non-zero (or with any ``). Repeatable ([Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)) | | `--rot1 ` | PONI detector rotation 1 (rad). On stills, `--rot1` or `--rot2` holds the tilt (not refined); without them stills refine the tilt where held-out frames agree | | `--rot2 ` | PONI detector rotation 2 (rad) | diff --git a/docs/RUGNUX_FORMATS.md b/docs/RUGNUX_FORMATS.md index 25a172b1b..c5b68b052 100644 --- a/docs/RUGNUX_FORMATS.md +++ b/docs/RUGNUX_FORMATS.md @@ -156,38 +156,48 @@ turn; `--panel-refine=off` keeps them where they were given, a CrystFEL `.geom`' where that predicts held-out spots better, module by module; modules move only relative to one another, the beam centre, distance and crystal orientations carrying what all of them share. The placement is used from there on, and written to `_modules.h5` (read back with -`--module-geometry`) and `_panels.json` (read back with `--panels`, and taken by the broker's -`/start`). +`--module-geometry`); on stills of a detector of several modules `_module_placement.json` is +written whenever placement is not turned off - also where every module was held, as given being a +placement too - read back with `--module-placement` and taken by the broker's `/start`. -#### Module placement file (`_panels.json`) +#### Module placement file (`_module_placement.json`) ```json { - "module_geometry": [ - {"x0": 0, "y0": 0, "width": 1030, "height": 514, - "ox": 0.31, "oy": -0.42, "fx": 1.0, "fy": 0.0002, "sx": -0.0002, "sy": 1.0}, + "module_placement": [ + {"box_height": 514, "box_width": 1030, "box_x": 0, "box_y": 0, + "plane_x": 0.31, "plane_y": -0.42, + "x_step_x": 1.0, "x_step_y": 0.0002, "y_step_x": -0.0002, "y_step_y": 1.0}, ... ] } ``` +The file is a fragment of the broker's `dataset_settings` (`jfjoch_api.yaml`, `detector_module_placement`), +read and written through the same OpenAPI model the broker uses; the keys come in alphabetical order. + One entry per module, in pixels of the image as it is stored (for a JUNGFRAU: as the broker assembles it, after `mirror_y`; a pixel's centre at its index): -* `x0`, `y0`, `width`, `height` - the module's box in the image (NXmx `data_origin`, `data_size`). They - name the module: entries match modules by their box, not by their place in the list, so the order of - a broker's `custom_geometry`, a master's `module_NN` groups or a reader's panels does not matter. -* `ox`, `oy`, `fx`, `fy`, `sx`, `sy` - where the box lies on the detector plane: image pixel (x, y) of the - module is at (ox, oy) + (x - x0) (fx, fy) + (y - y0) (sx, sy). As assembled, `ox = x0`, `oy = y0`, - `(fx, fy) = (1, 0)`, `(sx, sy) = (0, 1)`; the refinement's fraction of a pixel and small turn are kept. +* `box_x`, `box_y`, `box_width`, `box_height` - the module's box in the image (NXmx `data_origin`, + `data_size`): `box_x`, `box_y` are its lowest image x and y, whatever way its readout runs, so not the + readout corner `x0`, `y0` of a broker's `custom_geometry`. They name the module: entries match modules + by their box, not by their place in the list, so the order of a broker's `custom_geometry`, a master's + `module_NN` groups or a reader's panels does not matter. rugnux's log names modules the same way. +* `plane_x`, `plane_y`, `x_step_x`, `x_step_y`, `y_step_x`, `y_step_y` - where the box lies on the + detector plane: image pixel (x, y) of the module is at + (plane_x, plane_y) + (x - box_x) (x_step_x, x_step_y) + (y - box_y) (y_step_x, y_step_y). + As assembled, `plane_x = box_x`, `plane_y = box_y`, `(x_step_x, x_step_y) = (1, 0)`, + `(y_step_x, y_step_y) = (0, 1)`; the refinement's fraction of a pixel and small turn are kept. -A placement is valid only for the image layout it was measured on. `--panels ` and `/start` refuse +A placement is valid only for the image layout it was measured on. `--module-placement ` and `/start` refuse it, naming the module, where an entry matches no module of the image or a module has no entry - another detector size, other gaps, a binned image or a raw (unassembled) JUNGFRAU stream. The beam centre and distance are not in the file: they belong to each dataset (`--beam-x`, `--detector-distance`, or the `/start` settings), and the placement only moves the modules relative to one another. -With the broker, the file's `module_geometry` array goes into the `/start` dataset settings as it stands. +With the broker, the file's `module_placement` is merged into the `/start` dataset settings as it stands; `/start` +answers `400` where it does not name exactly the detector's modules. The image is still assembled as `custom_geometry` lays it out, at whole pixels; the placement goes into the start message and the master's `module_NN` groups, and the broker's own analysis uses it. diff --git a/docs/RUGNUX_TUTORIAL.md b/docs/RUGNUX_TUTORIAL.md index f0575a2c6..d048d448a 100644 --- a/docs/RUGNUX_TUTORIAL.md +++ b/docs/RUGNUX_TUTORIAL.md @@ -324,9 +324,10 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke [The results report](RUGNUX_REPORT.md#the-results-report). - `_log.txt` — the full timestamped log of the run, at debug level, written by every run with an output prefix; send it with the report when asking about a run. -- `_modules.h5`, `_panels.json` — serial stills on a detector of several modules, where - the geometry refinement placed the modules: where it placed each one, readable back with - `--module-geometry` / `--panels` (the JSON also by the broker's `/start`; see +- `_modules.h5`, `_module_placement.json` — serial stills on a detector of several + modules: where the geometry refinement placed each one (the `.h5` only where it moved some; the JSON + whenever placement is not off), readable back with `--module-geometry` / `--module-placement` (the + JSON also by the broker's `/start`, merged into its dataset settings; see [Module geometry](RUGNUX_FORMATS.md#module-geometry)). - `_plot.txt` — **the per-image table**, one row per processed image, whitespace-separated, with a single `#` legend line over its own columns so gnuplot plots it as it stands diff --git a/rugnux/CMakeLists.txt b/rugnux/CMakeLists.txt index 3a5b90302..78b183b96 100644 --- a/rugnux/CMakeLists.txt +++ b/rugnux/CMakeLists.txt @@ -49,7 +49,7 @@ TARGET_LINK_LIBRARIES(JFJochRugnuxOutput JFJochRugnux JFJochWriter JFJochModelWr # rugnux is the single offline analysis CLI; --mode picks what it does: mx (the full pipeline, the # default), azint, scale (re-scale/merge stored reflections) or calibration (powder-ring geometry). ADD_EXECUTABLE(rugnux rugnux_cli.cpp RugnuxConsole.cpp RugnuxConsole.h) -TARGET_LINK_LIBRARIES(rugnux JFJochRugnuxOutput JFJochRugnux JFJochReader JFJochImageAnalysis JFJochWriter) +TARGET_LINK_LIBRARIES(rugnux JFJochRugnuxOutput JFJochRugnux JFJochReader JFJochImageAnalysis JFJochWriter JFJochAPIConvert) INSTALL(TARGETS rugnux RUNTIME COMPONENT rugnux) # macOS: sign the installed binary with the Developer ID (hardened runtime, secure timestamp) so the # .tgz can be notarized - see JFJOCH_MACOS_SIGNING_IDENTITY in the top-level CMakeLists.txt. The diff --git a/rugnux/RugnuxStills.cpp b/rugnux/RugnuxStills.cpp index 4fa5f5616..fb9386261 100644 --- a/rugnux/RugnuxStills.cpp +++ b/rugnux/RugnuxStills.cpp @@ -515,8 +515,8 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_ start[m].dx = scan[m].dx; start[m].dy = scan[m].dy; if (std::hypot(scan[m].dx, scan[m].dy) >= 1.0) - logger.Info("Module {} at ({:.0f}, {:.0f}): {} spots on the lattices as placed, {} shifted by " - "({:+.1f}, {:+.1f}) px", m, start[m].x0, start[m].y0, scan[m].on_lattice_at_zero, + logger.Info("Module {} at box ({}, {}): {} spots on the lattices as placed, {} shifted by " + "({:+.1f}, {:+.1f}) px", m, given.Get()[m].x0, given.Get()[m].y0, scan[m].on_lattice_at_zero, scan[m].on_lattice_at_best, scan[m].dx, scan[m].dy); } scan_mark.Stop(); @@ -596,8 +596,8 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_ if (moved_m && !(moved_t.module[m].Rms() < held_t.module[m].Rms())) { hold[m] = true; held_back++; - logger.Info("Module {} at ({:.0f}, {:.0f}): held as given - held-out residual {:.3f} px placed, " - "{:.3f} px as given ({} spots)", m, centres[m].x0, centres[m].y0, + logger.Info("Module {} at box ({}, {}): held as given - held-out residual {:.3f} px placed, " + "{:.3f} px as given ({} spots)", m, given.Get()[m].x0, given.Get()[m].y0, moved_t.module[m].Rms(), held_t.module[m].Rms(), moved_t.module[m].n); } } @@ -613,8 +613,8 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_ if (apply) { for (size_t m = 0; m < rm.modules.size(); m++) { const auto &ms = rm.modules[m]; - logger.Info("Module {} at ({:.0f}, {:.0f}): shift ({:+.2f}, {:+.2f}) px, turn {:+.3f} deg, {} spots", - m, ms.x0, ms.y0, ms.dx, ms.dy, ms.rot_rad * 180.0 / PI, ms.spots); + logger.Info("Module {} at box ({}, {}): shift ({:+.2f}, {:+.2f}) px, turn {:+.3f} deg, {} spots", + m, given.Get()[m].x0, given.Get()[m].y0, ms.dx, ms.dy, ms.rot_rad * 180.0 / PI, ms.spots); } // The move is a step from where the modules were given; the experiment now has them there, // so every spot found and every prediction integrated from here on uses it. The image is diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 11d1e2fb6..446a1e9d4 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -12,7 +12,7 @@ #include #include #include -#include +#include "../broker/ModulePlacementConvert.h" #include #include #include @@ -117,39 +117,20 @@ enum class RugnuxSweepsMode { Joint, Separate }; // buys nothing and runs into the 65534-bin limit on q x azimuth, so 32 is the default. constexpr int CALIBRATION_AZIM_BINS_DEFAULT = 32; -// A module placement file (docs/RUGNUX_FORMATS.md, Module placement file): {"module_geometry": [module, ...]}, each module its box -// in the image (x0, y0, width, height) and where that box lies on the detector plane (ox, oy, fx, fy, sx, sy: -// DetectorModule). The same array is what the broker's /start takes as module_geometry. -std::vector ReadModulePlacementJson(const std::string &path) { +// A module placement file (docs/RUGNUX_FORMATS.md, Module placement file): the dataset_settings fragment +// {"module_placement": [...]} of the broker's /start, read and written through the OpenAPI model. +std::vector ReadModulePlacementFile(const std::string &path) { std::ifstream in(path); if (!in) throw std::runtime_error("cannot open the file"); - const auto j = nlohmann::json::parse(in); - std::vector out; - for (const auto &e: j.at("module_geometry")) { - DetectorModule m; - m.x0 = e.at("x0"); - m.y0 = e.at("y0"); - m.width = e.at("width"); - m.height = e.at("height"); - m.ox = e.at("ox"); - m.oy = e.at("oy"); - m.fx = e.at("fx"); - m.fy = e.at("fy"); - m.sx = e.at("sx"); - m.sy = e.at("sy"); - out.push_back(m); - } - return out; + std::stringstream text; + text << in.rdbuf(); + return ParseModulePlacement(text.str()); } -void WriteModulePlacementJson(const std::string &path, const DetectorModules &modules) { - nlohmann::ordered_json list = nlohmann::ordered_json::array(); - for (const auto &m: modules.Get()) - list.push_back({{"x0", m.x0}, {"y0", m.y0}, {"width", m.width}, {"height", m.height}, - {"ox", m.ox}, {"oy", m.oy}, {"fx", m.fx}, {"fy", m.fy}, {"sx", m.sx}, {"sy", m.sy}}); +void WriteModulePlacementFile(const std::string &path, const DetectorModules &modules) { std::ofstream out(path); - out << nlohmann::ordered_json{{"module_geometry", list}}.dump(2) << "\n"; + out << ModulePlacementJson(modules); if (!out) throw std::runtime_error("write failed"); } @@ -323,7 +304,7 @@ enum { OPT_MASK_BAD, OPT_REFINE_MODULES, OPT_MODULE_GEOMETRY, - OPT_PANELS, + OPT_MODULE_PLACEMENT, OPT_ROT1, OPT_ROT2, OPT_ROT3, @@ -420,7 +401,7 @@ static option long_options[] = { {"refine-geometry", optional_argument, nullptr, OPT_REFINE_GEOMETRY}, {"panel-refine", optional_argument, nullptr, OPT_REFINE_MODULES}, {"module-geometry", required_argument, nullptr, OPT_MODULE_GEOMETRY}, - {"panels", required_argument, nullptr, OPT_PANELS}, + {"module-placement", required_argument, nullptr, OPT_MODULE_PLACEMENT}, {"detect-beam-stop", optional_argument, nullptr, OPT_DETECT_BEAM_STOP}, {"beam-center-check", optional_argument, nullptr, OPT_BEAM_CENTER_CHECK}, @@ -982,7 +963,7 @@ struct RugnuxCli { bool force_modules = false; // --panel-refine: every placeable module moved, no held-out tests bool module_diagnostics = false; // --panel-refine=diagnose std::string module_geometry_file; // --module-geometry - std::string panels_file; // --panels + std::string module_placement_file; // --module-placement std::optional bandwidth_fwhm; // relative FWHM of dlambda/lambda @@ -1191,8 +1172,8 @@ int RugnuxCli::Parse(int argc, char **argv) { case OPT_MODULE_GEOMETRY: module_geometry_file = optarg; break; - case OPT_PANELS: - panels_file = optarg; + case OPT_MODULE_PLACEMENT: + module_placement_file = optarg; break; case OPT_FORCE_ROTATION_LATTICE: { if (rotation_indexing) { @@ -2981,12 +2962,13 @@ int RugnuxInputRun::PrepareRugnuxInput() { } // Where a module placement file puts them (docs/RUGNUX_FORMATS.md, Module placement file): the modules are named by their boxes // in the image, so it has to be the same detector, assembled the same way. - if (!panels_file.empty()) { + if (!module_placement_file.empty()) { try { - experiment.Modules(ModulesPlaced(experiment.GetModules(), ReadModulePlacementJson(panels_file))); - logger.Info("--panels: {} modules placed from {}", experiment.GetModules().size(), panels_file); + experiment.Modules(ModulesPlaced(experiment.GetModules(), ReadModulePlacementFile(module_placement_file))); + logger.Info("--module-placement: {} modules placed from {}", experiment.GetModules().size(), + module_placement_file); } catch (const std::exception &e) { - logger.Error("Cannot use --panels {}: {}", panels_file, e.what()); + logger.Error("Cannot use --module-placement {}: {}", module_placement_file, e.what()); throw RugnuxInputFailed{EXIT_FAILURE}; } } @@ -3763,11 +3745,15 @@ int RugnuxInputRun::RunPreparedInput() { } catch (const std::exception &e) { logger.Warning("Cannot write {}_modules.h5: {}", output_prefix, e.what()); } - // The same, for --panels and the broker's /start (module_geometry). + } + // The same, for --module-placement and the broker's /start: on stills of a detector of several modules + // whenever placement was not turned off, also where every module was held - as given is a placement too. + if (!output_prefix.empty() && refine_modules && result.modules.size() >= 2 + && (force_still || !RugnuxRotationData(experiment))) { try { - WriteModulePlacementJson(output_prefix + "_panels.json", result.modules); + WriteModulePlacementFile(output_prefix + "_module_placement.json", result.modules); } catch (const std::exception &e) { - logger.Warning("Cannot write {}_panels.json: {}", output_prefix, e.what()); + logger.Warning("Cannot write {}_module_placement.json: {}", output_prefix, e.what()); } }