From 138abafa1460e9b79a77fa502c7acf088137b94d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sun, 18 Oct 2026 13:04:54 +0200 Subject: [PATCH 01/53] RotationScaleMerge: Anderson-accelerate the per-frame scaling loops (RESULT-CHANGING) The partials and fulls scaling loops are alternating least squares that converge linearly: over the rc176 battery's 2527 loops a round shrinks the step by a median 0.86 (partials) and 0.93 (fulls), up to 0.98, so the 1e-3 step tolerance stopped them 0.6% / 1.4% (median) and up to 17% (fulls p90) short of their fixed point, and 313 of them were stopped as "stalled" when they were only slow - a stalled fulls loop also skipped ShrinkToRestrained. Each round is now accelerated in log G by type-II Anderson mixing (Walker & Ni 2011), depth 5, each frame weighted by its observation count as in the step test; frames below the credible floor take the plain round's scale. The history is dropped when the weighted residual grows or the set of fitted frames changes (new AndersonMixing class, unit-tested on a synthetic slow ALS loop). The mixed scales are gauge-pinned again before the rescale, and the stop test is unchanged: it reads the round's residual F(x) - x, so "stalled" now means a loop that does not settle under acceleration either. Measured (CPU, -N 8): 4nwv 311 -> 82 iterations over 9 loops, 9v16 76 iterations over 11 loops, the same counts as the env-gated prototype on tail-solver; no history restart fired on either. Depth 3/5/8 give 92/91/90 iterations on 4nwv. A restart on a change of the fulls loop's cross-validated lambda was tried and gave nothing (83 vs 82). Same space group and merging statistics on 4nwv and 9v16. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- docs/ACKNOWLEDGEMENT.md | 7 ++ docs/CPU_DATA_ANALYSIS.md | 1 + docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 2 + image_analysis/scale_merge/AndersonMixing.cpp | 47 +++++++ image_analysis/scale_merge/AndersonMixing.h | 40 ++++++ image_analysis/scale_merge/CMakeLists.txt | 2 + .../scale_merge/RotationScaleMerge.cpp | 111 +++++++++++++---- tests/AndersonMixingTest.cpp | 115 ++++++++++++++++++ tests/CMakeLists.txt | 1 + 9 files changed, 299 insertions(+), 27 deletions(-) create mode 100644 image_analysis/scale_merge/AndersonMixing.cpp create mode 100644 image_analysis/scale_merge/AndersonMixing.h create mode 100644 tests/AndersonMixingTest.cpp diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 722b3e894..45afd765c 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -523,6 +523,13 @@ corrected once by the exact remainder, which returns the correctly rounded quoti P. W. Markstein, "Computation of elementary functions on the IBM RISC System/6000 processor" (1990), IBM J. Res. Dev. 34, 111-119. +The per-frame scaling loops of the rotation merge are accelerated by Anderson acceleration in the +type-II form analysed by Walker and Ni: H. F. Walker and P. Ni, "Anderson acceleration for fixed-point +iterations" (2011), SIAM J. Numer. Anal. 49, 1715-1735 +[doi:10.1137/10078356X](https://doi.org/10.1137/10078356X), after D. G. Anderson, "Iterative +procedures for nonlinear integral equations" (1965), J. ACM 12, 547-560 +[doi:10.1145/321296.321305](https://doi.org/10.1145/321296.321305). + The single-precision exp the Bragg integration's profile fit shares between the CPU and an Apple GPU (`BraggExp.h`) is built from textbook error-free floating-point arithmetic: the argument reduction of W. J. Cody and W. Waite, "Software Manual for the Elementary Functions" (1980), Prentice-Hall; the diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index de13ffdcc..7189d903c 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -57,6 +57,7 @@ The methods draw on, and in places reimplement, solutions from: - S. French & K. Wilson, "On the treatment of negative intensity observations", *Acta Cryst.* **A34** (1978), 517-525 (Bayesian amplitude estimation from intensities), and CCP4's ctruncate (C. Ballard & N. Stein), whose anisotropic Wilson prior §10.8 follows, cited through M. D. Winn et al., "Overview of the CCP4 suite and current developments", *Acta Cryst.* **D67** (2011), 235-242. - A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free cross-validation). - P. H. C. Eilers, "A perfect smoother", *Anal. Chem.* **75** (2003), 3631-3636, after E. T. Whittaker, "On a new method of graduation", *Proc. Edinburgh Math. Soc.* **41** (1923), 63-75 (the penalised smoother of the fulls' per-frame scale, §10.6). +- H. F. Walker & P. Ni, "Anderson acceleration for fixed-point iterations", *SIAM J. Numer. Anal.* **49** (2011), 1715-1735, after D. G. Anderson, "Iterative procedures for nonlinear integral equations", *J. ACM* **12** (1965), 547-560 (the acceleration of the per-frame scaling loops, §10.6). - R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", *Biometrika* **10** (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared). - M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (model / structure-factor / map machinery used in §14). - J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and [github.com/jonwright/bslz4decoders](https://github.com/jonwright/bslz4decoders) (device-side decoding of bitshuffle+LZ4 images, §0). diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 2541337ad..9be9ef481 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -388,6 +388,8 @@ Before the combine a per-frame scale can also be fitted on the partials themselv How smooth that scale is over the rotation is left to the data rather than to a fixed window. Each round fits every frame on its own fulls against the current reference, giving a scale $G_f$ and its information $D_f=\sum w^2c^2$; the scale is then the curve $x=\log G$ minimising $\sum_f J_f\,(x_f-y_f)^2+\lambda\sum_f(\Delta^2 x)_f^2$, with $y_f$ the frame's own fit in log scale and $J_f$ its information carried there — a penalised (Whittaker–Eilers) smoother, solved as a five-band linear system. $\lambda$ is chosen by cross-validation: blocks of frames one rocking curve wide are left out in turn and predicted from the curve through the rest (neighbours closer than a rocking curve share their measurement, since a full sums those frames). A frame of hundreds of fulls is then followed frame by frame, a frame of two or three is carried by its neighbours, a stretch with none is bridged by a straight line, and a scale that falls a hundredfold over a few degrees — an absorbing crystal turning edge-on — is followed where a window would average across it. Once the curve settles, one free fit is shrunk toward it frame by frame by how much of each frame's deviation its neighbour shares (the lag-1 covariance), which hands back a real per-frame systematic and discards fit noise. +Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the step grows or the set of fitted frames changes. This changes the path, not the fixed point; the loops stop at the same tolerance (an rms step of $10^{-3}$ in $\log G$) in about a quarter of the rounds and closer to the converged scales than the plain iteration stopped there. + After scale-fulls, five **correction surfaces** are fitted on the combined fulls (rotation path, **on by default**; disable all with `--no-scaling-corrections`), each an alternating multiplicative refinement of the per-full scale against the merged reference: - **Decay.** Radiation damage weakens later frames more at higher resolution — a resolution×time (Debye–Waller) systematic the resolution-flat per-image scale cannot capture. A single global relative-$B$ rate is fitted, $\ln(I_\mathrm{ref}/I_\mathrm{obs}) = 2\,(\mathrm{d}B/\mathrm{d}n)\,(n-\bar n)\,s^2$ (frame $n$, $s^2 = 1/4d^2$), and folded into the scale. It engages only when the total relative-$B$ over the run exceeds a physical floor (2 Ų); below that the decay is negligible and "correcting" it only spreads symmetry equivalents (same $s^2$, different frames). An optional **per-batch relative-$B$** (`--relative-b[=deg]`, off unless requested; 10°-of-rotation batches by default) extends the single global rate to a smooth $B(n)$ curve — the same $s^2$-weighted decay fit solved independently over short frame batches, curvature-penalized so it cannot over-fit and cross-validated like the surfaces below — for crystals whose decay is non-linear in dose. Its cross-validation splits on **ASU-group parity**, not the frame parity the surfaces below use: a per-batch parameter owns whole frames and so cannot be scored on a held-out frame, whereas splitting the symmetry equivalents tests whether a batch's $B$ generalises to reflections it was not fitted on. diff --git a/image_analysis/scale_merge/AndersonMixing.cpp b/image_analysis/scale_merge/AndersonMixing.cpp new file mode 100644 index 000000000..076a462de --- /dev/null +++ b/image_analysis/scale_merge/AndersonMixing.cpp @@ -0,0 +1,47 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "AndersonMixing.h" + +std::vector AndersonMixing::Next(const std::vector &x, const std::vector &fx, + const std::vector &w) { + // Anderson acceleration, type II, after Walker & Ni (2011) SIAM J. Numer. Anal. 49, 1715-1735 + // (Anderson (1965) J. ACM 12, 547-560): with dX and dR the differences of the states and of the + // residuals of consecutive rounds, gamma = argmin |r - dR gamma|_w and the next state is + // F(x) - (dX + dR) gamma. + const int n = static_cast(x.size()); + const Eigen::Map xv(x.data(), n), fxv(fx.data(), n), wv(w.data(), n); + const Eigen::VectorXd r = fxv - xv; + std::vector support(n); + for (int i = 0; i < n; ++i) support[i] = w[i] > 0.0 ? 1 : 0; + const double r_norm = (wv.array() * r.array().square()).sum(); + + if (x_last.size() == n && support == support_last && r_norm <= r_norm_last) { + dx.push_back(xv - x_last); + dr.push_back(r - r_last); + if (static_cast(dx.size()) > depth) { + dx.pop_front(); + dr.pop_front(); + } + } else if (!dx.empty()) { + dx.clear(); + dr.clear(); + ++restarts; + } + x_last = xv; + r_last = r; + support_last = std::move(support); + r_norm_last = r_norm; + + std::vector out = fx; + const int m = static_cast(dr.size()); + if (m == 0) return out; + const Eigen::VectorXd sqrt_w = wv.cwiseMax(0.0).cwiseSqrt(); + Eigen::MatrixXd M(n, m); + for (int j = 0; j < m; ++j) M.col(j) = sqrt_w.cwiseProduct(dr[j]); + const Eigen::VectorXd gamma = M.colPivHouseholderQr().solve(sqrt_w.cwiseProduct(r)); + for (int j = 0; j < m; ++j) + for (int i = 0; i < n; ++i) + if (w[i] > 0.0) out[i] -= gamma[j] * (dx[j][i] + dr[j][i]); + return out; +} diff --git a/image_analysis/scale_merge/AndersonMixing.h b/image_analysis/scale_merge/AndersonMixing.h new file mode 100644 index 000000000..8cc18a16d --- /dev/null +++ b/image_analysis/scale_merge/AndersonMixing.h @@ -0,0 +1,40 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include + +#include + +// Acceleration of a fixed-point iteration x -> F(x) that converges linearly. Each round hands in the +// state x it started from and F(x), where one plain round took it; the residual r = F(x) - x is what +// the plain iteration would step. The state the next round starts from is F(x) corrected by the +// combination of the last few rounds' steps that best cancels r, so the slow directions the plain +// iteration creeps along - a handful of them, whose contraction is close to one - are taken in a few +// rounds. Where r = 0 the correction is 0: the fixed point is the plain iteration's. +// +// Each component carries a weight in the norm r is measured in; a component of weight 0 takes F(x) +// unchanged and is not part of the fit. The history is dropped, and the round is a plain one, when +// the residual grew - the history no longer describes the map from here - or when the set of +// components of non-zero weight changed, since the differences of the history are then differences +// of different things. +class AndersonMixing { +public: + explicit AndersonMixing(int depth) : depth(depth) {} + + std::vector Next(const std::vector &x, const std::vector &fx, + const std::vector &w); + + int Restarts() const { return restarts; } + +private: + int depth; + int restarts = 0; + std::deque dx, dr; // x and r differences of consecutive rounds, newest last + Eigen::VectorXd x_last, r_last; + std::vector support_last; + double r_norm_last = 0.0; +}; diff --git a/image_analysis/scale_merge/CMakeLists.txt b/image_analysis/scale_merge/CMakeLists.txt index ae658c62f..bae06d3d2 100644 --- a/image_analysis/scale_merge/CMakeLists.txt +++ b/image_analysis/scale_merge/CMakeLists.txt @@ -37,6 +37,8 @@ ADD_LIBRARY(JFJochScaleMerge WilsonOutliers.h ErrorModel.cpp ErrorModel.h + AndersonMixing.cpp + AndersonMixing.h SublatticeRelabel.cpp SublatticeRelabel.h) TARGET_LINK_LIBRARIES(JFJochScaleMerge Ceres::ceres Eigen3::Eigen JFJochCommon fftw3f) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index fa02a8d48..c8606697d 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -26,6 +26,7 @@ #include #include +#include "AndersonMixing.h" #include "HKLKey.h" #include "RfreeFlags.h" #include "FrenchWilson.h" @@ -60,6 +61,10 @@ namespace { // The per-frame scaling loop has settled when the scales moved less than this between two // iterations, as the rms of |log(G_new/G_old)| over the frames fitted in both. constexpr double SCALING_TOLERANCE = 1e-3; + // The rounds of history the per-frame scaling loops are accelerated with (AndersonMixing). Their slow + // directions are a few smooth modes over the sweep: measured on a 4nwv-class sweep, 3, 5 and 8 rounds + // take its ten loops in 92, 91 and 90 iterations, against 364 without. + constexpr int SCALING_ANDERSON_DEPTH = 5; // The per-frame scale is smoothed over at least this many rocking curves (see Run). constexpr int SMOOTH_G_MIN_ROCKING_EVENTS = 6; @@ -1846,12 +1851,22 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( // and handing any of it back re-opens it (measured: ISa 23.8 -> 22.1 on a 360 deg, 14x merge). The // fulls, each a whole rocking event, have their own loop (RunFullsScalingLoop). // + // The iteration converges linearly, and slowly: a round takes 5-15% off the step on most sweeps and + // 2% on the slowest, so a loop stopped on its step stopped several steps' worth short of where it + // was going. Each round's scales are therefore accelerated (AndersonMixing, in log G, each frame at + // its observation count): the plain round's answer corrected by the combination of the last few + // rounds' steps that best cancels this one. That moves no fixed point - only the path to it - and + // it takes the loop to its answer in a quarter of the rounds and closer: on one sweep the merged + // intensities ended 0.02% rms from the converged point, against 0.12% for the plain loop stopped at + // the same tolerance. A frame below the credible floor (below) takes the plain round's scale. + // // The loop stops when the relative scales stop moving - the rms of |log(G_new/G_old)| over the // frames, each weighted by its observation count and nothing else (a weight that vanishes with G // is what let a collapsing sweep report itself settled) - or when they stop settling: a step that // has not fallen below nine tenths of its smallest value for five rounds is a loop on a flat or // rising valley, and more rounds of it can only walk further. Both leave the frames where they are - // and say so. + // and say so. The step is the residual of the round, F(x) - x at the state the round started from, + // so the test reads the same whether that state was a plain round's or an accelerated one. ScalingLoopOutcome out; std::vector scaled_any(n_frames, 0); std::vector g_prev(n_frames, 0.0), g_smooth; @@ -1859,6 +1874,8 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( std::vector ratio(n_frames, 1.0); double best_step = INFINITY; int rounds_since_best = 0; + AndersonMixing anderson(SCALING_ANDERSON_DEPTH); + std::vector log_g_prev(n_frames), log_g(n_frames), weight(n_frames); for (int it = 0; it < scaling_iter; ++it) { iterate(); out.iterations = it + 1; @@ -1874,15 +1891,15 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( if (scaled_any[f] && std::isfinite(g_smooth[f]) && g_smooth[f] > 0.0) g[f] = g_smooth[f]; } - for (int f = 0; f < n_frames; ++f) - n_obs[f] = scaled_any[f] && std::isfinite(g[f]) && g[f] > 0.0 ? frame_obs_count[f] : 0; - const double g_ref = TypicalFrameScale(g, n_obs); + std::vector usable(n_frames); for (int f = 0; f < n_frames; ++f) { - if (!scaled_any[f] || !(std::isfinite(g[f]) && g[f] > 0.0)) { ratio[f] = 1.0; continue; } - g[f] /= g_ref; - ratio[f] /= g[f]; + usable[f] = scaled_any[f] && std::isfinite(g[f]) && g[f] > 0.0 ? 1 : 0; + n_obs[f] = usable[f] ? frame_obs_count[f] : 0; } - rescale(scaled_any, ratio); + const double g_ref = TypicalFrameScale(g, n_obs); + for (int f = 0; f < n_frames; ++f) + if (usable[f]) g[f] /= g_ref; + bool stop = false; if (it > 0) { // A frame already below the credible floor is not part of the fit's answer: the window // leaves it out (ComputeSmoothGWindow) and the guard after the loop drops it. Left in the @@ -1899,28 +1916,47 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( } double ss = 0.0, sw = 0.0; int n = 0; - for (int f = 0; f < n_frames; ++f) + for (int f = 0; f < n_frames; ++f) { + weight[f] = 0.0; if (scaled_any[f] && g_prev[f] > 0.0 && g[f] > 0.0 && g[f] >= g_floor) { - const double d = std::log(g[f] / g_prev[f]); + log_g_prev[f] = std::log(g_prev[f]); + log_g[f] = std::log(g[f]); + const double d = log_g[f] - log_g_prev[f]; ss += frame_obs_count[f] * d * d; sw += frame_obs_count[f]; + weight[f] = frame_obs_count[f]; ++n; + } else { + log_g_prev[f] = log_g[f] = 0.0; } + } out.step = sw > 0.0 ? std::sqrt(ss / sw) : 0.0; logger->Debug("Per-frame scaling ({}): iteration {}, rms |dlogG| {:.2e} over {} frames, " - "gauge {:.4f}", what, it + 1, out.step, n, g_ref); + "gauge {:.4f}, {} restarts", what, it + 1, out.step, n, g_ref, anderson.Restarts()); if (out.step < SCALING_TOLERANCE) { out.converged = true; - break; - } - if (out.step < 0.9 * best_step) { + stop = true; + } else if (out.step < 0.9 * best_step) { best_step = out.step; rounds_since_best = 0; } else if (++rounds_since_best >= 5) { out.stalled = true; - break; + stop = true; + } + if (!stop) { + // The next round starts from the accelerated scales, gauge-pinned again. + const std::vector next = anderson.Next(log_g_prev, log_g, weight); + for (int f = 0; f < n_frames; ++f) + if (weight[f] > 0.0) g[f] *= std::exp(next[f] - log_g[f]); + const double g_ref_next = TypicalFrameScale(g, n_obs); + for (int f = 0; f < n_frames; ++f) + if (usable[f]) g[f] /= g_ref_next; } } + for (int f = 0; f < n_frames; ++f) + ratio[f] = usable[f] ? ratio[f] / g[f] : 1.0; + rescale(scaled_any, ratio); + if (stop) break; g_prev = g; } frame_scaled_scratch = scaled_any; @@ -1966,7 +2002,9 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( // zero, where DropCollapsedFullScales finds it, without one round's noise throwing it there. // // A frame below the credible floor is left out of the step, as in RunScalingLoop: it walks - // toward zero for as long as the loop runs, which is the answer, not a failure to settle. + // toward zero for as long as the loop runs, which is the answer, not a failure to settle. The + // rounds are accelerated as RunScalingLoop's are, in log G with each frame at its count of fulls; + // the frame below the floor takes the plain round's scale. ScalingLoopOutcome out; std::vector y(n_frames), J(n_frames), g_new(n_frames), ratio(n_frames); std::vector n_obs(n_frames); @@ -1977,6 +2015,8 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( } double best_step = INFINITY, lambda = 0.0; int rounds_since_best = 0; + AndersonMixing anderson(SCALING_ANDERSON_DEPTH); + std::vector log_g(n_frames), log_g_new(n_frames), weight(n_frames); for (int it = 0; it < scaling_iter; ++it) { fit(); out.iterations = it + 1; @@ -1995,33 +2035,50 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( std::vector scales; for (int f = 0; f < n_frames; ++f) { g_new[f] /= g_ref; - ratio[f] = g[f] / g_new[f]; if (has_fulls[f]) scales.push_back(g_new[f]); } - rescale(has_fulls, ratio); const double g_floor = scales.empty() ? 0.0 : median_of(scales) * MIN_CREDIBLE_SCALE_RATIO; double ss = 0.0, sw = 0.0; - for (int f = 0; f < n_frames; ++f) + for (int f = 0; f < n_frames; ++f) { + weight[f] = 0.0; if (has_fulls[f] && g_new[f] >= g_floor) { - const double d = std::log(g_new[f] / g[f]); + log_g[f] = std::log(g[f]); + log_g_new[f] = std::log(g_new[f]); + const double d = log_g_new[f] - log_g[f]; ss += frame_obs_count[f] * d * d; sw += frame_obs_count[f]; + weight[f] = frame_obs_count[f]; + } else { + log_g[f] = log_g_new[f] = 0.0; } - g = g_new; + } out.step = sw > 0.0 ? std::sqrt(ss / sw) : 0.0; - logger->Debug("Per-frame scaling (fulls): iteration {}, rms |dlogG| {:.2e}, lambda {:.3g}, gauge {:.4f}", - it + 1, out.step, lambda, g_ref); + logger->Debug("Per-frame scaling (fulls): iteration {}, rms |dlogG| {:.2e}, lambda {:.3g}, gauge {:.4f}, " + "{} restarts", it + 1, out.step, lambda, g_ref, anderson.Restarts()); + bool stop = false; if (out.step < SCALING_TOLERANCE) { out.converged = true; - break; - } - if (out.step < 0.9 * best_step) { + stop = true; + } else if (out.step < 0.9 * best_step) { best_step = out.step; rounds_since_best = 0; } else if (++rounds_since_best >= 5) { out.stalled = true; - break; + stop = true; } + if (!stop) { + // The next round starts from the accelerated scales, gauge-pinned again. + const std::vector next = anderson.Next(log_g, log_g_new, weight); + for (int f = 0; f < n_frames; ++f) + if (weight[f] > 0.0) g_new[f] *= std::exp(next[f] - log_g_new[f]); + const double g_ref_next = TypicalFrameScale(g_new, n_obs); + for (int f = 0; f < n_frames; ++f) g_new[f] /= g_ref_next; + } + for (int f = 0; f < n_frames; ++f) + ratio[f] = g[f] / g_new[f]; + rescale(has_fulls, ratio); + g = g_new; + if (stop) break; } // The curve is the part of the scale the frames share over a rocking curve and more; what a // frame's own fulls say beyond it is handed back by how much of it is real (ShrinkToRestrained) - diff --git a/tests/AndersonMixingTest.cpp b/tests/AndersonMixingTest.cpp new file mode 100644 index 000000000..255b8b76f --- /dev/null +++ b/tests/AndersonMixingTest.cpp @@ -0,0 +1,115 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include + +#include "../image_analysis/scale_merge/AndersonMixing.h" + +namespace { + // A noiseless sweep of 24 frames whose reflections each span three consecutive frames, scaled by the + // alternating least squares of the per-frame scaling loop: the reference intensities from the + // scales, the scales from the references, the gauge pinned at a geometric mean of 1. A frame is + // tied only to its neighbours, so a smooth error of the scales over the sweep decays by a few + // percent a round - the slow linear convergence of the real loop. + struct SlowScalingLoop { + static constexpr int n_frames = 24, span = 3; + std::vector first; // reflection h is on frames first[h] .. first[h] + span - 1 + std::vector I_true, G_true; + + SlowScalingLoop() { + for (int h = 0; h + span <= n_frames; ++h) + for (int rep = 0; rep < 3; ++rep) { + first.push_back(h); + I_true.push_back(1.0 + 0.5 * std::sin(1.7 * h + rep)); + } + for (int f = 0; f < n_frames; ++f) + G_true.push_back(1.0 + 0.4 * std::sin(6.283 * f / n_frames) + 0.2 * std::cos(2.1 * f)); + } + + std::vector Round(const std::vector &g) const { + std::vector num(n_frames, 0.0), den(n_frames, 0.0), out(n_frames); + for (size_t h = 0; h < first.size(); ++h) { + double mean = 0.0; + for (int k = 0; k < span; ++k) mean += G_true[first[h] + k] * I_true[h] / g[first[h] + k]; + mean /= span; + for (int k = 0; k < span; ++k) { + num[first[h] + k] += G_true[first[h] + k] * I_true[h] * mean; + den[first[h] + k] += mean * mean; + } + } + double s = 0.0; + for (int f = 0; f < n_frames; ++f) { out[f] = num[f] / den[f]; s += std::log(out[f]); } + for (int f = 0; f < n_frames; ++f) out[f] /= std::exp(s / n_frames); + return out; + } + + // Rounds to a step below `tolerance`, and the largest |log G - log G_true| there (gauge-pinned). + std::pair Run(int depth, double tolerance) const { + std::vector g(n_frames, 1.0), log_g(n_frames), log_g_new(n_frames), w(n_frames, 1.0); + AndersonMixing anderson(depth); + int rounds = 0; + for (double step = INFINITY; step >= tolerance && rounds < 100000; ++rounds) { + std::vector g_new = Round(g); + double ss = 0.0; + for (int f = 0; f < n_frames; ++f) { + log_g[f] = std::log(g[f]); + log_g_new[f] = std::log(g_new[f]); + ss += (log_g_new[f] - log_g[f]) * (log_g_new[f] - log_g[f]); + } + step = std::sqrt(ss / n_frames); + if (depth > 0) { + const std::vector next = anderson.Next(log_g, log_g_new, w); + double s = 0.0; + for (double v : next) s += v; + for (int f = 0; f < n_frames; ++f) g_new[f] = std::exp(next[f] - s / n_frames); + } + g = g_new; + } + double s = 0.0, err = 0.0; + for (double v : G_true) s += std::log(v); + for (int f = 0; f < n_frames; ++f) + err = std::max(err, std::fabs(std::log(g[f]) - (std::log(G_true[f]) - s / n_frames))); + return {rounds, err}; + } + }; +} + +TEST_CASE("AndersonMixing_SlowScalingLoop", "[RotationScale]") { + const SlowScalingLoop loop; + const auto [plain_rounds, plain_err] = loop.Run(0, 1e-6); + const auto [aa_rounds, aa_err] = loop.Run(5, 1e-6); + // The plain loop creeps: hundreds of rounds, and stopped on its step it still sits a hundred + // times the tolerance from the answer. + CHECK(plain_rounds > 300); + CHECK(plain_err > 5e-5); + // Accelerated: the same fixed point, in a tenth of the rounds, and closer to it at the same stop. + CHECK(aa_rounds * 10 < plain_rounds); + CHECK(aa_err < 0.5 * plain_err); +} + +TEST_CASE("AndersonMixing_ZeroWeightTakesThePlainStep", "[RotationScale]") { + // x -> 0.9 x on two components, the second not part of the fit. + AndersonMixing anderson(5); + std::vector x{1.0, 1.0}; + const std::vector w{1.0, 0.0}; + for (int it = 0; it < 4; ++it) { + const std::vector fx{0.9 * x[0], 0.9 * x[1]}; + const std::vector next = anderson.Next(x, fx, w); + CHECK(next[1] == fx[1]); + x = next; + } + // A linear map on the fitted component is solved by the first mixed step. + CHECK(std::fabs(x[0]) < 1e-12); + CHECK(anderson.Restarts() == 0); +} + +TEST_CASE("AndersonMixing_RestartsWhenTheResidualGrows", "[RotationScale]") { + AndersonMixing anderson(5); + const std::vector w{1.0}; + CHECK(anderson.Next({1.0}, {0.5}, w)[0] == 0.5); // no history: the plain step + anderson.Next({0.5}, {0.25}, w); // residual fell: history kept + CHECK(anderson.Restarts() == 0); + CHECK(anderson.Next({0.0}, {2.0}, w)[0] == 2.0); // residual grew: history dropped, plain step + CHECK(anderson.Restarts() == 1); +} diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 8a670baf0..6f7b1bb10 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -111,6 +111,7 @@ ADD_EXECUTABLE(jfjoch_test DiagnosticOutputTest.cpp PostRefineTest.cpp RotationScaleWalkTest.cpp + AndersonMixingTest.cpp RugnuxLargeTest.cpp TestData.h MovingAverageTest.cpp -- 2.54.0 From 089423067193dc23de3deca295955f10f548b9ca Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 08:18:06 +0200 Subject: [PATCH 02/53] Anderson: accelerate a scaling loop only near its fixed point, and never again once its step grows (RESULT-CHANGING) The Anderson commit before this one changed three rc176 battery results, which is why it was reverted there. One of them is the acceleration's own doing: on a split myoglobin sweep (P2_1, ISa 2.5) the partials loops never settle - one walks at a step of ~2e-2 for as long as it runs - and the fulls loops walk from their first round (their cross-validated lambda moves every round, so the map itself is not fixed). Restarting the history whenever the step grew only rebuilt a one- or two-round history and extrapolated along the walk: the stalled loops ended somewhere else, the pre-pass fulls smoothing window went from ~7 frames to ~1, and CC1/2 fell 0.701 -> 0.595. Two rules now confine the acceleration to the regime it is for - a map contracting to its fixed point: - a loop is accelerated only once its plain step is below ten times the tolerance (1e-2); - the first round whose weighted residual grew turns the acceleration off for the rest of the loop (AndersonMixing::Stopped), instead of restarting it. A loop that walks therefore stays the plain loop it was in rc176. The growth rule alone gave CC1/2 0.625; with the onset, 0.701, as rc176. Measured against rc176 bc191bb80 (GPU, workstation): - myob_x06da_split: P2_1, CC1/2 0.701, d 1.81 A - as rc176; 481 -> 257 scaling iterations. - 7qij 434 -> 126 iterations, 9v16 273 -> 101, same space group and statistics. - ci tier (29 sets, no null model): 28 unchanged beyond noise; sum of times 580 -> 585 s (9gdj +9 s, below). The two other changes are NOT Anderson's and stay: - insu_I_x06da_weak I23 -> I222: the all-observations search refuses I23 on a merge chi2 ratio 2.08 against a 1.85 bound; plain scaling converged to 1e-5 gives 2.07-2.09 and refuses it too. Only the plain loop stopped at 1e-3 passed - the bound was set on unconverged scaling. - 9gdj +9 s, same answer: the early beam-centre comparison's "loses beyond noise" margin (-0.002 against 2 sigma 0.0016) moves by ~0.07 between scaling paths, so the converged loop no longer takes the shortcut and both centres are judged on all frames; the file's centre is kept either way. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 2 +- image_analysis/scale_merge/AndersonMixing.cpp | 6 +++++- image_analysis/scale_merge/AndersonMixing.h | 12 +++++++++--- .../scale_merge/RotationScaleMerge.cpp | 17 ++++++++++++----- tests/AndersonMixingTest.cpp | 9 ++++++--- 5 files changed, 33 insertions(+), 13 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 9be9ef481..cdd8736a6 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -388,7 +388,7 @@ Before the combine a per-frame scale can also be fitted on the partials themselv How smooth that scale is over the rotation is left to the data rather than to a fixed window. Each round fits every frame on its own fulls against the current reference, giving a scale $G_f$ and its information $D_f=\sum w^2c^2$; the scale is then the curve $x=\log G$ minimising $\sum_f J_f\,(x_f-y_f)^2+\lambda\sum_f(\Delta^2 x)_f^2$, with $y_f$ the frame's own fit in log scale and $J_f$ its information carried there — a penalised (Whittaker–Eilers) smoother, solved as a five-band linear system. $\lambda$ is chosen by cross-validation: blocks of frames one rocking curve wide are left out in turn and predicted from the curve through the rest (neighbours closer than a rocking curve share their measurement, since a full sums those frames). A frame of hundreds of fulls is then followed frame by frame, a frame of two or three is carried by its neighbours, a stretch with none is bridged by a straight line, and a scale that falls a hundredfold over a few degrees — an absorbing crystal turning edge-on — is followed where a window would average across it. Once the curve settles, one free fit is shrunk toward it frame by frame by how much of each frame's deviation its neighbour shares (the lag-1 covariance), which hands back a real per-frame systematic and discards fit noise. -Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the step grows or the set of fitted frames changes. This changes the path, not the fixed point; the loops stop at the same tolerance (an rms step of $10^{-3}$ in $\log G$) in about a quarter of the rounds and closer to the converged scales than the plain iteration stopped there. +Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the set of fitted frames changes. Acceleration starts only once the plain step has fallen below ten times the tolerance, and stops for the rest of the loop the first time the step grows: Anderson mixing models a map near its fixed point, and a loop further out, or one whose step grows, may be walking rather than converging - accelerated, it walks faster and ends somewhere else. This changes the path, not the fixed point; the loops stop at the same tolerance (an rms step of $10^{-3}$ in $\log G$) in about a quarter of the rounds and closer to the converged scales than the plain iteration stopped there. After scale-fulls, five **correction surfaces** are fitted on the combined fulls (rotation path, **on by default**; disable all with `--no-scaling-corrections`), each an alternating multiplicative refinement of the per-full scale against the merged reference: diff --git a/image_analysis/scale_merge/AndersonMixing.cpp b/image_analysis/scale_merge/AndersonMixing.cpp index 076a462de..2d307eb3d 100644 --- a/image_analysis/scale_merge/AndersonMixing.cpp +++ b/image_analysis/scale_merge/AndersonMixing.cpp @@ -16,7 +16,11 @@ std::vector AndersonMixing::Next(const std::vector &x, const std for (int i = 0; i < n; ++i) support[i] = w[i] > 0.0 ? 1 : 0; const double r_norm = (wv.array() * r.array().square()).sum(); - if (x_last.size() == n && support == support_last && r_norm <= r_norm_last) { + if (stopped || (x_last.size() == n && r_norm > r_norm_last)) { + stopped = true; + return fx; + } + if (x_last.size() == n && support == support_last) { dx.push_back(xv - x_last); dr.push_back(r - r_last); if (static_cast(dx.size()) > depth) { diff --git a/image_analysis/scale_merge/AndersonMixing.h b/image_analysis/scale_merge/AndersonMixing.h index 8cc18a16d..1f44ea370 100644 --- a/image_analysis/scale_merge/AndersonMixing.h +++ b/image_analysis/scale_merge/AndersonMixing.h @@ -18,9 +18,13 @@ // // Each component carries a weight in the norm r is measured in; a component of weight 0 takes F(x) // unchanged and is not part of the fit. The history is dropped, and the round is a plain one, when -// the residual grew - the history no longer describes the map from here - or when the set of -// components of non-zero weight changed, since the differences of the history are then differences -// of different things. +// the set of components of non-zero weight changed, since the differences of the history are then +// differences of different things. +// +// Acceleration is for a map that contracts. A residual that grows from one round to the next says the +// iteration is not on its way to a fixed point here - it walks, or the map itself moved - and +// extrapolating from its history takes it further along the walk. From that round on every round is +// a plain one. class AndersonMixing { public: explicit AndersonMixing(int depth) : depth(depth) {} @@ -29,10 +33,12 @@ public: const std::vector &w); int Restarts() const { return restarts; } + bool Stopped() const { return stopped; } private: int depth; int restarts = 0; + bool stopped = false; std::deque dx, dr; // x and r differences of consecutive rounds, newest last Eigen::VectorXd x_last, r_last; std::vector support_last; diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index c8606697d..19581fc40 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -65,6 +65,9 @@ namespace { // directions are a few smooth modes over the sweep: measured on a 4nwv-class sweep, 3, 5 and 8 rounds // take its ten loops in 92, 91 and 90 iterations, against 364 without. constexpr int SCALING_ANDERSON_DEPTH = 5; + // The loops are accelerated only once their step is within this of the tolerance: Anderson mixing + // models the map near its fixed point, and a loop still further out is as likely to be walking. + constexpr double SCALING_ANDERSON_ONSET = 10.0 * SCALING_TOLERANCE; // The per-frame scale is smoothed over at least this many rocking curves (see Run). constexpr int SMOOTH_G_MIN_ROCKING_EVENTS = 6; @@ -1858,7 +1861,9 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( // rounds' steps that best cancels this one. That moves no fixed point - only the path to it - and // it takes the loop to its answer in a quarter of the rounds and closer: on one sweep the merged // intensities ended 0.02% rms from the converged point, against 0.12% for the plain loop stopped at - // the same tolerance. A frame below the credible floor (below) takes the plain round's scale. + // the same tolerance. A frame below the credible floor (below) takes the plain round's scale. A + // loop whose step grows is not contracting - it walks, and acceleration would only walk it faster - + // so from that round on its rounds are plain ones. // // The loop stops when the relative scales stop moving - the rms of |log(G_new/G_old)| over the // frames, each weighted by its observation count and nothing else (a weight that vanishes with G @@ -1932,7 +1937,8 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( } out.step = sw > 0.0 ? std::sqrt(ss / sw) : 0.0; logger->Debug("Per-frame scaling ({}): iteration {}, rms |dlogG| {:.2e} over {} frames, " - "gauge {:.4f}, {} restarts", what, it + 1, out.step, n, g_ref, anderson.Restarts()); + "gauge {:.4f}, {} restarts{}", what, it + 1, out.step, n, g_ref, anderson.Restarts(), + anderson.Stopped() ? ", plain" : ""); if (out.step < SCALING_TOLERANCE) { out.converged = true; stop = true; @@ -1943,7 +1949,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( out.stalled = true; stop = true; } - if (!stop) { + if (!stop && out.step < SCALING_ANDERSON_ONSET) { // The next round starts from the accelerated scales, gauge-pinned again. const std::vector next = anderson.Next(log_g_prev, log_g, weight); for (int f = 0; f < n_frames; ++f) @@ -2054,7 +2060,8 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( } out.step = sw > 0.0 ? std::sqrt(ss / sw) : 0.0; logger->Debug("Per-frame scaling (fulls): iteration {}, rms |dlogG| {:.2e}, lambda {:.3g}, gauge {:.4f}, " - "{} restarts", it + 1, out.step, lambda, g_ref, anderson.Restarts()); + "{} restarts{}", it + 1, out.step, lambda, g_ref, anderson.Restarts(), + anderson.Stopped() ? ", plain" : ""); bool stop = false; if (out.step < SCALING_TOLERANCE) { out.converged = true; @@ -2066,7 +2073,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( out.stalled = true; stop = true; } - if (!stop) { + if (!stop && out.step < SCALING_ANDERSON_ONSET) { // The next round starts from the accelerated scales, gauge-pinned again. const std::vector next = anderson.Next(log_g, log_g_new, weight); for (int f = 0; f < n_frames; ++f) diff --git a/tests/AndersonMixingTest.cpp b/tests/AndersonMixingTest.cpp index 255b8b76f..6eac50194 100644 --- a/tests/AndersonMixingTest.cpp +++ b/tests/AndersonMixingTest.cpp @@ -104,12 +104,15 @@ TEST_CASE("AndersonMixing_ZeroWeightTakesThePlainStep", "[RotationScale]") { CHECK(anderson.Restarts() == 0); } -TEST_CASE("AndersonMixing_RestartsWhenTheResidualGrows", "[RotationScale]") { +TEST_CASE("AndersonMixing_PlainOnceTheResidualGrows", "[RotationScale]") { AndersonMixing anderson(5); const std::vector w{1.0}; CHECK(anderson.Next({1.0}, {0.5}, w)[0] == 0.5); // no history: the plain step anderson.Next({0.5}, {0.25}, w); // residual fell: history kept + CHECK(!anderson.Stopped()); + CHECK(anderson.Next({0.0}, {2.0}, w)[0] == 2.0); // residual grew: the plain step + CHECK(anderson.Stopped()); + CHECK(anderson.Next({2.0}, {2.1}, w)[0] == 2.1); // and plain from then on, though it fell again + CHECK(anderson.Next({2.1}, {2.15}, w)[0] == 2.15); CHECK(anderson.Restarts() == 0); - CHECK(anderson.Next({0.0}, {2.0}, w)[0] == 2.0); // residual grew: history dropped, plain step - CHECK(anderson.Restarts() == 1); } -- 2.54.0 From e724f3d26a62cd4663d64e1c52d644726631f73f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 13:18:48 +0200 Subject: [PATCH 03/53] Beam-centre gates: decide on the strong-reflection count, not on how a scaling loop stopped MeasuredCentreClearlyLoses, different-lattice branch: the measured centre loses on the pre-pass subset when it measures clearly fewer strong reflections (Poisson, 2 sigma) than the file's centre. The subset CC1/2 is no longer read there (still logged). A third of the frames in five blocks merges to a multiplicity of ~1.09 in P1, so its CC1/2 reads the relative scale of the blocks: on 9gdj the file centre's subset CC1/2 moved 0.899 / 0.880 / 0.832 / 0.846 with the scaling path (plain 1e-3 / 1e-5, Anderson 1e-3 / 1e-5), ~50x the Fisher noise the gate allowed, and flipped the early verdict. The count is comparable across lattice classes (taken on the primitive lattice) and stable across paths: 36.6-38.6k against 103.4-103.6k on 9gdj. MeasuredCentreWins, same-lattice branch: the measured centre has to be ahead on the strong count by more than 2 Poisson sigmas; within that the file's centre stays (it adopted at 1874 vs 1873 on one in-house small-molecule set, and at 0.4 sigma on 9zmu). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- rugnux/Rugnux.h | 9 +++++---- rugnux/RugnuxPasses.cpp | 31 +++++++++++++++++++++---------- tests/RugnuxTest.cpp | 15 ++++++++++----- 3 files changed, 36 insertions(+), 19 deletions(-) diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index cf08a2cee..4af5023fd 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -839,10 +839,11 @@ bool ArmsHoldSameLattice(const ProcessResult &a, const ProcessResult &b); std::string MeasuredCentreWins(const ProcessResult &file_arm, const ProcessResult &measured_arm); // The search-merge numbers of a pass, as MeasuredCentreWins reads them. SearchMergeSignal SearchSignal(const ProcessResult &r); -// Whether the measured centre loses on every line MeasuredCentreWins draws by more than `sigmas` -// times that line's noise - the Poisson noise of a count, the standard error of a CC1/2 - with both -// arms merged from the same frames. The run asks it at the near-tie distance of its first passes -// (FirstPassRun::NEAR_TIE_SIGMA); anything else, a near tie or a win, is decided on all the frames. +// Whether the measured centre is behind on the strong-reflection count by more than `sigmas` times +// its Poisson noise - and, on the same lattice, also behind on the inner-shell CC1/2 line +// MeasuredCentreWins draws - with both arms merged from the same frames. The run asks it at the +// near-tie distance of its first passes (FirstPassRun::NEAR_TIE_SIGMA); anything else, a near tie or +// a win, is decided on all the frames. bool MeasuredCentreClearlyLoses(const SearchMergeSignal &file_arm, const SearchMergeSignal &measured_arm, bool same_lattice, double sigmas); diff --git a/rugnux/RugnuxPasses.cpp b/rugnux/RugnuxPasses.cpp index a0776553b..7256ab640 100644 --- a/rugnux/RugnuxPasses.cpp +++ b/rugnux/RugnuxPasses.cpp @@ -89,6 +89,9 @@ namespace { // How much better the alternative beam centre's search merge has to be for the run to move off // the centre the file gave it (the beam-centre arbitration in RunAllPasses). constexpr double MAX_CC_HALF_LOSS = 0.05; + // How far, in Poisson sigmas, the alternative's strong-reflection count has to be ahead on the same + // lattice - the near-tie distance of the first passes (FirstPassRun::NEAR_TIE_SIGMA). + constexpr double STRONG_COUNT_SIGMAS = 2.0; // Say so rather than printing a zero that reads as a measurement. ("not compared", not "not // measured": completeness IS measured and reported; what did not happen is its use in arbitrating @@ -292,7 +295,11 @@ std::string MeasuredCentreWins(const ProcessResult &file_arm, const ProcessResul // two arms here integrate with the same settings, so the geometry is the only difference, and the // file's centre has no standing the measured one lacks - neither indexed the whole spot list. if (ArmsHoldSameLattice(file_arm, measured_arm)) { - if (measured_arm.search_merge_strong_reflections > file_arm.search_merge_strong_reflections) + // Ahead by more than the Poisson noise of the two counts; within it the file's centre stays. + const int64_t strong_m = measured_arm.search_merge_strong_reflections; + const int64_t strong_f = file_arm.search_merge_strong_reflections; + if (static_cast(strong_m - strong_f) + > STRONG_COUNT_SIGMAS * std::sqrt(static_cast(strong_m + strong_f))) return fmt::format("the measured centre measures more of the crystal - {} reflections at " "I/sigma >= 2 before corrections against {}", measured_arm.search_merge_strong_reflections, @@ -331,15 +338,19 @@ bool MeasuredCentreClearlyLoses(const SearchMergeSignal &file_arm, const SearchM const auto loses = [sigmas](double margin, double noise) { return std::isfinite(margin) && std::isfinite(noise) && margin < -sigmas * noise; }; - if (same_lattice) - return loses(static_cast(measured_arm.strong - file_arm.strong), - std::sqrt(static_cast(measured_arm.strong + file_arm.strong))) - && loses(measured_arm.inner_cc_half - file_arm.inner_cc_half - MAX_CC_HALF_LOSS, - std::hypot(cc_se(measured_arm.inner_cc_half, measured_arm.inner_reflections), - cc_se(file_arm.inner_cc_half, file_arm.inner_reflections))); - return loses(measured_arm.cc_half - file_arm.cc_half - MAX_CC_HALF_LOSS, - std::hypot(cc_se(measured_arm.cc_half, measured_arm.reflections), - cc_se(file_arm.cc_half, file_arm.reflections))); + // Both arms integrate the same frames with the same settings, and the strong reflections are + // counted on the primitive lattice, so the counts compare across lattice classes as well. The + // subset's CC1/2 does not decide the different-lattice case: a third of the frames in blocks merges + // to a multiplicity near one in P1, its CC1/2 then reads the relative scale of the blocks, and + // measured it moved by 0.07 with how far the scaling loop ran - fifty times its Fisher noise. + const bool behind_on_signal = loses(static_cast(measured_arm.strong - file_arm.strong), + std::sqrt(static_cast(measured_arm.strong + file_arm.strong))); + if (!same_lattice) + return behind_on_signal; + return behind_on_signal + && loses(measured_arm.inner_cc_half - file_arm.inner_cc_half - MAX_CC_HALF_LOSS, + std::hypot(cc_se(measured_arm.inner_cc_half, measured_arm.inner_reflections), + cc_se(file_arm.inner_cc_half, file_arm.inner_reflections))); } ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { diff --git a/tests/RugnuxTest.cpp b/tests/RugnuxTest.cpp index 8308608fa..c85c7b86e 100644 --- a/tests/RugnuxTest.cpp +++ b/tests/RugnuxTest.cpp @@ -645,6 +645,8 @@ TEST_CASE("MeasuredCentreWins", "[process]") { // Same lattice, same CC1/2: the arm that measured more signal wins, whichever centre it is... CHECK(!MeasuredCentreWins(file, arm(swapped, tri, 0.87, 9500)).empty()); CHECK(MeasuredCentreWins(file, arm(swapped, tri, 0.87, 8500)).empty()); + // ...but not by less than twice the Poisson noise of the two counts (sqrt(18100) = 135)... + CHECK(MeasuredCentreWins(file, arm(swapped, tri, 0.87, 9100)).empty()); // ...a tie keeps the file's centre, and a higher CC1/2 alone does not move it. CHECK(MeasuredCentreWins(file, arm(swapped, tri, 0.99, 9000)).empty()); @@ -676,12 +678,15 @@ TEST_CASE("MeasuredCentreClearlyLoses", "[process]") { m.inner_reflections = 20; CHECK_FALSE(MeasuredCentreClearlyLoses(inconsistent, m, true, 2.0)); - // Different lattices: the CC1/2 line, 0.05 above the file's centre. + // Different lattices: the count alone - the subset's CC1/2 is not read. SearchMergeSignal d = file; - d.cc_half = 0.80; + d.strong = 3000; + d.cc_half = 0.99; CHECK(MeasuredCentreClearlyLoses(file, d, false, 2.0)); - d.cc_half = 0.95; // exactly on the line + d.strong = 8900; // 100 behind: within two sigmas + d.cc_half = 0.50; CHECK_FALSE(MeasuredCentreClearlyLoses(file, d, false, 2.0)); - // Nothing measured is no loss: it is decided on all the frames. - CHECK_FALSE(MeasuredCentreClearlyLoses(file, SearchMergeSignal{}, false, 2.0)); + // A merge with no strong reflection in it measured none of the crystal. (An arm without a merge + // is never asked: RunAllPasses judges it on all the frames.) + CHECK(MeasuredCentreClearlyLoses(file, SearchMergeSignal{}, false, 2.0)); } -- 2.54.0 From 37c4f152e13c3295c61b651f127b5e1dcb26496c Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 13:45:14 +0200 Subject: [PATCH 04/53] SearchSpaceGroup: the chi^2-ratio and systematic-b gates are not asked of a merge whose L-test reads untwinned (RESULT-CHANGING) Both statistics divide the disagreement a promotion's added operators bring by the P1 search merge's sigmas, which carry the error model the P1 scaling fitted. P1 scaling never ties symmetry mates together, so the better the per-frame scaling converges, the smaller those sigmas and the larger the same real systematic floor between symmetry mates reads: the ratios tend to functions of the sigma-free disagreement H measures directly (chi^2 ratio ~ H ratio^2, b ratio ~ H ratio). Measured on the weak in-house cubic insulin set (true I23), all-observation arm, 23 over 222: plain, tol 1e-3 chi^2 2.024 b ratio 1.438 (bound 1.44, rescued) -> I23 plain, tol 3e-4 chi^2 2.072 b ratio 1.453 -> I222 plain, tol 1e-5 chi^2 2.094 b ratio 1.460 -> I222 Anderson 5, 1e-3 chi^2 2.077 b ratio 1.453 -> I222 chi2_ref 4.16-4.18 (> chi2_ref_reliable, so the b ratio alone decided); H ratio 1.46-1.50, every sigma-free gate passing. The late re-merge comparison (R_meas 0.1388 -> 0.1419, ISa 21.5 -> 19.5) then kept I222: a genuine cubic step with a 3-fold systematic floor always costs a little. These two tests exist for the twin law H cannot tell from real symmetry. A twin compresses the intensity distribution; this crystal's <|L|> is 0.48, the twins they refuse read 0.39-0.43 (including a trigonal twin, the only corpus case so far refused by them alone). So they are now skipped where the merge reads untwinned (<|L|> >= 0.44, AnalyzeTwinning's bar), and the sigma-free gates decide there alone. With it the set gives I23 under plain scaling at 1e-3, 3e-4 and 1e-5 and under Anderson at 1e-3 and 1e-5. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- .../scale_merge/SearchSpaceGroup.cpp | 31 ++++++++++++++----- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/image_analysis/scale_merge/SearchSpaceGroup.cpp b/image_analysis/scale_merge/SearchSpaceGroup.cpp index 7625c9d39..6cf37304d 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.cpp +++ b/image_analysis/scale_merge/SearchSpaceGroup.cpp @@ -1635,17 +1635,23 @@ SearchSpaceGroupResult SearchSpaceGroup( const double rho = std::min(1.0, cc_outside / (cc_own / n_own)); return 0.5 * (1.0 - std::sqrt((1.0 - rho) / (1.0 + rho))); }; + // The merge's <|L|>, measured once, on first use (NaN without L-test pairs). + std::optional mean_abs_l; + const auto merge_mean_abs_l = [&] { + if (!mean_abs_l) { + const auto t = AnalyzeTwinning(merged, nullptr, 20, nullptr, opt.lattice_centring); + mean_abs_l = t.l_test_pairs > 0 ? t.mean_abs_l : std::numeric_limits::quiet_NaN(); + } + return *mean_abs_l; + }; // Whether the merge reads twinned, for the twin-immune zone gate: its <|L|> in the band a partial // twin puts it in, from the perfect twin's 0.375 up to AnalyzeTwinning's bar of 0.44, the band the - // L-test gate below reads too. Measured once, on first use. - std::optional twinned; + // L-test gate below reads too. const auto merge_reads_twinned = [&] { - if (!twinned) { - const auto t = AnalyzeTwinning(merged, nullptr, 20, nullptr, opt.lattice_centring); - twinned = t.l_test_pairs > 0 && t.mean_abs_l >= 0.375 && t.mean_abs_l < 0.44; - } - return *twinned; + return merge_mean_abs_l() >= 0.375 && merge_mean_abs_l() < 0.44; }; + // Whether it reads untwinned: <|L|> at or above that same bar. + const auto merge_reads_untwinned = [&] { return merge_mean_abs_l() >= 0.44; }; // Whether the odd class of one principal axial row reads extinct - a 2(1) on that axis - judged // exactly as Stage B judges that zone: the odd reflections, de-modulated for a pseudo-translation, // in units of the even ones' mean E^2, through TrimmedZoneSum and ScrewZoneEvidence, with the same @@ -1905,6 +1911,17 @@ SearchSpaceGroupResult SearchSpaceGroup( && (miscalibrated || (std::isfinite(c.chi2) && std::isfinite(chi2_ref) && c.chi2 <= chi2_ref * opt.max_merge_chi2_rescue))) consistent = true; + // Both tests above divide the added operators' disagreement by the P1 merge's sigmas, and those + // carry the error model the P1 scaling fitted - which never ties symmetry mates together. So the + // better the scaling converges, the smaller the sigmas and the larger the same real systematic + // floor between symmetry mates reads: on a weak cubic crystal the b ratio of the true 23 read + // 1.438 against its 1.44 bound with the scaling loop stopped at 1e-3, and 1.453-1.460 once it + // ran on or was accelerated. What these two tests are for is the twin law that H cannot tell + // from real symmetry, so they are not asked of a merge whose L-test reads untwinned (measured: + // the twins they refuse read <|L|> 0.39-0.43, that cubic crystal 0.48). There the sigma-free + // gates below decide alone. + if (!consistent && merge_reads_untwinned()) + consistent = true; // The H test is a necessary condition for promotion where it can be computed: it is the only // statistic measured to separate genuine symmetry from a merohedral twin across data amounts. const bool h_refused = std::isfinite(h_ratio) && h_ratio > opt.max_operator_h_ratio; -- 2.54.0 From 7349832309eada78027844b069a6893b24524a5e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 19:35:54 +0200 Subject: [PATCH 05/53] Live rugnux timing: from the puller's receive time, and from the first image The live panel showed ~80 Hz on a 100 Hz collection. Its clock started when the receiver had applied and forwarded the START message, so the START -> first image gap counted as collection time; the images were stamped when a forward thread dequeued them, and the last image only after END and the thread joins; and N images were divided by a span holding N-1 intervals. - ImagePullerOutput carries `received`, a steady_clock time set by the TCP and ZeroMQ pullers as the message comes off the socket, before it is queued or deserialised (TestImagePuller stamps it on Put when unset). - RugnuxLiveStatus takes those times: START's for its origin, each image's for the first and last image (min/max, as forward threads report out of order). last_image_s is the last data image's puller time. - Zero of every reported time (elapsed_s, steps' started_s, probe_s, last_image_s, cut_s) is now the FIRST IMAGE, as the frontend already said; before the first image arrives "now" is 0 and earlier steps are negative. The START -> first image time is recorded on its own as start_to_first_image_s (snapshot, OpenAPI rugnux_live_status, regenerated C++ model and frontend client, OpenAPIConvert). - The frontend rate is (images_received - 1) / (last - first image); the panel headers show "START -> first image X.X s". - The broker logs "rugnux live: START -> first image ..., N images in T s from the first to the last (R Hz)"; "done X s after the last image" and last_image_to_written_s run from the puller time of the last image. The deadline timer is unchanged (it still starts once the forward threads have joined). No analysis result changes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/OpenAPIConvert.cpp | 2 + broker/gen/model/Rugnux_live_status.cpp | 31 ++++++++++- broker/gen/model/Rugnux_live_status.h | 19 +++++-- broker/gen/model/Rugnux_live_step.h | 2 +- broker/jfjoch_api.yaml | 17 ++++-- frontend/src/client/types.gen.ts | 17 ++++-- frontend/src/client/zod.gen.ts | 6 ++- frontend/src/components/RugnuxPage.tsx | 1 + frontend/src/components/RugnuxParts.tsx | 2 +- frontend/src/components/RugnuxStatus.tsx | 1 + frontend/src/components/RugnuxTimeline.tsx | 8 +-- image_puller/ImagePuller.h | 4 ++ image_puller/TCPImagePuller.cpp | 1 + image_puller/TestImagePuller.cpp | 6 ++- image_puller/ZMQImagePuller.cpp | 1 + receiver/JFJochReceiverRugnux.cpp | 35 +++++++------ receiver/JFJochReceiverRugnux.h | 3 -- receiver/RugnuxLiveStatus.cpp | 37 ++++++++++--- receiver/RugnuxLiveStatus.h | 24 ++++++--- tests/CMakeLists.txt | 1 + tests/RugnuxLiveStatusTest.cpp | 60 ++++++++++++++++++++++ tests/TCPImagePusherTest.cpp | 1 + tests/ZMQImagePusherTest.cpp | 2 + 23 files changed, 226 insertions(+), 55 deletions(-) create mode 100644 tests/RugnuxLiveStatusTest.cpp diff --git a/broker/OpenAPIConvert.cpp b/broker/OpenAPIConvert.cpp index 5f8a85376..b6427be90 100644 --- a/broker/OpenAPIConvert.cpp +++ b/broker/OpenAPIConvert.cpp @@ -1238,6 +1238,8 @@ org::openapitools::server::model::Rugnux_live_status Convert(const RugnuxLiveSna ret.setProbe(input.probe); if (input.probe_s) ret.setProbeS(input.probe_s.value()); + if (input.start_to_first_image_s) + ret.setStartToFirstImageS(input.start_to_first_image_s.value()); if (input.step_progress) ret.setStepProgress(input.step_progress.value()); diff --git a/broker/gen/model/Rugnux_live_status.cpp b/broker/gen/model/Rugnux_live_status.cpp index dd338ca1e..997da533c 100644 --- a/broker/gen/model/Rugnux_live_status.cpp +++ b/broker/gen/model/Rugnux_live_status.cpp @@ -25,6 +25,8 @@ Rugnux_live_status::Rugnux_live_status() m_Probe = ""; m_Probe_s = 0.0; m_Probe_sIsSet = false; + m_Start_to_first_image_s = 0.0; + m_Start_to_first_image_sIsSet = false; m_Message = ""; m_Images_expected = 0L; m_Images_received = 0L; @@ -63,7 +65,7 @@ bool Rugnux_live_status::validate(std::stringstream& msg, const std::string& pat bool success = true; const std::string _pathPrefix = pathPrefix.empty() ? "Rugnux_live_status" : pathPrefix; - + if (stepProgressIsSet()) { const float& value = m_Step_progress; @@ -163,6 +165,9 @@ bool Rugnux_live_status::operator==(const Rugnux_live_status& rhs) const ((!probeSIsSet() && !rhs.probeSIsSet()) || (probeSIsSet() && rhs.probeSIsSet() && getProbeS() == rhs.getProbeS())) && + + ((!startToFirstImageSIsSet() && !rhs.startToFirstImageSIsSet()) || (startToFirstImageSIsSet() && rhs.startToFirstImageSIsSet() && getStartToFirstImageS() == rhs.getStartToFirstImageS())) && + (getMessage() == rhs.getMessage()) && @@ -220,6 +225,8 @@ void to_json(nlohmann::json& j, const Rugnux_live_status& o) j["probe"] = o.m_Probe; if(o.probeSIsSet()) j["probe_s"] = o.m_Probe_s; + if(o.startToFirstImageSIsSet()) + j["start_to_first_image_s"] = o.m_Start_to_first_image_s; j["message"] = o.m_Message; j["images_expected"] = o.m_Images_expected; j["images_received"] = o.m_Images_received; @@ -251,6 +258,11 @@ void from_json(const nlohmann::json& j, Rugnux_live_status& o) j.at("probe_s").get_to(o.m_Probe_s); o.m_Probe_sIsSet = true; } + if(j.find("start_to_first_image_s") != j.end()) + { + j.at("start_to_first_image_s").get_to(o.m_Start_to_first_image_s); + o.m_Start_to_first_image_sIsSet = true; + } j.at("message").get_to(o.m_Message); j.at("images_expected").get_to(o.m_Images_expected); j.at("images_received").get_to(o.m_Images_received); @@ -321,6 +333,23 @@ void Rugnux_live_status::unsetProbe_s() { m_Probe_sIsSet = false; } +double Rugnux_live_status::getStartToFirstImageS() const +{ + return m_Start_to_first_image_s; +} +void Rugnux_live_status::setStartToFirstImageS(double const value) +{ + m_Start_to_first_image_s = value; + m_Start_to_first_image_sIsSet = true; +} +bool Rugnux_live_status::startToFirstImageSIsSet() const +{ + return m_Start_to_first_image_sIsSet; +} +void Rugnux_live_status::unsetStart_to_first_image_s() +{ + m_Start_to_first_image_sIsSet = false; +} std::string Rugnux_live_status::getMessage() const { return m_Message; diff --git a/broker/gen/model/Rugnux_live_status.h b/broker/gen/model/Rugnux_live_status.h index 3a80b2a8e..292004157 100644 --- a/broker/gen/model/Rugnux_live_status.h +++ b/broker/gen/model/Rugnux_live_status.h @@ -12,7 +12,7 @@ /* * Rugnux_live_status.h * - * Live rugnux processing of the last collection; kept until the next collection starts + * Live rugnux processing of the last collection; kept until the next collection starts. Times are taken when the image puller received a message, and are in seconds since the first image (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. */ #ifndef Rugnux_live_status_H_ @@ -29,7 +29,7 @@ namespace org::openapitools::server::model { /// -/// Live rugnux processing of the last collection; kept until the next collection starts +/// Live rugnux processing of the last collection; kept until the next collection starts. Times are taken when the image puller received a message, and are in seconds since the first image (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. /// class Rugnux_live_status { @@ -72,13 +72,20 @@ public: std::string getProbe() const; void setProbe(std::string const& value); /// - /// When the probe answered, since the collection started + /// When the probe answered, since the first image /// double getProbeS() const; void setProbeS(double const value); bool probeSIsSet() const; void unsetProbe_s(); /// + /// From the START message to the first image; set once the first image arrives + /// + double getStartToFirstImageS() const; + void setStartToFirstImageS(double const value); + bool startToFirstImageSIsSet() const; + void unsetStart_to_first_image_s(); + /// /// Why the run failed or was skipped /// std::string getMessage() const; @@ -111,7 +118,7 @@ public: std::vector getSteps() const; void setSteps(std::vector const& value); /// - /// When the last image arrived (and the cutoff timer started), since the collection started + /// When the last image arrived, since the first image; set once the collection is over /// double getLastImageS() const; void setLastImageS(double const value); @@ -125,7 +132,7 @@ public: bool cutIsSet() const; void unsetCut(); /// - /// When the analysis was cut, since the collection started + /// When the analysis was cut, since the first image /// double getCutS() const; void setCutS(double const value); @@ -168,6 +175,8 @@ protected: double m_Probe_s; bool m_Probe_sIsSet; + double m_Start_to_first_image_s; + bool m_Start_to_first_image_sIsSet; std::string m_Message; int64_t m_Images_expected; diff --git a/broker/gen/model/Rugnux_live_step.h b/broker/gen/model/Rugnux_live_step.h index ccc995e9a..e99a86926 100644 --- a/broker/gen/model/Rugnux_live_step.h +++ b/broker/gen/model/Rugnux_live_step.h @@ -65,7 +65,7 @@ public: std::string getTitle() const; void setTitle(std::string const& value); /// - /// Start of the step, since the collection started + /// Start of the step, since the first image (negative if before it) /// double getStartedS() const; void setStartedS(double const value); diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index 7cba4f220..1b03fd825 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -2747,7 +2747,7 @@ components: started_s: type: number format: double - description: Start of the step, since the collection started + description: Start of the step, since the first image (negative if before it) duration_s: type: number format: double @@ -2769,7 +2769,10 @@ components: type: string rugnux_live_status: type: object - description: Live rugnux processing of the last collection; kept until the next collection starts + description: | + Live rugnux processing of the last collection; kept until the next collection starts. + Times are taken when the image puller received a message, and are in seconds since the first image + (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. required: - state - message @@ -2792,7 +2795,11 @@ components: probe_s: type: number format: double - description: When the probe answered, since the collection started + description: When the probe answered, since the first image + start_to_first_image_s: + type: number + format: double + description: From the START message to the first image; set once the first image arrives message: type: string description: Why the run failed or was skipped @@ -2818,7 +2825,7 @@ components: last_image_s: type: number format: double - description: When the last image arrived (and the cutoff timer started), since the collection started + description: When the last image arrived, since the first image; set once the collection is over cut: type: string description: | @@ -2827,7 +2834,7 @@ components: cut_s: type: number format: double - description: When the analysis was cut, since the collection started + description: When the analysis was cut, since the first image last_image_to_written_s: type: number format: double diff --git a/frontend/src/client/types.gen.ts b/frontend/src/client/types.gen.ts index 1f4e0aa5d..d52d49350 100644 --- a/frontend/src/client/types.gen.ts +++ b/frontend/src/client/types.gen.ts @@ -1924,7 +1924,7 @@ export type rugnux_status = { export type rugnux_live_step = { title: string; /** - * Start of the step, since the collection started + * Start of the step, since the first image (negative if before it) */ started_s: number; /** @@ -1943,7 +1943,10 @@ export type rugnux_summary_row = { }; /** - * Live rugnux processing of the last collection; kept until the next collection starts + * Live rugnux processing of the last collection; kept until the next collection starts. + * Times are taken when the image puller received a message, and are in seconds since the first image + * (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. + * */ export type rugnux_live_status = { state: 'Idle' | 'Running' | 'Done' | 'Failed' | 'Cancelled' | 'Skipped'; @@ -1952,9 +1955,13 @@ export type rugnux_live_status = { */ probe: string; /** - * When the probe answered, since the collection started + * When the probe answered, since the first image */ probe_s?: number; + /** + * From the START message to the first image; set once the first image arrives + */ + start_to_first_image_s?: number; /** * Why the run failed or was skipped */ @@ -1968,7 +1975,7 @@ export type rugnux_live_status = { step_progress?: number; steps: Array; /** - * When the last image arrived (and the cutoff timer started), since the collection started + * When the last image arrived, since the first image; set once the collection is over */ last_image_s?: number; /** @@ -1978,7 +1985,7 @@ export type rugnux_live_status = { */ cut?: string; /** - * When the analysis was cut, since the collection started + * When the analysis was cut, since the first image */ cut_s?: number; /** diff --git a/frontend/src/client/zod.gen.ts b/frontend/src/client/zod.gen.ts index a47c44e67..31206d66b 100644 --- a/frontend/src/client/zod.gen.ts +++ b/frontend/src/client/zod.gen.ts @@ -824,7 +824,10 @@ export const zRugnuxSummaryRow = z.object({ }); /** - * Live rugnux processing of the last collection; kept until the next collection starts + * Live rugnux processing of the last collection; kept until the next collection starts. + * Times are taken when the image puller received a message, and are in seconds since the first image + * (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. + * */ export const zRugnuxLiveStatus = z.object({ state: z.enum([ @@ -837,6 +840,7 @@ export const zRugnuxLiveStatus = z.object({ ]), probe: z.string(), probe_s: z.number().optional(), + start_to_first_image_s: z.number().optional(), message: z.string(), images_expected: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }), images_received: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }), diff --git a/frontend/src/components/RugnuxPage.tsx b/frontend/src/components/RugnuxPage.tsx index 62aeaa923..81ce7ac64 100644 --- a/frontend/src/components/RugnuxPage.tsx +++ b/frontend/src/components/RugnuxPage.tsx @@ -42,6 +42,7 @@ function Header({s}: { s: rugnux_live_status }) { {s.images_received} / {s.images_expected} images {lastImage !== undefined && ` · collected in ${lastImage.toFixed(1)} s`} {` · ${s.elapsed_s.toFixed(1)} s since the first image`} + {s.start_to_first_image_s !== undefined && ` · START → first image ${s.start_to_first_image_s.toFixed(1)} s`} diff --git a/frontend/src/components/RugnuxParts.tsx b/frontend/src/components/RugnuxParts.tsx index 88875bc9f..cbd6502a0 100644 --- a/frontend/src/components/RugnuxParts.tsx +++ b/frontend/src/components/RugnuxParts.tsx @@ -16,7 +16,7 @@ export const verdictSeverity = (verdict: string) => { } }; -// When the last image arrived, in seconds since the collection started. Older brokers do not report +// When the last image arrived, in seconds since the first image. Older brokers do not report // it; there it follows from the latency (every result written `latency` after the last image). export const lastImageTime = (s: rugnux_live_status) => s.last_image_s ?? (s.last_image_to_written_s !== undefined ? s.elapsed_s - s.last_image_to_written_s : undefined); diff --git a/frontend/src/components/RugnuxStatus.tsx b/frontend/src/components/RugnuxStatus.tsx index a1dc88dd8..bac6497db 100644 --- a/frontend/src/components/RugnuxStatus.tsx +++ b/frontend/src/components/RugnuxStatus.tsx @@ -52,6 +52,7 @@ function RugnuxStatus() { {s.images_received} / {s.images_expected} images · {s.elapsed_s.toFixed(0)} s + {s.start_to_first_image_s !== undefined && ` · START → first image ${s.start_to_first_image_s.toFixed(1)} s`} {latency !== undefined && ( diff --git a/frontend/src/components/RugnuxTimeline.tsx b/frontend/src/components/RugnuxTimeline.tsx index 22b499d20..c9f0ca328 100644 --- a/frontend/src/components/RugnuxTimeline.tsx +++ b/frontend/src/components/RugnuxTimeline.tsx @@ -5,7 +5,8 @@ import {indigo, lime, lightBlue, orange, grey, deepOrange} from "@mui/material/c import {rugnux_live_status} from "../client"; import {lastImageTime} from "./RugnuxParts"; -// The overlap of collection and processing on one time axis (seconds since the first image): the +// The overlap of collection and processing on one time axis (seconds since the first image, as the +// broker's puller took it off the socket): the // detector lane fills while images arrive, the rugnux lane shows its steps where they ran, and the // gap between "last image" and "all written" is the headline latency. @@ -100,8 +101,9 @@ function RugnuxTimeline({s, deadline_s}: { s: rugnux_live_status, deadline_s: nu const lastImage = lastImageTime(s); const collecting = lastImage === undefined && s.state === 'Running'; const collectionEnd = lastImage ?? s.elapsed_s; - const rate = collectionEnd > 0 ? s.images_received / collectionEnd : 0; - const expectedEnd = rate > 0 ? s.images_expected / rate : 0; + // n images span n - 1 intervals from the first to the last + const rate = s.images_received > 1 && collectionEnd > 0 ? (s.images_received - 1) / collectionEnd : 0; + const expectedEnd = rate > 0 ? (s.images_expected - 1) / rate : 0; const done = s.state !== 'Running'; // The deadline matters only while the analysis runs; once it is over the axis is the run's own. diff --git a/image_puller/ImagePuller.h b/image_puller/ImagePuller.h index 2d1d433dc..b399fb63e 100644 --- a/image_puller/ImagePuller.h +++ b/image_puller/ImagePuller.h @@ -4,6 +4,7 @@ #pragma once #include +#include #include #include @@ -37,6 +38,9 @@ struct ImagePullerOutput { std::shared_ptr raw; std::shared_ptr tcp_msg; std::shared_ptr cbor; + // When the message came off the socket - before it was queued or deserialised, so a consumer that + // falls behind does not shift it. + std::chrono::steady_clock::time_point received{}; }; struct ImagePullerStatistics { diff --git a/image_puller/TCPImagePuller.cpp b/image_puller/TCPImagePuller.cpp index 122729127..2f27b2ec9 100644 --- a/image_puller/TCPImagePuller.cpp +++ b/image_puller/TCPImagePuller.cpp @@ -394,6 +394,7 @@ void TCPImagePuller::ReceiverThread() { continue; } } + out.received = std::chrono::steady_clock::now(); cbor_fifo.PutBlocking(out); } diff --git a/image_puller/TestImagePuller.cpp b/image_puller/TestImagePuller.cpp index b16e8b08e..7a207a560 100644 --- a/image_puller/TestImagePuller.cpp +++ b/image_puller/TestImagePuller.cpp @@ -6,7 +6,11 @@ TestImagePuller::TestImagePuller(size_t fifo_size) : ImagePuller(fifo_size) {} void TestImagePuller::Put(const ImagePullerOutput &output) { - outside_fifo.Put(output); + // Put stands for the socket: a message without a receive time gets it here, as a real puller would. + ImagePullerOutput out = output; + if (out.received == std::chrono::steady_clock::time_point{}) + out.received = std::chrono::steady_clock::now(); + outside_fifo.Put(out); } void TestImagePuller::Disconnect() { diff --git a/image_puller/ZMQImagePuller.cpp b/image_puller/ZMQImagePuller.cpp index 282c6fcfb..e606d4d4d 100644 --- a/image_puller/ZMQImagePuller.cpp +++ b/image_puller/ZMQImagePuller.cpp @@ -81,6 +81,7 @@ void ZMQImagePuller::PullerThread() { } } ImagePullerOutput ret; + ret.received = std::chrono::steady_clock::now(); if (ring.GetCapacity() == 0) { // No ring: the ZeroMQ message itself is the buffer, kept alive by the region. No copy. ret.raw = std::shared_ptr(new ReceiveBufferRegion{const_cast(msg->data()), msg->size()}, diff --git a/receiver/JFJochReceiverRugnux.cpp b/receiver/JFJochReceiverRugnux.cpp index af225b037..21e1e023f 100644 --- a/receiver/JFJochReceiverRugnux.cpp +++ b/receiver/JFJochReceiverRugnux.cpp @@ -33,13 +33,8 @@ namespace { return budget > taken ? budget - taken : 1; } - int64_t SteadyNowNs() { - return std::chrono::duration_cast( - std::chrono::steady_clock::now().time_since_epoch()).count(); - } - - double SecondsSince(int64_t ns) { - return static_cast(SteadyNowNs() - ns) / 1e9; + double SecondsSince(std::chrono::steady_clock::time_point t) { + return std::chrono::duration(std::chrono::steady_clock::now() - t).count(); } // A run reachable for Cut while it exists: registered in `slot` (under `m`, cancelled at once where the @@ -188,7 +183,7 @@ void JFJochReceiverRugnux::MeasurementThread() { // Rugnux reads the images back from the buffer, so it needs one slot per image: a slot taken // again would hand it a later image. Without that the collection is only forwarded. const auto slots = image_buffer.GetStatus().total_slots; - live_status.Start(experiment.GetImageNum()); + live_status.Start(experiment.GetImageNum(), msg->received); if (slots < experiment.GetImageNum()) { skipped_warning = fmt::format("no rugnux analysis: the image buffer holds {} images, the collection " "has {}", slots, experiment.GetImageNum()); @@ -214,9 +209,17 @@ void JFJochReceiverRugnux::MeasurementThread() { f.get(); reader.Ended(); images_done = true; - live_status.LastImage(); - const int64_t last_image = last_image_ns.load(); + // The latency of the demonstration runs from the last image in (as the puller took it off the socket) + // to every result in place. + const auto last_image = live_status.LastImage(); logger.Info("Receiving finished, {} images", images_collected.load()); + { + const auto live = live_status.Get(); + if (live.start_to_first_image_s && live.last_image_s && live.images_received > 1) + logger.Info("rugnux live: START -> first image {:.2f} s, {} images in {:.2f} s from the first to the last ({:.1f} Hz)", + live.start_to_first_image_s.value(), live.images_received, live.last_image_s.value(), + static_cast(live.images_received - 1) / live.last_image_s.value()); + } if (rugnux_done.valid()) { // The fixed cutoff: the analysis gets deadline_s after the last image, then gives what it has. @@ -224,7 +227,8 @@ void JFJochReceiverRugnux::MeasurementThread() { == std::future_status::timeout) Cut("deadline"); rugnux_done.wait(); - logger.Info("rugnux live: done {:.2f} s after the last image", SecondsSince(last_image)); + if (last_image) + logger.Info("rugnux live: done {:.2f} s after the last image", SecondsSince(last_image.value())); } if (probe_done.valid()) probe_done.wait(); @@ -233,7 +237,7 @@ void JFJochReceiverRugnux::MeasurementThread() { current_status.SetStatus(GetStatus()); SendEndMessage(); - const double end_sent_s = SecondsSince(last_image); + const double end_sent_s = last_image ? SecondsSince(last_image.value()) : 0.0; if (!image_buffer.CheckIfBufferReturned(SendFinalizeTimeout)) { logger.Error("Send commands not finalized in {} seconds", SendFinalizeTimeout.count()); @@ -242,8 +246,8 @@ void JFJochReceiverRugnux::MeasurementThread() { } if (push_images_to_writer) writer_error = image_pusher.Finalize(); - if (last_image > 0 && rugnux_done.valid()) { - const double written_s = SecondsSince(last_image); + if (last_image && rugnux_done.valid()) { + const double written_s = SecondsSince(last_image.value()); live_status.Latency(written_s); logger.Info("rugnux live: last image -> all written {:.2f} s (end message sent at {:.2f} s)", written_s, end_sent_s); @@ -438,8 +442,7 @@ void JFJochReceiverRugnux::ForwardThread(uint32_t id) { if (push_images_to_writer) image_pusher.SendCalibration(msg->cbor->calibration.value()); } else if (msg->cbor->data_message) { - last_image_ns = SteadyNowNs(); - live_status.ImageReceived(); + live_status.ImageReceived(msg->received); DataMessage data_msg = msg->cbor->data_message.value(); ++images_collected; diff --git a/receiver/JFJochReceiverRugnux.h b/receiver/JFJochReceiverRugnux.h index 74281555a..ee795c049 100644 --- a/receiver/JFJochReceiverRugnux.h +++ b/receiver/JFJochReceiverRugnux.h @@ -47,9 +47,6 @@ class JFJochReceiverRugnux : public JFJochReceiver { bool cut = false; std::atomic images_done{false}; // the forward threads have ended - // The latency of the demonstration: from the last image in to every result in place. - std::atomic last_image_ns{0}; - void MeasurementThread(); // Opens the reader on the collection and starts the rugnux and probe threads; a failure is the // analysis's alone (live status Failed), never the collection's. diff --git a/receiver/RugnuxLiveStatus.cpp b/receiver/RugnuxLiveStatus.cpp index ebebcd1da..884203831 100644 --- a/receiver/RugnuxLiveStatus.cpp +++ b/receiver/RugnuxLiveStatus.cpp @@ -3,8 +3,12 @@ #include "RugnuxLiveStatus.h" +double RugnuxLiveStatus::Since(std::chrono::steady_clock::time_point t) const { + return std::chrono::duration(t - start).count(); +} + double RugnuxLiveStatus::Now() const { - return std::chrono::duration(std::chrono::steady_clock::now() - start).count(); + return Since(std::chrono::steady_clock::now()); } void RugnuxLiveStatus::CloseStep() { @@ -12,17 +16,24 @@ void RugnuxLiveStatus::CloseStep() { s.steps.back().duration_s = Now() - s.steps.back().started_s; } -void RugnuxLiveStatus::Start(int64_t images_expected) { +void RugnuxLiveStatus::Start(int64_t images_expected, std::chrono::steady_clock::time_point start_received) { std::unique_lock ul(m); s = RugnuxLiveSnapshot{}; s.state = RugnuxLiveState::Running; s.images_expected = images_expected; - start = std::chrono::steady_clock::now(); + start = start_received; + last_image.reset(); } -void RugnuxLiveStatus::ImageReceived() { +void RugnuxLiveStatus::ImageReceived(std::chrono::steady_clock::time_point received) { std::unique_lock ul(m); ++s.images_received; + // Several forward threads report, so the images need not come here in the order they arrived. + const double t = Since(received); + if (!s.start_to_first_image_s || t < s.start_to_first_image_s.value()) + s.start_to_first_image_s = t; + if (!last_image || received > last_image.value()) + last_image = received; } void RugnuxLiveStatus::Step(const std::string &title) { @@ -69,9 +80,11 @@ void RugnuxLiveStatus::Latency(double last_image_to_written_s) { s.last_image_to_written_s = last_image_to_written_s; } -void RugnuxLiveStatus::LastImage() { +std::optional RugnuxLiveStatus::LastImage() { std::unique_lock ul(m); - s.last_image_s = Now(); + if (last_image) + s.last_image_s = Since(last_image.value()); + return last_image; } void RugnuxLiveStatus::Cut(const std::string &reason) { @@ -96,5 +109,17 @@ RugnuxLiveSnapshot RugnuxLiveStatus::Get() const { RugnuxLiveSnapshot ret = s; if (s.state == RugnuxLiveState::Running) ret.elapsed_s = Now(); + + // Held since START, reported since the first image - or, before it has come, since now. + const double zero = s.start_to_first_image_s.value_or(s.state == RugnuxLiveState::Running ? ret.elapsed_s : 0.0); + ret.elapsed_s -= zero; + for (auto &step: ret.steps) + step.started_s -= zero; + if (ret.probe_s) + *ret.probe_s -= zero; + if (ret.last_image_s) + *ret.last_image_s -= zero; + if (ret.cut_s) + *ret.cut_s -= zero; return ret; } diff --git a/receiver/RugnuxLiveStatus.h b/receiver/RugnuxLiveStatus.h index ce8026c2f..550a36356 100644 --- a/receiver/RugnuxLiveStatus.h +++ b/receiver/RugnuxLiveStatus.h @@ -14,11 +14,15 @@ // taken with what each found ("lattice found", "space group P 21 21 21"), the step it is on and how far // into its images, and - once it is over - its verdict and summary as the report states them, and the // latency from the last image to every result written. +// +// Times are those of the image puller - when a message came off the socket - and every time reported is +// in seconds since the first image arrived; until it has, since now. The START message comes earlier, by +// start_to_first_image_s. The collection's frame rate is (images_received - 1) / last_image_s. enum class RugnuxLiveState { Idle, Running, Done, Failed, Cancelled, Skipped }; struct RugnuxLiveStep { std::string title; - double started_s = 0.0; // since the collection started + double started_s = 0.0; // since the first image std::optional duration_s; // set once the next step starts or the run ends std::vector notes; }; @@ -34,10 +38,11 @@ struct RugnuxLiveSnapshot { // The lattice probe on the first 60 degrees (JFJochReceiverRugnux::ProbeThread): an early answer // to "is there a crystal", seconds into the collection; the run's own answer comes later. std::string probe; - std::optional probe_s; // when it answered, since the collection started - std::optional last_image_s; // when the last image arrived, since the collection started + std::optional probe_s; // when it answered + std::optional start_to_first_image_s; // from the START message to the first image + std::optional last_image_s; // when the last image arrived; set once the collection is over // The analysis was cut - "deadline" (the broker's deadline_s after the last image) or "cancel" - at - // cut_s since the collection started; what it had established went out, the steps after it were left out. + // cut_s; what it had established went out, the steps after it were left out. std::string cut; std::optional cut_s; std::optional last_image_to_written_s; @@ -50,12 +55,16 @@ struct RugnuxLiveSnapshot { class RugnuxLiveStatus { mutable std::mutex m; RugnuxLiveSnapshot s; + // Kept since the START message, shifted to the first image by Get(). std::chrono::steady_clock::time_point start; + std::optional last_image; + double Since(std::chrono::steady_clock::time_point t) const; double Now() const; void CloseStep(); public: - void Start(int64_t images_expected); - void ImageReceived(); + // start_received, received: when the START message and the image came off the socket (ImagePullerOutput) + void Start(int64_t images_expected, std::chrono::steady_clock::time_point start_received); + void ImageReceived(std::chrono::steady_clock::time_point received); void Step(const std::string &title); void Note(const std::string &text); void Progress(uint64_t done, uint64_t total); @@ -64,7 +73,8 @@ public: const std::vector> &summary, const std::vector &warnings); void Latency(double last_image_to_written_s); - void LastImage(); + // The collection is over: last_image_s is set, and the last image's time is returned (none if no image came). + std::optional LastImage(); void Cut(const std::string &reason); void Probe(const std::string &text); void Warning(const std::string &text); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 6f7b1bb10..48b37353b 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -100,6 +100,7 @@ ADD_EXECUTABLE(jfjoch_test ZMQMetadataSocketTest.cpp JFJochReaderTest.cpp LiveCollectionTest.cpp + RugnuxLiveStatusTest.cpp SweepLayoutTest.cpp ShadowFinderTest.cpp HotPixelFinderTest.cpp diff --git a/tests/RugnuxLiveStatusTest.cpp b/tests/RugnuxLiveStatusTest.cpp new file mode 100644 index 000000000..96ee22844 --- /dev/null +++ b/tests/RugnuxLiveStatusTest.cpp @@ -0,0 +1,60 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include + +#include "../receiver/RugnuxLiveStatus.h" +#include "../image_puller/TestImagePuller.h" + +using namespace std::chrono_literals; + +TEST_CASE("RugnuxLiveStatus_TimesFromFirstImage", "[RugnuxLive]") { + // 11 images at 100 Hz, the first 3 s after START, reported out of order as several forward threads do + const auto start = std::chrono::steady_clock::now() - 10s; + const auto first = start + 3s; + + RugnuxLiveStatus status; + status.Start(11, start); + for (int i: {1, 0, 2, 3, 4, 5, 6, 7, 8, 10, 9}) + status.ImageReceived(first + i * 10ms); + + auto s = status.Get(); + REQUIRE(s.images_received == 11); + REQUIRE(s.start_to_first_image_s.value() == Catch::Approx(3.0)); + CHECK(!s.last_image_s); + // "now" is 10 s after START, 7 s after the first image + CHECK(s.elapsed_s == Catch::Approx(7.0).margin(0.5)); + + REQUIRE(status.LastImage() == first + 100ms); + s = status.Get(); + REQUIRE(s.last_image_s.value() == Catch::Approx(0.1)); + CHECK((s.images_received - 1) / s.last_image_s.value() == Catch::Approx(100.0)); +} + +TEST_CASE("RugnuxLiveStatus_BeforeFirstImage", "[RugnuxLive]") { + RugnuxLiveStatus status; + status.Start(10, std::chrono::steady_clock::now() - 2s); + status.Step("waiting"); + + const auto s = status.Get(); + CHECK(!s.start_to_first_image_s); + CHECK(s.elapsed_s == 0.0); + REQUIRE(s.steps.size() == 1); + CHECK(s.steps[0].started_s <= 0.0); + CHECK(!status.LastImage()); +} + +TEST_CASE("TestImagePuller_SetsReceiveTime", "[RugnuxLive]") { + TestImagePuller puller; + const auto before = std::chrono::steady_clock::now(); + puller.Put(ImagePullerOutput{}); + const auto given = before - 5s; + puller.Put(ImagePullerOutput{.received = given}); + + auto out = puller.PollImage(); + REQUIRE(out); + CHECK(out->received >= before); + out = puller.PollImage(); + REQUIRE(out); + CHECK(out->received == given); +} diff --git a/tests/TCPImagePusherTest.cpp b/tests/TCPImagePusherTest.cpp index d19a67a80..caae26f18 100644 --- a/tests/TCPImagePusherTest.cpp +++ b/tests/TCPImagePusherTest.cpp @@ -104,6 +104,7 @@ TEST_CASE("TCPImageCommTest_2Writers_WithAck", "[TCP]") { REQUIRE(out.has_value()); REQUIRE(out->cbor != nullptr); REQUIRE(out->tcp_msg != nullptr); + REQUIRE(out->received != std::chrono::steady_clock::time_point{}); const auto &h = out->tcp_msg->header; diff --git a/tests/ZMQImagePusherTest.cpp b/tests/ZMQImagePusherTest.cpp index f6e541e52..6c34ebd18 100644 --- a/tests/ZMQImagePusherTest.cpp +++ b/tests/ZMQImagePusherTest.cpp @@ -47,6 +47,8 @@ void test_puller(ZMQImagePuller *puller, diff_content[writer_id]++; if (img->cbor->data_message->image.GetCompressionAlgorithm() != CompressionAlgorithm::NO_COMPRESSION) diff_content[writer_id]++; + if (img->received == std::chrono::steady_clock::time_point{}) + diff_content[writer_id]++; nimages[writer_id]++; } img = puller->PollImage(timeout); -- 2.54.0 From 7bb8bc48125420adfe1a6390ebe77615434e7533 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 14:21:08 +0200 Subject: [PATCH 06/53] RotationScaleMerge: the fulls loop keeps its cross-validated smoothness once within 10x of its tolerance (RESULT-CHANGING) The fulls loop re-chose its smoothing lambda by cross-validation every round, so every round was a different map and the loop had no fixed point: on 149 of the 245 fulls loops that ended unsettled in the rc176 full battery lambda jumped >= 2x over the last rounds, and one sweep cycled between 1.2e5 and 3.8e7. The choice is now kept from the round whose step falls below SCALING_ANDERSON_ONSET (10x the tolerance) - the round acceleration starts on, which models one map too. In the audit (env knob RUGNUX_FREEZE_LAMBDA, non-sticky) this took settled fulls loops 12/30 -> 21/30 on five sets. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- image_analysis/scale_merge/RotationScaleMerge.cpp | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 19581fc40..be3b9aa95 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -2011,6 +2011,13 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( // toward zero for as long as the loop runs, which is the answer, not a failure to settle. The // rounds are accelerated as RunScalingLoop's are, in log G with each frame at its count of fulls; // the frame below the floor takes the plain round's scale. + // + // The smoothness is chosen afresh each round only while the loop is still far out; once its step is + // within SCALING_ANDERSON_ONSET - the round acceleration starts on - it is kept for every round after. + // A choice made every round makes each round a different map, which has no fixed point to settle on: + // on 149 of the 245 fulls loops that ended unsettled over one battery the choice was still jumping + // 2x or more at the end, one sweep's cycling between two choices 300x apart for as long as the loop + // ran. The acceleration, too, models one map. ScalingLoopOutcome out; std::vector y(n_frames), J(n_frames), g_new(n_frames), ratio(n_frames); std::vector n_obs(n_frames); @@ -2020,6 +2027,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( has_fulls[f] = frame_obs_count[f] > 0 ? 1 : 0; } double best_step = INFINITY, lambda = 0.0; + bool lambda_kept = false; int rounds_since_best = 0; AndersonMixing anderson(SCALING_ANDERSON_DEPTH); std::vector log_g(n_frames), log_g_new(n_frames), weight(n_frames); @@ -2033,7 +2041,8 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( if (fitted) y[f] += g_fit[f] > 0.0 ? std::max(std::log(g_fit[f] / g[f]), -1.0) : -1.0; } - lambda = ChooseLogScaleSmoothing(y, J, std::max(1, rocking_event_frames_at_start), nthreads); + if (!lambda_kept) + lambda = ChooseLogScaleSmoothing(y, J, std::max(1, rocking_event_frames_at_start), nthreads); const std::vector x = SmoothLogScale(y, J, lambda); for (int f = 0; f < n_frames; ++f) g_new[f] = std::exp(x[f]); @@ -2062,6 +2071,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( logger->Debug("Per-frame scaling (fulls): iteration {}, rms |dlogG| {:.2e}, lambda {:.3g}, gauge {:.4f}, " "{} restarts{}", it + 1, out.step, lambda, g_ref, anderson.Restarts(), anderson.Stopped() ? ", plain" : ""); + if (out.step < SCALING_ANDERSON_ONSET) lambda_kept = true; bool stop = false; if (out.step < SCALING_TOLERANCE) { out.converged = true; -- 2.54.0 From dd75c6eb2b8a694a3f6b889f7029e6ed1111dfbf Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 14:21:22 +0200 Subject: [PATCH 07/53] RotationScaleMerge: the fulls' own deviation is handed back however the loop stopped (RESULT-CHANGING) ShrinkToRestrained ran only off a fulls loop that reported itself converged, so whether each frame kept what its own fulls say beyond the smooth curve depended on whether the last step happened to cross the tolerance: one sweep's ISa read 35.09 or 34.46 by how its loop stopped. The hand-back is a continuous decision already - each frame keeps the fraction its observations support, read off the lag-1 covariance of the sweep's deviations - so it is now applied after every fulls loop. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- .../scale_merge/RotationScaleMerge.cpp | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index be3b9aa95..f8e574d64 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -2098,25 +2098,24 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( if (stop) break; } // The curve is the part of the scale the frames share over a rocking curve and more; what a - // frame's own fulls say beyond it is handed back by how much of it is real (ShrinkToRestrained) - - // only off a loop that settled, since a loop still walking has measured nothing to hand back. - if (out.converged) { - fit(); - std::vector fitted(n_frames); - for (int f = 0; f < n_frames; ++f) - fitted[f] = frame_scaled_scratch[f] && frame_info_scratch[f] > 0.0 ? 1 : 0; - g_new = g; - ShrinkToRestrained("fulls", g, fitted, frame_obs_count, g_fit); - for (int f = 0; f < n_frames; ++f) - if (fitted[f] && std::isfinite(g_fit[f]) && g_fit[f] > 0.0) g_new[f] = g_fit[f]; - const double g_ref = TypicalFrameScale(g_new, n_obs); - for (int f = 0; f < n_frames; ++f) { - g_new[f] /= g_ref; - ratio[f] = g[f] / g_new[f]; - } - rescale(has_fulls, ratio); - g = g_new; + // frame's own fulls say beyond it is handed back by how much of it is real (ShrinkToRestrained), + // however the loop stopped: what is real is read off the frames themselves, and a hand-back that + // depended on whether the last step crossed the tolerance moved ISa by 0.6 on one sweep. + fit(); + std::vector fitted(n_frames); + for (int f = 0; f < n_frames; ++f) + fitted[f] = frame_scaled_scratch[f] && frame_info_scratch[f] > 0.0 ? 1 : 0; + g_new = g; + ShrinkToRestrained("fulls", g, fitted, frame_obs_count, g_fit); + for (int f = 0; f < n_frames; ++f) + if (fitted[f] && std::isfinite(g_fit[f]) && g_fit[f] > 0.0) g_new[f] = g_fit[f]; + const double g_ref = TypicalFrameScale(g_new, n_obs); + for (int f = 0; f < n_frames; ++f) { + g_new[f] /= g_ref; + ratio[f] = g[f] / g_new[f]; } + rescale(has_fulls, ratio); + g = g_new; frame_scaled_scratch = has_fulls; out.best_step = best_step; std::vector positive; -- 2.54.0 From b5d9488ceb31246ac51ebe6bf333a9f4901d4cd6 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 14:44:01 +0200 Subject: [PATCH 08/53] RotationScaleMerge: a scaling loop is stalled only when its steps stop falling, and ends on its best round (RESULT-CHANGING) The stall rule ('step not below 0.9 x its best for five rounds') called a steady slow fall a stall - 82 of the 245 loops that ended unsettled in the rc176 full battery were still falling at < 2% a round - and every unsettled loop returned its LAST state: a fulls loop that diverged from round 1 handed on the state after five rounds of walking. Now a loop is stalled when the median of the pairwise slopes of log(step) over its last six rounds (Theil-Sen) is not negative - a median, because a least-squares slope let one round's jump (the smoothness re-chosen five decades away) end a loop falling by a quarter a round - and however a loop stops (settled, stalled or at the cap) it ends on the plain round with the smallest step. On a settled loop that is the last round, so nothing changes there; on any other it is the state closest to a fixed point the loop reached. The partials' loop restores it through the round's own rescale (ratio fitted G / best G), so the corr bookkeeping of the CPU and GPU paths is unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- .../scale_merge/RotationScaleMerge.cpp | 79 +++++++++++++------ 1 file changed, 57 insertions(+), 22 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index f8e574d64..4ce0dea01 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -68,6 +68,10 @@ namespace { // The loops are accelerated only once their step is within this of the tolerance: Anderson mixing // models the map near its fixed point, and a loop still further out is as likely to be walking. constexpr double SCALING_ANDERSON_ONSET = 10.0 * SCALING_TOLERANCE; + // A scaling loop has stopped settling when the trend of log(step) over its last this many rounds is + // not falling (StepsStoppedFalling): a steady slow fall is a loop still contracting, a flat or + // rising trend is not. + constexpr int SCALING_STALL_WINDOW = 6; // The per-frame scale is smoothed over at least this many rocking curves (see Run). constexpr int SMOOTH_G_MIN_ROCKING_EVENTS = 6; @@ -304,6 +308,21 @@ namespace { std::nth_element(v.begin(), v.begin() + v.size() / 2, v.end()); return v[v.size() / 2]; } + + // Whether the steps of a scaling loop have stopped falling: the median of the pairwise slopes of + // log(step) over the last SCALING_STALL_WINDOW rounds (Theil-Sen) is not negative. A median, so that + // one round's jump - the smoothness re-chosen five decades away, on a loop that was falling by a + // quarter a round - does not end a loop that is still contracting. + bool StepsStoppedFalling(const std::vector &steps) { + const int n = SCALING_STALL_WINDOW; + if (static_cast(steps.size()) < n) return false; + const double *s = steps.data() + steps.size() - n; + std::vector slopes; + for (int i = 0; i < n; ++i) + for (int j = i + 1; j < n; ++j) + slopes.push_back(std::log(s[j] / s[i]) / (j - i)); + return median_of(slopes) >= 0.0; + } } RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment, @@ -1867,10 +1886,14 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( // // The loop stops when the relative scales stop moving - the rms of |log(G_new/G_old)| over the // frames, each weighted by its observation count and nothing else (a weight that vanishes with G - // is what let a collapsing sweep report itself settled) - or when they stop settling: a step that - // has not fallen below nine tenths of its smallest value for five rounds is a loop on a flat or - // rising valley, and more rounds of it can only walk further. Both leave the frames where they are - // and say so. The step is the residual of the round, F(x) - x at the state the round started from, + // is what let a collapsing sweep report itself settled) - or when they stop settling: steps that + // have not fallen over the last SCALING_STALL_WINDOW rounds, by the median trend of their logarithm, + // are a loop on a flat or rising valley, and more rounds of it can only walk further. A steady slow + // fall is not a stall (a rule of 'not below nine tenths of the best for five rounds' stopped a third + // of the unsettled loops of one battery while they were still falling at 2% a round). However the + // loop stops, it ends on the plain round with the smallest step - on a settled loop that is the + // last round, on any other the state closest to a fixed point it reached, not wherever its walk + // had got to. The step is the residual of the round, F(x) - x at the state the round started from, // so the test reads the same whether that state was a plain round's or an accelerated one. ScalingLoopOutcome out; std::vector scaled_any(n_frames, 0); @@ -1878,7 +1901,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( std::vector n_obs(n_frames, 0); std::vector ratio(n_frames, 1.0); double best_step = INFINITY; - int rounds_since_best = 0; + std::vector steps, g_best; AndersonMixing anderson(SCALING_ANDERSON_DEPTH); std::vector log_g_prev(n_frames), log_g(n_frames), weight(n_frames); for (int it = 0; it < scaling_iter; ++it) { @@ -1939,17 +1962,22 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( logger->Debug("Per-frame scaling ({}): iteration {}, rms |dlogG| {:.2e} over {} frames, " "gauge {:.4f}, {} restarts{}", what, it + 1, out.step, n, g_ref, anderson.Restarts(), anderson.Stopped() ? ", plain" : ""); + steps.push_back(out.step); + if (out.step < best_step) { + best_step = out.step; + g_best = g; + } if (out.step < SCALING_TOLERANCE) { out.converged = true; stop = true; - } else if (out.step < 0.9 * best_step) { - best_step = out.step; - rounds_since_best = 0; - } else if (++rounds_since_best >= 5) { + } else if (StepsStoppedFalling(steps)) { out.stalled = true; stop = true; } - if (!stop && out.step < SCALING_ANDERSON_ONSET) { + if (stop || it + 1 == scaling_iter) { + for (int f = 0; f < n_frames; ++f) + if (usable[f]) g[f] = g_best[f]; + } else if (out.step < SCALING_ANDERSON_ONSET) { // The next round starts from the accelerated scales, gauge-pinned again. const std::vector next = anderson.Next(log_g_prev, log_g, weight); for (int f = 0; f < n_frames; ++f) @@ -1977,14 +2005,16 @@ void RotationScaleMerge::LogScalingOutcome(const char *what, const ScalingLoopOu out.iterations, out.step); else if (out.stalled) logger->Warning("Per-frame scaling ({}): stopped after {} iterations - the scales stopped settling " - "(rms |dlogG| {:.1e}, not below {:.1e} for five rounds); more rounds would only walk " - "them, so everything read off this merge is read off an unsettled state", what, - out.iterations, out.step, out.best_step); + "(rms |dlogG| {:.1e}, not falling over the last {} rounds); they are left at the " + "round with the smallest step ({:.1e}), and everything read off this merge is read " + "off an unsettled state", what, out.iterations, out.step, SCALING_STALL_WINDOW, + out.best_step); else logger->Warning("Per-frame scaling ({}): NOT settled after the cap of {} iterations - the scales " - "were still moving by rms |dlogG| {:.1e} (tolerance {:.0e}); everything read off " - "this merge is read off an unsettled state", what, out.iterations, out.step, - SCALING_TOLERANCE); + "were still moving by rms |dlogG| {:.1e} (tolerance {:.0e}); they are left at the " + "round with the smallest step ({:.1e}), and everything read off this merge is read " + "off an unsettled state", what, out.iterations, out.step, SCALING_TOLERANCE, + out.best_step); } RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( @@ -2028,7 +2058,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( } double best_step = INFINITY, lambda = 0.0; bool lambda_kept = false; - int rounds_since_best = 0; + std::vector steps, g_best; AndersonMixing anderson(SCALING_ANDERSON_DEPTH); std::vector log_g(n_frames), log_g_new(n_frames), weight(n_frames); for (int it = 0; it < scaling_iter; ++it) { @@ -2072,18 +2102,23 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( "{} restarts{}", it + 1, out.step, lambda, g_ref, anderson.Restarts(), anderson.Stopped() ? ", plain" : ""); if (out.step < SCALING_ANDERSON_ONSET) lambda_kept = true; + // Stopped and ended on the best round as RunScalingLoop's loop is. + steps.push_back(out.step); + if (out.step < best_step) { + best_step = out.step; + g_best = g_new; + } bool stop = false; if (out.step < SCALING_TOLERANCE) { out.converged = true; stop = true; - } else if (out.step < 0.9 * best_step) { - best_step = out.step; - rounds_since_best = 0; - } else if (++rounds_since_best >= 5) { + } else if (StepsStoppedFalling(steps)) { out.stalled = true; stop = true; } - if (!stop && out.step < SCALING_ANDERSON_ONSET) { + if (stop || it + 1 == scaling_iter) { + g_new = g_best; + } else if (out.step < SCALING_ANDERSON_ONSET) { // The next round starts from the accelerated scales, gauge-pinned again. const std::vector next = anderson.Next(log_g, log_g_new, weight); for (int f = 0; f < n_frames; ++f) -- 2.54.0 From 638794158c0f668a6c85a7902a6000fabc90c5ef Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 20:45:43 +0200 Subject: [PATCH 09/53] rugnux: the banner's Compute line names the card count once ("4x NVIDIA L4", not "4 x 4x NVIDIA L4") get_gpu_description already collapses identical cards into "x "; the banner prefixed the count a second time. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- rugnux/rugnux_cli.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 03522cc7d..0f28f5403 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -1691,8 +1691,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { // job - is worth knowing about while there is still time to stop and fix it. console->Field("Compute", no_gpu ? fmt::format("CPU only (--no-gpu), {} threads", nthreads) : provenance.gpu_count > 0 - ? fmt::format("{}{}, {} threads", provenance.gpu_count > 1 ? fmt::format("{} x ", provenance.gpu_count) : "", - provenance.gpu_description, nthreads) + ? fmt::format("{}, {} threads", provenance.gpu_description, nthreads) : !metal_device.empty() ? fmt::format("{} CPU threads, Bragg integration on {} (Metal)", nthreads, metal_device) : fmt::format("no GPU visible - {} CPU threads, which is much slower", nthreads)); -- 2.54.0 From 831b05aebd2b75d785e18b1814c502ea6ae669f1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 20:47:51 +0200 Subject: [PATCH 10/53] Live rugnux panel: the timeline spans the collection's set-up length from the first poll The broker's rugnux live status carries expected_collection_s = images x image time (DiffractionExperiment::GetImageTime, summation included), set at START. The frontend's timeline uses it as the expected end until a frame rate is measured, and as the axis floor while collecting, so the axis no longer starts at 10 s and rescales as images arrive. An older broker without the field keeps the 10 s floor. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/OpenAPIConvert.cpp | 2 ++ broker/gen/model/Rugnux_live_status.cpp | 31 +++++++++++++++++++++- broker/gen/model/Rugnux_live_status.h | 9 +++++++ broker/jfjoch_api.yaml | 4 +++ frontend/src/client/types.gen.ts | 4 +++ frontend/src/client/zod.gen.ts | 1 + frontend/src/components/RugnuxTimeline.tsx | 8 ++++-- receiver/JFJochReceiverRugnux.cpp | 7 ++++- receiver/RugnuxLiveStatus.cpp | 4 ++- receiver/RugnuxLiveStatus.h | 7 +++-- tests/RugnuxLiveStatusTest.cpp | 6 +++-- 11 files changed, 74 insertions(+), 9 deletions(-) diff --git a/broker/OpenAPIConvert.cpp b/broker/OpenAPIConvert.cpp index b6427be90..c97040847 100644 --- a/broker/OpenAPIConvert.cpp +++ b/broker/OpenAPIConvert.cpp @@ -1233,6 +1233,8 @@ org::openapitools::server::model::Rugnux_live_status Convert(const RugnuxLiveSna } ret.setMessage(input.message); ret.setImagesExpected(input.images_expected); + if (input.expected_collection_s) + ret.setExpectedCollectionS(input.expected_collection_s.value()); ret.setImagesReceived(input.images_received); ret.setElapsedS(input.elapsed_s); ret.setProbe(input.probe); diff --git a/broker/gen/model/Rugnux_live_status.cpp b/broker/gen/model/Rugnux_live_status.cpp index 997da533c..21b3c0b01 100644 --- a/broker/gen/model/Rugnux_live_status.cpp +++ b/broker/gen/model/Rugnux_live_status.cpp @@ -29,6 +29,8 @@ Rugnux_live_status::Rugnux_live_status() m_Start_to_first_image_sIsSet = false; m_Message = ""; m_Images_expected = 0L; + m_Expected_collection_s = 0.0; + m_Expected_collection_sIsSet = false; m_Images_received = 0L; m_Elapsed_s = 0.0; m_Step_progress = 0.0f; @@ -65,7 +67,7 @@ bool Rugnux_live_status::validate(std::stringstream& msg, const std::string& pat bool success = true; const std::string _pathPrefix = pathPrefix.empty() ? "Rugnux_live_status" : pathPrefix; - + if (stepProgressIsSet()) { const float& value = m_Step_progress; @@ -174,6 +176,9 @@ bool Rugnux_live_status::operator==(const Rugnux_live_status& rhs) const (getImagesExpected() == rhs.getImagesExpected()) && + + ((!expectedCollectionSIsSet() && !rhs.expectedCollectionSIsSet()) || (expectedCollectionSIsSet() && rhs.expectedCollectionSIsSet() && getExpectedCollectionS() == rhs.getExpectedCollectionS())) && + (getImagesReceived() == rhs.getImagesReceived()) && @@ -229,6 +234,8 @@ void to_json(nlohmann::json& j, const Rugnux_live_status& o) j["start_to_first_image_s"] = o.m_Start_to_first_image_s; j["message"] = o.m_Message; j["images_expected"] = o.m_Images_expected; + if(o.expectedCollectionSIsSet()) + j["expected_collection_s"] = o.m_Expected_collection_s; j["images_received"] = o.m_Images_received; j["elapsed_s"] = o.m_Elapsed_s; if(o.stepProgressIsSet()) @@ -265,6 +272,11 @@ void from_json(const nlohmann::json& j, Rugnux_live_status& o) } j.at("message").get_to(o.m_Message); j.at("images_expected").get_to(o.m_Images_expected); + if(j.find("expected_collection_s") != j.end()) + { + j.at("expected_collection_s").get_to(o.m_Expected_collection_s); + o.m_Expected_collection_sIsSet = true; + } j.at("images_received").get_to(o.m_Images_received); j.at("elapsed_s").get_to(o.m_Elapsed_s); if(j.find("step_progress") != j.end()) @@ -366,6 +378,23 @@ void Rugnux_live_status::setImagesExpected(int64_t const value) { m_Images_expected = value; } +double Rugnux_live_status::getExpectedCollectionS() const +{ + return m_Expected_collection_s; +} +void Rugnux_live_status::setExpectedCollectionS(double const value) +{ + m_Expected_collection_s = value; + m_Expected_collection_sIsSet = true; +} +bool Rugnux_live_status::expectedCollectionSIsSet() const +{ + return m_Expected_collection_sIsSet; +} +void Rugnux_live_status::unsetExpected_collection_s() +{ + m_Expected_collection_sIsSet = false; +} int64_t Rugnux_live_status::getImagesReceived() const { return m_Images_received; diff --git a/broker/gen/model/Rugnux_live_status.h b/broker/gen/model/Rugnux_live_status.h index 292004157..9107a5682 100644 --- a/broker/gen/model/Rugnux_live_status.h +++ b/broker/gen/model/Rugnux_live_status.h @@ -96,6 +96,13 @@ public: int64_t getImagesExpected() const; void setImagesExpected(int64_t const value); /// + /// How long the collection is set up to take - images_expected times the image time; absent if not known + /// + double getExpectedCollectionS() const; + void setExpectedCollectionS(double const value); + bool expectedCollectionSIsSet() const; + void unsetExpected_collection_s(); + /// /// /// int64_t getImagesReceived() const; @@ -181,6 +188,8 @@ protected: int64_t m_Images_expected; + double m_Expected_collection_s; + bool m_Expected_collection_sIsSet; int64_t m_Images_received; double m_Elapsed_s; diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index 1b03fd825..d142b8ede 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -2806,6 +2806,10 @@ components: images_expected: type: integer format: int64 + expected_collection_s: + type: number + format: double + description: How long the collection is set up to take - images_expected times the image time; absent if not known images_received: type: integer format: int64 diff --git a/frontend/src/client/types.gen.ts b/frontend/src/client/types.gen.ts index d52d49350..edc0b9d43 100644 --- a/frontend/src/client/types.gen.ts +++ b/frontend/src/client/types.gen.ts @@ -1967,6 +1967,10 @@ export type rugnux_live_status = { */ message: string; images_expected: number; + /** + * How long the collection is set up to take - images_expected times the image time; absent if not known + */ + expected_collection_s?: number; images_received: number; elapsed_s: number; /** diff --git a/frontend/src/client/zod.gen.ts b/frontend/src/client/zod.gen.ts index 31206d66b..505fe6a74 100644 --- a/frontend/src/client/zod.gen.ts +++ b/frontend/src/client/zod.gen.ts @@ -843,6 +843,7 @@ export const zRugnuxLiveStatus = z.object({ start_to_first_image_s: z.number().optional(), message: z.string(), images_expected: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }), + expected_collection_s: z.number().optional(), images_received: z.coerce.bigint().min(BigInt('-9223372036854775808'), { error: 'Invalid value: Expected int64 to be >= -9223372036854775808' }).max(BigInt('9223372036854775807'), { error: 'Invalid value: Expected int64 to be <= 9223372036854775807' }), elapsed_s: z.number(), step_progress: z.number().gte(0).lte(1).optional(), diff --git a/frontend/src/components/RugnuxTimeline.tsx b/frontend/src/components/RugnuxTimeline.tsx index c9f0ca328..fb98ea398 100644 --- a/frontend/src/components/RugnuxTimeline.tsx +++ b/frontend/src/components/RugnuxTimeline.tsx @@ -103,12 +103,16 @@ function RugnuxTimeline({s, deadline_s}: { s: rugnux_live_status, deadline_s: nu const collectionEnd = lastImage ?? s.elapsed_s; // n images span n - 1 intervals from the first to the last const rate = s.images_received > 1 && collectionEnd > 0 ? (s.images_received - 1) / collectionEnd : 0; - const expectedEnd = rate > 0 ? (s.images_expected - 1) / rate : 0; + // Until a rate is measured, the collection's set-up length stands in for it + const expectedEnd = rate > 0 ? (s.images_expected - 1) / rate : (s.expected_collection_s ?? 0); + // While collecting, the axis spans at least the set-up length, so it does not rescale as images + // arrive; an older broker does not send it, and then 10 s is the floor as before. + const floor = s.expected_collection_s === undefined ? 10 : collecting ? s.expected_collection_s : 0; const done = s.state !== 'Running'; // The deadline matters only while the analysis runs; once it is over the axis is the run's own. const deadline = !done && lastImage !== undefined && deadline_s > 0 ? lastImage + deadline_s : undefined; - const span = Math.max(s.elapsed_s, expectedEnd, deadline ?? 0, 10) * 1.06; + const span = Math.max(s.elapsed_s, expectedEnd, deadline ?? 0, floor, 1) * 1.06; const pct = (t: number) => 100 * Math.min(Math.max(t, 0), span) / span; const tick = niceStep(span); const ticks = Array.from({length: Math.floor(span / tick) + 1}, (_, i) => i * tick); diff --git a/receiver/JFJochReceiverRugnux.cpp b/receiver/JFJochReceiverRugnux.cpp index 21e1e023f..45f2e1767 100644 --- a/receiver/JFJochReceiverRugnux.cpp +++ b/receiver/JFJochReceiverRugnux.cpp @@ -183,7 +183,12 @@ void JFJochReceiverRugnux::MeasurementThread() { // Rugnux reads the images back from the buffer, so it needs one slot per image: a slot taken // again would hand it a later image. Without that the collection is only forwarded. const auto slots = image_buffer.GetStatus().total_slots; - live_status.Start(experiment.GetImageNum(), msg->received); + // The image time as the detector produces images, summation included + const double image_time_s = std::chrono::duration(experiment.GetImageTime()).count(); + std::optional expected_collection_s; + if (image_time_s > 0) + expected_collection_s = static_cast(experiment.GetImageNum()) * image_time_s; + live_status.Start(experiment.GetImageNum(), expected_collection_s, msg->received); if (slots < experiment.GetImageNum()) { skipped_warning = fmt::format("no rugnux analysis: the image buffer holds {} images, the collection " "has {}", slots, experiment.GetImageNum()); diff --git a/receiver/RugnuxLiveStatus.cpp b/receiver/RugnuxLiveStatus.cpp index 884203831..b209ec9c6 100644 --- a/receiver/RugnuxLiveStatus.cpp +++ b/receiver/RugnuxLiveStatus.cpp @@ -16,11 +16,13 @@ void RugnuxLiveStatus::CloseStep() { s.steps.back().duration_s = Now() - s.steps.back().started_s; } -void RugnuxLiveStatus::Start(int64_t images_expected, std::chrono::steady_clock::time_point start_received) { +void RugnuxLiveStatus::Start(int64_t images_expected, std::optional expected_collection_s, + std::chrono::steady_clock::time_point start_received) { std::unique_lock ul(m); s = RugnuxLiveSnapshot{}; s.state = RugnuxLiveState::Running; s.images_expected = images_expected; + s.expected_collection_s = expected_collection_s; start = start_received; last_image.reset(); } diff --git a/receiver/RugnuxLiveStatus.h b/receiver/RugnuxLiveStatus.h index 550a36356..73c03f0e7 100644 --- a/receiver/RugnuxLiveStatus.h +++ b/receiver/RugnuxLiveStatus.h @@ -31,6 +31,7 @@ struct RugnuxLiveSnapshot { RugnuxLiveState state = RugnuxLiveState::Idle; std::string message; // why it failed or was skipped int64_t images_expected = 0; + std::optional expected_collection_s; // images_expected x the image time, as set up int64_t images_received = 0; double elapsed_s = 0.0; std::optional step_progress; // fraction of the current pass's images @@ -62,8 +63,10 @@ class RugnuxLiveStatus { double Now() const; void CloseStep(); public: - // start_received, received: when the START message and the image came off the socket (ImagePullerOutput) - void Start(int64_t images_expected, std::chrono::steady_clock::time_point start_received); + // start_received, received: when the START message and the image came off the socket (ImagePullerOutput); + // expected_collection_s: how long the collection is set up to take, none if not known + void Start(int64_t images_expected, std::optional expected_collection_s, + std::chrono::steady_clock::time_point start_received); void ImageReceived(std::chrono::steady_clock::time_point received); void Step(const std::string &title); void Note(const std::string &text); diff --git a/tests/RugnuxLiveStatusTest.cpp b/tests/RugnuxLiveStatusTest.cpp index 96ee22844..2c3f25d80 100644 --- a/tests/RugnuxLiveStatusTest.cpp +++ b/tests/RugnuxLiveStatusTest.cpp @@ -14,11 +14,12 @@ TEST_CASE("RugnuxLiveStatus_TimesFromFirstImage", "[RugnuxLive]") { const auto first = start + 3s; RugnuxLiveStatus status; - status.Start(11, start); + status.Start(11, 0.11, start); for (int i: {1, 0, 2, 3, 4, 5, 6, 7, 8, 10, 9}) status.ImageReceived(first + i * 10ms); auto s = status.Get(); + CHECK(s.expected_collection_s.value() == Catch::Approx(0.11)); REQUIRE(s.images_received == 11); REQUIRE(s.start_to_first_image_s.value() == Catch::Approx(3.0)); CHECK(!s.last_image_s); @@ -33,11 +34,12 @@ TEST_CASE("RugnuxLiveStatus_TimesFromFirstImage", "[RugnuxLive]") { TEST_CASE("RugnuxLiveStatus_BeforeFirstImage", "[RugnuxLive]") { RugnuxLiveStatus status; - status.Start(10, std::chrono::steady_clock::now() - 2s); + status.Start(10, std::nullopt, std::chrono::steady_clock::now() - 2s); status.Step("waiting"); const auto s = status.Get(); CHECK(!s.start_to_first_image_s); + CHECK(!s.expected_collection_s); CHECK(s.elapsed_s == 0.0); REQUIRE(s.steps.size() == 1); CHECK(s.steps[0].started_s <= 0.0); -- 2.54.0 From aa05882c46fc30c3a528278404f4baea208ce36f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 21:20:15 +0200 Subject: [PATCH 11/53] Battery: in-house set cytc_x10sa_mr - cytochrome C from the production server's live test EIGER2 16M, 1800 images of 0.2 deg, the same P 32 2 1 three-copy form as the other cytochrome C sets (c 86.5). XDS reference made with XDS + the Durin plugin (FRIEDEL'S_LAW=FALSE, CORRECT in 152 like the siblings, INCLUDE_RESOLUTION_RANGE 50-1.80 A where CC1/2 has fallen to 5%): ISa 22.1, CC1/2 99.9, refs rule cc_half_0.30 -> d_min 1.98 A. Same tags, hand override and rationale as cytc_x10sa; not in the ci tier. Reference model (beside the data, outside the repository): cytc_x10sa's model re-refined with phenix.refine (rigid body, sites, ADP, TLS, waters) against this set's XDS data with rugnux's free flags; XDS indexed it in the other setting of the 321 ambiguity, so the data were reindexed -h,-k,l. R-work 0.269, R-free 0.319. battery --only cytc_x10sa_mr (rc178 build): pass, merged in P 32 2 1 via the model, d_min 1.72 A, placed-model R-free 0.337. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- tools/battery/README.md | 2 +- tools/battery/inhouse.json | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/battery/README.md b/tools/battery/README.md index 8efe76e70..6054d3d35 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -138,7 +138,7 @@ the reference: | `thau*` (P4₁2₁2) | 5A47 | thaumatin from *T. daniellii*, 1.2 Å, a 57.9 c 150.6 (6G89, 2.4 Å: R-free +0.06) | | `insu_I*` (I2₁3) | 2BN3 | bovine insulin, cubic, 1.4 Å, 100 K, a 77.9 (9INS, pig, a 78.9, no R-free deposited: R-free +0.07) | | `insu_H*` (H3) | 4E7T | bovine T6 2Zn insulin, 1.4 Å, a 81.0 c 33.5 (1MSO, human: R-free +0.08 - our insulin is bovine) | - | `cytc*` (P3₂21) | `reference_model.cif` beside each set's data | bovine heart cytochrome c in a form with no deposited entry: P3₂21, three molecules in the asymmetric unit (two related by a pseudo-translation 0.30 0.70 0.33), c 86.0-88.6 between crystals. Our own model, placed by molecular replacement with 6FF5 and refined against each set's XDS data with rugnux's free flags - one per set, because the 3% spread in c costs a single model ~5 points of R-free in the other crystals. The data cannot tell the hand: the model's settles it. `rfree_deposited` is the R-free of that refinement | + | `cytc*` (P3₂21) | `reference_model.cif` beside each set's data | bovine heart cytochrome c in a form with no deposited entry: P3₂21, three molecules in the asymmetric unit (two related by a pseudo-translation 0.30 0.70 0.33), c 86.0-88.6 between crystals. Our own model, placed by molecular replacement with 6FF5 and refined against each set's XDS data with rugnux's free flags - one per set, because the 3% spread in c costs a single model ~5 points of R-free in the other crystals. The data cannot tell the hand: the model's settles it. `rfree_deposited` is the R-free of that refinement. `cytc_x10sa_mr` (c 86.5) started from `cytc_x10sa`'s model instead of a new molecular replacement; XDS indexed it in the other setting of the 321 ambiguity, so its data were reindexed -h,-k,l for the refinement | | `myob*` (P2₁) | 4DC8 | horse heart myoglobin (b 28.8 Å; sperm whale's P2₁ form has b 30.8), met form with sulfate, 1.5 Å, 100 K, a 35.4 b 28.8 c 63.1 β 106.1 - our setting; the unexposed control of its deposition, so no ligand or modified heme (5ZZE, same form, R-free 0.235 vs 0.233) | - **`--report-resolution`** (XDS arms): a second statistics table from the same merge, binned over diff --git a/tools/battery/inhouse.json b/tools/battery/inhouse.json index dcc8bc1d7..726d3061e 100644 --- a/tools/battery/inhouse.json +++ b/tools/battery/inhouse.json @@ -30,6 +30,7 @@ {"id": "cytc_x10sa", "input": "cytc_x10sa/cytC_10_002_master.h5", "wavelength": 0.99989, "ref": {"sgno": 152, "cell": [83.659, 83.659, 86.011, 90.0, 90.0, 120.0], "anomalous": true, "isa": 31.75, "completeness": 99.8, "r_meas": 0.232, "cc_half": 0.999, "multiplicity": 10.65, "dmin_low": 6.05, "r_meas_low": 0.036, "cc_half_high": 0.046, "dmin": 2.267, "dmin_rule": "cc_half_0.30", "dmin_xds": 2.039, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"], "model": "reference_model.cif", "tiers": {"ci": "trigonal P3121 with tNCS; the hand comes from the reference model"}, "ref_override": {"sg": "P 32 2 1", "sgno": 154}, "ref_override_why": "The reference model (P 32 2 1, three copies) fixes the hand that XDS left arbitrary: the 2-fold of P 32 2 1 holds to measurement noise in every shell and a P 32 six-copy model, twinned or not, fits no better on the same data (in-house analysis 2026-10-13)"}, {"id": "cytc_x06da_1", "input": "cytc_x06da_1/cytC_2_0770b5_master.h5", "wavelength": 0.95375, "ref": {"sgno": 152, "cell": [83.836, 83.836, 86.733, 90.0, 90.0, 120.0], "anomalous": true, "isa": 24.49, "completeness": 99.9, "r_meas": 0.084, "cc_half": 0.999, "multiplicity": 7.67, "dmin_low": 5.58, "r_meas_low": 0.033, "cc_half_high": 0.366, "dmin": 1.875, "dmin_rule": "xds_range", "dmin_xds": 1.875, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"], "model": "reference_model.cif", "ref_override": {"sg": "P 32 2 1", "sgno": 154}, "ref_override_why": "The reference model (P 32 2 1, three copies) fixes the hand that XDS left arbitrary: the 2-fold of P 32 2 1 holds to measurement noise in every shell and a P 32 six-copy model, twinned or not, fits no better on the same data (in-house analysis 2026-10-13)"}, {"id": "cytc_x06da_2", "input": "cytc_x06da_2/cytC_3_874285_master.h5", "wavelength": 0.95375, "ref": {"sgno": 152, "cell": [83.719, 83.719, 88.617, 90.0, 90.0, 120.0], "anomalous": true, "isa": 26.98, "completeness": 99.9, "r_meas": 0.097, "cc_half": 0.999, "multiplicity": 7.82, "dmin_low": 5.03, "r_meas_low": 0.032, "cc_half_high": 0.236, "dmin": 1.69, "dmin_rule": "xds_range", "dmin_xds": 1.69, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"], "model": "reference_model.cif", "ref_override": {"sg": "P 32 2 1", "sgno": 154}, "ref_override_why": "The reference model (P 32 2 1, three copies) fixes the hand that XDS left arbitrary: the 2-fold of P 32 2 1 holds to measurement noise in every shell and a P 32 six-copy model, twinned or not, fits no better on the same data (in-house analysis 2026-10-13)"}, + {"id": "cytc_x10sa_mr", "input": "cytc_x10sa_mr/CytoC4_2_001_master.h5", "wavelength": 0.99988, "ref": {"sgno": 152, "cell": [83.864, 83.864, 86.511, 90.0, 90.0, 120.0], "anomalous": true, "isa": 22.06, "completeness": 99.9, "r_meas": 0.139, "cc_half": 0.999, "multiplicity": 10.48, "dmin_low": 5.36, "r_meas_low": 0.037, "cc_half_high": 0.053, "dmin": 1.977, "dmin_rule": "cc_half_0.30", "dmin_xds": 1.8, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"], "model": "reference_model.cif", "ref_override": {"sg": "P 32 2 1", "sgno": 154}, "ref_override_why": "The reference model (P 32 2 1, three copies) fixes the hand that XDS left arbitrary: the 2-fold of P 32 2 1 holds to measurement noise in every shell and a P 32 six-copy model, twinned or not, fits no better on the same data (in-house analysis 2026-10-13)"}, {"id": "myob_x10sa", "input": "myob_x10sa/MyoB_13_110ee0_master.h5", "wavelength": 0.99988, "ref": {"sgno": 4, "cell": [35.229, 28.461, 63.076, 90.0, 106.504, 90.0], "anomalous": true, "isa": 5.21, "completeness": 97.7, "r_meas": 0.247, "cc_half": 0.965, "multiplicity": 3.29, "dmin_low": 5.16, "r_meas_low": 0.186, "cc_half_high": 0.359, "dmin": 1.742, "dmin_rule": "xds_range", "dmin_xds": 1.742, "dmax": 50.0}, "tags": ["h5", "myoglobin"], "model": "4dc8", "tiers": {"ci": "monoclinic P21, Eiger"}}, {"id": "myob_x06da", "input": "myob_x06da/MyoB_9_446b70_master.h5", "wavelength": 0.95365, "ref": {"sgno": 3, "cell": [35.182, 28.544, 62.83, 90.0, 106.138, 90.0], "anomalous": true, "isa": 7.59, "completeness": 99.4, "r_meas": 0.232, "cc_half": 0.987, "multiplicity": 3.47, "dmin_low": 4.23, "r_meas_low": 0.105, "cc_half_high": 0.267, "dmin": 1.422, "dmin_rule": "xds_range", "dmin_xds": 1.422, "dmax": 50.0}, "tags": ["h5", "myoglobin"], "tiers": {"smoke": "monoclinic myoglobin, screw not arbitrable by XDS"}, "ref_override": {"sg": "P 1 21 1", "sgno": 4}, "ref_override_why": "myoglobin, P 1 21 1. XDS does not judge screw axes (it reports P 2)", "model": "4dc8"}, {"id": "myob_x06da_powder_1", "input": "myob_x06da_powder_1/MyoB2-1_75979a_master.h5", "wavelength": 0.95373, "ref": {"sgno": 1, "cell": [28.705, 34.796, 64.373, 74.314, 89.879, 90.057], "anomalous": false, "isa": 5.54, "completeness": 74.2, "r_meas": 0.493, "cc_half": 0.966, "multiplicity": 2.87, "dmin_low": 4.44, "r_meas_low": 0.125, "cc_half_high": 0.953, "dmin": 1.495, "dmin_rule": "xds_range", "dmin_xds": 1.495, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"], "tiers": {"smoke": "known hard: triclinic twin"}, "ref_override": {"sg": "P 1 21 1", "sgno": 4, "cell": [35.14, 28.88, 64.64, 90.0, 105.9, 90.0]}, "ref_override_why": "myoglobin, P 1 21 1. XDS indexed this sweep (a powder-contaminated crystal) in P 1 on a reduced cell (28.7 34.8 64.4 74.3 89.9 90.1); the monoclinic cell is that cell's conventional setting", "model": "4dc8"}, -- 2.54.0 From c2e5e4a4e8c90a6df601ba6fbe975596bac91357 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 22:11:51 +0200 Subject: [PATCH 12/53] broker: initialize clears the image buffer, turns live rugnux off and removes its model Initialize is the fresh start between users, so nothing of the previous one's session is left: the image buffer is cleared, live rugnux processing is disabled and the uploaded MR model is dropped. It is done at the start of Initialize, under m then rugnux_mutex (the order the other setters take), so it happens even if configuring the detector then fails. The receiver is always Idle when the state machine is not running, so the buffer clear has no unmet precondition. The dashboard's rugnux panel read the settings only once, on mount, so after an initialize from any client it kept showing the previous run; it now polls them like the Rugnux tab does. The /initialize description in jfjoch_api.yaml says so; the frontend client is regenerated (description only). The python client docs and redoc follow at the release. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/JFJochStateMachine.cpp | 12 ++++++++++++ broker/jfjoch_api.yaml | 1 + .../src/client/@tanstack/react-query.gen.ts | 1 + frontend/src/client/sdk.gen.ts | 1 + frontend/src/components/RugnuxStatus.tsx | 5 +++-- tests/JFJochStateMachineTest.cpp | 19 +++++++++++++++++++ 6 files changed, 37 insertions(+), 2 deletions(-) diff --git a/broker/JFJochStateMachine.cpp b/broker/JFJochStateMachine.cpp index ccadf4949..ddc009829 100644 --- a/broker/JFJochStateMachine.cpp +++ b/broker/JFJochStateMachine.cpp @@ -329,6 +329,18 @@ void JFJochStateMachine::Initialize() { ResetError(); // Clear error, we don't care what was it start_exception = nullptr; // Re-initialising discards a pending start failure + // A fresh start: no images from before, and no live rugnux analysis or model left over from the + // previous user. + services.ClearImageBuffer(); + { + std::unique_lock ul2(rugnux_mutex); + RugnuxLiveSettings s = services.GetRugnuxLive(); + s.enable = false; + s.model = nullptr; + s.model_size_bytes = 0; + services.SetRugnuxLive(s); + } + logger.Info("Initialize"); SetState(JFJochState::Busy, "Configuring indexing threads", BrokerStatus::MessageSeverity::Info); try { diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index d142b8ede..4e9768dae 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -2917,6 +2917,7 @@ paths: This operation will reconfigure network interface of the detector. During operation of the detector it is recommended to use the `POST /pedestal` operation instead. If storage cells are used, the execution time might be few minutes. + It also clears the image buffer, disables live rugnux analysis and removes the rugnux model. This is async function - one needs to use `POST /wait_till_done` to ensure operation is done. responses: diff --git a/frontend/src/client/@tanstack/react-query.gen.ts b/frontend/src/client/@tanstack/react-query.gen.ts index db804e38d..04691c492 100644 --- a/frontend/src/client/@tanstack/react-query.gen.ts +++ b/frontend/src/client/@tanstack/react-query.gen.ts @@ -16,6 +16,7 @@ import type { deleteConfigRugnuxModelData, deleteConfigRugnuxModelError, getConf * This operation will reconfigure network interface of the detector. * During operation of the detector it is recommended to use the `POST /pedestal` operation instead. * If storage cells are used, the execution time might be few minutes. + * It also clears the image buffer, disables live rugnux analysis and removes the rugnux model. * * This is async function - one needs to use `POST /wait_till_done` to ensure operation is done. * diff --git a/frontend/src/client/sdk.gen.ts b/frontend/src/client/sdk.gen.ts index 9e0922bea..1da0a08e2 100644 --- a/frontend/src/client/sdk.gen.ts +++ b/frontend/src/client/sdk.gen.ts @@ -28,6 +28,7 @@ export type Options Date: Mon, 19 Oct 2026 22:04:58 +0200 Subject: [PATCH 13/53] broker: the journal keeps operational events; rugnux's processing log goes to it only when verbose The live rugnux run wrote its whole processing log into the broker's journal - every [Rugnux] line of the library (tables included) plus a "rugnux:" line per step and note - about 300 lines per collection, all of them results the user already gets through /rugnux/status, the end message and the written files. - Logger::ServiceLevel(service, level): every Logger of that name made afterwards keeps only that level and above. The library makes its own Logger("Rugnux") (and Logger("ScaleOnTheFly")) wherever it logs, so this is the one switch: the broker sets them to error, or to debug with the existing `verbose` setting. The per-Logger "Version" line goes with it. The rugnux CLI routes every Logger to its own sinks (RouteAll) and never calls this, so its console and _log.txt are unchanged. - The live observer feeds steps and notes to the live status only. - The run's outcome is one line at the end: state, START -> first image and frame rate, done after the last image, last image -> all written. The probe result, the warm-up time and the cut reason go to debug (the cut is in the outcome line). - Per-collection step chatter goes to debug: receiver init/teardown, thread start/stop, "Wait for receiver done" and friends, the per-image mean processing time. The JFJochBroker logger now follows `verbose` too, so these are available again with it. - Kept: Start request, measurement start, receiving started/finished, receiver efficiency (and the per-module packet counts on loss), calibration sequence, all warnings and errors. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/JFJochBrokerHttp.cpp | 2 +- broker/JFJochServices.cpp | 18 ++++---- broker/JFJochStateMachine.cpp | 2 +- broker/jfjoch_broker.cpp | 11 +++-- common/Logger.cpp | 9 ++++ common/Logger.h | 5 ++ receiver/JFJochReceiver.cpp | 8 ++-- receiver/JFJochReceiverFPGA.cpp | 18 ++++---- receiver/JFJochReceiverLite.cpp | 12 ++--- receiver/JFJochReceiverRugnux.cpp | 76 ++++++++++++++++++------------- 10 files changed, 96 insertions(+), 65 deletions(-) diff --git a/broker/JFJochBrokerHttp.cpp b/broker/JFJochBrokerHttp.cpp index f0b638ba3..722301af0 100644 --- a/broker/JFJochBrokerHttp.cpp +++ b/broker/JFJochBrokerHttp.cpp @@ -490,7 +490,7 @@ void JFJochBrokerHttp::wait_till_done_post(const std::optional &timeout } else status = state_machine.WaitTillMeasurementDone(std::chrono::seconds(timeout.value())); - logger.Info("Wait till done"); + logger.Debug("Wait till done"); switch (status.state) { case JFJochState::Idle: diff --git a/broker/JFJochServices.cpp b/broker/JFJochServices.cpp index b38d4e5dc..503643236 100644 --- a/broker/JFJochServices.cpp +++ b/broker/JFJochServices.cpp @@ -26,11 +26,11 @@ void JFJochServices::Start(const DiffractionExperiment& experiment, cannot_stop_detector = false; if (receiver == nullptr) { - logger.Info(" Done!"); + logger.Debug(" Done!"); return; } - logger.Info(" ... receiver start"); + logger.Debug(" ... receiver start"); if (image_puller) image_puller->ResumeAndClear(); receiver->Start(experiment, pixel_mask, calibration, image_puller); @@ -42,7 +42,7 @@ void JFJochServices::Start(const DiffractionExperiment& experiment, try { std::shared_lock ul(detector_mutex); if (detector && !experiment.IsUsingInternalPacketGen()) { - logger.Info(" ... detector start"); + logger.Debug(" ... detector start"); detector->Start(experiment); } } catch (const std::exception &e) { @@ -65,7 +65,7 @@ void JFJochServices::Start(const DiffractionExperiment& experiment, throw; } - logger.Info(" Done!"); + logger.Debug(" Done!"); } void JFJochServices::Off() { @@ -140,7 +140,7 @@ JFJochServicesOutput JFJochServices::Stop() { { std::shared_lock ul(detector_mutex); if (detector) { - logger.Info("Wait for detector idle"); + logger.Debug("Wait for detector idle"); DetectorState state = detector->GetState(); while ((!cannot_stop_detector) && ((state == DetectorState::WAITING) || (state == DetectorState::BUSY))) { @@ -149,7 +149,7 @@ JFJochServicesOutput JFJochServices::Stop() { state = detector->GetState(); } if (state == DetectorState::IDLE) { - logger.Info(" ... detector idle"); + logger.Debug(" ... detector idle"); receiver->Cancel(true); // cancel silently } else { logger.Error(" ... detector not idle: state {}{}; {}", to_string(state), @@ -161,12 +161,12 @@ JFJochServicesOutput JFJochServices::Stop() { } } - logger.Info("Wait for receiver done"); + logger.Debug("Wait for receiver done"); ret.receiver_output = receiver->Stop(); if (image_puller) image_puller->Suspend(); - logger.Info(" ... Receiver efficiency: {} % Max delay: {} Compression ratio {}x", + logger.Info("Receiver efficiency: {} % Max delay: {} Compression ratio {}x", static_cast(ret.receiver_output.efficiency * 100.0), ret.receiver_output.status.max_receive_delay.value_or(0), static_cast(std::round(ret.receiver_output.status.compressed_ratio.value_or(1)))); @@ -183,7 +183,7 @@ JFJochServicesOutput JFJochServices::Stop() { // reported to the caller via receiver_output.writer_err (the state machine surfaces it as // an error message) - it is not a detector fault, so it does not need re-initialisation. - logger.Info(" ... finished with success"); + logger.Debug(" ... finished with success"); } catch (const JFJochException &e) { logger.Error(" ... finished with error {}", e.what()); exception = std::current_exception(); diff --git a/broker/JFJochStateMachine.cpp b/broker/JFJochStateMachine.cpp index ddc009829..182f35170 100644 --- a/broker/JFJochStateMachine.cpp +++ b/broker/JFJochStateMachine.cpp @@ -538,7 +538,7 @@ void JFJochStateMachine::MeasurementThread() { scan_result = tmp_output.receiver_output.scan_result; auto image_mean_time = tmp_output.receiver_output.processing_time; - logger.Info("Per-image mean processing time (microseconds): compression {:.0f} preprocess {:.0f} azint {:.0f} spot finding {:.0f} indexing {:.0f} refinement {:.0f} indexing analysis {:.0f} prediction {:.0f} integration {:.0f} total {:.0f}", + logger.Debug("Per-image mean processing time (microseconds): compression {:.0f} preprocess {:.0f} azint {:.0f} spot finding {:.0f} indexing {:.0f} refinement {:.0f} indexing analysis {:.0f} prediction {:.0f} integration {:.0f} total {:.0f}", image_mean_time.compression * 1e6, image_mean_time.preprocessing * 1e6, image_mean_time.azint * 1e6, diff --git a/broker/jfjoch_broker.cpp b/broker/jfjoch_broker.cpp index 6badfbc2c..29fca9145 100644 --- a/broker/jfjoch_broker.cpp +++ b/broker/jfjoch_broker.cpp @@ -79,10 +79,13 @@ int main(int argc, char **argv) { if (settings.numaPolicyIsSet()) logger.Warning("NUMA policy is deprecated and ignored - processed are split over all the CPUs/GPUs in the system"); - if (settings.verboseIsSet() && settings.isVerbose()) - logger.Verbose(true); - else - logger.Verbose(false); + const bool verbose = settings.verboseIsSet() && settings.isVerbose(); + logger.Verbose(verbose); + Logger::ServiceLevel("JFJochBroker", verbose ? spdlog::level::debug : spdlog::level::info); + // What rugnux logs as it processes is its result, and that reaches the user through the live status, + // the end message and the written files: the journal keeps its errors only, and all of it when verbose. + for (const auto &service: {"Rugnux", "ScaleOnTheFly"}) + Logger::ServiceLevel(service, verbose ? spdlog::level::debug : spdlog::level::err); std::unique_ptr image_pusher = ParseImagePusher(settings); if (image_pusher) diff --git a/common/Logger.cpp b/common/Logger.cpp index 82cd01e62..e48564d7a 100644 --- a/common/Logger.cpp +++ b/common/Logger.cpp @@ -6,6 +6,8 @@ #include "spdlog/sinks/base_sink.h" #include "spdlog/fmt/bin_to_hex.h" +#include + #include "Logger.h" #include "GitInfo.h" @@ -29,12 +31,17 @@ void NvtxRangeEnd(uint64_t) {} namespace { spdlog::sink_ptr routed_sink; // set once, at start-up, before any thread logs + std::map service_levels; // the same } void Logger::RouteAll(spdlog::sink_ptr sink) { routed_sink = std::move(sink); } +void Logger::ServiceLevel(const std::string &service, spdlog::level::level_enum level) { + service_levels[service] = level; +} + Logger::Logger(const std::string &service_name, const std::string &file_name) { if (routed_sink) { spdlog_logger = std::make_shared(service_name, routed_sink); @@ -55,6 +62,8 @@ Logger::Logger(const std::string &service_name, const std::string &file_name) { sinks.push_back(std::make_shared(file_name, 23, 59)); spdlog_logger = std::make_shared(service_name, std::begin(sinks), std::end(sinks)); + if (const auto it = service_levels.find(service_name); it != service_levels.end()) + spdlog_logger->set_level(it->second); if (jfjoch_git_sha1().empty()) spdlog_logger->info("Version {} {}", jfjoch_version(), variant); else diff --git a/common/Logger.h b/common/Logger.h index 4549aeb4c..94b94c5c3 100644 --- a/common/Logger.h +++ b/common/Logger.h @@ -63,6 +63,11 @@ public: // next to its outputs and shows only warnings and errors among its own step lines. static void RouteAll(spdlog::sink_ptr sink); + // Every Logger named `service` made after this call keeps only lines of `level` and above. For a program + // that runs a library whose log is not its own: the broker keeps rugnux's processing log out of its + // journal. Set at start-up, before any thread logs. + static void ServiceLevel(const std::string &service, spdlog::level::level_enum level); + // A logger that keeps its lines instead of printing them, for work run beside other work that // logs: ReplayInto hands them to `to` afterwards, in order and as one block, so the two do not // interleave. diff --git a/receiver/JFJochReceiver.cpp b/receiver/JFJochReceiver.cpp index b7d0d1cdc..516f48900 100644 --- a/receiver/JFJochReceiver.cpp +++ b/receiver/JFJochReceiver.cpp @@ -32,12 +32,12 @@ JFJochReceiver::JFJochReceiver(const DiffractionExperiment &in_experiment, // retain_outcomes=false for now: online scaling/merge at the end is not enabled yet, so the // whole-run integration_outcome is unused here. Flip back to true when online scaling lands. indexer(experiment, indexing_thread_pool, false, /*real_time=*/true) { - logger.Info("Initializing receiver"); + logger.Debug("Initializing receiver"); // Ensure there is nothing running for now if (!image_buffer.Finalize(std::chrono::seconds(1))) throw JFJochException(JFJochExceptionCategory::WrongDAQState, "There are unfinished preview/sending jobs in the buffer"); - logger.Info("Image buffer from previous run finalized"); + logger.Debug("Image buffer from previous run finalized"); current_status.SetProgress(0); current_status.SetEfficiency({}); @@ -47,7 +47,7 @@ JFJochReceiver::JFJochReceiver(const DiffractionExperiment &in_experiment, az_int_mapping = std::make_unique(experiment, pixel_mask); auto end_time_point = std::chrono::steady_clock::now(); auto duration = std::chrono::duration(end_time_point - start_time_point); - logger.Info("Azimuthal integration mapping done in {:.5f} s with {} threads", duration.count(), az_int_mapping->GetNThreads()); + logger.Debug("Azimuthal integration mapping done in {:.5f} s with {} threads", duration.count(), az_int_mapping->GetNThreads()); plots.Setup(experiment, *az_int_mapping); @@ -203,7 +203,7 @@ void JFJochReceiver::SendEndMessage() { if (!image_pusher.EndDataCollection(message)) logger.Error("End message not sent via ZeroMQ (time-out)"); - logger.Info("Disconnected from writers"); + logger.Debug("Disconnected from writers"); } // The result blocks (AddToEndMessage) are megabytes only the writer needs. diff --git a/receiver/JFJochReceiverFPGA.cpp b/receiver/JFJochReceiverFPGA.cpp index 4e2c775e7..d22f64de4 100644 --- a/receiver/JFJochReceiverFPGA.cpp +++ b/receiver/JFJochReceiverFPGA.cpp @@ -79,7 +79,7 @@ JFJochReceiverFPGA::JFJochReceiverFPGA(const DiffractionExperiment &in_experimen if (experiment.IsCPUSummation()) expected_packets_per_image *= experiment.GetSummation(); - logger.Info("Data acquisition devices ready"); + logger.Debug("Data acquisition devices ready"); if ((experiment.GetDetectorMode() == DetectorMode::PedestalG0) || (experiment.GetDetectorMode() == DetectorMode::PedestalG1) @@ -116,7 +116,7 @@ JFJochReceiverFPGA::JFJochReceiverFPGA(const DiffractionExperiment &in_experimen &JFJochReceiverFPGA::MeasurePedestalThread, this, d, m, s, n, ignore)); } - logger.Info("Pedestal threads ready ({} threads/module*SC)", pedestal_nthreads); + logger.Debug("Pedestal threads ready ({} threads/module*SC)", pedestal_nthreads); } else if (experiment.GetImageNum() > 0) { SendStartMessage(); SendCalibration(); @@ -131,9 +131,9 @@ JFJochReceiverFPGA::JFJochReceiverFPGA(const DiffractionExperiment &in_experimen frame_transformation_futures.emplace_back(std::move(handle)); } - logger.Info("Image compression/forwarding threads started ({} threads)", frame_transformation_nthreads); + logger.Debug("Image compression/forwarding threads started ({} threads)", frame_transformation_nthreads); frame_transformation_ready.wait(); - logger.Info("Image compression/forwarding threads ready"); + logger.Debug("Image compression/forwarding threads ready"); } for (int d = 0; d < ndatastreams; d++) @@ -205,7 +205,7 @@ void JFJochReceiverFPGA::AcquireThread(uint16_t data_stream) { logger.Warning("Device thread {} done due to an error", data_stream); return; } - logger.Info("Device thread {} done", data_stream); + logger.Debug("Device thread {} done", data_stream); } @@ -595,7 +595,7 @@ void JFJochReceiverFPGA::FinalizeMeasurement() { for (auto &future: frame_transformation_futures) future.get(); - logger.Info("All processing threads done"); + logger.Debug("All processing threads done"); } current_status.SetProgress(1.0); @@ -613,7 +613,7 @@ void JFJochReceiverFPGA::FinalizeMeasurement() { for (auto &future: data_acquisition_futures) future.get(); - logger.Info("Devices stopped"); + logger.Debug("Devices stopped"); if (!image_buffer.CheckIfBufferReturned(SendFinalizeTimeout)) { logger.Error("Send commands not finalized in {} seconds", SendFinalizeTimeout.count()); @@ -628,7 +628,7 @@ void JFJochReceiverFPGA::FinalizeMeasurement() { current_status.SetProgress({}); current_status.SetStatus(GetStatus()); - logger.Info("Writing process finalized"); + logger.Debug("Writing process finalized"); } void JFJochReceiverFPGA::SetSpotFindingSettings(const SpotFindingSettings &in_spot_finding_settings) { @@ -642,7 +642,7 @@ void JFJochReceiverFPGA::SetSpotFindingSettings(const SpotFindingSettings &in_sp void JFJochReceiverFPGA::StopReceiver() { if (measurement.valid()) { measurement.get(); - logger.Info("Receiver stopped"); + logger.Debug("Receiver stopped"); } } diff --git a/receiver/JFJochReceiverLite.cpp b/receiver/JFJochReceiverLite.cpp index a4544d008..c105e1216 100644 --- a/receiver/JFJochReceiverLite.cpp +++ b/receiver/JFJochReceiverLite.cpp @@ -91,7 +91,7 @@ JFJochReceiverLite::JFJochReceiverLite(const DiffractionExperiment &in_experimen data_analysis_started(data_analysis_nthreads), measurement_started(1), dark_mask_analysis(in_experiment.GetDarkMaskSettings(), in_experiment.GetPixelsNum()) { - logger.Info("Starting {} data analysis threads", data_analysis_nthreads); + logger.Debug("Starting {} data analysis threads", data_analysis_nthreads); if (experiment.GetDetectorMode() == DetectorMode::DarkMask) { for (int i = 0; i < data_analysis_nthreads; i++) @@ -109,7 +109,7 @@ JFJochReceiverLite::JFJochReceiverLite(const DiffractionExperiment &in_experimen measurement = std::async(std::launch::async, &JFJochReceiverLite::MeasurementThread, this); data_analysis_started.wait(); - logger.Info("Data analysis threads ready"); + logger.Debug("Data analysis threads ready"); } JFJochReceiverLite::~JFJochReceiverLite() { @@ -175,7 +175,7 @@ void JFJochReceiverLite::MeasurementThread() { for (auto &f: data_analysis_futures) f.get(); - logger.Info("Data analysis threads finished"); + logger.Debug("Data analysis threads finished"); current_status.SetProgress(1.0); current_status.SetStatus(GetStatus()); @@ -189,12 +189,12 @@ void JFJochReceiverLite::MeasurementThread() { + std::to_string(SendFinalizeTimeout.count()) + " seconds"); } - logger.Info("All images sent through ZeroMQ"); + logger.Debug("All images sent through ZeroMQ"); if (push_images_to_writer) writer_error = image_pusher.Finalize(); - logger.Info("Writing process finalized"); + logger.Debug("Writing process finalized"); end_time = std::chrono::system_clock::now(); @@ -430,7 +430,7 @@ void JFJochReceiverLite::Cancel(bool silent) { void JFJochReceiverLite::StopReceiver() { if (measurement.valid()) { measurement.get(); - logger.Info("Receiver stopped"); + logger.Debug("Receiver stopped"); } } diff --git a/receiver/JFJochReceiverRugnux.cpp b/receiver/JFJochReceiverRugnux.cpp index 45f2e1767..cf77cdd7c 100644 --- a/receiver/JFJochReceiverRugnux.cpp +++ b/receiver/JFJochReceiverRugnux.cpp @@ -58,18 +58,17 @@ namespace { // What rugnux reports as it goes: the per-image results of its canonical pass into the receiver's // plots and scan result - the plots accumulate, so an image reported by the pre-pass as well would be - // counted twice - and its steps into the log. + // counted twice - and its steps into the live status (not the log: they are processing results). class LiveObserver : public RugnuxObserver { JFJochReceiverPlots &plots; ScanResultGenerator &scan_result; const AzimuthalIntegrationMapping &mapping; RugnuxLiveStatus &status; - Logger &logger; std::atomic canonical{false}; public: LiveObserver(JFJochReceiverPlots &plots, ScanResultGenerator &scan_result, - const AzimuthalIntegrationMapping &mapping, RugnuxLiveStatus &status, Logger &logger) - : plots(plots), scan_result(scan_result), mapping(mapping), status(status), logger(logger) {} + const AzimuthalIntegrationMapping &mapping, RugnuxLiveStatus &status) + : plots(plots), scan_result(scan_result), mapping(mapping), status(status) {} void OnImagePass(bool c) override { canonical = c; } void OnImageProcessed(const DataMessage &msg) override { @@ -79,15 +78,21 @@ namespace { scan_result.Add(msg); } void OnProgress(uint64_t done, uint64_t total) override { status.Progress(done, total); } - void OnStep(const std::string &title) override { - status.Step(title); - logger.Info("rugnux: {}", title); - } - void OnNote(const std::string &text) override { - status.Note(text); - logger.Info("rugnux: {}", text); - } + void OnStep(const std::string &title) override { status.Step(title); } + void OnNote(const std::string &text) override { status.Note(text); } }; + + std::string StateName(RugnuxLiveState state) { + switch (state) { + case RugnuxLiveState::Idle: return "idle"; + case RugnuxLiveState::Running: return "running"; + case RugnuxLiveState::Done: return "done"; + case RugnuxLiveState::Failed: return "failed"; + case RugnuxLiveState::Cancelled: return "cancelled"; + case RugnuxLiveState::Skipped: return "skipped"; + } + return "unknown"; + } } JFJochReceiverRugnux::JFJochReceiverRugnux(const DiffractionExperiment &in_experiment, @@ -125,7 +130,7 @@ JFJochReceiverRugnux::JFJochReceiverRugnux(const DiffractionExperiment &in_exper // analysis. Never destroyed - at exit the CUDA runtime may already be gone. [[maybe_unused]] static const FFTIndexerReuse *fft_indexers = new FFTIndexerReuse; [[maybe_unused]] static const FFTPlanReuse *fft_plans = new FFTPlanReuse; - logger.Info("Starting {} forward threads; rugnux runs on {} threads while the images arrive, " + logger.Info("rugnux live: starting {} forward threads; rugnux runs on {} threads while the images arrive, " "planning against {:.1f} GB of host memory", forward_nthreads, rugnux_nthreads, static_cast(host_budget) / 1e9); for (int i = 0; i < forward_nthreads; i++) @@ -218,14 +223,8 @@ void JFJochReceiverRugnux::MeasurementThread() { // to every result in place. const auto last_image = live_status.LastImage(); logger.Info("Receiving finished, {} images", images_collected.load()); - { - const auto live = live_status.Get(); - if (live.start_to_first_image_s && live.last_image_s && live.images_received > 1) - logger.Info("rugnux live: START -> first image {:.2f} s, {} images in {:.2f} s from the first to the last ({:.1f} Hz)", - live.start_to_first_image_s.value(), live.images_received, live.last_image_s.value(), - static_cast(live.images_received - 1) / live.last_image_s.value()); - } + std::optional done_s; if (rugnux_done.valid()) { // The fixed cutoff: the analysis gets deadline_s after the last image, then gives what it has. if (deadline_s > 0 && rugnux_done.wait_for(std::chrono::duration(deadline_s)) @@ -233,7 +232,7 @@ void JFJochReceiverRugnux::MeasurementThread() { Cut("deadline"); rugnux_done.wait(); if (last_image) - logger.Info("rugnux live: done {:.2f} s after the last image", SecondsSince(last_image.value())); + done_s = SecondsSince(last_image.value()); } if (probe_done.valid()) probe_done.wait(); @@ -251,11 +250,26 @@ void JFJochReceiverRugnux::MeasurementThread() { } if (push_images_to_writer) writer_error = image_pusher.Finalize(); - if (last_image && rugnux_done.valid()) { - const double written_s = SecondsSince(last_image.value()); - live_status.Latency(written_s); - logger.Info("rugnux live: last image -> all written {:.2f} s (end message sent at {:.2f} s)", - written_s, end_sent_s); + if (rugnux_done.valid()) { + // The run's outcome, in one line: its results are in the live status, the end message and the files. + const auto live = live_status.Get(); + std::string outcome = "rugnux live: " + StateName(live.state); + if (!live.message.empty()) + outcome += " (" + live.message + ")"; + if (!live.cut.empty()) + outcome += " (cut: " + live.cut + ")"; + if (live.start_to_first_image_s && live.last_image_s && live.images_received > 1) + outcome += fmt::format("; START -> first image {:.2f} s, {} images in {:.2f} s ({:.1f} Hz)", + live.start_to_first_image_s.value(), live.images_received, + live.last_image_s.value(), + static_cast(live.images_received - 1) / live.last_image_s.value()); + if (done_s) { + const double written_s = SecondsSince(last_image.value()); + live_status.Latency(written_s); + outcome += fmt::format("; done {:.2f} s after the last image, last image -> all written {:.2f} s " + "(end message sent at {:.2f} s)", done_s.value(), written_s, end_sent_s); + } + logger.Info("{}", outcome); } end_time = std::chrono::system_clock::now(); @@ -290,14 +304,14 @@ void JFJochReceiverRugnux::RugnuxThread() { try { const auto t0 = std::chrono::steady_clock::now(); WarmUpRugnuxLiveGPU(setup); - logger.Info("rugnux live: FFT indexers and plans ready in {:.2f} s", std::chrono::duration( + logger.Debug("rugnux live: FFT indexers and plans ready in {:.2f} s", std::chrono::duration( std::chrono::steady_clock::now() - t0).count()); } catch (const std::exception &e) { logger.Warning("rugnux live: building the FFT indexers ahead failed, they are built when used - {}", e.what()); } }); - LiveObserver observer(plots, scan_result, *az_int_mapping, live_status, logger); + LiveObserver observer(plots, scan_result, *az_int_mapping, live_status); ProcessResult result; { Rugnux process(reader, setup.experiment, *dataset->pixel_mask, setup.config); @@ -365,7 +379,7 @@ void JFJochReceiverRugnux::ProbeThread() { } catch (const std::exception &e) { logger.Warning("rugnux live: the lattice probe failed: {}", e.what()); } - logger.Info("rugnux live: lattice probe - {}", live_status.Get().probe); + logger.Debug("rugnux live: lattice probe - {}", live_status.Get().probe); } void JFJochReceiverRugnux::AddToEndMessage(EndMessage &message) { @@ -530,13 +544,13 @@ void JFJochReceiverRugnux::Cut(const std::string &reason) { if (probe != nullptr) probe->Cancel(); live_status.Cut(reason); - logger.Info("rugnux live: analysis cut ({}) - the end message carries what is established", reason); + logger.Debug("rugnux live: analysis cut ({}) - the end message carries what is established", reason); } void JFJochReceiverRugnux::StopReceiver() { if (measurement.valid()) { measurement.get(); - logger.Info("Receiver stopped"); + logger.Debug("Receiver stopped"); } } -- 2.54.0 From 02e5bc387f9cd16b4003b00a6d5ab5750e831bd8 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 01:00:54 +0200 Subject: [PATCH 14/53] broker: initialize and start clear the last collection's live rugnux status /rugnux/status kept answering with the previous collection's rugnux run after POST /initialize, and also after a /start whose collection does not run rugnux - the latter under the new run's tokens. RugnuxLiveStatus gets a Clear(), called from JFJochReceiverService::ClearResults(), which /start already uses to drop the previous run's status, plots and buffered images before the tokens change. Initialize now calls ClearResults() instead of ClearImageBuffer(), so a fresh start also drops the previous run's receiver status and plots. Authentication: every rugnux endpoint (GET/PUT /config/rugnux, PUT/DELETE /config/rugnux/model, GET /rugnux/status) was already in the broker's bearer-protected set and carries bearerAuth in jfjoch_api.yaml; the HTTP auth test now checks all five. The yaml descriptions of /initialize and rugnux_live_status say so; the C++ model and frontend client are regenerated (comments only). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/JFJochServices.h | 2 +- broker/JFJochStateMachine.cpp | 6 ++--- broker/gen/model/Rugnux_live_status.h | 4 +-- broker/jfjoch_api.yaml | 4 +-- .../src/client/@tanstack/react-query.gen.ts | 2 +- frontend/src/client/sdk.gen.ts | 2 +- frontend/src/client/types.gen.ts | 2 +- frontend/src/client/zod.gen.ts | 2 +- receiver/JFJochReceiverService.cpp | 1 + receiver/JFJochReceiverService.h | 4 +-- receiver/RugnuxLiveStatus.cpp | 6 +++++ receiver/RugnuxLiveStatus.h | 2 ++ tests/BrokerHttpAuthTest.cpp | 10 ++++++- tests/RugnuxLiveStatusTest.cpp | 26 +++++++++++++++++++ 14 files changed, 58 insertions(+), 15 deletions(-) diff --git a/broker/JFJochServices.h b/broker/JFJochServices.h index d4274dc29..4db201dae 100644 --- a/broker/JFJochServices.h +++ b/broker/JFJochServices.h @@ -80,7 +80,7 @@ public: bool GetImageFromBuffer(std::vector &v, int64_t image_number = -1); ImageBufferStatus GetImageBufferStatus() const; void ClearImageBuffer() const; - void ClearResults() const; // previous run's status, plots and buffered images + void ClearResults() const; // previous run's status, plots, live rugnux status and buffered images void LoadDetectorPixelMask(PixelMask &mask); diff --git a/broker/JFJochStateMachine.cpp b/broker/JFJochStateMachine.cpp index 182f35170..cdae150db 100644 --- a/broker/JFJochStateMachine.cpp +++ b/broker/JFJochStateMachine.cpp @@ -329,9 +329,9 @@ void JFJochStateMachine::Initialize() { ResetError(); // Clear error, we don't care what was it start_exception = nullptr; // Re-initialising discards a pending start failure - // A fresh start: no images from before, and no live rugnux analysis or model left over from the - // previous user. - services.ClearImageBuffer(); + // A fresh start: no images, plots or live rugnux status from before, and no live rugnux analysis + // or model left over from the previous user. + services.ClearResults(); { std::unique_lock ul2(rugnux_mutex); RugnuxLiveSettings s = services.GetRugnuxLive(); diff --git a/broker/gen/model/Rugnux_live_status.h b/broker/gen/model/Rugnux_live_status.h index 9107a5682..cab0e75e8 100644 --- a/broker/gen/model/Rugnux_live_status.h +++ b/broker/gen/model/Rugnux_live_status.h @@ -12,7 +12,7 @@ /* * Rugnux_live_status.h * - * Live rugnux processing of the last collection; kept until the next collection starts. Times are taken when the image puller received a message, and are in seconds since the first image (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. + * Live rugnux processing of the last collection; kept until the next collection starts or `POST /initialize`. Times are taken when the image puller received a message, and are in seconds since the first image (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. */ #ifndef Rugnux_live_status_H_ @@ -29,7 +29,7 @@ namespace org::openapitools::server::model { /// -/// Live rugnux processing of the last collection; kept until the next collection starts. Times are taken when the image puller received a message, and are in seconds since the first image (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. +/// Live rugnux processing of the last collection; kept until the next collection starts or `POST /initialize`. Times are taken when the image puller received a message, and are in seconds since the first image (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. /// class Rugnux_live_status { diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index 4e9768dae..58da1644d 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -2770,7 +2770,7 @@ components: rugnux_live_status: type: object description: | - Live rugnux processing of the last collection; kept until the next collection starts. + Live rugnux processing of the last collection; kept until the next collection starts or `POST /initialize`. Times are taken when the image puller received a message, and are in seconds since the first image (before it arrives, since now). The frame rate is `(images_received - 1) / last_image_s`. required: @@ -2917,7 +2917,7 @@ paths: This operation will reconfigure network interface of the detector. During operation of the detector it is recommended to use the `POST /pedestal` operation instead. If storage cells are used, the execution time might be few minutes. - It also clears the image buffer, disables live rugnux analysis and removes the rugnux model. + It also clears the image buffer, the previous collection's plots and live rugnux status, disables live rugnux analysis and removes the rugnux model. This is async function - one needs to use `POST /wait_till_done` to ensure operation is done. responses: diff --git a/frontend/src/client/@tanstack/react-query.gen.ts b/frontend/src/client/@tanstack/react-query.gen.ts index 04691c492..616b11276 100644 --- a/frontend/src/client/@tanstack/react-query.gen.ts +++ b/frontend/src/client/@tanstack/react-query.gen.ts @@ -16,7 +16,7 @@ import type { deleteConfigRugnuxModelData, deleteConfigRugnuxModelError, getConf * This operation will reconfigure network interface of the detector. * During operation of the detector it is recommended to use the `POST /pedestal` operation instead. * If storage cells are used, the execution time might be few minutes. - * It also clears the image buffer, disables live rugnux analysis and removes the rugnux model. + * It also clears the image buffer, the previous collection's plots and live rugnux status, disables live rugnux analysis and removes the rugnux model. * * This is async function - one needs to use `POST /wait_till_done` to ensure operation is done. * diff --git a/frontend/src/client/sdk.gen.ts b/frontend/src/client/sdk.gen.ts index 1da0a08e2..ae9c88485 100644 --- a/frontend/src/client/sdk.gen.ts +++ b/frontend/src/client/sdk.gen.ts @@ -28,7 +28,7 @@ export type Optionsbody.find("protected_run") == std::string::npos); REQUIRE(client.Get(path, wrong)->status == 401); } + // Live rugnux settings and its model are the dataset's too: changed only with a token. + REQUIRE(client.Put("/config/rugnux", R"({"enable":false})", "application/json")->status == 401); + REQUIRE(client.Put("/config/rugnux/model", "", "application/octet-stream")->status == 401); + REQUIRE(client.Delete("/config/rugnux/model")->status == 401); + REQUIRE(client.Delete("/config/rugnux/model", wrong)->status == 401); } SECTION("either token opens them") { @@ -86,6 +92,8 @@ TEST_CASE("JFJochBrokerHttp_BearerTokens", "[broker]") { REQUIRE(res->body.find("protected_run") != std::string::npos); REQUIRE(client.Get("/statistics/data_collection", other)->status == 200); REQUIRE(client.Get("/preview/plot?type=bkg_estimate", good)->status == 200); + REQUIRE(client.Get("/rugnux/status", good)->status == 200); + REQUIRE(client.Get("/config/rugnux", other)->status == 200); } SECTION("the aggregate statistics drop the measurement block instead of refusing") { diff --git a/tests/RugnuxLiveStatusTest.cpp b/tests/RugnuxLiveStatusTest.cpp index 2c3f25d80..24c8be717 100644 --- a/tests/RugnuxLiveStatusTest.cpp +++ b/tests/RugnuxLiveStatusTest.cpp @@ -46,6 +46,32 @@ TEST_CASE("RugnuxLiveStatus_BeforeFirstImage", "[RugnuxLive]") { CHECK(!status.LastImage()); } +TEST_CASE("RugnuxLiveStatus_Clear", "[RugnuxLive]") { + RugnuxLiveStatus status; + status.Start(10, 1.0, std::chrono::steady_clock::now() - 2s); + status.ImageReceived(std::chrono::steady_clock::now() - 1s); + status.Step("indexing"); + status.Probe("lattice found"); + status.Result("OK", "fine", {{"Space group", "P 1"}}, {"a warning"}); + status.Finish(RugnuxLiveState::Done); + REQUIRE(status.LastImage()); + + status.Clear(); + const auto s = status.Get(); + CHECK(s.state == RugnuxLiveState::Idle); + CHECK(s.images_expected == 0); + CHECK(s.images_received == 0); + CHECK(s.steps.empty()); + CHECK(s.probe.empty()); + CHECK(s.verdict.empty()); + CHECK(s.summary.empty()); + CHECK(s.warnings.empty()); + CHECK(!s.start_to_first_image_s); + CHECK(!s.last_image_s); + CHECK(s.elapsed_s == 0.0); + CHECK(!status.LastImage()); +} + TEST_CASE("TestImagePuller_SetsReceiveTime", "[RugnuxLive]") { TestImagePuller puller; const auto before = std::chrono::steady_clock::now(); -- 2.54.0 From 6936b04d5fb9afe05808faef60ab8a91447ea4e7 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 01:36:19 +0200 Subject: [PATCH 15/53] jfjoch_broker and rugnux check the CUDA JIT cache at start and warn when cuFFT plans are slow cuFFT ships its transforms as PTX for every card (cuFFT 11 and 12 alike), so each plan has the driver compile kernels, and a second plan of the same shape compiles them again. From the driver's JIT cache that costs ~0.5 ms a plan; without it 45-130 ms, under cuFFT's process-wide lock - which, over the ~100 plans of molecular replacement, matches the stage pattern of the slow MR in the production broker. The cache is unusable in more ways than CUDA_CACHE_DISABLE: with no HOME at all (a systemd system service without User= gets none, and the driver does not fall back to the passwd entry), on a read-only or unwritable directory, on a filesystem whose fcntl locks fail (NFS without locking) - each measured as slow as no cache - or on a slow network filesystem (each plan opens and locks the cache ~10 times). - common/CudaJitCache: the path the driver uses (CUDA_CACHE_PATH, else $HOME/.nv/ComputeCache), its filesystem type, whether it is writable, whether its index opens read-write and takes the driver's fcntl read lock, CUDA_CACHE_DISABLE and a CUDA_CACHE_MAXSIZE below 64 MiB. One line: info when fine, a warning with the cause and the remedy (CUDA_CACHE_PATH on a local disk) when not, or when a plan takes over 10 ms. - WarmUpCuFFT returns the fastest of three 64^3 R2C plan makes after a first one that pays the initialisation (and a cold-cache compile): ~2 ms in all with a cache, ~0.3 s without. - jfjoch_broker runs it after its version line (0.15 s at start, cuFFT init and card 0's context included); rugnux on its existing background warm-up thread. - CUDAMemHelpers.h: the modules are PTX on every platform, not only on Windows. No result changes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/jfjoch_broker.cpp | 5 + common/CMakeLists.txt | 1 + common/CudaJitCache.cpp | 124 ++++++++++++++++++ common/CudaJitCache.h | 26 ++++ image_analysis/indexing/CUDAMemHelpers.h | 6 +- image_analysis/indexing/IndexerThreadPool.cpp | 25 +++- image_analysis/indexing/IndexerThreadPool.h | 6 +- rugnux/rugnux_cli.cpp | 6 +- 8 files changed, 188 insertions(+), 11 deletions(-) create mode 100644 common/CudaJitCache.cpp create mode 100644 common/CudaJitCache.h diff --git a/broker/jfjoch_broker.cpp b/broker/jfjoch_broker.cpp index 29fca9145..852b9972b 100644 --- a/broker/jfjoch_broker.cpp +++ b/broker/jfjoch_broker.cpp @@ -9,10 +9,12 @@ #include +#include "../common/CudaJitCache.h" #include "../common/GitInfo.h" #include "../common/Logger.h" #include "../common/print_license.h" #include "../detector_control/DectrisSimplonClient.h" +#include "../image_analysis/indexing/IndexerThreadPool.h" #include "../writer/HDF5Objects.h" #include "JFJochBrokerHttp.h" @@ -64,6 +66,9 @@ int main(int argc, char **argv) { Logger logger("jfjoch_broker"); logger.Info("jfjoch_broker {} (git {}), built with {}", jfjoch_version(), jfjoch_git_sha1().substr(0, 6), jfjoch_build_info(true)); + // cuFFT's initialisation, and card 0's context, ahead of the first collection; the plan it times shows + // whether the CUDA JIT cache works - without one, every plan the analysis makes is compiled again. + LogCudaJitCache(logger, WarmUpCuFFT()); org::openapitools::server::model::Jfjoch_settings settings; std::ifstream file(argv[1]); diff --git a/common/CMakeLists.txt b/common/CMakeLists.txt index bac24abd8..e91172f1d 100644 --- a/common/CMakeLists.txt +++ b/common/CMakeLists.txt @@ -86,6 +86,7 @@ ADD_LIBRARY(JFJochCommon STATIC DetectorModuleGeometry.cpp DetectorModuleGeometry.h DetectorSetup.h DetectorSetup.cpp ZeroCopyReturnValue.h Histogram.h DiffractionGeometry.h CUDAWrapper.cpp CUDAWrapper.h + CudaJitCache.cpp CudaJitCache.h HostMemory.cpp HostMemory.h ThreadAffinity.cpp ThreadAffinity.h ADUHistogram.cpp ADUHistogram.h diff --git a/common/CudaJitCache.cpp b/common/CudaJitCache.cpp new file mode 100644 index 000000000..6967a72d3 --- /dev/null +++ b/common/CudaJitCache.cpp @@ -0,0 +1,124 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "CudaJitCache.h" + +#include +#include +#include + +#ifdef __linux__ +#include +#include +#include +#include +#include +#endif + +#ifdef __linux__ +namespace { + // statfs f_type of the filesystems a home directory is likely to be on. A network one is not wrong in + // itself, but each cuFFT plan opens and locks the cache about ten times: at 1 ms a round trip that is + // 15-20 ms a plan, at 5 ms as slow as no cache. + struct FsType { + long magic; + const char *name; + }; + + constexpr FsType fs_types[] = { + {0x6969, "nfs"}, {0x5346414F, "afs"}, {0x6B414653, "afs"}, {0x47504653, "gpfs"}, + {0x0BD00BD0, "lustre"}, {0x00C36400, "ceph"}, {0x19830326, "beegfs"}, {0xFF534D42, "cifs"}, + {0xFE534D42, "smb2"}, {0x517B, "smb"}, {0x01021997, "9p"}, {0x65735546, "fuse"}, + {0x58465342, "xfs"}, {0xEF53, "ext4"}, {0x9123683E, "btrfs"}, {0x01021994, "tmpfs"}, + {0x794C7630, "overlay"}, {0x2FC12FC1, "zfs"}, + }; +} +#endif + +CudaJitCache InspectCudaJitCache() { + CudaJitCache c; + const char *path = std::getenv("CUDA_CACHE_PATH"); + const char *disable = std::getenv("CUDA_CACHE_DISABLE"); + const char *maxsize = std::getenv("CUDA_CACHE_MAXSIZE"); + + if (path != nullptr && *path != '\0') + c.path = path; +#ifdef _WIN32 + else + c.path = "%APPDATA%\\NVIDIA\\ComputeCache"; +#else + else if (const char *home = std::getenv("HOME"); home != nullptr && *home != '\0') + c.path = std::string(home) + "/.nv/ComputeCache"; +#endif + + if (disable != nullptr && std::string(disable) == "1") + c.problem = "CUDA_CACHE_DISABLE=1 turns it off"; + else if (maxsize != nullptr && std::strtoull(maxsize, nullptr, 10) < 64 * 1024 * 1024) + c.problem = fmt::format("CUDA_CACHE_MAXSIZE={} keeps too little of it", maxsize); + else if (c.path.empty()) + c.problem = "neither CUDA_CACHE_PATH nor HOME is set (a systemd service without User= has no HOME)"; + +#ifdef __linux__ + if (c.path.empty()) + return c; + // The driver makes the directory when it is missing, so the nearest one that exists must be writable. + std::error_code ec; + std::filesystem::path dir = c.path; + while (!std::filesystem::exists(dir, ec) && dir.has_parent_path() && dir != dir.parent_path()) + dir = dir.parent_path(); + + struct statfs s{}; + if (statfs(dir.c_str(), &s) == 0) { + c.filesystem = fmt::format("{:#x}", static_cast(s.f_type)); + for (const auto &t: fs_types) + if (static_cast(s.f_type) == static_cast(t.magic)) + c.filesystem = t.name; + } + + if (!c.problem.empty()) + return c; + if (access(dir.c_str(), W_OK | X_OK) != 0) { + c.problem = fmt::format("{} is not writable ({})", dir.string(), std::strerror(errno)); + return c; + } + // Each lookup opens the index read-write and read-locks it with fcntl: where either fails (a read-only + // cache, a filesystem without POSIX locks) the driver compiles every time, though the cache is there. + const std::string index = c.path + "/index"; + if (std::filesystem::exists(index, ec)) { + const int fd = open(index.c_str(), O_RDWR); + if (fd < 0) { + c.problem = fmt::format("its index cannot be opened for writing ({})", std::strerror(errno)); + return c; + } + struct flock lock{}; + lock.l_type = F_RDLCK; + lock.l_whence = SEEK_SET; + if (fcntl(fd, F_SETLK, &lock) != 0 && errno != EAGAIN && errno != EACCES) + c.problem = fmt::format("its index cannot be locked ({})", std::strerror(errno)); + lock.l_type = F_UNLCK; + fcntl(fd, F_SETLK, &lock); + close(fd); + } +#endif + return c; +} + +void LogCudaJitCache(Logger &logger, double plan_ms) { + if (std::isnan(plan_ms)) + return; + const CudaJitCache c = InspectCudaJitCache(); + // Measured: 0.4-1.7 ms from a working cache, 60-130 ms with none. + const bool slow = plan_ms > 10.0; + const std::string where = c.path.empty() ? std::string("none") + : c.filesystem.empty() ? c.path : fmt::format("{} ({})", c.path, c.filesystem); + if (c.problem.empty() && !slow) { + logger.Info("CUDA JIT cache {}: a cuFFT plan takes {:.1f} ms", where, plan_ms); + return; + } + logger.Warning("CUDA JIT cache {}: {}a cuFFT plan takes {:.0f} ms{}, which can make molecular " + "replacement several times slower. Give the process a cache " + "on a local disk that only its user can write: CUDA_CACHE_PATH=, for instance " + "Environment=CUDA_CACHE_PATH=/var/cache/jfjoch/nv in a systemd unit", + where, c.problem.empty() ? std::string() : c.problem + "; ", plan_ms, + slow ? " (~1 ms from a working cache)" : ""); +} diff --git a/common/CudaJitCache.h b/common/CudaJitCache.h new file mode 100644 index 000000000..72eb2168f --- /dev/null +++ b/common/CudaJitCache.h @@ -0,0 +1,26 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include + +#include "Logger.h" + +// cuFFT ships its transforms as PTX (cuFFT 11 and 12 alike, for every card), so every plan it makes has +// the driver compile kernels - and makes them again for a second plan of the same shape. The driver keeps +// what it compiled in its JIT cache, CUDA_CACHE_PATH or else $HOME/.nv/ComputeCache: from there a plan +// takes about 1 ms, without it 60-130 ms, all under cuFFT's process-wide lock. A run that makes a hundred +// plans (molecular replacement) is then seconds slower, with nothing else to show for it. +struct CudaJitCache { + std::string path; // where the driver keeps it; empty when it keeps none + std::string filesystem; // the type of the filesystem it is on (Linux only) + std::string problem; // why the driver cannot use it, or only slowly; empty when none is seen +}; + +CudaJitCache InspectCudaJitCache(); + +// One line for a program's start: the cache, and `plan_ms` - how long one cuFFT plan took to make once +// cuFFT was initialised (WarmUpCuFFT). A warning, saying what to do, when the cache is unusable or a plan +// is slow. Nothing when plan_ms is NaN: there is no GPU, so no cuFFT. +void LogCudaJitCache(Logger &logger, double plan_ms); diff --git a/image_analysis/indexing/CUDAMemHelpers.h b/image_analysis/indexing/CUDAMemHelpers.h index 5c9d680d7..91b052439 100644 --- a/image_analysis/indexing/CUDAMemHelpers.h +++ b/image_analysis/indexing/CUDAMemHelpers.h @@ -352,9 +352,9 @@ struct CudaFFTShape { // and shape, and lent to the next user of that shape; when the last scope ends they are destroyed, and // outside any scope a plan is destroyed as soon as its user is done. Making a plan is costly and // serialised: cuFFT holds a process-wide lock while it loads the plan's kernel modules (each plan loads -// its own, two plans of one shape included), which took over a second a plan on a busy card, and on -// Windows compiles them from PTX first; and it frees and loads them all again for the next plan of the -// same shape. A plan is never shared by two users at once - cuFFT is thread safe only across distinct +// its own, two plans of one shape included), which took over a second a plan on a busy card. The modules +// are PTX on every platform, compiled by the driver - ~1 ms from its JIT cache, 60-130 ms without one +// (CudaJitCache.h); and it frees and loads them all again for the next plan of the same shape. A plan is never shared by two users at once - cuFFT is thread safe only across distinct // plans. // // In `rugnux` the scope is one processing run, so nothing is kept from one run for the next, nor past the diff --git a/image_analysis/indexing/IndexerThreadPool.cpp b/image_analysis/indexing/IndexerThreadPool.cpp index 53f0b2d82..bd8e2696d 100644 --- a/image_analysis/indexing/IndexerThreadPool.cpp +++ b/image_analysis/indexing/IndexerThreadPool.cpp @@ -3,6 +3,9 @@ #include "IndexerThreadPool.h" +#include +#include +#include #include #include @@ -18,13 +21,27 @@ #include "FFTIndexerCPU.h" #endif -void WarmUpCuFFT() { +double WarmUpCuFFT() { #ifdef JFJOCH_USE_CUDA if (get_gpu_count() == 0) - return; - cufftHandle plan = 0; - if (cufftPlan1d(&plan, 1024, CUFFT_C2C, 1) == CUFFT_SUCCESS) + return NAN; + // The first plan pays the initialisation, and compiles its kernels if the JIT cache does not have them + // yet; the next ones show what a plan costs from then on. The fastest of three, so that a stall + // beside (another card's context being made) is not taken for a slow plan. + double fastest_ms = INFINITY; + for (int i = 0; i < 4; i++) { + const auto t0 = std::chrono::steady_clock::now(); + cufftHandle plan = 0; + if (cufftPlan3d(&plan, 64, 64, 64, CUFFT_R2C) != CUFFT_SUCCESS) + return NAN; cufftDestroy(plan); + const double ms = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + if (i > 0) + fastest_ms = std::min(fastest_ms, ms); + } + return fastest_ms; +#else + return NAN; #endif } diff --git a/image_analysis/indexing/IndexerThreadPool.h b/image_analysis/indexing/IndexerThreadPool.h index 61b6a145a..023f6971d 100644 --- a/image_analysis/indexing/IndexerThreadPool.h +++ b/image_analysis/indexing/IndexerThreadPool.h @@ -37,8 +37,10 @@ enum class IndexerConstruction { Preconstruct, OnFirstUse }; // Force cuFFT's one-time library initialisation - 0.3 s measured, paid by whichever call happens to // be first - so it can be got out of the way on a background thread at startup rather than landing in -// the middle of the first pass with nothing overlapping it. No-op without a GPU. -void WarmUpCuFFT(); +// the middle of the first pass with nothing overlapping it. Returns how long a 3D plan takes to make +// after that, in ms - about 1 from a working CUDA JIT cache, 60 or more without (see CudaJitCache.h) - +// and NaN without a GPU. +double WarmUpCuFFT(); // cuFFT plans are kept and reused for as long as one of these lives, and destroyed when the last one // ends (CudaFFTPlanScope): one per processing run in `rugnux`, so no plan outlives the run that made it; diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 0f28f5403..1e546c24b 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -37,6 +37,7 @@ #include #include +#include "../common/CudaJitCache.h" #include "../common/Logger.h" #include "../common/ThreadAffinity.h" #include "../common/GitInfo.h" @@ -1701,8 +1702,9 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { logger.Info("Using space group {} (number {})", space_group->hm, space_group->number); // Off the critical path, so cuFFT's one-time initialisation overlaps the file open, the mask read - // and the beam-stop pre-scan instead of landing inside the first pass. - auto cufft_warmup = std::async(std::launch::async, WarmUpCuFFT); + // and the beam-stop pre-scan instead of landing inside the first pass. It also times a plan, which + // shows whether the CUDA JIT cache works. + auto cufft_warmup = std::async(std::launch::async, [&logger] { LogCudaJitCache(logger, WarmUpCuFFT()); }); // The same for the CUDA contexts of the other cards (the warm-up above creates card 0's), which // the pre-scan does not use: created one after another at start-up, they cost a fifth of a second // per card before the run began. -- 2.54.0 From 97ef0936b225fc07c9ba73125197d2e5178ef78b Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 01:50:17 +0200 Subject: [PATCH 16/53] CudaFFTPlan: one cuFFT plan per card and shape, shared by its users; rigid body transforms one grid at a time Every cuFFT plan has the driver compile cuFFT's PTX kernels: cufftMakePlanMany -> cuModuleLoadData -> the PTX JIT, under cuFFT's process-wide lock. A second plan of the same shape pays again, even while the first is alive. The CUDA JIT cache makes that ~1 ms. Without one (CUDA_CACHE_DISABLE, no writable $HOME) a plan costs 20-250 ms. That holds for every size (64^3 and 1D 256 included), every transform type and batch, on sm_75 and sm_120, with cuFFT 11.2, 12.0 and 12.4. No CUFFT_* setting changes it, so a choice of grid sizes cannot avoid it; only the number of plans made can be cut. Before this change, a plan was lent to one user at a time, so each concurrent user of a shape on a card made its own: 9-24 rigid-body engines, the translation slices and the score engines. Now a plan is made once per card and shape and shared. Each user keeps its own stream and work area, and CudaFFTPlan::R2C/C2R set both and launch under the plan's mutex. Shared plans give the same bits as private ones: 8 threads on one plan with their own work areas, 9 shapes x 160 transforms, 0 mismatches on RTX 5080 and T4. A shared work area, the negative control, gives 2-13 mismatches per shape where the plan has a work area. The rigid-body engine also dropped its batch-3 plan: the Jacobian transforms its three grids one at a time on the batch-1 plan. That gives the same bits (10 zone-grid shapes, sm_75 and sm_120) and one plan less per zone and card. The cytochrome C and thaumatin MR test sets, AF models (MR step, s): 4x T4 cytc cache on 2.22 -> 2.22 cache off 8.41 -> 4.00 plans 86 -> 24 4x T4 thau cache on 1.58 -> 1.61 cache off 11.13 -> 3.08 plans 131 -> 23 1x T4 cytc cache on 2.87 -> 2.81 cache off 8.82 -> 3.84 plans 80 -> 12 1x T4 thau cache on 2.41 -> 2.39 cache off 10.42 -> 3.17 plans 104 -> 11 RTX5080 cytc cache on 1.41 -> 1.12 cache off 6.5-10.7 -> 2.02 RTX5080 thau cache on 0.90 -> 0.85 cache off 8.03 -> 1.60 Outputs are bit-identical before and after, cache on or off: model, merged hkl, maps, and every MR and validation log line. The mtz/cif differ only in the command line they record. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- .../geom_refinement/BeamCenterFFTGPU.cu | 8 +- image_analysis/indexing/CUDAMemHelpers.h | 193 +++++++++++------- image_analysis/indexing/FFTIndexerGPU.cu | 2 +- .../ModelStructureFactorsGPU.cu | 6 +- .../MolecularReplacementGPU.cu | 7 +- .../structure_refinement/RigidBodyGPU.cu | 47 ++--- .../structure_refinement/RigidBodyGPUEngine.h | 2 +- tests/CUDAMemHelpersTest.cpp | 36 ++-- 8 files changed, 173 insertions(+), 128 deletions(-) diff --git a/image_analysis/geom_refinement/BeamCenterFFTGPU.cu b/image_analysis/geom_refinement/BeamCenterFFTGPU.cu index 3b49d6eae..bdf7ba5a4 100644 --- a/image_analysis/geom_refinement/BeamCenterFFTGPU.cu +++ b/image_analysis/geom_refinement/BeamCenterFFTGPU.cu @@ -210,7 +210,7 @@ PointSurfacesDevice TransformPoint(const std::vector &a, const std::vecto auto inverse_product = [&](const cufftComplex *u, const cufftComplex *v, cufftComplex *dst) { MultiplySpectra<<>>(u, v, dst, ncomplex); CheckLastKernel("BeamCenterFFT: spectrum product failed"); - Check(cufftExecC2R(inv, dst, out.real), "BeamCenterFFT: inverse transform failed"); + Check(inv.C2R(dst, out.real), "BeamCenterFFT: inverse transform failed"); }; auto take_corner = [&](CudaDevicePtr &dst) { dst = CudaDevicePtr(ncrop, ALLOC); @@ -222,11 +222,11 @@ PointSurfacesDevice TransformPoint(const std::vector &a, const std::vecto // a.a goes where m's spectrum will be, before it is there; a.m in the place of a's spectrum, the last // to read it; m.m there too; and the spectrum of a^2 there after that. upload(a); - Check(cufftExecR2C(fwd, out.real, d_a), "BeamCenterFFT: forward transform failed"); + Check(fwd.R2C(out.real, d_a), "BeamCenterFFT: forward transform failed"); inverse_product(d_a, d_a, d_m); take_corner(out.C); upload(m); - Check(cufftExecR2C(fwd, out.real, d_m), "BeamCenterFFT: forward transform failed"); + Check(fwd.R2C(out.real, d_m), "BeamCenterFFT: forward transform failed"); inverse_product(d_a, d_m, d_a); take_corner(out.S); inverse_product(d_m, d_m, d_a); @@ -234,7 +234,7 @@ PointSurfacesDevice TransformPoint(const std::vector &a, const std::vecto upload(a); SquareInPlace<<>>(out.real.get(), nreal); CheckLastKernel("BeamCenterFFT: squaring failed"); - Check(cufftExecR2C(fwd, out.real, d_a), "BeamCenterFFT: forward transform failed"); + Check(fwd.R2C(out.real, d_a), "BeamCenterFFT: forward transform failed"); inverse_product(d_a, d_m, d_a); ScaleCorner<<>>(out.real.get(), nx, 2 * h, 2 * w, norm); CheckLastKernel("BeamCenterFFT: surface scaling failed"); diff --git a/image_analysis/indexing/CUDAMemHelpers.h b/image_analysis/indexing/CUDAMemHelpers.h index 91b052439..fcbe55d72 100644 --- a/image_analysis/indexing/CUDAMemHelpers.h +++ b/image_analysis/indexing/CUDAMemHelpers.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -348,33 +349,45 @@ struct CudaFFTShape { int batch = 1; }; -// While a CudaFFTPlanScope is alive, cuFFT plans are kept when their user is done with them, by device -// and shape, and lent to the next user of that shape; when the last scope ends they are destroyed, and -// outside any scope a plan is destroyed as soon as its user is done. Making a plan is costly and -// serialised: cuFFT holds a process-wide lock while it loads the plan's kernel modules (each plan loads -// its own, two plans of one shape included), which took over a second a plan on a busy card. The modules -// are PTX on every platform, compiled by the driver - ~1 ms from its JIT cache, 60-130 ms without one -// (CudaJitCache.h); and it frees and loads them all again for the next plan of the same shape. A plan is never shared by two users at once - cuFFT is thread safe only across distinct -// plans. +// A cuFFT plan is made once per device and shape and shared by all its users at once: each user +// transforms on its own stream with its own work area (CudaFFTPlan), and a transform sets both and +// launches under the plan's lock, since cuFFT is thread safe only across distinct plans. While a +// CudaFFTPlanScope is alive a plan is kept when its last user is done, for the next user of that shape; +// when the last scope ends they are destroyed, and outside any scope a plan is destroyed as soon as its +// last user is done. +// +// Making a plan is costly and serialised: cuFFT holds a process-wide lock while it loads the plan's kernel +// modules, and it ships those as PTX, compiled for the card at every plan made - two plans of one shape +// included. The CUDA JIT cache makes that ~1 ms a plan; without one (CUDA_CACHE_DISABLE, no writable +// $HOME; see CudaJitCache.h) it is 20-250 ms a plan whatever the size, on every card and every cuFFT version tried; on a +// busy card it took over a second. So the number of plans made is what counts, not their size. // // In `rugnux` the scope is one processing run, so nothing is kept from one run for the next, nor past the -// device reset at the end of a run; jfjoch_broker, which resets no device, keeps one for its lifetime. A kept plan holds no work area: the user supplies one (CudaFFTPlan), -// so an idle plan holds only its tables and modules. +// device reset at the end of a run; jfjoch_broker, which resets no device, keeps one for its lifetime. A +// plan holds no work area: each user supplies one, so a kept plan holds only its tables and modules. namespace jfjoch_cufft_plans { struct Plan { cufftHandle handle = 0; size_t work_bytes = 0; }; - struct Idle { + // A plan and its users; its handle is used only under `exec`. + struct Shared { + Plan plan; + std::mutex exec; + int users = 0; + }; + + struct Kept { std::mutex m; - std::map, std::vector> plans; // by key() + std::mutex make; // one plan made at a time, so that two users of a new shape make it once + std::map, std::unique_ptr> plans; // by key() int scopes = 0; }; - inline Idle &idle() { - static Idle *i = new Idle; - return *i; + inline Kept &kept_plans() { + static Kept *k = new Kept; + return *k; } // The current device and the shape. @@ -422,18 +435,27 @@ namespace jfjoch_cufft_plans { return p; } - // An idle plan of this key, or a new one. - inline Plan borrow(const std::vector &k, const CudaFFTShape &s) { - { - std::lock_guard lock(idle().m); - auto it = idle().plans.find(k); - if (it != idle().plans.end() && !it->second.empty()) { - const Plan p = it->second.back(); - it->second.pop_back(); - return p; - } - } - return make(s); + // The plan of this key, made if there is none, with one more user. + inline Shared *borrow(const std::vector &k, const CudaFFTShape &s) { + Kept &kp = kept_plans(); + auto find = [&]() -> Shared * { + std::lock_guard lock(kp.m); + auto it = kp.plans.find(k); + if (it == kp.plans.end()) + return nullptr; + it->second->users++; + return it->second.get(); + }; + if (Shared *p = find()) + return p; + std::lock_guard make_lock(kp.make); + if (Shared *p = find()) + return p; + auto p = std::make_unique(); + p->plan = make(s); + p->users = 1; + std::lock_guard lock(kp.m); + return kp.plans.emplace(k, std::move(p)).first->second.get(); } // On the plan's own device (the first element of its key). @@ -445,24 +467,24 @@ namespace jfjoch_cufft_plans { cudaSetDevice(current); } - inline void give_back(const std::vector &k, const Plan &p) { + inline void give_back(const std::vector &k) { + Plan p; { - std::lock_guard lock(idle().m); - if (idle().scopes > 0) { - idle().plans[k].push_back(p); + Kept &kp = kept_plans(); + std::lock_guard lock(kp.m); + auto it = kp.plans.find(k); + if (--it->second->users > 0 || kp.scopes > 0) return; - } + p = it->second->plan; + kp.plans.erase(it); } destroy(k, p); } - // How many plans are kept, idle. + // How many plans there are, in use or kept. inline size_t kept() { - std::lock_guard lock(idle().m); - size_t n = 0; - for (const auto &[k, plans] : idle().plans) - n += plans.size(); - return n; + std::lock_guard lock(kept_plans().m); + return kept_plans().plans.size(); } } @@ -470,68 +492,93 @@ namespace jfjoch_cufft_plans { class CudaFFTPlanScope { public: CudaFFTPlanScope() { - std::lock_guard lock(jfjoch_cufft_plans::idle().m); - jfjoch_cufft_plans::idle().scopes++; + std::lock_guard lock(jfjoch_cufft_plans::kept_plans().m); + jfjoch_cufft_plans::kept_plans().scopes++; } + // The plans nobody uses go; one still in use goes when its last user is done. ~CudaFFTPlanScope() { - std::map, std::vector> plans; + std::vector, jfjoch_cufft_plans::Plan>> unused; { - std::lock_guard lock(jfjoch_cufft_plans::idle().m); - if (--jfjoch_cufft_plans::idle().scopes == 0) - plans.swap(jfjoch_cufft_plans::idle().plans); + jfjoch_cufft_plans::Kept &kp = jfjoch_cufft_plans::kept_plans(); + std::lock_guard lock(kp.m); + if (--kp.scopes == 0) + for (auto it = kp.plans.begin(); it != kp.plans.end();) { + if (it->second->users == 0) { + unused.emplace_back(it->first, it->second->plan); + it = kp.plans.erase(it); + } else + ++it; + } } - for (const auto &[k, list] : plans) - for (const auto &p : list) - jfjoch_cufft_plans::destroy(k, p); + for (const auto &[k, p] : unused) + jfjoch_cufft_plans::destroy(k, p); } CudaFFTPlanScope(const CudaFFTPlanScope &) = delete; CudaFFTPlanScope &operator=(const CudaFFTPlanScope &) = delete; }; -// A cuFFT plan of `shape` on the current device, borrowed from the plans kept above (or made) for as long -// as this lives, its transforms on `stream`. Its work area is `work_area` (at least WorkBytes() long), or, given -// none, one of its own. The plan goes back once the stream has finished with it. +// The cuFFT plan of `shape` on the current device (see above), used for as long as this lives, its +// transforms on `stream`. Its work area is `work_area` (at least WorkBytes() long), or, given none, one of +// its own. class CudaFFTPlan { std::vector key_; - jfjoch_cufft_plans::Plan plan_; + jfjoch_cufft_plans::Shared *plan_ = nullptr; cudaStream_t stream_ = nullptr; + void *work_area_ = nullptr; CudaDevicePtr work_; void release() { - if (!plan_.handle) + if (!plan_) return; cudaStreamSynchronize(stream_); work_ = CudaDevicePtr(); - jfjoch_cufft_plans::give_back(key_, plan_); - plan_ = {}; + jfjoch_cufft_plans::give_back(key_); + plan_ = nullptr; + } + + // The plan set to this user's work area and stream, under its lock for the transform `f` launches. + template + cufftResult Exec(F f) const { + std::lock_guard lock(plan_->exec); + cufftResult r = cufftSetStream(plan_->plan.handle, stream_); + if (r == CUFFT_SUCCESS) + r = cufftSetWorkArea(plan_->plan.handle, work_area_); + if (r == CUFFT_SUCCESS) + r = f(plan_->plan.handle); + return r; } public: CudaFFTPlan() = default; explicit CudaFFTPlan(const CudaFFTShape &shape, cudaStream_t stream = nullptr, void *work_area = nullptr) - : key_(jfjoch_cufft_plans::key(shape)), plan_(jfjoch_cufft_plans::borrow(key_, shape)), stream_(stream) { + : key_(jfjoch_cufft_plans::key(shape)), plan_(jfjoch_cufft_plans::borrow(key_, shape)), stream_(stream), + work_area_(work_area) { try { - if (work_area == nullptr) { - work_ = CudaDevicePtr(std::max(plan_.work_bytes, 1)); - work_area = work_; + if (work_area_ == nullptr) { + work_ = CudaDevicePtr(std::max(plan_->plan.work_bytes, 1)); + work_area_ = work_; } - if (cufftSetWorkArea(plan_.handle, work_area) != CUFFT_SUCCESS - || cufftSetStream(plan_.handle, stream_) != CUFFT_SUCCESS) - throw JFJochException(JFJochExceptionCategory::GPUCUDAError, "Setting up the CUFFT plan failed"); } catch (...) { release(); throw; } } - // The work area a plan of `shape` on the current device needs - read from the kept plan, which is - // made here if there is none, so that within a scope the plan asked is the plan used. + // The work area a plan of `shape` on the current device needs - read from the plan, which is made here + // if there is none, so that within a scope the plan asked is the plan used. static size_t WorkBytes(const CudaFFTShape &shape) { const std::vector k = jfjoch_cufft_plans::key(shape); - const jfjoch_cufft_plans::Plan p = jfjoch_cufft_plans::borrow(k, shape); - jfjoch_cufft_plans::give_back(k, p); - return p.work_bytes; + const size_t bytes = jfjoch_cufft_plans::borrow(k, shape)->plan.work_bytes; + jfjoch_cufft_plans::give_back(k); + return bytes; + } + + cufftResult R2C(cufftReal *in, cufftComplex *out) const { + return Exec([&](cufftHandle h) { return cufftExecR2C(h, in, out); }); + } + cufftResult C2R(cufftComplex *in, cufftReal *out) const { + return Exec([&](cufftHandle h) { return cufftExecC2R(h, in, out); }); } ~CudaFFTPlan() { @@ -539,8 +586,9 @@ public: } // Move-only type CudaFFTPlan(CudaFFTPlan&& other) noexcept - : key_(std::move(other.key_)), plan_(other.plan_), stream_(other.stream_), work_(std::move(other.work_)) { - other.plan_ = {}; + : key_(std::move(other.key_)), plan_(other.plan_), stream_(other.stream_), work_area_(other.work_area_), + work_(std::move(other.work_)) { + other.plan_ = nullptr; } CudaFFTPlan& operator=(CudaFFTPlan&& other) noexcept { if (this != &other) { @@ -548,17 +596,18 @@ public: key_ = std::move(other.key_); plan_ = other.plan_; stream_ = other.stream_; + work_area_ = other.work_area_; work_ = std::move(other.work_); - other.plan_ = {}; + other.plan_ = nullptr; } return *this; } CudaFFTPlan(const CudaFFTPlan&) = delete; CudaFFTPlan& operator=(const CudaFFTPlan&) = delete; - size_t WorkBytes() const { return plan_.work_bytes; } - operator cufftHandle() const { return plan_.handle; } - cufftHandle get() const { return plan_.handle; } + size_t WorkBytes() const { return plan_->plan.work_bytes; } + // Which plan this is - for tests; a transform goes through R2C/C2R. + cufftHandle Handle() const { return plan_->plan.handle; } }; // Page-locked host memory is slow to get and to give back: cudaHostAlloc and cudaFreeHost map and lock diff --git a/image_analysis/indexing/FFTIndexerGPU.cu b/image_analysis/indexing/FFTIndexerGPU.cu index cc72cd2a4..234820928 100644 --- a/image_analysis/indexing/FFTIndexerGPU.cu +++ b/image_analysis/indexing/FFTIndexerGPU.cu @@ -312,7 +312,7 @@ void FFTIndexerGPU::ExecuteFFT(const std::vector &coord, size_t nspots) { } cuda_err(cudaGetLastError()); - cuda_err(cufftExecR2C(n == FFT_BATCH_DIRECTIONS || n == nDirections ? plan : plan_last, + cuda_err((n == FFT_BATCH_DIRECTIONS || n == nDirections ? plan : plan_last).R2C( d_input_fft, d_output_fft)); const dim3 tr_grid((out_len + 31) / 32, (n + 31) / 32); diff --git a/image_analysis/structure_refinement/ModelStructureFactorsGPU.cu b/image_analysis/structure_refinement/ModelStructureFactorsGPU.cu index aa1b036d0..274214da8 100644 --- a/image_analysis/structure_refinement/ModelStructureFactorsGPU.cu +++ b/image_analysis/structure_refinement/ModelStructureFactorsGPU.cu @@ -280,7 +280,7 @@ void ModelStructureFactorsGPUEngine::Compute(const std::vector cudaMemcpyHostToDevice, e.stream)); e.density->Compute(e.pos, e.grid); - cufft_err(cufftExecR2C(e.forward, e.grid, reinterpret_cast(e.spectrum.get()))); + cufft_err(e.forward.R2C(e.grid, reinterpret_cast(e.spectrum.get()))); if (rows > 0) { compose_kernel<<>>(e.spectrum, e.Norm(), g.nu, g.nv, g.nw, e.ops, static_cast(e.setup.sym_ops.size()), e.setup.den, @@ -289,7 +289,7 @@ void ModelStructureFactorsGPUEngine::Compute(const std::vector } e.mask->Compute(e.mask_atoms, static_cast(mask_atoms.size()), e.grid); - cufft_err(cufftExecR2C(e.forward, e.grid, reinterpret_cast(e.spectrum.get()))); + cufft_err(e.forward.R2C(e.grid, reinterpret_cast(e.spectrum.get()))); if (rows > 0) { fmask_kernel<<>>(e.spectrum, e.Norm(), e.row_hkl, rows, g.nu, g.nv, g.nw, e.fmask); @@ -320,7 +320,7 @@ void ModelStructureFactorsGPUEngine::Map(int nu, int nv, int nw, double volume, e.stream)); conjugate_kernel<<>>(e.spectrum, complex_points); cuda_err(cudaGetLastError()); - cufft_err(cufftExecC2R(plan, reinterpret_cast(e.spectrum.get()), + cufft_err(plan.C2R(reinterpret_cast(e.spectrum.get()), reinterpret_cast(e.spectrum.get()))); transpose_kernel<<>>(reinterpret_cast(e.spectrum.get()), nu, nv, nw, static_cast(1.0 / volume), e.grid); diff --git a/image_analysis/structure_refinement/MolecularReplacementGPU.cu b/image_analysis/structure_refinement/MolecularReplacementGPU.cu index f405855ca..bb1fe7b88 100644 --- a/image_analysis/structure_refinement/MolecularReplacementGPU.cu +++ b/image_analysis/structure_refinement/MolecularReplacementGPU.cu @@ -386,7 +386,7 @@ gemmi::FPhiGrid MapToFPhiGPU(const gemmi::Grid &map, int reach) { CudaDevicePtr d_half(static_cast(map.nu / 2 + 1) * map.nv * map.nw); CudaDevicePtr d_out(hkl.data.size()); CudaFFTPlan plan(CudaFFTShape{.n = {map.nw, map.nv, map.nu}, .type = CUFFT_R2C}, stream); - cufft_err(cufftExecR2C(plan, d_map.get(), d_half.get())); + cufft_err(plan.R2C(d_map.get(), d_half.get())); half_u_to_half_w<<((hkl.data.size() + THREADS - 1) / THREADS), THREADS, 0, stream>>>( d_half, map.nu, map.nv, map.nw, reach, norm, d_out); cuda_err(cudaGetLastError()); @@ -658,9 +658,8 @@ std::vector MolecularReplacementGPU::TranslationFunction( CudaDevicePtr d_best(nstat); CudaDevicePtr d_best_index(nstat); - const cufftHandle plan = p.plan; + const CudaFFTPlan &plan = p.plan; // on `stream` { - cufft_err(cufftSetStream(plan, stream)); for (size_t r = 0; r < rotations.size(); r++) { const Rot rot = transposed(rotations[r]); translation_copies<<<(terms + THREADS - 1) / THREADS, THREADS, 0, stream>>>( @@ -673,7 +672,7 @@ std::vector MolecularReplacementGPU::TranslationFunction( translation_gather_cross<<((cells + THREADS - 1) / THREADS), THREADS, 0, stream>>>( p.cross_offset, p.cross_entry, copies, scaled, d_known, p.cross_term, p.cross_op, p.cross_conj, nops, cells, coef); - cufft_err(cufftExecC2R(plan, reinterpret_cast(coef.get()), map.get())); + cufft_err(plan.C2R(reinterpret_cast(coef.get()), map.get())); translation_stats<<>>( map, points, d_sum.get() + r * stat_blocks, d_sum2.get() + r * stat_blocks, d_best.get() + r * stat_blocks, d_best_index.get() + r * stat_blocks); diff --git a/image_analysis/structure_refinement/RigidBodyGPU.cu b/image_analysis/structure_refinement/RigidBodyGPU.cu index 962221384..54a26a2a3 100644 --- a/image_analysis/structure_refinement/RigidBodyGPU.cu +++ b/image_analysis/structure_refinement/RigidBodyGPU.cu @@ -295,10 +295,9 @@ int blocks(size_t n) { return static_cast((n + THREADS - 1) / THREADS); } -// An (nu, nv, nw) R2C transform, u fastest, `batch` grids back to back. -CudaFFTShape GridShape(int nu, int nv, int nw, int batch) { - return {.n = {nw, nv, nu}, .idist = nu * nv * nw, .odist = (nu / 2 + 1) * nv * nw, .type = CUFFT_R2C, - .batch = batch}; +// An (nu, nv, nw) R2C transform, u fastest. +CudaFFTShape GridShape(int nu, int nv, int nw) { + return {.n = {nw, nv, nu}, .type = CUFFT_R2C}; } int GreatestStreamPriority() { @@ -356,7 +355,9 @@ struct RigidBodyGPUEngineImpl { CudaDevicePtr point_fcmol, point_fmask; CudaDevicePtr products, x; - std::map, CudaFFTPlan> plans; + // The zone grid's transform, one grid at a time: a batch of three gives the same bits, but would be one more + // plan per zone and card to make (CudaFFTPlan). + CudaFFTPlan plan; std::unique_ptr density; std::unique_ptr mask; std::unique_ptr scale; @@ -422,15 +423,10 @@ struct RigidBodyGPUEngineImpl { cudaStreamSynchronize(stream); } - cufftHandle Plan(int batch) { - const std::array k{geom.nu, geom.nv, geom.nw, batch}; - auto it = plans.find(k); - if (it != plans.end()) - return it->second; - CudaFFTPlan plan(GridShape(geom.nu, geom.nv, geom.nw, batch), stream, fft_work); - if (plan.WorkBytes() > cap.fft_work_bytes) - throw JFJochException(JFJochExceptionCategory::GPUCUDAError, "rigid body: cuFFT work area over the reserve"); - return plans.emplace(k, std::move(plan)).first->second; + // Grid `slot`'s transform into its spectrum. + void Transform(int slot) { + cufft_err(plan.R2C(grid.get() + slot * GridPoints(), + reinterpret_cast(spectrum.get()) + slot * ComplexPoints())); } size_t GridPoints() const { return static_cast(geom.nu) * geom.nv * geom.nw; } @@ -489,10 +485,7 @@ int RigidBodyGPUEngine::CurrentDevice() { } size_t RigidBodyGPUEngine::FFTWorkBytes(int nu, int nv, int nw) { - size_t largest = 0; - for (int batch : {1, 3}) - largest = std::max(largest, CudaFFTPlan::WorkBytes(GridShape(nu, nv, nw, batch))); - return largest; + return CudaFFTPlan::WorkBytes(GridShape(nu, nv, nw)); } size_t RigidBodyGPUEngine::DeviceBytes(const RigidBodyGPUCapacity &c) { @@ -596,10 +589,9 @@ void RigidBodyGPUEngine::SetZone(const RigidBodyGPUZone &zone) { e.mask->SetGrid(mask_grid, images); e.density->SetAtoms(DensityGrid(zone), zone.atoms); e.scale->SetPoints(zone.point_hkl, zone.point_stol2, zone.point_fobs, zone.point_sigma, zone.constraints, zone.frac); - // The previous zone's plans go back to be lent to another engine; this one takes the zone's own. - e.plans.clear(); - e.Plan(1); - e.Plan(3); + e.plan = CudaFFTPlan(GridShape(e.geom.nu, e.geom.nv, e.geom.nw), e.stream, e.fft_work); + if (e.plan.WorkBytes() > e.cap.fft_work_bytes) + throw JFJochException(JFJochExceptionCategory::GPUCUDAError, "rigid body: cuFFT work area over the reserve"); cuda_err(cudaStreamSynchronize(e.stream)); } @@ -607,7 +599,7 @@ void RigidBodyGPUEngine::Fcalc(const double rotation[9], const double translatio RigidBodyGPUEngineImpl &e = *impl_; const DeviceGuard guard(e.device); e.Density(rotation, translation, 0); - cufft_err(cufftExecR2C(e.Plan(1), e.grid, reinterpret_cast(e.spectrum.get()))); + e.Transform(0); e.Compose(0, e.f, e.df_dt); } @@ -633,7 +625,7 @@ void RigidBodyGPUEngine::SetFixedCopies(const std::vector cuda_err(cudaGetLastError()); if (c < fixed.size()) { e.density->Compute(e.pos, e.grid.get()); - cufft_err(cufftExecR2C(e.Plan(1), e.grid, reinterpret_cast(e.spectrum.get()))); + cufft_err(e.plan.R2C(e.grid, reinterpret_cast(e.spectrum.get()))); e.has_fixed = false; e.Compose(0, e.shifted, nullptr); add_kernel<<>>(e.shifted, static_cast(e.rows), e.fixed); @@ -644,7 +636,7 @@ void RigidBodyGPUEngine::SetFixedCopies(const std::vector cufftComplex *mask_spectrum = reinterpret_cast(e.spectrum.get() + e.ComplexPoints()); e.mask->Compute(reinterpret_cast(e.copies_mask_atoms.get()), static_cast(copies * e.n_mask_atoms), mask_grid); - cufft_err(cufftExecR2C(e.Plan(1), mask_grid, mask_spectrum)); + cufft_err(e.plan.R2C(mask_grid, mask_spectrum)); fmask_kernel<<>>(reinterpret_cast(mask_spectrum), e.Norm(), e.row_hkl, static_cast(e.rows), e.geom.nu, e.geom.nv, e.geom.nw, e.fmask); @@ -661,7 +653,7 @@ void RigidBodyGPUEngine::Fmask() { // The mask atoms Fcalc() placed, four doubles each, as ModelMaskAtom lays them out. e.mask->Compute(reinterpret_cast(e.mask_atoms.get()), static_cast(e.n_mask_atoms), mask_grid); - cufft_err(cufftExecR2C(e.Plan(1), mask_grid, mask_spectrum)); + cufft_err(e.plan.R2C(mask_grid, mask_spectrum)); fmask_kernel<<>>(reinterpret_cast(mask_spectrum), e.Norm(), e.row_hkl, static_cast(e.rows), e.geom.nu, e.geom.nv, e.geom.nw, e.fmask); @@ -720,7 +712,8 @@ void RigidBodyGPUEngine::Jacobian(const double rotation[3][9], const double tran const DeviceGuard guard(e.device); for (int j = 0; j < 3; j++) e.Density(rotation[j], translation[j], j); - cufft_err(cufftExecR2C(e.Plan(3), e.grid, reinterpret_cast(e.spectrum.get()))); + for (int j = 0; j < 3; j++) + e.Transform(j); for (int j = 0; j < 3; j++) e.Compose(j, e.shifted.get() + j * e.rows, nullptr); diff --git a/image_analysis/structure_refinement/RigidBodyGPUEngine.h b/image_analysis/structure_refinement/RigidBodyGPUEngine.h index 1c1859e2c..683c5522b 100644 --- a/image_analysis/structure_refinement/RigidBodyGPUEngine.h +++ b/image_analysis/structure_refinement/RigidBodyGPUEngine.h @@ -83,7 +83,7 @@ class RigidBodyGPUEngine { public: // The bytes an engine of this capacity reserves on the device. static size_t DeviceBytes(const RigidBodyGPUCapacity &capacity); - // The largest cuFFT work area a (nu, nv, nw) grid needs, batch 1 or 3. + // The cuFFT work area a (nu, nv, nw) grid needs. static size_t FFTWorkBytes(int nu, int nv, int nw); // (brick, atom) pairs of the gather over at most, for a zone's atoms on its grid. static size_t PairBound(const RigidBodyGPUZone &zone); diff --git a/tests/CUDAMemHelpersTest.cpp b/tests/CUDAMemHelpersTest.cpp index 9c70f0c0f..6f5cf08fd 100644 --- a/tests/CUDAMemHelpersTest.cpp +++ b/tests/CUDAMemHelpersTest.cpp @@ -138,11 +138,11 @@ TEST_CASE("CudaPinnedPool_ReusedByTheNextEngine", "[CUDAMemHelpers]") { CHECK(cuda_release_pinned_pool() == 0); } -// Within a scope a cuFFT plan is kept once made and lent to one user at a time: two users at once get -// two plans, the next one after them gets one of those back, and a transform on a lent plan - another -// stream, a work area of the caller's - gives the same result bit for bit. When the scope ends, every -// kept plan is destroyed; outside a scope nothing is kept. -TEST_CASE("CudaFFTPlan_KeptAndLentToOneUserAtATime", "[CUDAMemHelpers]") { +// Within a scope a cuFFT plan is made once per shape and shared: two users at once get the same plan, +// each transforming on its own stream with its own work area, bit for bit as on a plan of its own; the +// plan is kept when they are done, and the next user gets it. When the scope ends, every kept plan is +// destroyed; outside a scope a plan goes with its last user. +TEST_CASE("CudaFFTPlan_SharedAndKept", "[CUDAMemHelpers]") { if (get_gpu_count() == 0) SKIP("No CUDA GPU present"); REQUIRE(jfjoch_cufft_plans::kept() == 0); @@ -154,11 +154,11 @@ TEST_CASE("CudaFFTPlan_KeptAndLentToOneUserAtATime", "[CUDAMemHelpers]") { for (size_t i = 0; i < input.size(); i++) input[i] = static_cast((i * 7919) % 1013) - 500.0f; CudaDevicePtr d_in(input.size()); - CudaDevicePtr d_out(nc * batch); REQUIRE(cudaMemcpy(d_in, input.data(), input.size() * sizeof(float), cudaMemcpyHostToDevice) == cudaSuccess); auto transform = [&](const CudaFFTPlan &plan, cudaStream_t stream) { - REQUIRE(cufftExecR2C(plan, d_in, d_out) == CUFFT_SUCCESS); + CudaDevicePtr d_out(nc * batch); + REQUIRE(plan.R2C(d_in, d_out) == CUFFT_SUCCESS); std::vector out(nc * batch); REQUIRE(cudaMemcpyAsync(out.data(), d_out, out.size() * sizeof(cufftComplex), cudaMemcpyDeviceToHost, stream) == cudaSuccess); @@ -166,30 +166,34 @@ TEST_CASE("CudaFFTPlan_KeptAndLentToOneUserAtATime", "[CUDAMemHelpers]") { return out; }; - CudaStream stream_a, stream_c; + CudaStream stream_a, stream_b, stream_c; std::vector first; - cufftHandle handle_a = 0; + cufftHandle handle = 0; { CudaFFTPlan a(shape, stream_a); - CudaFFTPlan b(shape); - handle_a = a.get(); - CHECK(handle_a != b.get()); + CudaFFTPlan b(shape, stream_b); + handle = a.Handle(); + CHECK(b.Handle() == handle); + CHECK(jfjoch_cufft_plans::kept() == 1); first = transform(a, stream_a); + const std::vector second = transform(b, stream_b); + CHECK(memcmp(second.data(), first.data(), first.size() * sizeof(cufftComplex)) == 0); } + CHECK(jfjoch_cufft_plans::kept() == 1); const size_t work = CudaFFTPlan::WorkBytes(shape); CudaDevicePtr work_area(std::max(work, 1)); CudaFFTPlan c(shape, stream_c, work_area); - CHECK(c.get() == handle_a); + CHECK(c.Handle() == handle); CHECK(c.WorkBytes() == work); const std::vector again = transform(c, stream_c); REQUIRE(again.size() == first.size()); CHECK(memcmp(again.data(), first.data(), first.size() * sizeof(cufftComplex)) == 0); - CHECK(jfjoch_cufft_plans::kept() == 1); // b; a is lent to c - c = CudaFFTPlan(); - CHECK(jfjoch_cufft_plans::kept() == 2); + // The scope ends while c still uses the plan: it goes with c. scope.reset(); + CHECK(jfjoch_cufft_plans::kept() == 1); + c = CudaFFTPlan(); CHECK(jfjoch_cufft_plans::kept() == 0); CHECK(CudaFFTPlan::WorkBytes(shape) == work); -- 2.54.0 From 4620130f7115a8a6d54cd7cd215e34fbdb3b0133 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 20:24:45 +0200 Subject: [PATCH 17/53] RotationScaleMerge: scaling loops run to an rms step of 1e-5, accelerated from 1e-2 (RESULT-CHANGING) The per-frame scaling tolerance goes from 1e-3 to 1e-5, and the Anderson onset (also the round the fulls loop keeps its cross-validated lambda) becomes an absolute 1e-2 instead of 10x the tolerance. Evaluated on 35 merged sets (ci tier + scaling-audit sets), on top of the loop fixes: - plain 1e-3: 8913 iterations, 38 s of loop time on GPU, 7 loops at the 100-round cap; - plain 1e-5: 35212 iterations, 120 s, 17 loops at the cap; - Anderson 1e-3 (previous default): 3715 iterations, 16 s, merged intensities as far from the converged ones as plain 1e-3 (median 0.13%, max 4.3%); - Anderson 1e-5 (this commit): 6507 iterations, 27 s, longest loop 89 rounds, merged intensities within 0.2% of plain 1e-5 and identical SG, d_min, CC1/2, ISa, R_meas, R_free. No space group or verdict changes in any mode. The converged merge is not measurably better against the references, but it no longer depends on where a loop happened to stop. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 2 +- image_analysis/scale_merge/RotationScaleMerge.cpp | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index cdd8736a6..f2eb05d06 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -388,7 +388,7 @@ Before the combine a per-frame scale can also be fitted on the partials themselv How smooth that scale is over the rotation is left to the data rather than to a fixed window. Each round fits every frame on its own fulls against the current reference, giving a scale $G_f$ and its information $D_f=\sum w^2c^2$; the scale is then the curve $x=\log G$ minimising $\sum_f J_f\,(x_f-y_f)^2+\lambda\sum_f(\Delta^2 x)_f^2$, with $y_f$ the frame's own fit in log scale and $J_f$ its information carried there — a penalised (Whittaker–Eilers) smoother, solved as a five-band linear system. $\lambda$ is chosen by cross-validation: blocks of frames one rocking curve wide are left out in turn and predicted from the curve through the rest (neighbours closer than a rocking curve share their measurement, since a full sums those frames). A frame of hundreds of fulls is then followed frame by frame, a frame of two or three is carried by its neighbours, a stretch with none is bridged by a straight line, and a scale that falls a hundredfold over a few degrees — an absorbing crystal turning edge-on — is followed where a window would average across it. Once the curve settles, one free fit is shrunk toward it frame by frame by how much of each frame's deviation its neighbour shares (the lag-1 covariance), which hands back a real per-frame systematic and discards fit noise. -Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the set of fitted frames changes. Acceleration starts only once the plain step has fallen below ten times the tolerance, and stops for the rest of the loop the first time the step grows: Anderson mixing models a map near its fixed point, and a loop further out, or one whose step grows, may be walking rather than converging - accelerated, it walks faster and ends somewhere else. This changes the path, not the fixed point; the loops stop at the same tolerance (an rms step of $10^{-3}$ in $\log G$) in about a quarter of the rounds and closer to the converged scales than the plain iteration stopped there. +Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the set of fitted frames changes. Acceleration starts only once the plain step has fallen below $10^{-2}$, and stops for the rest of the loop the first time the step grows: Anderson mixing models a map near its fixed point, and a loop further out, or one whose step grows, may be walking rather than converging - accelerated, it walks faster and ends somewhere else. This changes the path, not the fixed point. With it the loops are run to an rms step of $10^{-5}$ in $\log G$ - in fewer rounds than a plain loop took to $10^{-3}$ - so the merge no longer depends on where a loop happened to stop. After scale-fulls, five **correction surfaces** are fitted on the combined fulls (rotation path, **on by default**; disable all with `--no-scaling-corrections`), each an alternating multiplicative refinement of the per-full scale against the merged reference: diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 4ce0dea01..e096aeb1c 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -59,15 +59,17 @@ namespace { constexpr double MIN_EVENTS_PER_FRAME_SCALE_PARTIALS = 50.0; constexpr int64_t MIN_REFLECTIONS_FOR_IMAGE_CC = 20; // below this a frame's CC means nothing // The per-frame scaling loop has settled when the scales moved less than this between two - // iterations, as the rms of |log(G_new/G_old)| over the frames fitted in both. - constexpr double SCALING_TOLERANCE = 1e-3; + // iterations, as the rms of |log(G_new/G_old)| over the frames fitted in both. Tight enough that the + // merge no longer depends on where a loop stopped: with Anderson mixing a loop gets here in fewer + // rounds than a plain one took to 1e-3. + constexpr double SCALING_TOLERANCE = 1e-5; // The rounds of history the per-frame scaling loops are accelerated with (AndersonMixing). Their slow // directions are a few smooth modes over the sweep: measured on a 4nwv-class sweep, 3, 5 and 8 rounds // take its ten loops in 92, 91 and 90 iterations, against 364 without. constexpr int SCALING_ANDERSON_DEPTH = 5; - // The loops are accelerated only once their step is within this of the tolerance: Anderson mixing - // models the map near its fixed point, and a loop still further out is as likely to be walking. - constexpr double SCALING_ANDERSON_ONSET = 10.0 * SCALING_TOLERANCE; + // The loops are accelerated only once their step is below this: Anderson mixing models the map near + // its fixed point, and a loop still further out is as likely to be walking. + constexpr double SCALING_ANDERSON_ONSET = 1e-2; // A scaling loop has stopped settling when the trend of log(step) over its last this many rounds is // not falling (StepsStoppedFalling): a steady slow fall is a loop still contracting, a flat or // rising trend is not. -- 2.54.0 From 1949155c12377e752a82b6c9d31b426b83027c41 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 23:00:08 +0200 Subject: [PATCH 18/53] Rotation: a blank sweep end is trimmed before scaling, and a delta-CC1/2 stretch takes the sliver it leaves at an end (RESULT-CHANGING) Shutters that open late or close early, and beams that go off, leave frames at the ends of many sweeps with no exposure at all. Until now those frames went into the scaling like any other: on a CCD their noise, divided by an incident flux of ~0.005, made the partials loop walk their scales toward zero until the 50x collapse guard dropped them - or, if the loop stopped first, merged one as noise, which delta-CC1/2 then removed with its whole 10 deg batch and reported as SWEEP_GAPS (3p85, frames 338-359). On photon-counting detectors the blank frames fitted G = 0, escaped the guard and were reported as merged although nothing of them was (cytc_x10sa, myob_x10sa, thau_micromax_pink, lysoI_micromax_pink, 3p85). TrimBlankSweepEnds (RotationScaleMerge.h) walks in from each end and clears the reflections of every frame that - recorded no diffraction: as many of its reflections stand more than 3 sigma below zero as above it (a sign count, so no sigma calibration is needed and a slight background bias cannot hide the strong reflections), and that excess is 3 sigma clear of the run's typical frame - so a crystal too weak for one frame to show it is never trimmed; and - had no beam: its mean background is nearer to none than to the run's typical frame. That is what keeps a crystal leaving the beam, which also records nothing but still scatters, in the sweep ledger as a finding (insu_I_x06da_ref, 9zm0, 9fhc, 5reo are untouched). No stretch length: one blank frame is as blank as twenty. Partly exposed first/last frames recorded diffraction and stay (7tcd, 7q6j, 9z72, lyso_micromax_pink). It runs on the integration outcome before the overload count, so the merge, post-refinement and unmerged output all see the trimmed sweep, and in --mode scale on the stored reflections. The report gives FRAMES_BLANK_START/END and a prose line - information, no warning; the frames count as rejected. The sweep ledger looks for degraded stretches, and reads its running medians, only between the first and last frame with data. Delta-CC1/2: a convicted stretch that stops short of a sweep end by less than one rocking event takes the rest, where those frames read as the stretch does (5reo: 331-396 rejected, 397-399 were left in as a three-frame weak_diffraction range with its own warning). Battery --only, 31 sets, tol5 f217f180a -> this commit: no space group, d_min or verdict changes; merged statistics identical within 0.001 except 5reo (R_meas 0.236 -> 0.231, ISa 23.0 -> 23.3, R-free 0.2013 -> 0.2004, warnings 3 -> 2). Trimmed: 3p85 0/19 (+3 empty frames; collapse-guard drops 13 -> 0), 6w4h 0/5 (guard 5 -> 0), 7rji 0/4 (delta-CC1/2 no longer removes 3406-3412, two of which diffracted: 7 -> 0), cytc_x10sa 0/2, thau_x10sa_* 0/1-2, myob_x10sa 0/1, lyso_x10sa_strong 0/1, 8a1a 0/1, lyso_micromax_mono 1/0, lysoI_micromax_pink 2/0, thau_micromax_pink 1/0. Controls (11if, 5f6m, 6moj, 8u0i, 9fhc, 9z44, 9z72, 9zm0, 6iu6, insu_I_x06da_ref, myob_x06da_split, lyso_x06da_half_image, thau_bl1a_4p6keV) trim nothing and are unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- docs/RUGNUX_REPORT.md | 16 ++- .../scale_merge/RotationScaleMerge.cpp | 114 ++++++++++++++++-- .../scale_merge/RotationScaleMerge.h | 10 ++ rugnux/ResultReport.cpp | 9 ++ rugnux/Rugnux.h | 3 + rugnux/RugnuxScaleMerge.cpp | 8 ++ rugnux/rugnux_cli.cpp | 5 + tests/ResultReportTest.cpp | 5 + tests/RotationScaleWalkTest.cpp | 70 +++++++++++ 9 files changed, 228 insertions(+), 12 deletions(-) diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index bae385ed0..49736a946 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -370,6 +370,8 @@ FRAMES_DOWNGRADED= 101 FRAMES_REJECTED= 36 FRAMES_REJECTED_PCT= 2.00 ROTATION_REJECTED_DEG= 3.6 +FRAMES_BLANK_START= 0 +FRAMES_BLANK_END= 0 SWEEP_ROTATION= 180.0 FLUX_PEAK_TO_TROUGH= 1.03 SCALE_MODULATION_PEAK_TO_TROUGH= 1.00 @@ -382,8 +384,8 @@ SCALE_MODULATION_PEAK_TO_TROUGH= 1.00 ``` Of those keys, `SWEEP_QUALITY_COUNT` and the five disposition keys (`FRAMES_MERGED`, -`FRAMES_DOWNGRADED`, `FRAMES_REJECTED`, `FRAMES_REJECTED_PCT`, `ROTATION_REJECTED_DEG`) are in the -default report; `SWEEP_QUALITY_STATUS`, `SWEEP_QUALITY_REASONS`, `SWEEP_DISPOSITIONS`, +`FRAMES_DOWNGRADED`, `FRAMES_REJECTED`, `FRAMES_REJECTED_PCT`, `ROTATION_REJECTED_DEG`) and the two +blank-end counts (`FRAMES_BLANK_START`, `FRAMES_BLANK_END`) are in the default report; `SWEEP_QUALITY_STATUS`, `SWEEP_QUALITY_REASONS`, `SWEEP_DISPOSITIONS`, `SWEEP_ROTATION`, `FLUX_PEAK_TO_TROUGH` and `SCALE_MODULATION_PEAK_TO_TROUGH` appear with `--developer` (the default report states in prose whether the diagnostic ran). @@ -393,6 +395,16 @@ experiment was useless". It is reported beside `ROTATION_REJECTED_DEG` on purpos *frames* moves when the same experiment is re-sliced, and a percentage of the *rotation* does not. The prose headline above the table states both. +Frames at either **end** of a rotation sweep that recorded no diffraction at all - a shutter that +opened late or closed early, or a beam that went off - are left out before scaling and are not a +degraded range: they raise no `SWEEP_GAPS` warning. `FRAMES_BLANK_START` and `FRAMES_BLANK_END` count +them (they are also among `FRAMES_REJECTED`), and a prose line names them. A frame is blank when its +reflections show no diffraction - as many stand more than 3 sigma above zero as below it, and that +count is clearly below the run's typical frame - and its background says the beam was off (nearer to +none than to the run's typical frame). A partly exposed frame recorded something and stays in; a +crystal that has left the beam still scatters the beam into the background, so its frames stay too +and are reported in the table above. + A `SWEEP_GAPS` warning is given, one per range, where the degraded ranges together cover at least 1 % of the sweep's rotation; below that - a frame or two - they are in the table and the summary's `Sweep` line, and the warning lines in `--developer`. diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index e096aeb1c..70cc2f6a0 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -368,6 +368,86 @@ RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment, mosaicity_deg = s.GetDefaultMosaicity(); } +// The frames at either end of a rotation sweep that the beam never reached - a shutter that opened late +// or closed early, a beam that went off - which many sweeps carry. Their reflections are cleared, so +// nothing downstream sees them (the scaling, the merge, the post-refinement, the unmerged file) and the +// sweep reads as ending where the exposure did. Decided per frame, before anything is scaled, and with +// no stretch length: one blank frame at an end is as blank as twenty. +// +// Two things have to hold of such a frame, and each is read off its own integrated reflections. +// +// It recorded no diffraction. The reflections are counted by sign: those standing more than +// BLANK_TAIL_SIGMA above zero and those standing as far below it. Noise puts as many in one tail as in +// the other, however its sigmas are calibrated, so the excess of the upper tail is the diffraction the +// frame recorded, and the two counts are its own Poisson variance. Counted, not averaged: a mean I/sigma +// is carried by the many weak reflections, where a slight bias of the background estimate cancels what +// the few strong ones say - measured, the tail of a crystal leaving the beam read a mean below zero +// while its frames still correlated 0.3 with the merge, and a frame the shutter half exposed read as +// blank as one it never opened on. The frame has no diffraction when its excess is within +// BLANK_TAIL_SIGMA standard errors of zero AND that band lies wholly below the same band around the +// run's typical frame. The first alone would trim every frame of a crystal too weak for one frame to +// show it; the second makes a run that cannot tell a blank frame from a typical one trim nothing. +// +// And there was no beam: its background - the incident flux, see MeasureIncidentFlux - is nearer to none +// than to the run's typical frame. That is what separates a shutter from a crystal that has left the +// beam, which records no diffraction either but still scatters the beam into the background. Those are +// a finding about the crystal, and stay for the sweep ledger to report. +// +// A frame that recorded anything - partly exposed, the tail of a crystal leaving the beam - stays, for +// the scaling and the sweep ledger to judge as they judge any other. Only from the ends inward: a frame +// inside the sweep without beam is news, and is left to the ledger too. A frame with nothing integrated +// on it (missing from disk, not processed) is walked past and not counted. +std::pair TrimBlankSweepEnds(std::vector &outcomes, size_t nthreads) { + constexpr double BLANK_TAIL_SIGMA = 3.0; + const int n = static_cast(outcomes.size()); + std::vector upper(n, 0), lower(n, 0); + std::vector bkg(n, 0.0); + ParallelFor(n, nthreads, [&](int f) { + double bkg_sum = 0.0; + int64_t bkg_n = 0; + for (const auto &r : outcomes[f].reflections) { + if (std::isfinite(r.I) && std::isfinite(r.sigma) && r.sigma > 0.0f) { + upper[f] += r.I > BLANK_TAIL_SIGMA * r.sigma; + lower[f] += r.I < -BLANK_TAIL_SIGMA * r.sigma; + } + if (std::isfinite(r.bkg)) { bkg_sum += r.bkg; ++bkg_n; } + } + if (bkg_n > 0) bkg[f] = bkg_sum / bkg_n; + }); + std::vector excess, tails, background; + for (int f = 0; f < n; ++f) + if (!outcomes[f].reflections.empty()) { + excess.push_back(static_cast(upper[f] - lower[f])); + tails.push_back(static_cast(upper[f] + lower[f])); + background.push_back(bkg[f]); + } + if (excess.empty()) + return {0, 0}; + auto median = [](std::vector &v) { + std::nth_element(v.begin(), v.begin() + v.size() / 2, v.end()); + return v[v.size() / 2]; + }; + const double typical = median(excess); + const double typical_band = BLANK_TAIL_SIGMA * std::sqrt(median(tails)); + const double typical_bkg = median(background); + auto blank = [&](int f) { + const double e = static_cast(upper[f] - lower[f]); + const double band = BLANK_TAIL_SIGMA * std::sqrt(static_cast(upper[f] + lower[f])); + return e <= band && e + band < typical - typical_band && bkg[f] < typical_bkg - bkg[f]; + }; + auto walk = [&](int from, int step) { + int cleared = 0; + for (int f = from; f >= 0 && f < n && (outcomes[f].reflections.empty() || blank(f)); f += step) + if (!outcomes[f].reflections.empty()) { + outcomes[f].reflections.clear(); + ++cleared; + } + return cleared; + }; + const int at_start = walk(0, 1); + return {at_start, walk(n - 1, -1)}; +} + // Measure the incident flux per frame from the per-frame mean of the integrator's background estimate, // so it can be divided out of the scale before anything is fitted. // @@ -433,6 +513,11 @@ void RotationScaleMerge::IngestHost() { resident_ingest = gpu_active_ && !observation_dump; #endif + data_first = 0; + while (data_first < n_frames && partials_out[data_first].reflections.empty()) ++data_first; + data_last = n_frames - 1; + while (data_last >= 0 && partials_out[data_last].reflections.empty()) --data_last; + frame_cell_ok.assign(n_frames, 1); g_partial.assign(n_frames, 1.0); const float dist_tol = x.GetIndexingSettings().GetUnitCellDistTolerance(); @@ -2643,11 +2728,14 @@ namespace { // Running median over `window` frames. A median rather than a mean because the real per-frame scale // moves fast (its median frame-to-frame step in log is ~0.06) and one dead frame inside a good stretch // must not drag the window down. - std::vector RunningMedian(const std::vector &v, int window) { + // Over the frames [lo, hi] only - the sweep the data span, so that frames trimmed off its ends + // (empty, and reading 0) do not drag down the medians of the frames beside them. Outside the span + // a frame keeps its own value. + std::vector RunningMedian(const std::vector &v, int window, int lo, int hi) { const int n = static_cast(v.size()), half = window / 2; - std::vector out(n), buf; - for (int i = 0; i < n; ++i) { - buf.assign(v.begin() + std::max(0, i - half), v.begin() + std::min(n, i + half + 1)); + std::vector out(v), buf; + for (int i = std::max(0, lo); i <= std::min(n - 1, hi); ++i) { + buf.assign(v.begin() + std::max(lo, i - half), v.begin() + std::min(hi + 1, i + half + 1)); const size_t mid = buf.size() / 2; std::nth_element(buf.begin(), buf.begin() + mid, buf.end()); out[i] = buf[mid]; @@ -2795,7 +2883,8 @@ void RotationScaleMerge::MeasureSweepQuality(const std::vector &partial if (model[f] < model[trough]) trough = f; if (model[f] > model[peak]) peak = f; } - const auto qm = RunningMedian(quality, std::max(3, static_cast(std::lround(SWEEP_WINDOW_DEG / osc_deg)))); + const auto qm = RunningMedian(quality, std::max(3, static_cast(std::lround(SWEEP_WINDOW_DEG / osc_deg))), + data_first, data_last); // A modulation is a dimming, not a disappearance: the crystal has to be measurably worse at the // trough than at the peak, and still delivering data there. A sweep whose trough is simply dead // is a sweep with a dead arc, and is reported as one. @@ -2818,10 +2907,10 @@ void RotationScaleMerge::MeasureSweepQuality(const std::vector &partial // a stretch interrupted by a few good frames is one event, not two. const int window = std::max(3, static_cast(std::lround(SWEEP_WINDOW_DEG / osc_deg))); const int min_len = std::max(1, static_cast(std::lround(SWEEP_MIN_RANGE_DEG / osc_deg))); - const auto sm = RunningMedian(scale, window); - const auto qm = RunningMedian(quality, window); + const auto sm = RunningMedian(scale, window, data_first, data_last); + const auto qm = RunningMedian(quality, window, data_first, data_last); std::vector> found; - for (int f = 0; f < n_frames; ++f) { + for (int f = data_first; f <= data_last; ++f) { if (!(sm[f] < SWEEP_SCALE_FRACTION && qm[f] < SWEEP_CC_FRACTION)) continue; if (!found.empty() && f - found.back().second - 1 <= min_len) found.back().second = f; else found.emplace_back(f, f); @@ -3309,8 +3398,8 @@ void RotationScaleMerge::MeasureBatchDeltaCCHalf(int n_groups) { // nothing on its own. A frame at the run's typical scale AND at its typical agreement is a frame // nothing measured here says anything is wrong with. const int sweep_window = std::max(3, static_cast(std::lround(SWEEP_WINDOW_DEG / osc_deg))); - const auto scale_median = RunningMedian(sweep_scale, sweep_window); - const auto cc_median = RunningMedian(sweep_cc, sweep_window); + const auto scale_median = RunningMedian(sweep_scale, sweep_window, data_first, data_last); + const auto cc_median = RunningMedian(sweep_cc, sweep_window, data_first, data_last); auto normal_frame = [&](int f) { return scale_median[f] >= SWEEP_SCALE_FRACTION && cc_median[f] >= SWEEP_CC_FRACTION; }; @@ -3400,6 +3489,11 @@ void RotationScaleMerge::MeasureBatchDeltaCCHalf(int n_groups) { spent[worst] = 1; // nothing here any other channel can see; look elsewhere continue; } + // A stretch that stops short of an end of the sweep by less than one rocking event takes the rest + // of it, where those frames read as the stretch does: so few frames are below what this statistic + // can judge on their own, and left in they are the one piece of the defect nothing looks at again. + if (data_last - f1 < min_frames && corroborated(f1 + 1, data_last)) f1 = std::max(f1, data_last); + if (f0 - data_first < min_frames && corroborated(data_first, f0 - 1)) f0 = std::min(f0, data_first); int n_new = 0; for (int f = f0; f <= f1; ++f) n_new += frame_rejected[f] ? 0 : 1; if (n_rejected + n_new > max_rejected) { diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index fd60744ec..e3b4ab0d1 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -65,6 +65,11 @@ template using FillVector = std::vector>; // surface, both fitted on the host and pushed back to the resident (GPU) fulls before the merge. // External-reference scaling, the stills B-factor and wedge refinement are unsupported (caller rejects). // Stills use the per-image ScaleOnTheFly (fixed partiality) instead. +// +// Before any of it: TrimBlankSweepEnds clears the reflections of the frames at either end of the sweep +// that the beam never reached, and returns how many it cleared at the start and at the end. +std::pair TrimBlankSweepEnds(std::vector &outcomes, size_t nthreads); + class RotationScaleMerge { public: // The log scale x minimising sum_f J_f (x_f - y_f)^2 + lambda * sum_f (x_f - 2 x_{f+1} + x_{f+2})^2: @@ -656,6 +661,11 @@ private: // run median; 0 in both means the frame contributed nothing. std::vector sweep_scale, sweep_cc; + // The first and last frame with anything integrated on them. A sweep whose ends recorded no + // diffraction reaches this engine with those frames already empty (the caller trims them), so the + // sweep ledger looks for degraded stretches and edges between these two and not beyond. + int data_first = 0, data_last = -1; + // Frames whose observations delta-CC1/2 convicted (1 = out of the merge). Sized n_frames and all // zero until MeasureBatchDeltaCCHalf runs; Run() is what takes the observations out. std::vector frame_rejected; diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 61d8a5203..e1fc9a660 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -2065,6 +2065,8 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, Add(s, KeyInt("FRAMES_REJECTED", sq.frames_rejected)); Add(s, KeyReal("FRAMES_REJECTED_PCT", pct(sq.frames_rejected), "{:.2f}")); Add(s, KeyReal("ROTATION_REJECTED_DEG", sq.rejected_deg, "{:.1f}")); + Add(s, KeyInt("FRAMES_BLANK_START", result.blank_start_frames)); + Add(s, KeyInt("FRAMES_BLANK_END", result.blank_end_frames)); Add(s, KeyReal("SWEEP_ROTATION", sq.sweep_deg, "{:.1f}", true)); Add(s, KeyReal("FLUX_PEAK_TO_TROUGH", sq.flux_peak_to_trough, "{:.2f}", true)); Add(s, KeyReal("SCALE_MODULATION_PEAK_TO_TROUGH", sq.modulation_peak_to_trough, @@ -2082,6 +2084,13 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, sq.frames_merged, pct(sq.frames_merged), sq.frames_downgraded, pct(sq.frames_downgraded), sq.frames_rejected, pct(sq.frames_rejected), sq.rejected_deg))); + if (result.blank_start_frames + result.blank_end_frames > 0) + // Information, not a warning: a shutter that opened late or closed early is an ordinary + // artefact of the experiment and says nothing about the crystal. + Add(s, Prose(fmt::format( + " Sweep ends: {} frame(s) at the start and {} at the end recorded no diffraction - the\n" + " shutter or the beam was not on them - and were left out before scaling (counted as rejected).", + result.blank_start_frames, result.blank_end_frames))); if (sq.ranges.empty()) { // The empty table used to print its header and two rules around nothing on every // clean run - a table that says "no problem" by being blank. diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 4af5023fd..03eb8bab2 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -753,6 +753,9 @@ struct ProcessResult { // outcome before the merge; zero where nothing was integrated. int64_t integrated_reflections = 0, overloaded_reflections = 0; int64_t integrated_reflections_low_res = 0, overloaded_reflections_low_res = 0; + // Frames at the start and at the end of a rotation sweep that recorded no diffraction - a shutter + // or beam that was not on them - and were left out before scaling (TrimBlankSweepEnds). + int blank_start_frames = 0, blank_end_frames = 0; // Overlapping spots, as the integrator's overlap treatment measured them (BraggIntegrationCounts): // predicted reflections whose signal disk a nearer neighbour takes pixels from, and of those the // ones left with less than --overlap-minpk of it. Zero with --overlap off. diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 06cf4f0f8..54951ad7f 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -463,6 +463,14 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { // drops the whole rocking event) - they are the strongest reflections, mostly at low resolution, // and what a model's low-resolution terms rest on. The ice flags are set in the same sweep. Integer // counts, one image per task: the totals do not depend on the split. + if (experiment_.IsRotationIndexing()) { + const auto [at_start, at_end] = TrimBlankSweepEnds(indexer->GetIntegrationOutcome(), config_.nthreads); + result.blank_start_frames = at_start; + result.blank_end_frames = at_end; + if (at_start + at_end > 0) + logger.Info("Sweep ends: left out {} frame(s) at the start and {} at the end that recorded no " + "diffraction - a shutter or beam that was not on them", at_start, at_end); + } size_t ice_flagged = 0, ice_total = 0; { auto &outcomes = indexer->GetIntegrationOutcome(); diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 1e546c24b..f512e152b 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -2145,6 +2145,11 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Rotation scaling/merging (RotationScaleMerge) does not support reference " "scaling or wedge refinement"); + // The stored reflections still hold a blank sweep end (the file is written image by image, + // before anything is decided), so it is taken off here as the full pipeline takes it off. + if (const auto [at_start, at_end] = TrimBlankSweepEnds(reflections, nthreads); at_start + at_end > 0) + logger.Info("Sweep ends: left out {} frame(s) at the start and {} at the end that recorded " + "no diffraction", at_start, at_end); RotationScaleMerge rsm(experiment, reflections, experiment.GetUnitCell(), scaling_iter, nthreads, logger); rsm.Ingest(); diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index 5d94e077d..4359d4e46 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -87,6 +87,7 @@ TEST_CASE("ResultReport_Render", "[Diagnostics]") { result.used_distance_mm = 243.53f; result.has_merge_statistics = true; result.merge_statistics.sweep_quality = TestSweepQuality(); + result.blank_end_frames = 3; // a shutter that closed early: information, not a warning (below) const auto text = RenderResultReport("prefix", "in.h5", x, result); @@ -133,6 +134,10 @@ TEST_CASE("ResultReport_Render", "[Diagnostics]") { CHECK(text.find("\nROTATION_REJECTED_DEG= 5.0\n") != std::string::npos); CHECK(text.find("600 frames / 60.0 deg; 450 merged (75.0%), 100 downgraded (16.7%)") != std::string::npos); + CHECK(text.find("\nFRAMES_BLANK_START= 0\n") != std::string::npos); + CHECK(text.find("\nFRAMES_BLANK_END= 3\n") != std::string::npos); + CHECK(text.find("Sweep ends: 0 frame(s) at the start and 3 at the end recorded no diffraction") + != std::string::npos); // One table row per range, with the reason code verbatim. CHECK(text.find(" 100 149 50 5.0 crystal_out_of_beam ") diff --git a/tests/RotationScaleWalkTest.cpp b/tests/RotationScaleWalkTest.cpp index 7f942cca4..ddc2e4867 100644 --- a/tests/RotationScaleWalkTest.cpp +++ b/tests/RotationScaleWalkTest.cpp @@ -182,3 +182,73 @@ TEST_CASE("ValidationEvidenceLog10Tail_Exact", "[RotationScale]") { CHECK(ValidationEvidenceLog10Tail({.reflections = 1354, .reflections_by_chance = 777}) == Catch::Approx(-509.691).margin(1e-3)); CHECK(ValidationEvidenceLog10Tail({}) == 0.0); } + +namespace { + // A frame of `strong` reflections at I/sigma 10, `up` and `down` at +4 and -4 (noise in both + // tails), and enough at +-0.5 to make 200 in all, every one on a background of `bkg`. + IntegrationOutcome FrameOf(int strong, float bkg, int up = 0, int down = 0) { + IntegrationOutcome io; + auto add = [&](float z) { + Reflection r{}; + r.I = z; + r.sigma = 1.0f; + r.bkg = bkg; + io.reflections.push_back(r); + }; + for (int i = 0; i < strong; ++i) add(10.0f); + for (int i = 0; i < up; ++i) add(4.0f); + for (int i = 0; i < down; ++i) add(-4.0f); + for (int i = strong + up + down; i < 200; ++i) add(i % 2 ? 0.5f : -0.5f); + return io; + } +} + +TEST_CASE("TrimBlankSweepEnds_ShutterEndsOnly", "[RotationScale]") { + std::vector sweep; + sweep.push_back(FrameOf(0, 0.0f)); // 0: the shutter not yet open + sweep.push_back(FrameOf(20, 3.0f)); // 1: partly exposed - it recorded something, so it stays + for (int f = 2; f < 37; ++f) // 20: no beam inside the sweep - the ledger's, not ours + sweep.push_back(f == 20 ? FrameOf(0, 0.0f) : FrameOf(40, 10.0f)); + sweep.push_back(FrameOf(0, 2.0f)); // 37: closed; a dark level, as a CCD leaves + sweep.push_back(FrameOf(0, 0.0f, 4, 4)); // 38: closed, noise in both tails + sweep.emplace_back(); // 39: nothing integrated - passed over, not counted + + const auto [at_start, at_end] = TrimBlankSweepEnds(sweep, 2); + CHECK(at_start == 1); + CHECK(at_end == 2); + CHECK(sweep[0].reflections.empty()); + CHECK(sweep[1].reflections.size() == 200); + CHECK(sweep[20].reflections.size() == 200); + CHECK(sweep[36].reflections.size() == 200); + CHECK(sweep[37].reflections.empty()); + CHECK(sweep[38].reflections.empty()); +} + +TEST_CASE("TrimBlankSweepEnds_CrystalOutOfTheBeamStays", "[RotationScale]") { + // No diffraction at the end, but the beam was on - the background is still there. That is the crystal + // leaving the beam, which the sweep ledger reports; it is not a shutter. + std::vector sweep; + for (int f = 0; f < 36; ++f) + sweep.push_back(FrameOf(40, 10.0f)); + for (int f = 36; f < 40; ++f) + sweep.push_back(FrameOf(0, 8.0f)); + const auto [at_start, at_end] = TrimBlankSweepEnds(sweep, 2); + CHECK(at_start == 0); + CHECK(at_end == 0); + CHECK(sweep.back().reflections.size() == 200); +} + +TEST_CASE("TrimBlankSweepEnds_WeakCrystalKeepsItsEnds", "[RotationScale]") { + // Three strong reflections a frame: one frame cannot tell this crystal from no crystal, so even a + // frame without beam at an end is not called. + std::vector sweep; + sweep.push_back(FrameOf(0, 0.0f)); + for (int f = 1; f < 39; ++f) + sweep.push_back(FrameOf(3, 10.0f)); + sweep.push_back(FrameOf(0, 0.0f)); + const auto [at_start, at_end] = TrimBlankSweepEnds(sweep, 2); + CHECK(at_start == 0); + CHECK(at_end == 0); + CHECK(sweep.front().reflections.size() == 200); + CHECK(sweep.back().reflections.size() == 200); +} -- 2.54.0 From 2a5d303886749ea2ffacfe4a73c4a1d962d9230e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 22:51:44 +0200 Subject: [PATCH 19/53] RotationScaleMerge: the partials' credible-scale floor is a fraction of the typical frame, not of the median (RESULT-CHANGING) The floor below which a frame's partial scale is not credible (MIN_CREDIBLE_SCALE_RATIO = 0.02) was read off the MEDIAN fitted scale - in the loop's smoothing window, in its step, and in the guard after it (DropCollapsedScales). On a sweep that spent most of its turn out of the beam the median is itself a dead frame, and it walks: an empty frame's scale has no fixed point above zero, so the dead frames slide toward zero round after round, the median and the floor slide with them, and frames on the floor cross it, re-enter the window at their neighbours' scale (3x up in one round) and cross it again. The loop then has no fixed point, and which frames reach the merge depends on the round it stops on. The floor is now read off the run's typical frame (TypicalFrameScale), which is what the loop pins its gauge to - inside the loop it is the constant 0.02, so it stands still. The fulls' guard keeps the median (their scales are not gauge-pinned). Dropped frames still carry the median scale, so the sweep ledger reads them as before. lyso_x06da_half_image (1800 frames, two thirds weak or out of the beam; median frame at 1/5.7 of the typical one): traced, the final loop's floor fell 2.4x and every spike in its step was one frame re-entering the window; the R_meas 287% -> 685% of the rc178 loop fixes was the same non-settling loop ended by two stopping rules (delta-CC1/2 split the weak stretch differently, 206 frames at 1/20-1/50 of the median merged or not). The pre-pass P1 loop that crept to the cap is the geometry pre-pass, which commits nothing here. battery --only, before (rc178-aa-tol5 f217f180a) -> after, GPU: - lyso_x06da_half_image: R_meas 689% -> 81%, CC1/2 0.932 -> 0.971, REFRES CC1/2 0.484 -> 0.911, ISa 4.13 -> 4.22, model R-free 0.3302 -> 0.3271, CC_model 0.826 -> 0.829; final loop's smallest step 5.6e-3 -> 1.8e-4. --no-gpu gives the same numbers. - lcystine_x10sa_25keV: R_meas 0.396 -> 0.202, ISa 4.45 -> 8.49, CC1/2 0.9987 -> 0.9998. - lyso_x10sa_strong: R_meas 0.238 -> 0.151, ISa 5.51 -> 5.74, R-free 0.3653 -> 0.3658, CC_model 0.859 -> 0.857, d_min 1.354 -> 1.344. - insu_I_x06da_ref: R_meas 0.347 -> 0.290, R-free and ISa unchanged. - 7mzt: CC1/2 0.967 -> 0.983, CC_model 0.532 -> 0.548, R-free 0.4183 -> 0.4197, d_min 3.18 -> 3.21. - unchanged: lyso_x06da_ref, myob_x06da_split, myob_x06da_powder_2, thau_bl1a_4p6keV, 6o2h, 3meb, 9z72, 9o0h. Space groups identical on all 13. Tests: [RotationScale], AndersonMixing* pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- .../scale_merge/RotationScaleMerge.cpp | 75 ++++++++++--------- .../scale_merge/RotationScaleMerge.h | 6 +- 2 files changed, 41 insertions(+), 40 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 70cc2f6a0..3b9fbdbe3 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -83,15 +83,17 @@ namespace { // Interleaved folds of the cross-validation that sets the smoothness of the fulls' scale. constexpr int CV_FOLDS = 5; - // A fitted per-frame scale below this fraction of the run's median is not a measurement of anything: - // it would mean the frame received 2% of its neighbours' dose while still producing indexable spots. + // A fitted per-frame scale below this fraction of the run's typical frame (of the median, for the + // fulls - see DropCollapsedScales) is not a measurement of anything: it would mean the frame received + // 2% of its neighbours' dose while still producing indexable spots. // It matters because the scale enters as 1/G, so a G that collapses toward zero multiplies every // intensity on that frame without bound - and sigma by the identical factor, which is why no // sigma-based outlier test can see it. Measured over 12 rotation crystals in the default // configuration, the smallest LEGITIMATE min(G)/median(G) is 0.070; the case this bound exists for // sat at 0.0017 (a 546x amplification that put 99% of sum(I^2) into 0.66% of the observations and // took the merged CC1/2 from 93.7% to 17.2%). 0.02 is 3.5x below anything real and 12x above the - // failure. + // failure. Against the typical frame, over the 252 merged sets of one battery (read off their final + // per-frame scales), the smallest merged frame of a set that drops none sat at 0.036. constexpr double MIN_CREDIBLE_SCALE_RATIO = 0.02; // The same quantity, for the P1 merge that feeds the de-novo space-group search only. That merge @@ -1999,9 +2001,13 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( // What each frame's corr is multiplied by: the fitted G over the G it ends the round with. for (int f = 0; f < n_frames; ++f) ratio[f] = scaled_any[f] && std::isfinite(g[f]) && g[f] > 0.0 ? g[f] : 1.0; + // The credible floor is a fraction of the run's typical frame - the frame the gauge is pinned + // to below, so the floor stands still while the loop runs (see DropCollapsedScales). + for (int f = 0; f < n_frames; ++f) + n_obs[f] = scaled_any[f] && std::isfinite(g[f]) && g[f] > 0.0 ? frame_obs_count[f] : 0; if (window > 1) { frame_scaled_scratch = scaled_any; - ComputeSmoothGWindow(g, window, g_smooth); + ComputeSmoothGWindow(g, window, TypicalFrameScale(g, n_obs) * MIN_CREDIBLE_SCALE_RATIO, g_smooth); for (int f = 0; f < n_frames; ++f) if (scaled_any[f] && std::isfinite(g_smooth[f]) && g_smooth[f] > 0.0) g[f] = g_smooth[f]; @@ -2020,15 +2026,9 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( // leaves it out (ComputeSmoothGWindow) and the guard after the loop drops it. Left in the // step, a single such frame held the rms at 0.18 and the loop was stopped as unsettled after // 16 rounds; with it left out the same loop settles in 23. On that sweep every loop that - // stopped unsettled was one that went on to drop a frame. - std::vector fitted; - for (int f = 0; f < n_frames; ++f) - if (scaled_any[f] && g[f] > 0.0) fitted.push_back(g[f]); - double g_floor = 0.0; - if (!fitted.empty()) { - std::nth_element(fitted.begin(), fitted.begin() + fitted.size() / 2, fitted.end()); - g_floor = fitted[fitted.size() / 2] * MIN_CREDIBLE_SCALE_RATIO; - } + // stopped unsettled was one that went on to drop a frame. The scales are pinned to the + // typical frame by now, so the floor is the ratio itself. + const double g_floor = MIN_CREDIBLE_SCALE_RATIO; double ss = 0.0, sw = 0.0; int n = 0; for (int f = 0; f < n_frames; ++f) { @@ -4439,7 +4439,7 @@ void RotationScaleMerge::ApplyCellSurface(const std::vector &cell, int rounds_used, gauge_ramp, n_clamped, ncell); } -void RotationScaleMerge::ComputeSmoothGWindow(const std::vector &g, int window, +void RotationScaleMerge::ComputeSmoothGWindow(const std::vector &g, int window, double g_floor, std::vector &g_smooth) const { const int n = static_cast(g.size()); const int half = window / 2; @@ -4453,19 +4453,8 @@ void RotationScaleMerge::ComputeSmoothGWindow(const std::vector &g, int // stays NAN, which the caller reads as "leave this frame alone"), so what reaches // DropCollapsedScales is the scale the fit actually produced. // - // The floor is the guard's, read the same way: MIN_CREDIBLE_SCALE_RATIO of the MEDIAN fitted - // scale, not of the precision-weighted typical frame - see DropCollapsedScales for why the median. - std::vector fitted; - fitted.reserve(n); - for (int f = 0; f < n; ++f) - if (frame_scaled_scratch[f] && std::isfinite(g[f]) && g[f] > 0.0) - fitted.push_back(g[f]); - if (fitted.empty()) - return; - const size_t mid = fitted.size() / 2; - std::nth_element(fitted.begin(), fitted.begin() + mid, fitted.end()); - const double g_floor = fitted[mid] * MIN_CREDIBLE_SCALE_RATIO; - + // The floor is the guard's, read the same way: MIN_CREDIBLE_SCALE_RATIO of the run's typical frame, + // passed in by the loop. // Each frame's logarithm once, not once per window it falls in. std::vector log_g(n, 0.0); std::vector credible(n, 0); @@ -4488,7 +4477,7 @@ void RotationScaleMerge::ComputeSmoothGWindow(const std::vector &g, int } } -// Drop the observations of any frame whose scale fitted far below the run's median, reporting the +// Drop the observations of any frame whose scale fitted far below the run's typical frame, reporting the // per-frame factor the caller has to apply to its corr (1 where nothing changed, 0 where the frame is // dropped). Returns false when there was nothing to drop. // @@ -4510,11 +4499,19 @@ bool RotationScaleMerge::DropCollapsedScales(const std::vector &fitted_ apply.assign(n, 0); ratio.assign(n, 1.0); - // Against the MEDIAN frame, not the precision-weighted typical frame the sweep ledger reads: a - // frame that collapsed toward zero and was not quite dropped has its intensities amplified by - // 1/G, the fulls built from them are re-fitted with a scale of that size, and a cubic mean is then - // theirs - measured on a sweep 60% out of the beam, the fulls' precision-weighted typical scale - // sat on such frames and the floor dropped three quarters of every live frame's fulls. + // Against the run's typical frame (TypicalFrameScale), the frame the scaling loop pins its gauge to, + // not against the median frame. On a sweep that spent most of its turn out of the beam the median is + // itself a dead frame, and it walks: the dead frames' scales slide toward zero round after round + // (an empty frame's scale has no fixed point above zero), the median and the floor slide with them, + // and the frames on the floor cross it, re-enter the smoothing window at their neighbours' scale and + // cross it again - so the loop never settled, and which frames were merged depended on the round it + // stopped on. Measured on a lysozyme sweep two thirds out of the beam, where the median frame sat at + // a sixth of the typical one: the floor fell 2.4x in one loop, and the same loop ended by two + // stopping rules merged R_meas 287% and 685%; with the floor on the typical frame it merges 81%, + // CC1/2 0.93 -> 0.97, model R-free 0.330 -> 0.327. Where most of the sweep diffracts the + // median and the typical frame are the same frame and nothing changes. The fulls keep the median + // (DropCollapsedFullScales): their scales are not gauge-pinned, and amplified fulls of a frame that + // was not quite dropped pull a cubic mean up onto themselves. std::vector fitted; fitted.reserve(n); for (int f = 0; f < n; ++f) @@ -4525,7 +4522,11 @@ bool RotationScaleMerge::DropCollapsedScales(const std::vector &fitted_ const size_t mid = fitted.size() / 2; std::nth_element(fitted.begin(), fitted.begin() + mid, fitted.end()); - const double g_typ = fitted[mid]; + const double g_median = fitted[mid]; + std::vector n_obs(n, 0); + for (int f = 0; f < n; ++f) + if (fitted_mask[f] && std::isfinite(g[f]) && g[f] > 0.0) n_obs[f] = frame_count[f]; + const double g_typ = TypicalFrameScale(g, n_obs); const double g_floor = g_typ * MIN_CREDIBLE_SCALE_RATIO; int n_dropped = 0; @@ -4536,12 +4537,12 @@ bool RotationScaleMerge::DropCollapsedScales(const std::vector &fitted_ apply[f] = 1; ratio[f] = 0.0; // zero corr = out of the combine, the merge and the error model worst = std::max(worst, g_typ / g[f]); - g[f] = g_typ; // keep the median in g so this frame cannot drag a smoothing window + g[f] = g_median; // keep the median in g so this frame cannot drag a smoothing window ++n_dropped; } if (n_dropped > 0) - logger->Warning("Dropped {} frame(s) whose scale came out more than {:.0f}x below the run median " - "(worst {:.0f}x) - a scale that small says the frame holds no diffraction, and " + logger->Warning("Dropped {} frame(s) whose scale came out more than {:.0f}x below the run's typical " + "frame (worst {:.0f}x) - a scale that small says the frame holds no diffraction, and " "using it would amplify the frame's intensities, and its sigmas with them, " "by the same factor", n_dropped, 1.0 / MIN_CREDIBLE_SCALE_RATIO, worst); diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index e3b4ab0d1..4ef64c4fb 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -883,11 +883,11 @@ private: // The windowed geometric mean of G over frames; the scaling loop applies the resulting ratio to // the observations' corr (on the GPU, to the resident corr in a kernel). - void ComputeSmoothGWindow(const std::vector &g, int window, + void ComputeSmoothGWindow(const std::vector &g, int window, double g_floor, std::vector &g_smooth) const; - // Drop the observations of any frame whose fitted per-frame scale collapsed far below the run - // median, reporting the per-frame corr factor the caller has to apply. See the .cpp for why nothing + // Drop the observations of any frame whose fitted per-frame scale collapsed far below the run's + // typical frame, reporting the per-frame corr factor the caller has to apply. See the .cpp for why nothing // downstream can catch a collapsed scale on its own. bool DropCollapsedScales(const std::vector &fitted_mask, std::vector &g, std::vector &apply, std::vector &ratio) const; -- 2.54.0 From 456fd5d5cfedd205f550a882ae632851d8082267 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 01:52:20 +0200 Subject: [PATCH 20/53] RotationScaleMerge: the correction surfaces stop at an rms step of 1e-3, not at the per-frame scaling tolerance (RESULT-CHANGING) f217f180a tightened SCALING_TOLERANCE to 1e-5 for the per-frame scaling loops, but ApplyCellSurface used the same constant for its own stop. The surfaces then ran to their 30-round cap ("NOT settled"), and that - not the per-frame loops - is what lost ISa on the tol-1e-5 full battery (20261019-2106 vs 20261019-1726): on hepes, myob_x06da, 9rcs and 7bgt the per-frame loops settle on the same trajectory either way, while the modulation/absorption fits and their half-set gates changed. The surfaces' own fixed point is the worse one: run to 1e-5 with 300 rounds allowed, the detector-frame surfaces walk along the resolution direction the gauge projects out (7bgt modulation: 15 -> 283 rounds, 1.13x -> 8.47x ramp removed), their held-out half-set gain drops (hepes +0.026 -> -0.031, 6o2h -0.28 -> -0.78), and ISa falls (hepes 37.0 -> 18.3, myob_x06da 22.8 -> 19.6, 9rcs 7.4 -> 5.3, 7bgt 17.0 -> 15.3, 6o2h 31.0 -> 18.1). The surfaces' prior and gates were calibrated at 1e-3, so they get their own SURFACE_TOLERANCE = 1e-3 again; the per-frame loops stay at 1e-5. Tried and rejected (23 validation sets, runs 20261020-0040/0104/0129 cvfold-*): - leave-one-out reference in the surface fit: 7bgt R_meas 0.14 -> 0.31, R-free 0.207 -> 0.277, 9rcs CC1/2 0.994 -> 0.956. Regressing the observation on a mean that excludes it puts the rest of the group's noise in the regressor alone and attenuates the slope; the mean that includes it is matched by the same noise on both sides. - leave-one-out reference in the per-frame fit (CPU + GPU): same attenuation; 5reo R_meas 0.23 -> 0.49, 3meb 0.26 -> 0.37, myob_x06da_powder_2 d_min 1.54 -> 1.94 (aspirin and yag gained). - cross-validating the fulls smoothness on the held-out per-frame fit while scaling on the usual one: thau_bl1a_4p6keV ISa 34.3 -> 36.4, but kdp 23.9 -> 21.3, aspirin 35.5 -> 33.6, L-cystine R1 0.124 -> 0.137, 6o2h ISa 31.0 -> 25.7; no fewer stalled loops. Validation, 23 sets (small molecules, the four losers, stall sets, controls; run 20261020-0152 cvfold-final-sets): hepes, myob_x06da, 9rcs, 7bgt back to their tol-1e-3 values; the other 19 equal to the base within noise (aspirin 20 keV ISa 35.8 -> 35.5, L-cystine 25 keV ISa 8.5 -> 8.2 with SHELXL R1 0.1152 -> 0.1154). No SG, d_min or verdict change. ci tier (20261020-0152 cvfold-final-ci): 29/29 pass, every row within noise of both earlier full runs. [RotationScale] and AndersonMixing* pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- image_analysis/scale_merge/RotationScaleMerge.cpp | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 3b9fbdbe3..2f3de16a9 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -63,6 +63,15 @@ namespace { // merge no longer depends on where a loop stopped: with Anderson mixing a loop gets here in fewer // rounds than a plain one took to 1e-3. constexpr double SCALING_TOLERANCE = 1e-5; + // The correction surfaces (ApplyCellSurface) are NOT run to that tolerance: their fit stops once a + // round moves the cell factors by less than this, as the rms of |log| of the update. Their fixed + // point is worse than where this stops them: run to 1e-5 (up to 300 rounds), the detector-frame + // surfaces walked along the resolution direction the gauge projects out - 1.2x -> 8.5x of ramp + // removed on one set, 15 -> 283 rounds - their held-out half-set gain fell (+0.026 -> -0.031 per + // shell on one small-molecule set, -0.28 -> -0.78 on a protein), and ISa fell 37 -> 18, 22.8 -> 19.6, + // 7.4 -> 5.3, 17.0 -> 15.3 and 31.0 -> 18.1 on five of 23 sets, with SHELXL R1 0.0307 -> 0.0328 on + // one. The prior and the gates the surfaces carry were calibrated at this tolerance. + constexpr double SURFACE_TOLERANCE = 1e-3; // The rounds of history the per-frame scaling loops are accelerated with (AndersonMixing). Their slow // directions are a few smooth modes over the sweep: measured on a 4nwv-class sweep, 3, 5 and 8 rounds // take its ten loops in 92, 91 and 90 iterations, against 364 without. @@ -4202,8 +4211,8 @@ void RotationScaleMerge::ApplyCellSurface(const std::vector &cell, int // Fit the per-cell factor over the subset {frame&1 == parity} (parity < 0 = all fulls), alternating // rounds against that subset's own reference (a prior pull to 1, then gauge-fixed within each - // resolution shell where resolution_gauge, which also holds the overall scale) until the factors settle - the same - // tolerance as the per-frame scales, on the rms |log| of the round's update - or n_iter rounds + // resolution shell where resolution_gauge, which also holds the overall scale) until the factors settle - + // SURFACE_TOLERANCE on the rms |log| of the round's update - or n_iter rounds // have been spent. Returns the per-cell factor. int rounds_used = 0; bool settled = false; @@ -4342,7 +4351,7 @@ void RotationScaleMerge::ApplyCellSurface(const std::vector &cell, int } A[c] = next; } - if (n_step > 0 && std::sqrt(step2 / n_step) < SCALING_TOLERANCE) { + if (n_step > 0 && std::sqrt(step2 / n_step) < SURFACE_TOLERANCE) { settled = true; break; } -- 2.54.0 From 147f750c905f317fe4199773824c767ae85d014e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 07:43:47 +0200 Subject: [PATCH 21/53] broker: rugnux live settings and model need no bearer token GET/PUT /config/rugnux and PUT/DELETE /config/rugnux/model prepare the next collection and expose nothing of the current one, so they are open like the rest of the configuration. Protecting them meant the next user needed the previous dataset's token to set up their own run. /rugnux/status, which reports the current collection, stays protected. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/JFJochBrokerHttp.cpp | 5 ++--- broker/jfjoch_api.yaml | 32 -------------------------------- docs/JFJOCH_BROKER.md | 6 +++--- frontend/src/client/sdk.gen.ts | 4 ---- tests/BrokerHttpAuthTest.cpp | 13 ++++++------- 5 files changed, 11 insertions(+), 49 deletions(-) diff --git a/broker/JFJochBrokerHttp.cpp b/broker/JFJochBrokerHttp.cpp index 722301af0..4853b4b0a 100644 --- a/broker/JFJochBrokerHttp.cpp +++ b/broker/JFJochBrokerHttp.cpp @@ -143,7 +143,8 @@ JFJochBrokerHttp::HttpError JFJochBrokerHttp::handleOperationException(const std // The read endpoints that expose the current dataset (its name and unit cell, and its images). // While the dataset was started with tokens, they answer 401 to a request without one; the same // set carries the bearerAuth security scheme in jfjoch_api.yaml. Everything else - the control -// plane, instrument configuration, broker status - stays open. +// plane, instrument configuration (rugnux live settings and model included: they are set up for the +// next collection, not read from this one), broker status - stays open. static const std::set PROTECTED_PATHS{ "/statistics/data_collection", "/result/scan", @@ -154,8 +155,6 @@ static const std::set PROTECTED_PATHS{ "/preview/plot", "/preview/plot.bin", "/live/events", - "/config/rugnux", - "/config/rugnux/model", "/rugnux/status" }; diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index 58da1644d..c492e795b 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -3514,20 +3514,12 @@ paths: summary: Change rugnux live mode settings description: | This can only be done when detector is `Idle`, `Error` or `Inactive` states. - security: - - bearerAuth: [] requestBody: content: application/json: schema: $ref: '#/components/schemas/rugnux_settings' responses: - "401": - description: The current dataset is protected and no valid bearer token was presented. - content: - text/plain: - schema: - type: string "200": description: Everything OK "400": @@ -3546,15 +3538,7 @@ paths: get: summary: Get rugnux live mode settings and model status description: Can be done anytime - security: - - bearerAuth: [] responses: - "401": - description: The current dataset is protected and no valid bearer token was presented. - content: - text/plain: - schema: - type: string "200": description: Everything OK content: @@ -3569,8 +3553,6 @@ paths: The model (PDB or mmCIF, optionally gzipped; at most 64 MiB) is used for molecular replacement and validation of every following rugnux live collection, until it is removed with `DELETE /config/rugnux/model` or replaced by another upload. - security: - - bearerAuth: [] requestBody: content: application/octet-stream: @@ -3578,12 +3560,6 @@ paths: type: string format: binary responses: - "401": - description: The current dataset is protected and no valid bearer token was presented. - content: - text/plain: - schema: - type: string "200": description: All good "400": @@ -3600,15 +3576,7 @@ paths: summary: Remove the model used by rugnux live mode description: | This can only be done when detector is `Idle`, `Error` or `Inactive` states. - security: - - bearerAuth: [] responses: - "401": - description: The current dataset is protected and no valid bearer token was presented. - content: - text/plain: - schema: - type: string "200": description: All good "500": diff --git a/docs/JFJOCH_BROKER.md b/docs/JFJOCH_BROKER.md index 4d6258b91..b4350bba0 100644 --- a/docs/JFJOCH_BROKER.md +++ b/docs/JFJOCH_BROKER.md @@ -195,8 +195,8 @@ gets `413`, an empty one or one that cannot be read as a model `400`). The broke and keeps it in memory - nothing is written to disk - for every following collection until `DELETE /config/rugnux/model` removes it or another upload replaces it; a broker restart drops it. `GET /config/rugnux` reports whether a model is loaded and its size, never its contents. Changes are -refused during a collection, and while the current dataset was started with tokens, all of these -routes need its bearer token. +refused during a collection. Like the rest of the configuration these routes need no bearer token: +they prepare the next collection and expose nothing of the current one. The end message, and with it the end of the collection, waits for the analysis. `deadline_s` (default 60 s; 0 for no cutoff) bounds that wait: this long after the last image the analysis is cut. `POST /cancel` is @@ -212,7 +212,7 @@ state, the images received, the step it is on and how far into it, the steps so found, and once it is over the verdict, the summary, warnings and the time from the last image to every result written. The web frontend shows it on the dashboard and in its "Rugnux live" tab, which also changes the settings and uploads the model. It needs the dataset's bearer token -like the routes above. `GET /result/scan` carries the rotation lattice, unit cell and Bravais symbol +like the dataset routes (`/statistics/data_collection`, `/image_buffer/*`, `/preview/plot*`). `GET /result/scan` carries the rotation lattice, unit cell and Bravais symbol rugnux found. A collection in which rugnux finds no lattice still gets the spot count, resolution estimate and background of every image in `GET /result/scan` and the master file, marked as not indexed. diff --git a/frontend/src/client/sdk.gen.ts b/frontend/src/client/sdk.gen.ts index ae9c88485..9c5ed66ca 100644 --- a/frontend/src/client/sdk.gen.ts +++ b/frontend/src/client/sdk.gen.ts @@ -383,7 +383,6 @@ export const putConfigSelectDetector = (op * Can be done anytime */ export const getConfigRugnux = (options?: Options): RequestResult => (options?.client ?? client).get({ - security: [{ scheme: 'bearer', type: 'http' }], url: '/config/rugnux', ...options }); @@ -395,7 +394,6 @@ export const getConfigRugnux = (options?: * */ export const putConfigRugnux = (options?: Options): RequestResult => (options?.client ?? client).put({ - security: [{ scheme: 'bearer', type: 'http' }], url: '/config/rugnux', ...options, headers: { @@ -411,7 +409,6 @@ export const putConfigRugnux = (options?: * */ export const deleteConfigRugnuxModel = (options?: Options): RequestResult => (options?.client ?? client).delete({ - security: [{ scheme: 'bearer', type: 'http' }], url: '/config/rugnux/model', ...options }); @@ -427,7 +424,6 @@ export const deleteConfigRugnuxModel = (op */ export const putConfigRugnuxModel = (options?: Options): RequestResult => (options?.client ?? client).put({ bodySerializer: null, - security: [{ scheme: 'bearer', type: 'http' }], url: '/config/rugnux/model', ...options, headers: { diff --git a/tests/BrokerHttpAuthTest.cpp b/tests/BrokerHttpAuthTest.cpp index f6e39de3d..3af02f883 100644 --- a/tests/BrokerHttpAuthTest.cpp +++ b/tests/BrokerHttpAuthTest.cpp @@ -70,7 +70,7 @@ TEST_CASE("JFJochBrokerHttp_BearerTokens", "[broker]") { "/image_buffer/image.cbor?id=0", "/image_buffer/image.tiff?id=0", "/image_buffer/image.jpeg?id=0", "/preview/plot?type=bkg_estimate", "/preview/plot.bin?type=bkg_estimate", "/live/events", - "/config/rugnux", "/rugnux/status"}) { + "/rugnux/status"}) { INFO(path); auto res = client.Get(path); REQUIRE(res); @@ -79,11 +79,11 @@ TEST_CASE("JFJochBrokerHttp_BearerTokens", "[broker]") { REQUIRE(res->body.find("protected_run") == std::string::npos); REQUIRE(client.Get(path, wrong)->status == 401); } - // Live rugnux settings and its model are the dataset's too: changed only with a token. - REQUIRE(client.Put("/config/rugnux", R"({"enable":false})", "application/json")->status == 401); - REQUIRE(client.Put("/config/rugnux/model", "", "application/octet-stream")->status == 401); - REQUIRE(client.Delete("/config/rugnux/model")->status == 401); - REQUIRE(client.Delete("/config/rugnux/model", wrong)->status == 401); + // Live rugnux settings and its model prepare the next collection: open like the rest of the configuration. + REQUIRE(client.Get("/config/rugnux")->status == 200); + REQUIRE(client.Put("/config/rugnux", R"({"enable":false})", "application/json")->status == 200); + REQUIRE(client.Put("/config/rugnux/model", "", "application/octet-stream")->status == 400); + REQUIRE(client.Delete("/config/rugnux/model")->status == 200); } SECTION("either token opens them") { @@ -93,7 +93,6 @@ TEST_CASE("JFJochBrokerHttp_BearerTokens", "[broker]") { REQUIRE(client.Get("/statistics/data_collection", other)->status == 200); REQUIRE(client.Get("/preview/plot?type=bkg_estimate", good)->status == 200); REQUIRE(client.Get("/rugnux/status", good)->status == 200); - REQUIRE(client.Get("/config/rugnux", other)->status == 200); } SECTION("the aggregate statistics drop the measurement block instead of refusing") { -- 2.54.0 From ddf41e3a18b53b20058f8c08763ffc1f733be8ab Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 07:50:57 +0200 Subject: [PATCH 22/53] CUDA JIT cache: a private one under /var/tmp when the default is unusable A service started by systemd without User= runs as root with no HOME, so the driver keeps no JIT cache and every cuFFT plan is compiled again (50-130 ms each, under cuFFT's process-wide lock) - seconds of MR in the broker. Before anything touches CUDA, the broker and rugnux now point CUDA_CACHE_PATH at /var/tmp/jfjoch-cuda-cache- when the default cache is missing or unusable and the site has set no CUDA_CACHE_PATH. The directory is used only if it is a real directory of this user's, mode 0700; otherwise the start-up warning stays as before. Measured without HOME: a cuFFT plan 51 ms -> 0.5 ms. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/jfjoch_broker.cpp | 1 + common/CudaJitCache.cpp | 34 +++++++++++++++++++++++++++++++++- common/CudaJitCache.h | 6 ++++++ rugnux/rugnux_cli.cpp | 1 + 4 files changed, 41 insertions(+), 1 deletion(-) diff --git a/broker/jfjoch_broker.cpp b/broker/jfjoch_broker.cpp index 852b9972b..0f4606848 100644 --- a/broker/jfjoch_broker.cpp +++ b/broker/jfjoch_broker.cpp @@ -52,6 +52,7 @@ static void setUpUnixSignals(std::vector quitSignals) { } int main(int argc, char **argv) { + UsePrivateCudaJitCacheIfNeeded(); RegisterHDF5Filter(); print_license("jfjoch_broker"); diff --git a/common/CudaJitCache.cpp b/common/CudaJitCache.cpp index 6967a72d3..41e2db7bd 100644 --- a/common/CudaJitCache.cpp +++ b/common/CudaJitCache.cpp @@ -11,10 +11,16 @@ #include #include #include +#include #include #include #endif +namespace { + // Why UsePrivateCudaJitCacheIfNeeded moved the cache, for LogCudaJitCache; empty when it did not. + std::string private_cache_reason; +} + #ifdef __linux__ namespace { // statfs f_type of the filesystems a home directory is likely to be on. A network one is not wrong in @@ -103,6 +109,29 @@ CudaJitCache InspectCudaJitCache() { return c; } +void UsePrivateCudaJitCacheIfNeeded() { +#ifdef __linux__ + if (const char *path = std::getenv("CUDA_CACHE_PATH"); path != nullptr && *path != '\0') + return; + if (const char *disable = std::getenv("CUDA_CACHE_DISABLE"); disable != nullptr && std::string(disable) == "1") + return; + const CudaJitCache c = InspectCudaJitCache(); + if (c.problem.empty()) + return; + + // /var/tmp is shared with every user: the directory is used only if it is a real directory (not a + // symlink), this user's, and closed to everyone else. + const std::string dir = fmt::format("/var/tmp/jfjoch-cuda-cache-{}", getuid()); + if (mkdir(dir.c_str(), 0700) != 0 && errno != EEXIST) + return; + struct stat st{}; + if (lstat(dir.c_str(), &st) != 0 || !S_ISDIR(st.st_mode) || st.st_uid != getuid() || (st.st_mode & 077) != 0) + return; + setenv("CUDA_CACHE_PATH", dir.c_str(), 1); + private_cache_reason = c.problem; +#endif +} + void LogCudaJitCache(Logger &logger, double plan_ms) { if (std::isnan(plan_ms)) return; @@ -112,7 +141,10 @@ void LogCudaJitCache(Logger &logger, double plan_ms) { const std::string where = c.path.empty() ? std::string("none") : c.filesystem.empty() ? c.path : fmt::format("{} ({})", c.path, c.filesystem); if (c.problem.empty() && !slow) { - logger.Info("CUDA JIT cache {}: a cuFFT plan takes {:.1f} ms", where, plan_ms); + logger.Info("CUDA JIT cache {}{}: a cuFFT plan takes {:.1f} ms", where, + private_cache_reason.empty() ? std::string() + : fmt::format(", used because {}", private_cache_reason), + plan_ms); return; } logger.Warning("CUDA JIT cache {}: {}a cuFFT plan takes {:.0f} ms{}, which can make molecular " diff --git a/common/CudaJitCache.h b/common/CudaJitCache.h index 72eb2168f..f39093a95 100644 --- a/common/CudaJitCache.h +++ b/common/CudaJitCache.h @@ -20,6 +20,12 @@ struct CudaJitCache { CudaJitCache InspectCudaJitCache(); +// Where the driver would keep no usable cache - typically a service run as root by systemd without User=, +// so with no HOME - and CUDA_CACHE_PATH is not set, point CUDA_CACHE_PATH at a directory of this user's +// own, /var/tmp/jfjoch-cuda-cache- (0700). The driver reads it once, when it initialises, so this +// must run before anything touches CUDA. Linux only; a CUDA_CACHE_PATH set by the site is left alone. +void UsePrivateCudaJitCacheIfNeeded(); + // One line for a program's start: the cache, and `plan_ms` - how long one cuFFT plan took to make once // cuFFT was initialised (WarmUpCuFFT). A warning, saying what to do, when the cache is unusable or a plan // is slow. Nothing when plan_ms is NaN: there is no GPU, so no cuFFT. diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index f512e152b..ea281a597 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -3372,6 +3372,7 @@ int main(int argc, char **argv) { // waiting on the GPU then sleeps instead of spinning on a core - measured on a 16M rotation // run, a fifth of all CPU time went to that spinning, with no change in wall time either way. // --no-gpu is looked for here, ahead of the parser, because that already asks CUDA for the devices. + UsePrivateCudaJitCacheIfNeeded(); bool no_gpu = false; for (int i = 1; i < argc; i++) if (std::strcmp(argv[i], "--no-gpu") == 0) -- 2.54.0 From 13dd2a8710d245f2e367107cb195618ce25c5b73 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 20:12:00 +0200 Subject: [PATCH 23/53] MolecularReplacement: the choice's null and comparisons start beside the work that decides where it falls The placed candidate of lowest R-free is known once the groups' searches are done. Its null and its comparisons with the other candidates (the copies carried into another group beside tNCS, the paired R) used to start only after the tNCS searches again and the null, one after the other. They now start as soon as the groups are done, beside the searches again, and the choice takes each where it comes to the same placements (matched by the placement itself); anything the choice does not reach is waited for and dropped. A null and a comparison are the same number whenever they are computed, so nothing changes but when they run. All of them are waited for before a merge's rigid-body engines are given back. The rigid-body pool gets engines for those fits at once (a null's replicates, a carry and a group searched again: at least 13); the transform's layout is still decided on the search's own fits. Measured on 4x Tesla T4 (24 threads), MR step, two runs each, byte-identical p.hkl, p_model.pdb, p_maps.mtz and MR log lines against the base on 1 and 4 cards: - cytochrome C (AF model, 3 copies, tNCS, enantiomorph decision): 4 cards 2.18 -> 1.74 s, 1 card 2.91 -> 2.71 s; - thaumatin (AF model, one copy, no tNCS): unchanged (1.62 / 2.49 s), it has no comparisons to overlap; - 9s02 (AF model, 4 cards): 8.53 -> 8.39 s, identical. [MolecularReplacement] and RigidBody* pass on 4 cards. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- .../MolecularReplacement.cpp | 157 ++++++++++++++---- 1 file changed, 123 insertions(+), 34 deletions(-) diff --git a/image_analysis/structure_refinement/MolecularReplacement.cpp b/image_analysis/structure_refinement/MolecularReplacement.cpp index 5fc50422f..63b7689a0 100644 --- a/image_analysis/structure_refinement/MolecularReplacement.cpp +++ b/image_analysis/structure_refinement/MolecularReplacement.cpp @@ -2500,7 +2500,10 @@ MolecularReplacementResult search_model(const SearchModel &model, const std::vec need.other_budgets = other_budgets; need.transform = transform_bytes; need.rigid_body = size.rigid_body; - need.rigid_body_engines = size.rigid_body_engines; + // Engines for the fits the choice runs at once besides: a null's replicates, beside them a carry and a group + // searched again (search_merge's work ahead of the choice). The layout of the transform was decided on the + // search's own fits. + need.rigid_body_engines = std::max(size.rigid_body_engines, NULL_REPLICATES + 1 + RESCORED_PER_GROUP); need.structure_factors = size.structure_factors; const MolecularReplacementGPUPlan plan = PlanMolecularReplacementGPU(need); const int cards = get_gpu_count(); @@ -2599,10 +2602,101 @@ MolecularReplacementResult search_model(const SearchModel &model, const std::vec return TranslationFunction(ft, setup, rotations, known, nthreads); }; + // The null of a candidate (null_sigma) placed in the orientation `orientation` before the rigid body. + const auto null_of = [&](const MRCandidate &c, const gemmi::Mat33 &orientation) { + const int merge = c.merge; + // One copy: its replicates placed by the translation function, as it was. + std::function(const std::vector &)> translate; + if (c.copies == 1) + translate = [&](const std::vector &turned) { + const SearchReflections refl = MakeSearchReflections(*merges[merge].merged, + gemmi::UnitCell(merges[merge].cell), + merges[merge].space_group, + result.d_min_search, result.d_max_search); + const std::vector tf = translation_function( + MakeTranslationSetup(refl, gemmi::UnitCell(merges[merge].cell), c.space_group), turned, + nullptr, nullptr); + std::vector out; + for (const TranslationPeak &p : tf) + out.push_back(p.t); + return out; + }; + return null_sigma(*c.placed, c.copies, orientation, c.space_group, *merges[merge].merged, + result.d_min_score, rigid_body_pool(merge), nthreads, &scorers, translate); + }; + // The comparisons of a candidate `b` the choice falls on with another of its merge, `c`, as choose_in() makes + // them: b's copies carried into c's group, where c placed fewer beside tNCS, and the paired R, where the two are + // within the R-free margin with as many copies. + const auto needs_carry = [&](const MRCandidate &c, const MRCandidate &b) { + return result.tncs.present && c.copies < b.copies + && (TncsRelatesGroups(c.space_group, b.space_group, result.tncs.vector) + || enantiomorphs(c.space_group, b.space_group)); + }; + const auto needs_paired = [&](const MRCandidate &c, const MRCandidate &b) { + return !(c.r_free - b.r_free >= R_FREE_MARGIN) && c.copies == b.copies; + }; + const auto carry_of = [&](const MRCandidate &c, const MRCandidate &b) { + return carry_copies(*b.placed, b.copies, c.space_group, *merges[b.merge].merged, result.d_min_score, + rigid_body_pool(b.merge), nthreads, &scorers); + }; + const auto paired_of = [&](const MRCandidate &c, const MRCandidate &b) { + return paired_r_z(*c.placed, c.space_group, *b.placed, b.space_group, *merges[b.merge].merged, + result.d_min_score, nthreads, &scorers); + }; + // The work of the choice started before the choice comes to it, beside what decides where it falls: the null of + // the placement it is expected to fall on, and that placement's comparisons with the others. Each is taken where + // the choice comes to the same placements; a null or a comparison is the same number whenever it is computed, so + // this changes when they run, never what they give. All are waited for before a merge's rigid-body engines are + // given back. + struct Ahead { + std::shared_ptr placed; // the one the choice falls on + std::shared_ptr other; // the one it is compared with; none for its null + std::shared_future null_sigma, paired_z; + std::shared_future carried; + }; + std::vector ahead; + const auto find_ahead = [&](const MRCandidate &b, const MRCandidate *c) -> const Ahead * { + for (const Ahead &a : ahead) + if (a.placed == b.placed && a.other == (c != nullptr ? c->placed : nullptr)) + return &a; + return nullptr; + }; + // `b`'s null (with its orientation before the rigid body), where it is not already started, and its comparisons + // with `others` - those of its merge that pack. + const auto start_ahead = [&](const MRCandidate &b, const gemmi::Mat33 *orientation, + const std::vector &others) { + if (orientation != nullptr && find_ahead(b, nullptr) == nullptr) + ahead.push_back({b.placed, nullptr, + std::async(std::launch::async, [&, b, o = *orientation] { return null_of(b, o); }).share(), + {}, {}}); + for (const MRCandidate &c : others) { + if (c.placed == b.placed || c.clash_fraction > CLASH_MAX || find_ahead(b, &c) != nullptr) + continue; + Ahead a{b.placed, c.placed, {}, {}, {}}; + if (needs_carry(c, b)) + a.carried = std::async(std::launch::async, [&, b, c] { return carry_of(c, b); }).share(); + if (needs_paired(c, b)) + a.paired_z = std::async(std::launch::async, [&, b, c] { return paired_of(c, b); }).share(); + if (a.carried.valid() || a.paired_z.valid()) + ahead.push_back(std::move(a)); + } + }; + const auto wait_ahead = [&] { + for (Ahead &a : ahead) + for (const std::shared_future *f : {&a.null_sigma, &a.paired_z}) + if (f->valid()) + f->wait(); + for (Ahead &a : ahead) + if (a.carried.valid()) + a.carried.wait(); + ahead.clear(); + }; + std::vector orientations; // of each candidate's placement, before the rigid body // One merge: the rotation function over its point group, then for each of its groups the // translation function and the best placements through packing, the rigid body and R-free. const auto search_merge = [&](size_t m) { + wait_ahead(); // before plan_gpu() gives the last merge's engines back const MRMerge &merge = merges[m]; const gemmi::UnitCell cell(merge.cell); const double d_min = result.d_min_score; @@ -3158,6 +3252,15 @@ MolecularReplacementResult search_model(const SearchModel &model, const std::vec for (size_t g = 0; g < found.size(); g++) if (found[g].cand.verdict == "PLACED" && (lead < 0 || found[g].cand.r_free < found[lead].cand.r_free)) lead = static_cast(g); + // The choice falls first on the placed candidate of lowest R-free (choose_in), which is this one unless a + // search again below finds a better: its null and its comparisons with the others are started now, beside + // those searches. + if (lead >= 0) { + std::vector others; + for (const GroupResult &gr : found) + others.push_back(gr.cand); + start_ahead(found[lead].cand, &found[lead].orientation, others); + } if (lead >= 0 && result.tncs.present) { std::vector> again(found.size()); for (size_t g = 0; g < found.size(); g++) @@ -3192,28 +3295,6 @@ MolecularReplacementResult search_model(const SearchModel &model, const std::vec // The null is built for the candidate the choice would fall on, and only for it: a placed candidate // that does not beat its own null is set aside, and the choice falls on the next. const auto choose_in = [&](int merge) -> int { - // The null of candidate i. - const auto run_null = [&](int i) { - const MRCandidate &c = result.candidates[i]; - // One copy: its replicates placed by the translation function, as it was. - std::function(const std::vector &)> translate; - if (c.copies == 1) - translate = [&](const std::vector &turned) { - const SearchReflections refl = MakeSearchReflections(*merges[merge].merged, - gemmi::UnitCell(merges[merge].cell), - merges[merge].space_group, - result.d_min_search, result.d_max_search); - const std::vector tf = translation_function( - MakeTranslationSetup(refl, gemmi::UnitCell(merges[merge].cell), c.space_group), turned, - nullptr, nullptr); - std::vector out; - for (const TranslationPeak &p : tf) - out.push_back(p.t); - return out; - }; - return null_sigma(*c.placed, c.copies, orientations[i], c.space_group, *merges[merge].merged, - result.d_min_score, rigid_body_pool(merge), nthreads, &scorers, translate); - }; // The first candidate's null alone; where it fails, the nulls of all the other placed candidates side // by side, each taken where the choice comes to it. std::vector nulls(result.candidates.size(), NAN); @@ -3317,9 +3398,20 @@ MolecularReplacementResult search_model(const SearchModel &model, const std::vec if (static_cast(i) != best && result.candidates[i].merge == merge && result.candidates[i].verdict == "PLACED") todo.push_back(static_cast(i)); - std::vector> running; - for (int i : todo) - running.push_back(std::async(std::launch::async, run_null, i)); + std::vector> running; + for (int i : todo) { + const MRCandidate &ci = result.candidates[i]; + const Ahead *a = find_ahead(ci, nullptr); + running.push_back(a != nullptr ? a->null_sigma + : std::async(std::launch::async, null_of, std::cref(ci), + std::cref(orientations[i])).share()); + } + // Its comparisons, beside its null, where they were not started before. + std::vector others; + for (const MRCandidate &o : result.candidates) + if (o.merge == merge) + others.push_back(o); + start_ahead(c, nullptr, others); for (size_t k = 0; k < todo.size(); k++) nulls[todo[k]] = running[k].get(); result.t_null += seconds_since(t_null); @@ -3337,16 +3429,14 @@ MolecularReplacementResult search_model(const SearchModel &model, const std::vec // With tNCS one copy alone fits no better than at random, so a group whose search stopped at // fewer copies than the chosen one's has not been shown unable to hold them - where the tNCS // vector makes up the groups' screw translations, or where they are each other's enantiomorph. - if (result.tncs.present && c.copies < result.candidates[best].copies - && (TncsRelatesGroups(c.space_group, result.candidates[best].space_group, result.tncs.vector) - || enantiomorphs(c.space_group, result.candidates[best].space_group))) { + const Ahead *a = find_ahead(result.candidates[best], &c); + if (needs_carry(c, result.candidates[best])) { // Unless the chosen copies, carried into this group as they are and refined, build a crystal // that fits clearly worse: where the tNCS vector does make up the difference between the groups, // they build the same one. const MRCandidate &b = result.candidates[best]; - const PlacementScore carried = carry_copies(*b.placed, b.copies, c.space_group, *merges[merge].merged, - result.d_min_score, rigid_body_pool(merge), nthreads, - &scorers); + const PlacementScore carried = a != nullptr && a->carried.valid() ? a->carried.get() + : carry_of(c, b); logger.Info("Molecular replacement: {}'s {} copies carried into {}: R-free {:.3f} against {:.3f}", b.space_group.xhm(), b.copies, c.space_group.xhm(), carried.r_free, b.r_free); if (!(carried.r_free - b.r_free >= R_FREE_MARGIN) || !(b.r_free < CARRY_R_FREE_MAX)) { @@ -3361,8 +3451,7 @@ MolecularReplacementResult search_model(const SearchModel &model, const std::vec // the chosen placement fits every reflection better by PAIRED_Z_MIN standard errors. const MRCandidate &b = result.candidates[best]; if (c.copies == b.copies) { - const double z = paired_r_z(*c.placed, c.space_group, *b.placed, b.space_group, - *merges[merge].merged, result.d_min_score, nthreads, &scorers); + const double z = a != nullptr && a->paired_z.valid() ? a->paired_z.get() : paired_of(c, b); logger.Info("Molecular replacement: {} and {} within {:.2f} in R-free, {} copies each: {} fits every " "reflection better by {:.1f} standard errors", b.space_group.xhm(), c.space_group.xhm(), R_FREE_MARGIN, b.copies, b.space_group.xhm(), z); -- 2.54.0 From 6f5c235fd76183a55f22b4150087afb69f462f8e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 19:35:59 +0200 Subject: [PATCH 24/53] Live molecular replacement: warmed up at model upload, searched beside the second pass and carried to the answer's merge - WarmUpModelValidation did nothing for a prediction (no crystal cell: "no atoms or no unit cell"), so a broker's first molecular replacement paid every first use on the GPU. A prediction is now warmed up by a search of its own (WarmUpMolecularReplacement): its search model in a P 1 21 1 cell four radii on each edge, that crystal's intensities to 2.5 A, the search, and the validation of what it placed. RunMolecularReplacement waits for a warm-up still running, as ValidateAgainstModel does. --simulate-live warms the model up as the broker does at upload. Result-identical. - fast_live with a prediction (or --mr): the search starts on the geometry pre-pass's merge, averaged into a group from that merge's own space-group search (to 3 A, standard setting), beside the second pass, on card 1 % cards, a quarter of the threads, under GPUWorkBeside. At the answer's merge CarryMolecularReplacement takes it over where it holds - same group and a cell within 1%, every copy refined by the rigid body beside the others against the answer's merge and its R-free within R_FREE_MARGIN of the one it was found at - and the merge is searched itself otherwise (and always where the twin hypothesis' merge is searched too). RESULT-CHANGING only in the refinement end point of the placement. Sim-live (10 ms/frame, RTX 5080): molecular-replacement step thaumatin 0.96 -> 0.31 s, cytochrome C 1.19 -> 0.25 s; merged reflections identical; thaumatin placed model identical (0.000 A), R-free 0.3765 both; cytochrome C a symmetry-equivalent placement, CA rmsd 0.03-0.05 A per copy, R-free 0.4443 -> 0.4431. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- .../structure_refinement/ModelValidation.cpp | 29 +++- .../structure_refinement/ModelValidation.h | 4 + .../MolecularReplacement.cpp | 135 +++++++++++++++++- .../MolecularReplacement.h | 28 ++++ rugnux/Rugnux.h | 5 + rugnux/RugnuxScaleMerge.cpp | 130 ++++++++++++++++- rugnux/rugnux_cli.cpp | 3 + tests/MolecularReplacementTest.cpp | 48 +++++++ 8 files changed, 376 insertions(+), 6 deletions(-) diff --git a/image_analysis/structure_refinement/ModelValidation.cpp b/image_analysis/structure_refinement/ModelValidation.cpp index db415c610..7aa5a6692 100644 --- a/image_analysis/structure_refinement/ModelValidation.cpp +++ b/image_analysis/structure_refinement/ModelValidation.cpp @@ -41,6 +41,7 @@ #include "ModelFFT.h" #include "ModelGrid.h" // PutModelDensityOnGrid, PutMaskOnGrid #include "RigidBodyRefine.h" +#include "MolecularReplacement.h" // the warm-up of a prediction #include "SigmaA.h" #ifdef JFJOCH_USE_CUDA #include "ModelStructureFactorsGPU.h" @@ -1964,8 +1965,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector & const ModelValidationSchedule &schedule, const gemmi::Op *twin_law, const gemmi::Structure *placed) { - if (const auto w = LastWarmUp(); w.valid()) - w.wait(); + WaitForModelWarmUp(); #ifdef JFJOCH_USE_CUDA // A CUDA failure ends the validation, not the run, and nothing is moved to the CPU part way through: // the validation reports why it did not finish, and a validation that did not finish decides nothing, @@ -1989,6 +1989,11 @@ ModelValidationResult ValidateAgainstModel(const std::vector & #endif } +void WaitForModelWarmUp() { + if (const auto w = LastWarmUp(); w.valid()) + w.wait(); +} + void WarmUpModelValidation(std::shared_ptr model, size_t nthreads, Logger logger) { #ifdef JFJOCH_USE_CUDA if (!model || get_gpu_count() == 0) @@ -2002,6 +2007,26 @@ void WarmUpModelValidation(std::shared_ptr model, size_t nthr // Held for as long as it runs: work beside a run waits for it where the card is too small for both. GPUWorkBeside beside; Logger quiet = Logger::Buffered(); + if (const SearchModel search = PrepareSearchModel(*model); search.failure_reason.empty() && search.predicted) { + std::string failure; + try { + const MolecularReplacementWarmUp w = WarmUpMolecularReplacement(*model, nthreads); + failure = w.failure_reason; + if (w.placed) { + ModelValidationSchedule placed_elsewhere; + placed_elsewhere.fit_test = false; + failure = Validate(w.merged, w.cell, *model, quiet, &w.space_group, false, nthreads, 0.0, {}, + placed_elsewhere, nullptr, true, w.placed.get()).failure_reason; + } + } catch (const std::exception &e) { + cuda_clear_error(); + failure = e.what(); + } + logger.Info("Molecular replacement warm-up on the GPU: {:.2f} s{}", + std::chrono::duration(std::chrono::steady_clock::now() - start).count(), + failure.empty() ? "" : " (did not finish: " + failure + ")"); + return; + } const double d_min = model->structure.resolution > 0.0 ? model->structure.resolution : 2.0; std::vector fake = ModelReferenceIntensities(*model, std::nullopt, nullptr, d_min, quiet); for (size_t i = 0; i < fake.size(); i++) { diff --git a/image_analysis/structure_refinement/ModelValidation.h b/image_analysis/structure_refinement/ModelValidation.h index 0eebb653b..9551ff0f9 100644 --- a/image_analysis/structure_refinement/ModelValidation.h +++ b/image_analysis/structure_refinement/ModelValidation.h @@ -416,4 +416,8 @@ std::vector ModelReferenceIntensities(const RugnuxModel &model // calls it when a model is uploaded, before the collection starts. ValidateAgainstModel waits for a // warm-up still running, so a real validation never shares the cards with one and computes exactly // what it would have without it. Its time goes to `logger` when it ends. No-op without a GPU. +// A prediction, which a run places by molecular replacement and validates only as placed there, is warmed up by a +// search instead (WarmUpMolecularReplacement) and the validation of what it placed. void WarmUpModelValidation(std::shared_ptr model, size_t nthreads, Logger logger); +// Returns once the warm-up last started has ended (at once where there is none). +void WaitForModelWarmUp(); diff --git a/image_analysis/structure_refinement/MolecularReplacement.cpp b/image_analysis/structure_refinement/MolecularReplacement.cpp index 63b7689a0..816f0dc18 100644 --- a/image_analysis/structure_refinement/MolecularReplacement.cpp +++ b/image_analysis/structure_refinement/MolecularReplacement.cpp @@ -3713,8 +3713,10 @@ std::vector SplitTranslationFunction(const std::vector &merges, - Logger &logger, size_t nthreads, size_t gpu_resident_bytes) { +namespace { + +MolecularReplacementResult run_molecular_replacement(const RugnuxModel &input, const std::vector &merges, + Logger &logger, size_t nthreads, size_t gpu_resident_bytes) { const auto t0 = std::chrono::steady_clock::now(); // What the search may hold on the card: the budget model validation keeps to. What the stream-ordered pool // holds with no buffer in it - freed by the steps before - goes back to the driver first, as before a @@ -3822,6 +3824,135 @@ MolecularReplacementResult RunMolecularReplacement(const RugnuxModel &input, con return result; } +} // namespace + +MolecularReplacementResult RunMolecularReplacement(const RugnuxModel &model, const std::vector &merges, + Logger &logger, size_t nthreads, size_t gpu_resident_bytes) { + WaitForModelWarmUp(); + return run_molecular_replacement(model, merges, logger, nthreads, gpu_resident_bytes); +} + +std::optional CarryMolecularReplacement(const MolecularReplacementResult &early, + const MRMerge &searched, const MRMerge &merge, + Logger &logger, size_t nthreads, + size_t gpu_resident_bytes) { + if (!early.ok || early.chosen < 0 || early.candidates[early.chosen].merge != 0) { + logger.Info("Molecular replacement: the search on the {} placed nothing ({}) - searched again", + searched.name, early.ok ? early.decision : early.failure_reason); + return std::nullopt; + } + if (searched.space_group.xhm() != merge.space_group.xhm() || !merge.cell.is_close(searched.cell, 0.01, 1.0)) { + logger.Info("Molecular replacement: the search on the {} was made in {} ({:.2f} {:.2f} {:.2f}), this merge is in " + "{} ({:.2f} {:.2f} {:.2f}) - searched again", searched.name, searched.space_group.xhm(), + searched.cell.a, searched.cell.b, searched.cell.c, merge.space_group.xhm(), merge.cell.a, + merge.cell.b, merge.cell.c); + return std::nullopt; + } + const auto t0 = std::chrono::steady_clock::now(); + const MRCandidate &c = early.candidates[early.chosen]; + const gemmi::UnitCell cell(merge.cell); + const gemmi::SpaceGroup &sg = c.space_group; // the group it was placed in, one of the merge's (SameRotationGroups) + double data_d_min = 1e9; + for (const MergedReflection &r : *merge.merged) + if (r.d > 0) + data_d_min = std::min(data_d_min, r.d); + const double d_min = std::max(SCORE_D_MIN, data_d_min); + // Every copy at its fractional position in this cell. + std::vector body = SplitCopies(*c.placed, c.copies); + for (gemmi::Structure &b : body) { + std::vector pos = ModelPositions(b.models[0]); + for (gemmi::Position &x : pos) + x = cell.orthogonalize(b.cell.fractionalize(x)); + SetModelPositions(b.models[0], pos); + b.cell = cell; + b.spacegroup_hm = sg.xhm(); + b.setup_cell_images(); + } + RigidBodyGPUPool *pool = nullptr; + ScoreEngines scorers; +#ifdef JFJOCH_USE_CUDA + // One rigid-body engine and the score engine, on this card, as a search's own (size_merge_gpu). + std::unique_ptr engines; + if (get_gpu_count() > 0) { + const int device = RigidBodyGPUEngine::CurrentDevice(); + const size_t budget = ModelGPUBudget(get_gpu_total_memory(device), gpu_resident_bytes); + size_t observations = 0; + for (const MergedReflection &r : *merge.merged) + if (!std::isnan(r.F) && r.d >= RigidBodyLadder(d_min).back()) + ++observations; + Logger quiet = Logger::Buffered(); + const gemmi::Structure all = AssembleCopies(body); + engines = RigidBodyGPUPool::Plan(body[0].models[0], cell, sg, d_min, observations, quiet); + if (engines && engines->Cover(all.models[0], sg, quiet) && engines->EngineBytes() <= budget / 2) { + engines->Reserve({1}); + pool = engines.get(); + } + scorers.SetRoom({{device, budget - (pool != nullptr ? engines->EngineBytes() : 0)}}, device, true); + } +#endif + for (size_t k = 0; k < body.size(); k++) { + std::vector rest; + for (size_t o = 0; o < body.size(); o++) + if (o != k) + rest.push_back(body[o]); + if (rest.empty()) { + refine_rigid_body(body[k], sg, *merge.merged, d_min, pool, nthreads); + } else { + const gemmi::Structure fixed = AssembleCopies(rest); + refine_rigid_body(body[k], sg, *merge.merged, d_min, pool, nthreads, nullptr, &fixed.models[0]); + } + } + auto placed = std::make_shared(AssembleCopies(body)); + const PlacementScore score = score_placement(*placed, sg, *merge.merged, d_min, nthreads, &scorers); + const bool holds = score.r_free <= c.r_free + R_FREE_MARGIN; + logger.Info("Molecular replacement: the search on the {} ({}): its placement in {} ({} copies, R-free {:.3f} there) refined against " + "this merge: R-work {:.3f} R-free {:.3f} ({:.2f} s) - {}", searched.name, + early.decision, sg.xhm(), c.copies, c.r_free, score.r_work, score.r_free, + seconds_since(t0), holds ? "it holds, not searched again" : "it does not hold, searched again"); + if (!holds) + return std::nullopt; + MolecularReplacementResult out = early; + MRCandidate &chosen = out.candidates[out.chosen]; + out.r_free_searched = chosen.r_free; + chosen.placed = placed; + chosen.r_work = score.r_work; + chosen.r_free = score.r_free; + chosen.k_sol = score.k_sol; + chosen.b_sol = score.b_sol; + out.d_min_score = d_min; + return out; +} + +MolecularReplacementWarmUp WarmUpMolecularReplacement(const RugnuxModel &model, size_t nthreads) { + MolecularReplacementWarmUp out; + const SearchModel search = PrepareSearchModel(model); + if (!search.failure_reason.empty()) { + out.failure_reason = search.failure_reason; + return out; + } + const double edge = 4.0 * search.radius_A; + out.cell = UnitCell{static_cast(edge), static_cast(edge + 2.0), static_cast(edge + 4.0), + 90.0f, 100.0f, 90.0f}; + out.space_group = *gemmi::find_spacegroup_by_name("P 1 21 1"); + RugnuxModel crystal; + crystal.name = model.name; + crystal.structure = PlaceSearchModel(search, gemmi::Mat33(), gemmi::Fractional(0.1, 0.2, 0.3), + gemmi::UnitCell(out.cell), out.space_group); + Logger quiet = Logger::Buffered(); + out.merged = ModelReferenceIntensities(crystal, std::nullopt, nullptr, 2.5, quiet); + for (size_t i = 0; i < out.merged.size(); i++) { + out.merged[i].F = std::sqrt(out.merged[i].I); + out.merged[i].rfree_flag = i % 20 == 0; + } + const MolecularReplacementResult mr = run_molecular_replacement( + model, {{"MERGE", &out.merged, out.cell, out.space_group}}, quiet, nthreads, 0); + if (mr.chosen >= 0) + out.placed = mr.candidates[mr.chosen].placed; + else + out.failure_reason = mr.ok ? mr.decision : mr.failure_reason; + return out; +} + void CheckEnantiomorphHand(MolecularReplacementResult &result, const std::vector &merged, const gemmi::SpaceGroup &data_space_group, Logger &logger, size_t nthreads, size_t gpu_resident_bytes) { diff --git a/image_analysis/structure_refinement/MolecularReplacement.h b/image_analysis/structure_refinement/MolecularReplacement.h index 8c90974a3..d56d402f0 100644 --- a/image_analysis/structure_refinement/MolecularReplacement.h +++ b/image_analysis/structure_refinement/MolecularReplacement.h @@ -374,6 +374,9 @@ struct MolecularReplacementResult { // Seconds per step. double t_model = 0, t_transform = 0, t_rotation = 0, t_translation = 0, t_rescore = 0, t_null = 0; double t_domains = 0; // the PAE domains' own searches, wall clock (they run side by side) + // Searched on another merge of the crystal and carried over to this one (CarryMolecularReplacement): the + // chosen placement's R-free on the merge it was found on, NaN where this merge was searched itself. + double r_free_searched = NAN; }; // What a molecular-replacement search puts on its GPU, for one merge: the molecular transform the rotation and @@ -450,9 +453,34 @@ bool PackMolecularTransformGPU(const MolecularReplacementGPUNeed &need, size_t p // every other by an R-free margin; it chooses the twin hypothesis' merge only where none of the // answer's groups can hold the model at all (Matthews, packing) and a group of the subgroup can. // `gpu_resident_bytes`: what is held on the card meanwhile by what came before (a merge engine still at work). +// Waits for a model warm-up still running (WarmUpModelValidation), as a validation does. MolecularReplacementResult RunMolecularReplacement(const RugnuxModel &model, const std::vector &merges, Logger &logger, size_t nthreads, size_t gpu_resident_bytes = 0); +// A search made on another merge of the same crystal - `searched`, the geometry pre-pass's in a live run - taken +// over to `merge` where it holds there: the same space group, a cell within 1%, and the chosen placement - every +// copy refined by the rigid body beside the others against `merge`, and scored there - fitting it within +// R_FREE_MARGIN in R-free of how it fitted the merge it was found on. Its decision stands as it was taken; the +// placement and its R-factors are `merge`'s. Empty where it does not hold, and `merge` is searched itself. +std::optional CarryMolecularReplacement(const MolecularReplacementResult &early, + const MRMerge &searched, const MRMerge &merge, + Logger &logger, size_t nthreads, + size_t gpu_resident_bytes = 0); + +// A search for a prediction ahead of its run, which pays for what the GPU loads on first use - the kernels of the +// transform, the rotation and translation functions, the rigid body and the scores, on every card the fits are +// spread over. The crystal is made up from the model alone: its search model in one orientation in a P 1 21 1 cell +// four of its radii on each edge, and that crystal's own intensities to 2.5 A as the merge. The search is +// RunMolecularReplacement's, without the wait; `placed` is what it placed, null where it placed nothing. +struct MolecularReplacementWarmUp { + std::vector merged; + UnitCell cell; + gemmi::SpaceGroup space_group; + std::shared_ptr placed; + std::string failure_reason; +}; +MolecularReplacementWarmUp WarmUpMolecularReplacement(const RugnuxModel &model, size_t nthreads); + // The hand check of an ENANTIOMORPH decision, filled into result.hand_check: taken into the answer by // --model's rule for a model's enantiomorph - the model has to fit against its own null, which the choice // already required of it, and the anomalous map at its atoms (`merged`, the answer's merge in diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 03eb8bab2..776aa617c 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -1227,6 +1227,11 @@ class Rugnux { }; std::shared_ptr speculative_probe_; bool speculate_geometry_probe_ = false; + // fast_live, with a prediction (or --mr): its molecular replacement, started on the geometry pre-pass's merge + // and run beside the second pass; the answer's merge takes it over where it holds there + // (CarryMolecularReplacement) and searches itself where not. RugnuxScaleMerge.cpp. + struct EarlyMolecularReplacement; + std::shared_ptr early_mr_; // Whether this run is a probe pass running beside the main line under a GPUWorkBeside - the // speculative probe, the rotation-scale walk's indexing probes. Its indexer pools then wait for the // main line's image pass, never for that work, being that work themselves (IndexerThreadPool). diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 54951ad7f..661941763 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -347,6 +347,47 @@ namespace { } } +// The molecular replacement of a live run started on the geometry pre-pass's merge (Rugnux.h): the merge it +// searches - the pre-pass's P1 merge averaged into a group of its own search, on the axes the answer's files +// would be written on - and the search, its log held until the answer's merge reads it. +struct Rugnux::EarlyMolecularReplacement { + std::vector merged; + UnitCell cell; + gemmi::SpaceGroup space_group; + Logger log = Logger::Buffered(); + std::future run; +}; + +// A P1 merge averaged into the asymmetric unit of `sg`, each reflection weighted by 1/sigma^2; the +// reflections `sg` makes absent left out. +static std::vector AverageIntoGroup(const std::vector &p1, + const gemmi::SpaceGroup &sg, const UnitCell &cell) { + const HKLKeyGenerator key_gen(/*merge_friedel=*/true, sg); + std::map, std::pair> sums; // sum of w I, sum of w + for (const MergedReflection &r : p1) { + if (!std::isfinite(r.I) || !(r.sigma > 0.0f) || key_gen.IsSystematicallyAbsent(r.h, r.k, r.l)) + continue; + const HKLKey key = key_gen(r.h, r.k, r.l); + const double w = 1.0 / (static_cast(r.sigma) * r.sigma); + auto &[wi, ws] = sums[{key.h, key.k, key.l}]; + wi += w * r.I; + ws += w; + } + const gemmi::UnitCell uc(cell); + std::vector out; + for (const auto &[hkl, sum] : sums) { + MergedReflection m; + m.h = hkl[0]; + m.k = hkl[1]; + m.l = hkl[2]; + m.I = static_cast(sum.first / sum.second); + m.sigma = static_cast(1.0 / std::sqrt(sum.second)); + m.d = static_cast(uc.calculate_d({{m.h, m.k, m.l}})); + out.push_back(m); + } + return out; +} + // Proper rotations of a crystal system's holohedry - what a lattice of that class carries, and the // bound SearchSpaceGroup caps its point-group candidates at. static size_t HolohedryRotations(gemmi::CrystalSystem system) { @@ -1534,6 +1575,72 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { if (geometry_prepass && rsm && experiment_.IsRotationIndexing()) post_refine_geometry(); + // Live, the molecular replacement of a prediction (or of --mr) starts here, on this pass's merge, and runs + // beside the second pass on a card of its own where there is one, with a quarter of the threads. It is a + // hypothesis: the answer's merge takes its placement over only where it holds there (CarryMolecularReplacement) + // and searches itself otherwise. Its group is a quick one of its own, from this merge's search to the + // resolution the placements are scored at - the pre-pass decides no group for the run - so the merge searched is + // this one averaged into that group, on the axes the answer's files would be written on. + if (geometry_prepass && config_.fast_live && config_.model && result.consensus_cell + && config_.reference_data.empty() && (config_.force_mr || PrepareSearchModel(*config_.model).predicted)) { + early_mr_ = std::make_shared(); + early_mr_->run = std::async(std::launch::async, + [early = early_mr_.get(), p1 = sm.merged, cell = *result.consensus_cell, fixed_sg = initial_sg, + lattice = end_msg.rotation_lattice_type, isa = result.error_model_isa, model = config_.model, + rfree_fraction = experiment_.GetScalingSettings().GetRfreeFraction(), + nthreads = static_cast(std::max(1, config_.nthreads / 4))]() mutable { + GPUWorkBeside beside; + if (const int32_t cards = get_gpu_count(); cards > 0) + pin_gpu(1 % cards); + gemmi::SpaceGroup sg; + if (fixed_sg) { + sg = *fixed_sg; + } else { + SearchSpaceGroupOptions o; + o.nthreads = nthreads; + o.merge_friedel = true; + o.d_min_limit_A = 3.0; + if (lattice) { + o.lattice_system = lattice->crystal_system; + o.lattice_centring = lattice->centering; + } + o.cell = gemmi::UnitCell(cell); + o.enumerate_all_settings = true; + o.enumerate_all_rotation_sets = true; + o.merge_isa = isa; + const SearchSpaceGroupResult found = SearchSpaceGroup(p1, o); + if (found.best_space_group) + sg = *found.best_space_group; + else if (found.point_group_representative) + sg = *found.point_group_representative; + else { + MolecularReplacementResult none; + none.failure_reason = "no space group from the pre-pass merge"; + return none; + } + } + const gemmi::Op cob = ChooseOutputSetting(cell, sg); + if (const gemmi::SpaceGroup *moved = SpaceGroupInBasis(sg, cob); + !(cob == gemmi::Op::identity()) && moved != nullptr) { + p1 = ReindexReflections(p1, HklOperator(cob)); + cell = CellInBasis(cell, cob); + sg = *moved; + } + early->merged = AverageIntoGroup(p1, sg, cell); + early->cell = cell; + early->space_group = sg; + AssignRfreeFlags(early->merged, sg, rfree_fraction, 500, cell, nthreads); + FrenchWilsonOptions fw_opts; + fw_opts.num_threads = nthreads; + fw_opts.anisotropy_hkl = AnisotropyPriorHKL(early->merged, gemmi::UnitCell(cell), &sg); + ApplyFrenchWilson(early->merged, sg, fw_opts); + early->log.Info("Molecular replacement on the geometry pre-pass's merge, in {} from its own search, beside " + "the second pass", sg.xhm()); + return RunMolecularReplacement(*model, {{"geometry pre-pass's merge", &early->merged, cell, sg}}, + early->log, nthreads); + }); + } + // For a de-novo search, gate the P1 scaling+merge (the pass that feeds the space-group search) at // >= 1 over thin resolution shells: noise-dominated high-res shells otherwise mislead // the determination, where a single artifact (a zinger the box sum integrates whole, or a @@ -4444,8 +4551,27 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { if (model_twin) merges.push_back({"TWIN_SUBGROUP_MERGE", &model_twin->merged, model_twin->cell, model_twin->space_group, model_twin->law}); - MolecularReplacementResult mr = RunMolecularReplacement(*config_.model, merges, logger, - static_cast(config_.nthreads), merge_on_card); + // The search started on the geometry pre-pass's merge, where there is one, taken over where it holds on + // this one. Not where the twin hypothesis' merge is searched too: that search is this merge's own. + std::optional carried; + if (early_mr_ && !model_twin) { + TimingMark early_mark(logger, "early molecular replacement wait"); + try { + MolecularReplacementResult early = early_mr_->run.get(); + early_mark.Stop(); + early_mr_->log.ReplayInto(logger); + carried = CarryMolecularReplacement( + early, {"geometry pre-pass's merge", &early_mr_->merged, early_mr_->cell, early_mr_->space_group}, + merges[0], logger, static_cast(config_.nthreads), merge_on_card); + } catch (const std::exception &e) { + logger.Warning("Molecular replacement on the geometry pre-pass's merge failed ({}) - searched again", + e.what()); + } + } + MolecularReplacementResult mr = carried ? std::move(*carried) + : RunMolecularReplacement(*config_.model, merges, logger, + static_cast(config_.nthreads), + merge_on_card); if (mr.chosen >= 0) { const MRCandidate &c = mr.candidates[mr.chosen]; if (mr.decision == "ENANTIOMORPH") diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index ea281a597..2a1ff9f88 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -3186,6 +3186,9 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { if (simulate_live) logger.Info("Simulated live collection: image i readable (i+1) x {:.1f} ms after the run starts", simulated_frame_time.count() / 1000.0); + // As the broker does when a model is uploaded, before the collection it is for starts. + if (simulate_live && model) + WarmUpModelValidation(model, static_cast(config.nthreads), logger); simulated_start = std::chrono::steady_clock::now(); Rugnux process(reader, experiment, *dataset->pixel_mask, config); diff --git a/tests/MolecularReplacementTest.cpp b/tests/MolecularReplacementTest.cpp index 438144573..696e8fd06 100644 --- a/tests/MolecularReplacementTest.cpp +++ b/tests/MolecularReplacementTest.cpp @@ -646,6 +646,54 @@ TEST_CASE("MolecularReplacement_RigidBodyBesideFixedCopy", "[MolecularReplacemen #endif } +// A search made on one merge, carried over to another merge of the same crystal in a cell 0.5% longer: the +// placement, a degree and half an angstrom out where it was found, is refined against the new merge onto the +// truth and holds there; in another group it is not carried at all, and the merge is to be searched itself. +TEST_CASE("MolecularReplacement_CarryToAnotherMerge", "[MolecularReplacement]") { + const SearchModel model = synthetic_model(); + const gemmi::SpaceGroup &sg = gemmi::get_spacegroup_by_name("P 21 21 21"); + const gemmi::UnitCell found_cell(52, 60, 70, 90, 90, 90), cell(52.26, 60.3, 70.35, 90, 90, 90); + const gemmi::Mat33 r = axis_angle({1, 2, 3}, 50); + const gemmi::Fractional t(0.17, 0.31, 0.09); + const gemmi::Structure truth = PlaceSearchModel(model, r, t, cell, sg); + auto merged = observed(truth); + for (size_t i = 0; i < merged.size(); i++) + merged[i].rfree_flag = i % 10 == 0; + + MolecularReplacementResult early; + early.ok = true; + early.chosen = 0; + early.decision = "AGREES"; + MRCandidate c; + c.space_group = sg; + c.copies = 1; + c.r_free = 0.30; + c.placed = std::make_shared(PlaceSearchModel( + model, axis_angle({0, 0, 1}, 1).multiply(r), gemmi::Fractional(t.x + 0.5 / 52, t.y, t.z), found_cell, sg)); + early.candidates.push_back(c); + + const UnitCell found_uc{52, 60, 70, 90, 90, 90}, uc{52.26f, 60.3f, 70.35f, 90, 90, 90}; + const MRMerge searched{"first merge", &merged, found_uc, sg}; + Logger logger("MolecularReplacement_CarryToAnotherMerge"); + const auto carried = CarryMolecularReplacement(early, searched, {"MERGE", &merged, uc, sg}, logger, 4); + REQUIRE(carried); + CHECK(carried->decision == "AGREES"); + CHECK(carried->r_free_searched == Catch::Approx(0.30)); + CHECK(carried->candidates[0].placed->cell.a == Catch::Approx(52.26)); + const auto worst = [&](const gemmi::Structure &st) { + const auto a = ModelPositions(st.models[0]), b = ModelPositions(truth.models[0]); + double w = 0; + for (size_t i = 0; i < a.size(); i++) + w = std::max(w, a[i].dist(b[i])); + return w; + }; + CHECK(worst(*c.placed) > 0.5); + CHECK(worst(*carried->candidates[0].placed) < 0.2); + + const gemmi::SpaceGroup &other = gemmi::get_spacegroup_by_name("P 2 21 21"); + CHECK_FALSE(CarryMolecularReplacement(early, searched, {"MERGE", &merged, uc, other}, logger, 4)); +} + #ifdef JFJOCH_USE_CUDA TEST_CASE("MolecularReplacement_GPUMatchesCPU", "[MolecularReplacement]") { if (get_gpu_count() == 0) -- 2.54.0 From 9e09cd4b3d9449944afa041a441c8c6b5da3a4c2 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 12:16:32 +0200 Subject: [PATCH 25/53] broker: initialize drops the dataset's bearer tokens and everything they protected Initialize is the fresh start between users. It already cleared the images, plots, scan result and live rugnux status; it now also drops the tokens, resets the dataset settings (name, file prefix, unit cell, ...) to their defaults and forgets the last collection followed through /live/events, so after it nothing of the previous dataset is readable, with or without its token - the broker is as open and as empty as a freshly started one. The auth test checks that after /initialize the data-collection statistics and the /statistics aggregate are open and no longer carry the previous dataset's name. SECURITY.md also lists /rugnux/status among the protected endpoints. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- broker/JFJochStateMachine.cpp | 7 ++++++- broker/jfjoch_api.yaml | 1 + docs/SECURITY.md | 5 ++++- image_pusher/LiveCollection.cpp | 10 ++++++++++ image_pusher/LiveCollection.h | 2 ++ tests/BrokerHttpAuthTest.cpp | 18 ++++++++++++++++-- 6 files changed, 39 insertions(+), 4 deletions(-) diff --git a/broker/JFJochStateMachine.cpp b/broker/JFJochStateMachine.cpp index cdae150db..2ce46530e 100644 --- a/broker/JFJochStateMachine.cpp +++ b/broker/JFJochStateMachine.cpp @@ -330,8 +330,13 @@ void JFJochStateMachine::Initialize() { start_exception = nullptr; // Re-initialising discards a pending start failure // A fresh start: no images, plots or live rugnux status from before, and no live rugnux analysis - // or model left over from the previous user. + // or model left over from the previous user. The previous dataset's tokens go too, and with them + // everything they protected: its settings (name, cell, ...) and the record of its collection. services.ClearResults(); + tokens.Replace({}); + experiment.ImportDatasetSettings(DatasetSettings()); + if (auto *live = services.GetLiveCollection()) + live->Forget(); { std::unique_lock ul2(rugnux_mutex); RugnuxLiveSettings s = services.GetRugnuxLive(); diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index c492e795b..866d1a2e4 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -2918,6 +2918,7 @@ paths: During operation of the detector it is recommended to use the `POST /pedestal` operation instead. If storage cells are used, the execution time might be few minutes. It also clears the image buffer, the previous collection's plots and live rugnux status, disables live rugnux analysis and removes the rugnux model. + It drops the previous dataset's bearer tokens and resets its settings, so nothing of that dataset stays readable. This is async function - one needs to use `POST /wait_till_done` to ensure operation is done. responses: diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 4893b63a2..ee3ace785 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -101,7 +101,9 @@ users cannot read this one. No endpoint returns the tokens; the broker only comp An accepted `/start` also clears what the previous run left readable - its statistics, plots and buffered images - in the same step that installs the new tokens, so the previous run is never served under the new tokens, and the new run's name never under the old ones. A refused `/start` -(wrong state, invalid settings) changes nothing. +(wrong state, invalid settings) changes nothing. `/initialize` drops the tokens together with +everything they protected - the dataset's settings, results, images, plots, live rugnux status and +the record of its collection - so the broker is left as open and as empty as a freshly started one. | Endpoint | With tokens set | |----------|-----------------| @@ -110,6 +112,7 @@ served under the new tokens, and the new run's name never under the old ones. A | `/image_buffer/start.cbor`, `/image_buffer/image.cbor`, `/image_buffer/image.jpeg`, `/image_buffer/image.tiff` (the images and the start message) | 401 without a token | | `/preview/plot`, `/preview/plot.bin` (per-image plots, unit cell) | 401 without a token | | `/live/events` (the start message and the data files of the collection being written) | 401 without a token | +| `/rugnux/status` (live rugnux run of the collection: cell, space group, results) | 401 without a token | | `/statistics` (the aggregate the web UI polls) | 200, but the `measurement` block is omitted | | everything else (`/status`, `/config/*`, `/start`, `/cancel`, masks, pedestal, ...) | open | diff --git a/image_pusher/LiveCollection.cpp b/image_pusher/LiveCollection.cpp index d40f5ac56..6e40e3767 100644 --- a/image_pusher/LiveCollection.cpp +++ b/image_pusher/LiveCollection.cpp @@ -68,6 +68,16 @@ void LiveCollection::EndIfOpen(const std::string &error) { End(0, error); } +void LiveCollection::Forget() { + { + std::unique_lock ul(m); + state = Snapshot{}; + start_message = StartMessage{}; + start_cbor.clear(); + } + cv.notify_all(); +} + std::optional> LiveCollection::StartCBOR() const { std::unique_lock ul(m); if (!state.started) diff --git a/image_pusher/LiveCollection.h b/image_pusher/LiveCollection.h index d35cfbf67..4056c699d 100644 --- a/image_pusher/LiveCollection.h +++ b/image_pusher/LiveCollection.h @@ -51,6 +51,8 @@ public: void End(uint64_t images, const std::string &error = ""); // The measurement is over: ends a collection the image pusher did not end, with this error. void EndIfOpen(const std::string &error); + // Back to "no collection since the broker started" (/initialize): nothing of the last one is kept. + void Forget(); // The start message of the current collection; nothing before it is known. std::optional> StartCBOR() const; diff --git a/tests/BrokerHttpAuthTest.cpp b/tests/BrokerHttpAuthTest.cpp index 3af02f883..edebb0c0e 100644 --- a/tests/BrokerHttpAuthTest.cpp +++ b/tests/BrokerHttpAuthTest.cpp @@ -110,6 +110,20 @@ TEST_CASE("JFJochBrokerHttp_BearerTokens", "[broker]") { REQUIRE(client.Get("/live/events", good)->status == 409); } + SECTION("initialize drops the tokens and everything they protected") { + REQUIRE(client.Post("/initialize")->status == 200); + for (int i = 0; i < 120 && nlohmann::json::parse(client.Get("/status")->body).at("state") != "Idle"; i++) + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + auto res = client.Get("/statistics/data_collection"); + REQUIRE(res->status == 200); + REQUIRE(res->body.find("protected_run") == std::string::npos); + auto stats = client.Get("/statistics"); + REQUIRE(stats->status == 200); + REQUIRE(stats->body.find("protected_run") == std::string::npos); + REQUIRE(client.Get("/rugnux/status")->status == 200); + REQUIRE(client.Get("/result/scan")->status != 401); + } + SECTION("the control plane and the broker status stay open") { REQUIRE(client.Get("/status")->status == 200); REQUIRE(client.Get("/image_buffer/status")->status == 200); @@ -122,11 +136,11 @@ TEST_CASE("JFJochBrokerHttp_BearerTokens", "[broker]") { nlohmann::json body = {{"beam_x_pxl", 1000}, {"beam_y_pxl", 1000}, {"detector_distance_mm", 100}, {"incident_energy_keV", 12.4}}; REQUIRE(client.Post("/start", body.dump(), "application/json")->status == 500); - REQUIRE(client.Post("/initialize")->status == 200); REQUIRE(client.Get("/statistics/data_collection")->status == 401); REQUIRE(client.Get("/statistics/data_collection", good)->status == 200); - // The next start without tokens lifts the protection. + // Back from Inactive; the next start without tokens leaves the dataset open. + REQUIRE(client.Post("/initialize")->status == 200); start(nlohmann::json::object()); REQUIRE(client.Get("/statistics/data_collection")->status == 200); REQUIRE(client.Get("/statistics/data_collection", wrong)->status == 200); -- 2.54.0 From 923a21014270ec43404fe87d647e4ce889f30d70 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 12:04:39 +0200 Subject: [PATCH 26/53] WIP: SACLA MPCCD (Cheetah-SACLA HDF5) stills reader for rugnux - not built, not tested Reads the per-hit /tag-*/data int16 8192x512 stacks (eight 1024x512 sensors), places the sensors on one flat image from /metadata posx/posy/angle (or a refined CrystFEL .geom via --crystfel-geometry), masks gaps, 5 px sensor edges and the amplifier-shield rows, converts ADU to photons at 0.00125 ADU/eV, and decodes chunks (shuffle+deflate) outside the HDF5 lock. Co-Authored-By: Claude Opus 5.5 --- reader/CMakeLists.txt | 2 + reader/JFJochMPCCDReader.cpp | 350 +++++++++++++++++++++++++++++++++++ reader/JFJochMPCCDReader.h | 78 ++++++++ rugnux/rugnux_cli.cpp | 23 ++- 4 files changed, 451 insertions(+), 2 deletions(-) create mode 100644 reader/JFJochMPCCDReader.cpp create mode 100644 reader/JFJochMPCCDReader.h diff --git a/reader/CMakeLists.txt b/reader/CMakeLists.txt index 0d6350514..16f137359 100644 --- a/reader/CMakeLists.txt +++ b/reader/CMakeLists.txt @@ -12,6 +12,8 @@ ADD_LIBRARY(JFJochReader STATIC MarCCD.h JFJochMarCCDReader.cpp JFJochMarCCDReader.h + JFJochMPCCDReader.cpp + JFJochMPCCDReader.h SMV.cpp SMV.h JFJochSMVReader.cpp diff --git a/reader/JFJochMPCCDReader.cpp b/reader/JFJochMPCCDReader.cpp new file mode 100644 index 000000000..a783fab24 --- /dev/null +++ b/reader/JFJochMPCCDReader.cpp @@ -0,0 +1,350 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "JFJochMPCCDReader.h" + +#include +#include +#include +#include +#include +#include + +#include + +#include "../common/JFJochException.h" +#include "../common/Logger.h" +#include "../writer/HDF5Objects.h" + +namespace { + +constexpr size_t PANELS = 8; +constexpr size_t PANEL_FS = 512; // fast axis, the stored image's columns +constexpr size_t PANEL_SS = 1024; // slow axis: the eight sensors are stacked along it +constexpr size_t STACK_PIXELS = PANELS * PANEL_FS * PANEL_SS; + +// Pixels on each sensor that measure nothing usable, as the geometry files the SACLA Cheetah +// pipeline generates mask them: a 5-pixel frame on every side, and the last 23 rows of the slow +// axis, which sit under the amplifier shield. +constexpr size_t EDGE = 5; +constexpr size_t SHIELD_FIRST_SS = 1001; + +// The Cheetah-SACLA output is gain-corrected so that 1 eV deposited reads 0.00125 ADU - the +// adu_per_eV its geometry generator writes for every panel - about 10 ADU per 8 keV photon. +constexpr double ADU_PER_EV = 0.00125; + +bool IsMPCCDFile(const std::string &path) { + if (!std::filesystem::is_regular_file(path)) + return false; + bool ret = false; + std::unique_lock ul(hdf5_mutex); + H5E_BEGIN_TRY { + if (H5Fis_hdf5(path.c_str()) > 0) { + try { + HDF5ReadOnlyFile file(path); + ret = file.Exists("/metadata/sensor_id") && file.Exists("/metadata/posx_in_um") + && file.Exists("/metadata/angle_in_rad"); + } catch (...) { + ret = false; + } + } + } H5E_END_TRY; + return ret; +} + +std::vector CollectFiles(const std::string &path) { + std::vector ret; + if (std::filesystem::is_directory(path)) { + for (const auto &e: std::filesystem::directory_iterator(path)) { + const auto p = e.path().string(); + if (e.path().extension() == ".h5" && IsMPCCDFile(p)) + ret.push_back(p); + } + std::sort(ret.begin(), ret.end()); + } else if (IsMPCCDFile(path)) + ret.push_back(path); + return ret; +} + +// "+0.000390x -1.000000y" as CrystFEL writes a panel's fs and ss directions. +void ParseDirection(const std::string &txt, double &x, double &y) { + x = 0; + y = 0; + static const std::regex term(R"(([+-]?\s*[0-9.eE+-]*)\s*([xyz]))"); + for (auto it = std::sregex_iterator(txt.begin(), txt.end(), term); it != std::sregex_iterator(); ++it) { + std::string num = (*it)[1].str(); + num.erase(std::remove(num.begin(), num.end(), ' '), num.end()); + double v = 1.0; + if (num == "-") + v = -1.0; + else if (!num.empty() && num != "+") + v = std::stod(num); + if ((*it)[2] == "x") + x = v; + else if ((*it)[2] == "y") + y = v; + } +} + +} // namespace + +JFJochMPCCDReader::JFJochMPCCDReader() = default; +JFJochMPCCDReader::~JFJochMPCCDReader() { + Close(); +} + +bool JFJochMPCCDReader::CanRead(const std::string &path) { + if (std::filesystem::is_directory(path)) { + for (const auto &e: std::filesystem::directory_iterator(path)) + if (e.path().extension() == ".h5") + return IsMPCCDFile(e.path().string()); + return false; + } + return IsMPCCDFile(path); +} + +void JFJochMPCCDReader::ReadFiles(const std::string &path) { + Close(); + files_ = CollectFiles(path); + if (files_.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "No SACLA MPCCD (Cheetah) files found for " + path); + + std::array panels; + double distance_mm = 0, photon_energy_eV = 0; + std::vector file_mask; + { + std::unique_lock ul(hdf5_mutex); + for (size_t f = 0; f < files_.size(); f++) { + open_files_.emplace_back(std::make_unique(files_[f])); + for (const auto &name: open_files_.back()->FindLeafs("/")) + if (name.starts_with("tag-")) + shots_.push_back({f, "/" + name}); + } + if (shots_.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "No images (tag-* groups) in " + path); + + auto &file = *open_files_[0]; + const auto posx = file.ReadVector("/metadata/posx_in_um"); + const auto posy = file.ReadVector("/metadata/posy_in_um"); + // Degrees, whatever the name says: the sensors stand at +-90. + const auto angle = file.ReadVector("/metadata/angle_in_rad"); + const auto pixel = file.ReadVector("/metadata/pixelsizex_in_um"); + if (posx.size() != PANELS || posy.size() != PANELS || angle.size() != PANELS || pixel.size() != PANELS) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + files_[0] + " does not describe eight MPCCD sensors"); + // The same placement the SACLA pipeline's geometry generator writes for CrystFEL. + for (size_t i = 0; i < PANELS; i++) { + const double a = angle[i] * M_PI / 180.0; + panels[i].corner_x = -posx[i] / pixel[i]; + panels[i].corner_y = posy[i] / pixel[i]; + panels[i].fs_x = std::cos(a); + panels[i].fs_y = std::sin(a); + panels[i].ss_x = -std::sin(a); + panels[i].ss_y = std::cos(a); + } + distance_mm = HDF5DataSet(file, "/metadata/distance_in_mm").ReadScalar(); + photon_energy_eV = HDF5DataSet(file, shots_[0].group + "/photon_energy_ev").ReadScalar(); + if (file.Exists("/metadata/pixelmask")) + file_mask = file.ReadVector("/metadata/pixelmask"); + } + + if (!(photon_energy_eV > 0) || !(distance_mm > 0)) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + files_[0] + " states no photon energy or no camera length"); + Build(panels, distance_mm, photon_energy_eV, file_mask); +} + +void JFJochMPCCDReader::ApplyCrystFELGeometry(const std::string &geom_file) { + if (!dataset_) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "No MPCCD files read"); + + std::ifstream in(geom_file); + if (!in) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Cannot open " + geom_file); + + std::map keys; + std::string line; + while (std::getline(in, line)) { + line = line.substr(0, line.find(';')); + const auto eq = line.find('='); + if (eq == std::string::npos) + continue; + auto trim = [](std::string s) { + s.erase(0, s.find_first_not_of(" \t")); + s.erase(s.find_last_not_of(" \t\r") + 1); + return s; + }; + keys[trim(line.substr(0, eq))] = trim(line.substr(eq + 1)); + } + + std::array panels; + double coffset = 0; + for (size_t i = 0; i < PANELS; i++) { + const std::string q = "q" + std::to_string(i + 1) + "/"; + if (!keys.contains(q + "corner_x") || !keys.contains(q + "fs") || !keys.contains(q + "ss")) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + geom_file + " does not describe panels q1..q8"); + panels[i].corner_x = std::stod(keys[q + "corner_x"]); + panels[i].corner_y = std::stod(keys[q + "corner_y"]); + ParseDirection(keys[q + "fs"], panels[i].fs_x, panels[i].fs_y); + ParseDirection(keys[q + "ss"], panels[i].ss_x, panels[i].ss_y); + if (keys.contains(q + "coffset")) + coffset = std::stod(keys[q + "coffset"]); + } + if (!keys.contains("clen")) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, geom_file + " states no clen"); + const double distance_mm = (std::stod(keys["clen"]) + coffset) * 1000.0; + + std::vector file_mask; + { + std::unique_lock ul(hdf5_mutex); + if (open_files_[0]->Exists("/metadata/pixelmask")) + file_mask = open_files_[0]->ReadVector("/metadata/pixelmask"); + } + Build(panels, distance_mm, dataset_->experiment.GetIncidentEnergy_keV() * 1000.0, file_mask); +} + +// CrystFEL's frame has y up and x to the left looking downstream; the flat image has columns to the +// right and rows down. The two differ by a half turn about the beam, which keeps the handedness, so +// a position (X, Y) lands on column beam_x - X and row beam_y - Y. +void JFJochMPCCDReader::Build(const std::array &panels, double distance_mm, + double photon_energy_eV, const std::vector &file_mask) { + auto position = [&](size_t p, double fs, double ss) { + return std::pair{panels[p].corner_x + fs * panels[p].fs_x + ss * panels[p].ss_x, + panels[p].corner_y + fs * panels[p].fs_y + ss * panels[p].ss_y}; + }; + double max_x = -1e9, min_x = 1e9, max_y = -1e9, min_y = 1e9; + for (size_t p = 0; p < PANELS; p++) + for (double fs: {0.0, double(PANEL_FS)}) + for (double ss: {0.0, double(PANEL_SS)}) { + const auto [x, y] = position(p, fs, ss); + max_x = std::max(max_x, x); + min_x = std::min(min_x, x); + max_y = std::max(max_y, y); + min_y = std::min(min_y, y); + } + const double beam_x = std::ceil(max_x) + 1; + const double beam_y = std::ceil(max_y) + 1; + width_ = static_cast(beam_x - std::floor(min_x)) + 2; + height_ = static_cast(beam_y - std::floor(min_y)) + 2; + + // Everything starts as a gap; each sensor's usable pixels are then placed on top. + std::vector mask(width_ * height_, 1u << PixelMask::ModuleGapPixelBit); + destination_.assign(STACK_PIXELS, -1); + for (size_t p = 0; p < PANELS; p++) { + for (size_t ss = 0; ss < PANEL_SS; ss++) { + for (size_t fs = 0; fs < PANEL_FS; fs++) { + const size_t src = (p * PANEL_SS + ss) * PANEL_FS + fs; + const auto [x, y] = position(p, fs + 0.5, ss + 0.5); + const auto col = std::lround(beam_x - x); + const auto row = std::lround(beam_y - y); + const size_t dst = static_cast(row) * width_ + static_cast(col); + if (fs < EDGE || fs >= PANEL_FS - EDGE || ss < EDGE || ss >= SHIELD_FIRST_SS) + continue; + destination_[src] = static_cast(dst); + mask[dst] = (src < file_mask.size() && file_mask[src] != 0) ? (1u << PixelMask::ErrorPixelBit) : 0; + } + } + } + + adu_per_photon_ = ADU_PER_EV * photon_energy_eV; + + dataset_ = std::make_shared(); + dataset_->experiment = default_experiment; + DetectorSetup detector = DetDECTRIS(static_cast(width_), static_cast(height_), "SACLA MPCCD", {}); + detector.PixelSize_um(50.0f); + // A CCD: nothing for a parallax or sensor-efficiency correction to work with is stated in the file. + detector.SensorThickness_um(0); + // The pipeline stores signed 16-bit ADU, so that is where a pixel stops counting. + detector.SaturationLimit(static_cast(std::floor(INT16_MAX / adu_per_photon_))); + detector.BitDepthImage(32); + detector.MinFrameTime(std::chrono::microseconds(0)); + detector.MinCountTime(std::chrono::microseconds(0)); + detector.ReadOutTime(std::chrono::nanoseconds(0)); + dataset_->experiment.Detector(detector); + dataset_->experiment.BeamX_pxl(static_cast(beam_x)); + dataset_->experiment.BeamY_pxl(static_cast(beam_y)); + dataset_->experiment.DetectorDistance_mm(static_cast(distance_mm)); + dataset_->experiment.IncidentEnergy_keV(static_cast(photon_energy_eV / 1000.0)); + dataset_->experiment.ImagesPerTrigger(static_cast(shots_.size())); + dataset_->pixel_mask = std::make_shared(mask); + + Logger("MPCCDReader").Info("{} shots in {} files; {} x {} image, beam at ({}, {}) px, {:.2f} mm, " + "{:.1f} eV, {:.2f} ADU per photon", shots_.size(), files_.size(), + width_, height_, beam_x, beam_y, distance_mm, photon_energy_eV, adu_per_photon_); + SetStartMessage(dataset_); +} + +uint64_t JFJochMPCCDReader::GetNumberOfImages() const { + return shots_.size(); +} + +void JFJochMPCCDReader::Close() { + std::unique_lock ul(hdf5_mutex); + open_files_.clear(); + files_.clear(); + shots_.clear(); + dataset_.reset(); +} + +// The stored chunk is the whole image, byte-shuffled and deflated. Only reading it needs the HDF5 lock; +// it is inflated, unshuffled and placed outside it, so the decoding runs on as many threads as read. +template +void JFJochMPCCDReader::DecodeInto(int64_t image_number, Buffer &buffer) const { + if (image_number < 0 || static_cast(image_number) >= shots_.size()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Image number out of range"); + const auto &shot = shots_[image_number]; + + std::vector compressed; + { + std::unique_lock ul(hdf5_mutex); + HDF5DataSet dataset(*open_files_[shot.file], shot.group + "/data"); + dataset.ReadDirectChunk(compressed, {0, 0}); + } + + std::vector shuffled(STACK_PIXELS * sizeof(int16_t)); + uLongf out_size = shuffled.size(); + if (uncompress(shuffled.data(), &out_size, compressed.data(), compressed.size()) != Z_OK + || out_size != shuffled.size()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "Cannot inflate image " + shot.group + " of " + files_[shot.file]); + + buffer.assign(width_ * height_ * sizeof(int32_t), 0); + auto out = reinterpret_cast(buffer.data()); + for (size_t i = 0; i < STACK_PIXELS; i++) { + if (destination_[i] < 0) + continue; + const auto adu = static_cast(shuffled[i] | (shuffled[STACK_PIXELS + i] << 8)); + out[destination_[i]] = static_cast(std::lround(adu / adu_per_photon_)); + } +} + +bool JFJochMPCCDReader::LoadImage_i(std::shared_ptr &dataset, + DataMessage &message, + std::vector &buffer, + int64_t image_number, + bool update_dataset) { + (void) update_dataset; + if (!dataset || image_number < 0 || static_cast(image_number) >= shots_.size()) + return false; + DecodeInto(image_number, buffer); + message.image = CompressedImage(buffer.data(), buffer.size(), width_, height_, + CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION); + message.number = image_number; + return true; +} + +bool JFJochMPCCDReader::ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) { + if (image_number < 0 || static_cast(image_number) >= shots_.size()) + return false; + DecodeInto(image_number, image.image_buffer); + image.image = CompressedImage(image.image_buffer.data(), image.image_buffer.size(), width_, height_, + CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION); + return true; +} + +std::vector JFJochMPCCDReader::ReadSpots(int64_t) const { + return {}; +} diff --git a/reader/JFJochMPCCDReader.h b/reader/JFJochMPCCDReader.h new file mode 100644 index 000000000..bce7419c2 --- /dev/null +++ b/reader/JFJochMPCCDReader.h @@ -0,0 +1,78 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include +#include + +#include "JFJochReader.h" + +class HDF5ReadOnlyFile; + +// Reads serial (stills) data from the SACLA multi-port CCD (MPCCD) octal detector, as the facility's +// Cheetah pipeline writes it: one HDF5 file per run part, one /tag-/data group per hit, each +// an int16 8192 x 512 image of the eight 1024 x 512 sensors stacked along the slow axis. The pixels +// are dark-subtracted and gain-corrected ADU, so no pedestal is needed; they are converted to +// photons here. /metadata states each sensor's position and in-plane angle, from which the sensors +// are placed on one flat image with the gaps between them masked. +// +// A path is one such file, or a directory: every Cheetah-SACLA file in it is read as one run, in +// name order. +class JFJochMPCCDReader : public JFJochReader { +public: + // One sensor as CrystFEL describes a panel: the position of its first pixel's corner and the + // directions of its fast and slow axes, in pixels, in CrystFEL's laboratory frame. + struct Panel { + double corner_x = 0, corner_y = 0; + double fs_x = 0, fs_y = 0, ss_x = 0, ss_y = 0; + }; + +private: + struct Shot { + size_t file; + std::string group; + }; + + std::vector files_; + std::vector> open_files_; + std::vector shots_; + std::shared_ptr dataset_; + + size_t width_ = 0, height_ = 0; + // For every pixel of the 8192 x 512 stack, where it lands on the flat image (-1: not placed). + std::vector destination_; + double adu_per_photon_ = 1.0; + + void Build(const std::array &panels, double distance_mm, double photon_energy_eV, + const std::vector &file_mask); + + bool LoadImage_i(std::shared_ptr &dataset, + DataMessage &message, + std::vector &buffer, + int64_t image_number, + bool update_dataset) override; + + template + void DecodeInto(int64_t image_number, Buffer &buffer) const; + +public: + JFJochMPCCDReader(); + ~JFJochMPCCDReader() override; + + // True for a Cheetah-SACLA MPCCD file, or a directory holding at least one. + static bool CanRead(const std::string &path); + + void ReadFiles(const std::string &path); + + // Replaces the sensor placement and the camera length /metadata states with a refined CrystFEL + // geometry file for the same 512 x 8192 layout (panels q1..q8). + void ApplyCrystFELGeometry(const std::string &geom_file); + + [[nodiscard]] uint64_t GetNumberOfImages() const override; + void Close() override; + bool ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) override; + [[nodiscard]] std::vector ReadSpots(int64_t image) const override; +}; diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 2a1ff9f88..dc46c49f6 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -28,6 +28,7 @@ #include "../reader/JFJochCBFReader.h" #include "../reader/JFJochMarCCDReader.h" #include "../reader/JFJochSMVReader.h" +#include "../reader/JFJochMPCCDReader.h" #include "../reader/JFJochStartMessageReader.h" #include "BrokerFeed.h" #include "../writer/HDF5NXmx.h" @@ -147,6 +148,10 @@ void print_usage() { std::cout << " --no-refine-tilt Do not refine the detector tilt: hold rot1/rot2 at the header value and fit only the beam centre and the distance. For a calibration handed to a program that cannot express a tilted detector, e.g. XDS" << std::endl; std::cout << std::endl; + std::cout << " Detector geometry" << std::endl; + std::cout << " --crystfel-geometry SACLA MPCCD input only: place the eight sensors and set the camera length from this refined CrystFEL geometry file (panels q1..q8 of the 512x8192 layout) instead of the nominal placement the files' /metadata states" << std::endl; + std::cout << std::endl; + std::cout << " Detector mask" << std::endl; std::cout << " --detect-beam-stop[=N|off] Find the beam stop and its holder in a projection of N images and add them to the pixel mask (bit 9), so nothing shadowed by them is integrated. ON by default (60 images); =off disables. Reflections behind the stop are attenuated but not flagged, so they are integrated low with a plausible sigma and no existing rejection catches them" << std::endl; std::cout << " --beam-center-check[=off] Measure the beam centre from the isotropy of the scattered background on EVERY run, report how far the file's value is from it, and index a second first pass with it to see whether the two centres give the same lattice. The fit reads the projection --detect-beam-stop already builds, so it costs no extra frames. On a run that indexes, the measured centre is adopted where the file's centre indexes nothing and the measured one indexes a majority, and where the two centres return cells related by an integer volume factor, in either direction, and the measured centre's lattice carries materially more of the pooled spots against its own chance level - the axis harmonic a centre error along the spindle produces, which nothing downstream repairs. Where the indexing cannot tell the two centres apart - the same cell with a different metric symmetry, or the same lattice found at both only by reading less of each frame - the first pass is run at both and the run keeps the one that merges better (on the same lattice: the one that measures more reflections at I/sigma >= 2). ON by default; =off disables" << std::endl; @@ -303,6 +308,7 @@ enum { OPT_REJECT_OUTLIERS, OPT_REFERENCE_COLUMN, OPT_MODEL, + OPT_CRYSTFEL_GEOMETRY, OPT_MR, OPT_EXPORT_UNMERGED, OPT_NO_EXPORT_UNMERGED, @@ -369,6 +375,7 @@ static option long_options[] = { {"reference-mtz", required_argument, nullptr, 'z'}, {"reference-column", required_argument, nullptr, OPT_REFERENCE_COLUMN}, {"model", required_argument, nullptr, OPT_MODEL}, + {"crystfel-geometry", required_argument, nullptr, OPT_CRYSTFEL_GEOMETRY}, {"mr", required_argument, nullptr, OPT_MR}, {"export-unmerged", no_argument, nullptr, OPT_EXPORT_UNMERGED}, {"no-export-unmerged", no_argument, nullptr, OPT_NO_EXPORT_UNMERGED}, @@ -987,6 +994,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { float false_pixels_per_frame = SpotFindingSettings{}.false_pixels_per_frame; std::string ref_mtz; std::string ref_column; + std::string crystfel_geometry; // --crystfel-geometry: refined MPCCD sensor placement std::string model_pdb; // --model: PDB to validate merged intensities against (R-free + maps) std::string mr_model; // --mr: --model, placed by molecular replacement straight away bool export_unmerged = true; // on by default; --no-export-unmerged turns it off @@ -1215,6 +1223,9 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { case OPT_MODEL: model_pdb = optarg; break; + case OPT_CRYSTFEL_GEOMETRY: + crystfel_geometry = optarg; + break; case OPT_MR: mr_model = optarg; break; @@ -1718,17 +1729,19 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { SimulatedLive cbf_reader; SimulatedLive marccd_reader; SimulatedLive smv_reader; + SimulatedLive mpccd_reader; JFJochStartMessageReader live_reader; std::unique_ptr broker_feed; JFJochReader *reader_ptr = nullptr; // Which format this is. Asked INSIDE the try: these open and read the file - a directory whose // first plausible member cannot be read throws here - and an input the program cannot read has to // come out as the message below, not as an unhandled exception with nothing printed at all. - bool input_is_cbf = false, input_is_marccd = false, input_is_smv = false; + bool input_is_cbf = false, input_is_marccd = false, input_is_smv = false, input_is_mpccd = false; try { input_is_cbf = JFJochCBFReader::CanRead(input_file); input_is_marccd = !input_is_cbf && JFJochMarCCDReader::CanRead(input_file); input_is_smv = !input_is_cbf && !input_is_marccd && JFJochSMVReader::CanRead(input_file); + input_is_mpccd = !input_is_cbf && !input_is_marccd && !input_is_smv && JFJochMPCCDReader::CanRead(input_file); if (input_is_cbf) { cbf_reader.ReadFiles(input_file); reader_ptr = &cbf_reader; @@ -1744,6 +1757,12 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { reader_ptr = &smv_reader; logger.Info("Read {} SMV images of the sweep containing {}", smv_reader.GetNumberOfImages(), input_file); + } else if (input_is_mpccd) { + mpccd_reader.ReadFiles(input_file); + if (!crystfel_geometry.empty()) + mpccd_reader.ApplyCrystFELGeometry(crystfel_geometry); + reader_ptr = &mpccd_reader; + logger.Info("Read {} SACLA MPCCD images from {}", mpccd_reader.GetNumberOfImages(), input_file); } else if (std::filesystem::is_directory(input_file)) { // A directory is never an HDF5 master, so handing it to the HDF5 reader only buries the // real answer - that no one-file-per-image sweep was found in it - under a page of HDF5 @@ -1918,7 +1937,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { "runs no space-group search. Use --mode mx, which makes the P1 merge the " "ledger is folded from."); // Re-scaling reads reflections a previous run integrated, which only a _process.h5 holds. - if (input_is_cbf || input_is_marccd || input_is_smv) { + if (input_is_cbf || input_is_marccd || input_is_smv || input_is_mpccd) { logger.Error("--mode scale needs the integrated reflections in a _process.h5; " "a raw image sweep holds none. Run --mode mx on it first."); exit(EXIT_FAILURE); -- 2.54.0 From ce528f3a14981d0de1109de03c9e6f8df90886cc Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 13:03:12 +0200 Subject: [PATCH 27/53] rugnux: read stills through a CrystFEL geometry file (--crystfel-geometry) A CrystFEL .geom (current and old syntax) is parsed for panels (pixel ranges, fs/ss, corners), clen/coffset (a number, or an HDF5 path read as mm), res, adu_per_eV/adu_per_photon, max_adu, no_index, bad regions (panel, whole-frame or lab-plane) and the HDF5 mask (mask/mask_file, mask_good/mask_bad). JFJochCrystFELReader reads the frames the geometry's data path names - a 3D stack (Cheetah CXI), one 2D frame per file, or one group per shot (/%/data) - from a file or a directory, puts the panels on one flat image at their rounded positions with gaps and bad regions masked, and converts to photons where a gain is stated (otherwise the pre-scan measures it). One wavelength and one camera length per run: values taken from the files are averaged over the first file. Panel positions are used as given. Frames stored as one deflate(+shuffle) chunk are read raw under the HDF5 lock and decoded outside it. Replaces the MPCCD-specific reader of the previous commit. Co-Authored-By: Claude Opus 5.5 --- reader/CMakeLists.txt | 6 +- reader/CrystFELGeometry.cpp | 138 ++++++++++++ reader/CrystFELGeometry.h | 57 +++++ reader/JFJochCrystFELReader.cpp | 358 ++++++++++++++++++++++++++++++++ reader/JFJochCrystFELReader.h | 81 ++++++++ reader/JFJochMPCCDReader.cpp | 350 ------------------------------- reader/JFJochMPCCDReader.h | 78 ------- rugnux/rugnux_cli.cpp | 31 ++- 8 files changed, 653 insertions(+), 446 deletions(-) create mode 100644 reader/CrystFELGeometry.cpp create mode 100644 reader/CrystFELGeometry.h create mode 100644 reader/JFJochCrystFELReader.cpp create mode 100644 reader/JFJochCrystFELReader.h delete mode 100644 reader/JFJochMPCCDReader.cpp delete mode 100644 reader/JFJochMPCCDReader.h diff --git a/reader/CMakeLists.txt b/reader/CMakeLists.txt index 16f137359..fe0f0d7ec 100644 --- a/reader/CMakeLists.txt +++ b/reader/CMakeLists.txt @@ -12,8 +12,10 @@ ADD_LIBRARY(JFJochReader STATIC MarCCD.h JFJochMarCCDReader.cpp JFJochMarCCDReader.h - JFJochMPCCDReader.cpp - JFJochMPCCDReader.h + CrystFELGeometry.cpp + CrystFELGeometry.h + JFJochCrystFELReader.cpp + JFJochCrystFELReader.h SMV.cpp SMV.h JFJochSMVReader.cpp diff --git a/reader/CrystFELGeometry.cpp b/reader/CrystFELGeometry.cpp new file mode 100644 index 000000000..f377fbd17 --- /dev/null +++ b/reader/CrystFELGeometry.cpp @@ -0,0 +1,138 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "CrystFELGeometry.h" + +#include +#include +#include + +#include "../common/JFJochException.h" + +namespace crystfel { + +namespace { + +std::string Trim(const std::string &s) { + const auto b = s.find_first_not_of(" \t\r"); + if (b == std::string::npos) + return ""; + return s.substr(b, s.find_last_not_of(" \t\r") - b + 1); +} + +// "+0.000390x -1.000000y", "-x", "0.5x+0.5y": the in-plane direction of a panel axis. +void ParseDirection(const std::string &txt, double &x, double &y) { + x = 0; + y = 0; + static const std::regex term(R"(([+-]?)\s*([0-9.]*(?:[eE][+-]?[0-9]+)?)\s*([xyz]))"); + for (auto it = std::sregex_iterator(txt.begin(), txt.end(), term); it != std::sregex_iterator(); ++it) { + double v = (*it)[2].str().empty() ? 1.0 : std::stod((*it)[2].str()); + if ((*it)[1] == "-") + v = -v; + if ((*it)[3] == "x") + x = v; + else if ((*it)[3] == "y") + y = v; + } +} + +// Sets one panel property; false if the key is not one. +bool SetPanel(Panel &p, const std::string &key, const std::string &value) { + if (key == "min_fs") p.min_fs = std::stoll(value); + else if (key == "max_fs") p.max_fs = std::stoll(value); + else if (key == "min_ss") p.min_ss = std::stoll(value); + else if (key == "max_ss") p.max_ss = std::stoll(value); + else if (key == "fs") ParseDirection(value, p.fs_x, p.fs_y); + else if (key == "ss") ParseDirection(value, p.ss_x, p.ss_y); + else if (key == "corner_x") p.corner_x = std::stod(value); + else if (key == "corner_y") p.corner_y = std::stod(value); + else if (key == "res") p.res = std::stod(value); + else if (key == "clen") p.clen = value; + else if (key == "coffset") p.coffset = std::stod(value); + else if (key == "adu_per_eV") p.adu_per_eV = std::stod(value); + else if (key == "adu_per_photon") p.adu_per_photon = std::stod(value); + else if (key == "max_adu") p.max_adu = std::stod(value); + else if (key == "no_index") p.no_index = std::stoi(value) != 0; + else + return false; + return true; +} + +} // namespace + +Geometry Read(const std::string &filename) { + std::ifstream in(filename); + if (!in) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Cannot open " + filename); + + Geometry geom; + // Global panel keys hold for every panel that appears after them, as in the old syntax; a file in + // the current syntax states them first, so this reads both the same. + Panel defaults; + std::map panel_index; + std::map bad_index; + + std::string line; + while (std::getline(in, line)) { + line = line.substr(0, line.find(';')); + const auto eq = line.find('='); + if (eq == std::string::npos) + continue; + const std::string key = Trim(line.substr(0, eq)); + const std::string value = Trim(line.substr(eq + 1)); + const auto slash = key.find('/'); + + if (slash == std::string::npos) { + if (key == "data") geom.data = value; + else if (key == "photon_energy") geom.photon_energy = value; + else if (key == "mask") geom.mask = value; + else if (key == "mask_file") geom.mask_file = value; + else if (key == "mask_good") geom.mask_good = std::stoul(value, nullptr, 0); + else if (key == "mask_bad") geom.mask_bad = std::stoul(value, nullptr, 0); + else SetPanel(defaults, key, value); // anything else (rigid groups, dim, ...) is not needed + continue; + } + + const std::string name = key.substr(0, slash); + const std::string prop = key.substr(slash + 1); + + if (name.starts_with("bad")) { + if (!bad_index.contains(name)) { + bad_index[name] = geom.bad.size(); + geom.bad.push_back({}); + } + auto &b = geom.bad[bad_index[name]]; + if (prop == "panel") b.panel = value; + else if (prop == "min_x") { b.in_lab = true; b.min_x = std::stod(value); } + else if (prop == "max_x") { b.in_lab = true; b.max_x = std::stod(value); } + else if (prop == "min_y") { b.in_lab = true; b.min_y = std::stod(value); } + else if (prop == "max_y") { b.in_lab = true; b.max_y = std::stod(value); } + else if (prop == "min_fs") b.min_fs = std::stoll(value); + else if (prop == "max_fs") b.max_fs = std::stoll(value); + else if (prop == "min_ss") b.min_ss = std::stoll(value); + else if (prop == "max_ss") b.max_ss = std::stoll(value); + continue; + } + + if (!panel_index.contains(name)) { + panel_index[name] = geom.panels.size(); + geom.panels.push_back(defaults); + geom.panels.back().name = name; + } + SetPanel(geom.panels[panel_index[name]], prop, value); + } + + if (geom.panels.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, filename + " defines no panels"); + for (const auto &p: geom.panels) { + if (!(p.res > 0)) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + filename + ": panel " + p.name + " has no pixel size (res)"); + if (p.clen.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + filename + ": panel " + p.name + " has no camera length (clen)"); + } + return geom; +} + +} // namespace crystfel diff --git a/reader/CrystFELGeometry.h b/reader/CrystFELGeometry.h new file mode 100644 index 000000000..9253f9825 --- /dev/null +++ b/reader/CrystFELGeometry.h @@ -0,0 +1,57 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include +#include + +// A CrystFEL detector geometry file (.geom), as far as it places pixels and says what they are worth: +// the panels with their pixel ranges in the stored frame, fast/slow-scan directions and corners, the +// camera length, the pixel size, the gain and the regions to leave out. Both the current syntax and +// the older one - global keys that only take effect for the panels that follow them, bad regions +// stated in whole-frame coordinates - are read. +// +// Every length is in pixels except clen and coffset, which are in metres. A value CrystFEL takes from +// the image file instead (clen = /path, photon_energy = /path) is kept as that path, for the reader +// to resolve. +namespace crystfel { + +struct Panel { + std::string name; + int64_t min_fs = 0, max_fs = 0, min_ss = 0, max_ss = 0; // inclusive, in the stored frame + double fs_x = 1, fs_y = 0, ss_x = 0, ss_y = 1; + double corner_x = 0, corner_y = 0; + double res = 0; // pixels per metre + std::string clen; // metres, or an HDF5 path + double coffset = 0; // metres + std::optional adu_per_eV; + std::optional adu_per_photon; + std::optional max_adu; + bool no_index = false; +}; + +// A region to leave out: either a pixel range on one panel (or, in the old syntax, on the whole +// frame), or a box in the laboratory plane, in pixels. +struct BadRegion { + std::string panel; // empty: fs/ss are whole-frame coordinates + bool in_lab = false; + double min_x = 0, max_x = 0, min_y = 0, max_y = 0; + int64_t min_fs = 0, max_fs = 0, min_ss = 0, max_ss = 0; +}; + +struct Geometry { + std::vector panels; + std::vector bad; + std::string data = "/data/data"; + std::string photon_energy; // eV, or an HDF5 path + std::string mask; // HDF5 path, or empty + std::string mask_file; // file holding it, or empty for the image file itself + uint32_t mask_good = 0, mask_bad = 0; +}; + +Geometry Read(const std::string &filename); + +} // namespace crystfel diff --git a/reader/JFJochCrystFELReader.cpp b/reader/JFJochCrystFELReader.cpp new file mode 100644 index 000000000..77fb47f26 --- /dev/null +++ b/reader/JFJochCrystFELReader.cpp @@ -0,0 +1,358 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "JFJochCrystFELReader.h" + +#include +#include +#include +#include + +#include + +#include "../common/JFJochException.h" +#include "../common/Logger.h" +#include "../writer/HDF5Objects.h" + +namespace { + +bool IsPath(const std::string &s) { + return !s.empty() && s[0] == '/'; +} + +std::string ReplaceEvent(std::string path, const std::string &group) { + const auto pos = path.find('%'); + if (pos != std::string::npos) + path.replace(pos, 1, group); + return path; +} + +} // namespace + +JFJochCrystFELReader::JFJochCrystFELReader() = default; + +JFJochCrystFELReader::~JFJochCrystFELReader() { + Close(); +} + +// One number for the whole run from a value the geometry takes from the image files: a scalar, a +// per-frame vector, or one scalar per shot group. +double JFJochCrystFELReader::MeanOverFirstFile(const std::string &path) { + auto &file = *open_files_[0]; + std::vector values; + if (path.find('%') != std::string::npos) { + for (const auto &f: frames_) + if (f.file == 0) + values.push_back(HDF5DataSet(file, ReplaceEvent(path, f.group)).ReadScalar()); + } else if (file.GetDimension(path).empty()) + values.push_back(HDF5DataSet(file, path).ReadScalar()); + else + values = file.ReadVector(path); + if (values.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "No values in " + path); + return std::accumulate(values.begin(), values.end(), 0.0) / static_cast(values.size()); +} + +void JFJochCrystFELReader::ReadFiles(const std::string &path, const std::string &geometry_file) { + Close(); + geom_ = crystfel::Read(geometry_file); + + if (std::filesystem::is_directory(path)) { + for (const auto &e: std::filesystem::directory_iterator(path)) + if (e.path().extension() == ".h5" || e.path().extension() == ".cxi") + files_.push_back(e.path().string()); + std::sort(files_.begin(), files_.end()); + } else + files_.push_back(path); + if (files_.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "No .h5 or .cxi files in " + path); + + std::unique_lock ul(hdf5_mutex); + + // Every frame of every file, in file order. + const auto pct = geom_.data.find('%'); + for (size_t f = 0; f < files_.size(); f++) { + open_files_.emplace_back(std::make_unique(files_[f])); + auto &file = *open_files_.back(); + if (pct != std::string::npos) { + const std::string parent = geom_.data.substr(0, pct); + for (const auto &group: file.FindLeafs(parent.empty() ? "/" : parent)) { + const auto dataset = ReplaceEvent(geom_.data, group); + if (file.Exists(dataset)) + frames_.push_back({f, dataset, -1, group}); + } + } else { + const auto dim = file.GetDimension(geom_.data); + if (dim.size() == 3) { + for (hsize_t i = 0; i < dim[0]; i++) + frames_.push_back({f, geom_.data, static_cast(i), ""}); + } else if (dim.size() == 2) + frames_.push_back({f, geom_.data, -1, ""}); + } + } + if (frames_.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "No frames at " + geom_.data + " in " + path); + + const auto dim = open_files_[frames_[0].file]->GetDimension(frames_[0].dataset); + frame_height_ = dim[dim.size() - 2]; + frame_width_ = dim[dim.size() - 1]; + + const auto &p0 = geom_.panels[0]; + const double clen_m = IsPath(p0.clen) ? MeanOverFirstFile(p0.clen) * 1e-3 : std::stod(p0.clen); + const double distance_mm = (clen_m + p0.coffset) * 1000.0; + if (geom_.photon_energy.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + geometry_file + " states no photon_energy"); + const double photon_energy_eV = IsPath(geom_.photon_energy) ? MeanOverFirstFile(geom_.photon_energy) + : std::stod(geom_.photon_energy); + for (const auto &p: geom_.panels) + if (p.res != p0.res || p.clen != p0.clen || p.coffset != p0.coffset) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + geometry_file + ": panels at different distances or with different " + "pixel sizes cannot be put on one flat detector"); + + std::vector file_mask; + if (!geom_.mask.empty()) { + if (!geom_.mask_file.empty()) { + HDF5ReadOnlyFile mask_file(geom_.mask_file); + file_mask = mask_file.ReadVector(geom_.mask, {0, 0}, {frame_height_, frame_width_}); + } else + file_mask = open_files_[0]->ReadVector(geom_.mask, {0, 0}, {frame_height_, frame_width_}); + } + CheckDirectChunks(); + ul.unlock(); + + // CrystFEL's laboratory frame and the flat image's (column, row, beam) are both right-handed with + // the beam along z, so the point (X, Y) lands at column beam_x + X, row beam_y + Y - and CrystFEL's + // horizontal x stays the image's horizontal, where the polarisation correction expects it. A pixel + // centre is at its index, so beam_x is chosen to put the first panel's pixel centres on whole columns. + auto position = [](const crystfel::Panel &p, double fs, double ss) { + return std::pair{p.corner_x + fs * p.fs_x + ss * p.ss_x, p.corner_y + fs * p.fs_y + ss * p.ss_y}; + }; + double max_x = -1e9, min_x = 1e9, max_y = -1e9, min_y = 1e9; + for (const auto &p: geom_.panels) + for (double fs: {0.0, double(p.max_fs - p.min_fs + 1)}) + for (double ss: {0.0, double(p.max_ss - p.min_ss + 1)}) { + const auto [x, y] = position(p, fs, ss); + max_x = std::max(max_x, x); + min_x = std::min(min_x, x); + max_y = std::max(max_y, y); + min_y = std::min(min_y, y); + } + const auto [x0, y0] = position(p0, 0.5, 0.5); + const double beam_x = std::ceil(x0 - min_x) + 1 - x0; + const double beam_y = std::ceil(y0 - min_y) + 1 - y0; + width_ = static_cast(std::ceil(beam_x + max_x)) + 2; + height_ = static_cast(std::ceil(beam_y + max_y)) + 2; + + // Everything starts as a gap; the panels' pixels are put on top, less the bad regions. + std::vector mask(width_ * height_, 1u << PixelMask::ModuleGapPixelBit); + destination_.assign(frame_width_ * frame_height_, -1); + adu_per_photon_.assign(frame_width_ * frame_height_, 1.0f); + counts_in_photons_ = true; + std::optional saturation; + + for (const auto &p: geom_.panels) { + if (p.max_fs >= static_cast(frame_width_) || p.max_ss >= static_cast(frame_height_)) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "Panel " + p.name + " lies outside the stored frame"); + double gain = 1.0; + if (p.adu_per_photon.has_value()) + gain = p.adu_per_photon.value(); + else if (p.adu_per_eV.has_value()) + gain = p.adu_per_eV.value() * photon_energy_eV; + else + counts_in_photons_ = false; + if (p.max_adu.has_value()) + saturation = std::min(saturation.value_or(INT64_MAX), p.max_adu.value() / gain); + if (p.no_index) + continue; + + for (int64_t ss = p.min_ss; ss <= p.max_ss; ss++) { + for (int64_t fs = p.min_fs; fs <= p.max_fs; fs++) { + const auto [x, y] = position(p, fs - p.min_fs + 0.5, ss - p.min_ss + 0.5); + bool bad = false; + for (const auto &b: geom_.bad) { + if (b.in_lab) + bad |= x >= b.min_x && x <= b.max_x && y >= b.min_y && y <= b.max_y; + else if (b.panel.empty() || b.panel == p.name) + bad |= fs >= b.min_fs && fs <= b.max_fs && ss >= b.min_ss && ss <= b.max_ss; + } + if (bad) + continue; + + const size_t src = ss * frame_width_ + fs; + const auto col = std::lround(beam_x + x); + const auto row = std::lround(beam_y + y); + const size_t dst = static_cast(row) * width_ + static_cast(col); + destination_[src] = static_cast(dst); + adu_per_photon_[src] = static_cast(gain); + + const uint32_t m = src < file_mask.size() ? file_mask[src] : 0; + const bool masked = (geom_.mask_bad != 0 && (m & geom_.mask_bad) != 0) + || (geom_.mask_good != 0 && (m & geom_.mask_good) != geom_.mask_good); + mask[dst] = masked ? (1u << PixelMask::ErrorPixelBit) : 0; + } + } + } + + dataset_ = std::make_shared(); + dataset_->experiment = default_experiment; + DetectorSetup detector = DetDECTRIS(static_cast(width_), static_cast(height_), + "CrystFEL geometry", {}); + detector.PixelSize_um(static_cast(1e6 / p0.res)); + // Nothing a parallax or sensor-efficiency correction needs is stated in a geometry file. + detector.SensorThickness_um(0); + if (saturation.has_value()) + detector.SaturationLimit(static_cast(std::floor(saturation.value()))); + detector.BitDepthImage(32); + detector.MinFrameTime(std::chrono::microseconds(0)); + detector.MinCountTime(std::chrono::microseconds(0)); + detector.ReadOutTime(std::chrono::nanoseconds(0)); + dataset_->experiment.Detector(detector); + dataset_->experiment.BeamX_pxl(static_cast(beam_x)); + dataset_->experiment.BeamY_pxl(static_cast(beam_y)); + dataset_->experiment.DetectorDistance_mm(static_cast(distance_mm)); + dataset_->experiment.IncidentEnergy_keV(static_cast(photon_energy_eV / 1000.0)); + dataset_->experiment.ImagesPerTrigger(static_cast(frames_.size())); + dataset_->pixel_mask = std::make_shared(mask); + + Logger("CrystFELReader").Info("{} frames in {} files, {} panels; {} x {} image, beam at ({:.2f}, {:.2f}) px, " + "{:.3f} mm, {:.1f} eV, {}{}", frames_.size(), files_.size(), geom_.panels.size(), + width_, height_, beam_x, beam_y, distance_mm, photon_energy_eV, + counts_in_photons_ ? "converted to photons" : "in detector units (no gain stated)", + direct_chunks_ ? ", chunks decoded in parallel" : ""); + SetStartMessage(dataset_); +} + +void JFJochCrystFELReader::CheckDirectChunks() { + const auto &frame = frames_[0]; + HDF5DataSet dataset(*open_files_[frame.file], frame.dataset); + + const hid_t dcpl = H5Dget_create_plist(dataset.GetID()); + const int rank = frame.index >= 0 ? 3 : 2; + std::vector chunk(rank, 0); + bool ok = H5Pget_layout(dcpl) == H5D_CHUNKED && H5Pget_chunk(dcpl, rank, chunk.data()) == rank + && chunk[rank - 1] == frame_width_ && chunk[rank - 2] == frame_height_ && (rank == 2 || chunk[0] == 1); + bool deflate = false; + shuffled_ = false; + for (int i = 0; ok && i < H5Pget_nfilters(dcpl); i++) { + unsigned int flags; + size_t nelements = 0; + const H5Z_filter_t filter = H5Pget_filter2(dcpl, i, &flags, &nelements, nullptr, 0, nullptr, nullptr); + if (filter == H5Z_FILTER_DEFLATE) + deflate = true; + else if (filter == H5Z_FILTER_SHUFFLE) + shuffled_ = true; + else + ok = false; + } + H5Pclose(dcpl); + + const hid_t type = H5Dget_type(dataset.GetID()); + stored_size_ = H5Tget_size(type); + stored_float_ = H5Tget_class(type) == H5T_FLOAT; + stored_signed_ = H5Tget_sign(type) == H5T_SGN_2; + ok &= (H5Tget_class(type) == H5T_INTEGER && (stored_size_ == 2 || stored_size_ == 4)) + || (stored_float_ && stored_size_ == 4); + H5Tclose(type); + + direct_chunks_ = ok && deflate; +} + +bool JFJochCrystFELReader::CountsInPhotons() const { + return counts_in_photons_; +} + +uint64_t JFJochCrystFELReader::GetNumberOfImages() const { + return frames_.size(); +} + +void JFJochCrystFELReader::Close() { + std::unique_lock ul(hdf5_mutex); + open_files_.clear(); + files_.clear(); + frames_.clear(); + dataset_.reset(); +} + +template +void JFJochCrystFELReader::DecodeInto(int64_t image_number, Buffer &buffer) const { + const auto &frame = frames_.at(image_number); + std::vector stored(frame_width_ * frame_height_); + if (direct_chunks_) { + std::vector compressed; + { + std::unique_lock ul(hdf5_mutex); + HDF5DataSet dataset(*open_files_[frame.file], frame.dataset); + if (frame.index >= 0) + dataset.ReadDirectChunk(compressed, {static_cast(frame.index), 0, 0}); + else + dataset.ReadDirectChunk(compressed, {0, 0}); + } + const size_t n = stored.size(); + std::vector raw(n * stored_size_); + uLongf raw_size = raw.size(); + if (uncompress(raw.data(), &raw_size, compressed.data(), compressed.size()) != Z_OK || raw_size != raw.size()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "Cannot inflate " + frame.dataset + " of " + files_[frame.file]); + if (shuffled_) { + std::vector tmp(raw.size()); + for (size_t i = 0; i < n; i++) + for (size_t b = 0; b < stored_size_; b++) + tmp[i * stored_size_ + b] = raw[b * n + i]; + raw.swap(tmp); + } + for (size_t i = 0; i < n; i++) { + const uint8_t *v = raw.data() + i * stored_size_; + if (stored_float_) + stored[i] = *reinterpret_cast(v); + else if (stored_size_ == 2) + stored[i] = stored_signed_ ? *reinterpret_cast(v) : *reinterpret_cast(v); + else + stored[i] = stored_signed_ ? *reinterpret_cast(v) : *reinterpret_cast(v); + } + } else { + std::unique_lock ul(hdf5_mutex); + HDF5DataSet dataset(*open_files_[frame.file], frame.dataset); + if (frame.index >= 0) + dataset.ReadVector(stored, {static_cast(frame.index), 0, 0}, {1, frame_height_, frame_width_}); + else + dataset.ReadVector(stored, {0, 0}, {frame_height_, frame_width_}); + } + + buffer.assign(width_ * height_ * sizeof(int32_t), 0); + auto out = reinterpret_cast(buffer.data()); + for (size_t i = 0; i < stored.size(); i++) + if (destination_[i] >= 0) + out[destination_[i]] = static_cast(std::lround(stored[i] / adu_per_photon_[i])); +} + +bool JFJochCrystFELReader::LoadImage_i(std::shared_ptr &dataset, + DataMessage &message, + std::vector &buffer, + int64_t image_number, + bool update_dataset) { + (void) update_dataset; + if (!dataset || image_number < 0 || static_cast(image_number) >= frames_.size()) + return false; + DecodeInto(image_number, buffer); + message.image = CompressedImage(buffer.data(), buffer.size(), width_, height_, + CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION); + message.number = image_number; + return true; +} + +bool JFJochCrystFELReader::ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) { + if (image_number < 0 || static_cast(image_number) >= frames_.size()) + return false; + DecodeInto(image_number, image.image_buffer); + image.image = CompressedImage(image.image_buffer.data(), image.image_buffer.size(), width_, height_, + CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION); + return true; +} + +std::vector JFJochCrystFELReader::ReadSpots(int64_t) const { + return {}; +} diff --git a/reader/JFJochCrystFELReader.h b/reader/JFJochCrystFELReader.h new file mode 100644 index 000000000..4eb111382 --- /dev/null +++ b/reader/JFJochCrystFELReader.h @@ -0,0 +1,81 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include + +#include "JFJochReader.h" +#include "CrystFELGeometry.h" + +class HDF5ReadOnlyFile; + +// Reads serial (stills) frames from HDF5 files laid out the way CrystFEL reads them, placed on the +// detector by a CrystFEL geometry file: Cheetah CXI files with the frames stacked along the first +// dimension, files holding one frame each, or files with one group per shot (data = /%/data). The +// panels are put on one flat image at their geometry-file positions, rounded to whole pixels, with +// everything between them and every bad region masked. Pixel values are converted to photons with the +// geometry's gain where it states one, and are otherwise left in detector units. +// +// One wavelength and one camera length hold for the whole run: where the geometry takes either from +// the image files, it is the mean over the first file's frames. +// +// A path is one file, or a directory, every .h5/.cxi file of which is read in name order. +class JFJochCrystFELReader : public JFJochReader { + struct Frame { + size_t file; + std::string dataset; + int64_t index; // along the first dimension of a 3D dataset; -1 for a 2D one + std::string group; // the shot group the geometry's % stands for, if any + }; + + crystfel::Geometry geom_; + std::vector files_; + std::vector> open_files_; + std::vector frames_; + std::shared_ptr dataset_; + + size_t frame_width_ = 0, frame_height_ = 0; // the stored frame + size_t width_ = 0, height_ = 0; // the flat image + // For every stored pixel, where it lands on the flat image (-1: nowhere) and what one photon reads. + std::vector destination_; + std::vector adu_per_photon_; + bool counts_in_photons_ = false; + + // Where every frame is stored as one deflated (and optionally byte-shuffled) chunk of a plain + // number type - the Cheetah outputs - the chunk is read raw under the HDF5 lock and decoded + // outside it, so decoding runs on every reading thread instead of one at a time. + bool direct_chunks_ = false; + bool shuffled_ = false; + bool stored_float_ = false, stored_signed_ = false; + size_t stored_size_ = 0; + + void CheckDirectChunks(); + + double MeanOverFirstFile(const std::string &path); + + bool LoadImage_i(std::shared_ptr &dataset, + DataMessage &message, + std::vector &buffer, + int64_t image_number, + bool update_dataset) override; + + template + void DecodeInto(int64_t image_number, Buffer &buffer) const; + +public: + JFJochCrystFELReader(); + ~JFJochCrystFELReader() override; + + void ReadFiles(const std::string &path, const std::string &geometry_file); + + // False where the geometry states no gain, so the images are in detector units, not photons. + [[nodiscard]] bool CountsInPhotons() const; + + [[nodiscard]] uint64_t GetNumberOfImages() const override; + void Close() override; + bool ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) override; + [[nodiscard]] std::vector ReadSpots(int64_t image) const override; +}; diff --git a/reader/JFJochMPCCDReader.cpp b/reader/JFJochMPCCDReader.cpp deleted file mode 100644 index a783fab24..000000000 --- a/reader/JFJochMPCCDReader.cpp +++ /dev/null @@ -1,350 +0,0 @@ -// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute -// SPDX-License-Identifier: GPL-3.0-only - -#include "JFJochMPCCDReader.h" - -#include -#include -#include -#include -#include -#include - -#include - -#include "../common/JFJochException.h" -#include "../common/Logger.h" -#include "../writer/HDF5Objects.h" - -namespace { - -constexpr size_t PANELS = 8; -constexpr size_t PANEL_FS = 512; // fast axis, the stored image's columns -constexpr size_t PANEL_SS = 1024; // slow axis: the eight sensors are stacked along it -constexpr size_t STACK_PIXELS = PANELS * PANEL_FS * PANEL_SS; - -// Pixels on each sensor that measure nothing usable, as the geometry files the SACLA Cheetah -// pipeline generates mask them: a 5-pixel frame on every side, and the last 23 rows of the slow -// axis, which sit under the amplifier shield. -constexpr size_t EDGE = 5; -constexpr size_t SHIELD_FIRST_SS = 1001; - -// The Cheetah-SACLA output is gain-corrected so that 1 eV deposited reads 0.00125 ADU - the -// adu_per_eV its geometry generator writes for every panel - about 10 ADU per 8 keV photon. -constexpr double ADU_PER_EV = 0.00125; - -bool IsMPCCDFile(const std::string &path) { - if (!std::filesystem::is_regular_file(path)) - return false; - bool ret = false; - std::unique_lock ul(hdf5_mutex); - H5E_BEGIN_TRY { - if (H5Fis_hdf5(path.c_str()) > 0) { - try { - HDF5ReadOnlyFile file(path); - ret = file.Exists("/metadata/sensor_id") && file.Exists("/metadata/posx_in_um") - && file.Exists("/metadata/angle_in_rad"); - } catch (...) { - ret = false; - } - } - } H5E_END_TRY; - return ret; -} - -std::vector CollectFiles(const std::string &path) { - std::vector ret; - if (std::filesystem::is_directory(path)) { - for (const auto &e: std::filesystem::directory_iterator(path)) { - const auto p = e.path().string(); - if (e.path().extension() == ".h5" && IsMPCCDFile(p)) - ret.push_back(p); - } - std::sort(ret.begin(), ret.end()); - } else if (IsMPCCDFile(path)) - ret.push_back(path); - return ret; -} - -// "+0.000390x -1.000000y" as CrystFEL writes a panel's fs and ss directions. -void ParseDirection(const std::string &txt, double &x, double &y) { - x = 0; - y = 0; - static const std::regex term(R"(([+-]?\s*[0-9.eE+-]*)\s*([xyz]))"); - for (auto it = std::sregex_iterator(txt.begin(), txt.end(), term); it != std::sregex_iterator(); ++it) { - std::string num = (*it)[1].str(); - num.erase(std::remove(num.begin(), num.end(), ' '), num.end()); - double v = 1.0; - if (num == "-") - v = -1.0; - else if (!num.empty() && num != "+") - v = std::stod(num); - if ((*it)[2] == "x") - x = v; - else if ((*it)[2] == "y") - y = v; - } -} - -} // namespace - -JFJochMPCCDReader::JFJochMPCCDReader() = default; -JFJochMPCCDReader::~JFJochMPCCDReader() { - Close(); -} - -bool JFJochMPCCDReader::CanRead(const std::string &path) { - if (std::filesystem::is_directory(path)) { - for (const auto &e: std::filesystem::directory_iterator(path)) - if (e.path().extension() == ".h5") - return IsMPCCDFile(e.path().string()); - return false; - } - return IsMPCCDFile(path); -} - -void JFJochMPCCDReader::ReadFiles(const std::string &path) { - Close(); - files_ = CollectFiles(path); - if (files_.empty()) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - "No SACLA MPCCD (Cheetah) files found for " + path); - - std::array panels; - double distance_mm = 0, photon_energy_eV = 0; - std::vector file_mask; - { - std::unique_lock ul(hdf5_mutex); - for (size_t f = 0; f < files_.size(); f++) { - open_files_.emplace_back(std::make_unique(files_[f])); - for (const auto &name: open_files_.back()->FindLeafs("/")) - if (name.starts_with("tag-")) - shots_.push_back({f, "/" + name}); - } - if (shots_.empty()) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - "No images (tag-* groups) in " + path); - - auto &file = *open_files_[0]; - const auto posx = file.ReadVector("/metadata/posx_in_um"); - const auto posy = file.ReadVector("/metadata/posy_in_um"); - // Degrees, whatever the name says: the sensors stand at +-90. - const auto angle = file.ReadVector("/metadata/angle_in_rad"); - const auto pixel = file.ReadVector("/metadata/pixelsizex_in_um"); - if (posx.size() != PANELS || posy.size() != PANELS || angle.size() != PANELS || pixel.size() != PANELS) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - files_[0] + " does not describe eight MPCCD sensors"); - // The same placement the SACLA pipeline's geometry generator writes for CrystFEL. - for (size_t i = 0; i < PANELS; i++) { - const double a = angle[i] * M_PI / 180.0; - panels[i].corner_x = -posx[i] / pixel[i]; - panels[i].corner_y = posy[i] / pixel[i]; - panels[i].fs_x = std::cos(a); - panels[i].fs_y = std::sin(a); - panels[i].ss_x = -std::sin(a); - panels[i].ss_y = std::cos(a); - } - distance_mm = HDF5DataSet(file, "/metadata/distance_in_mm").ReadScalar(); - photon_energy_eV = HDF5DataSet(file, shots_[0].group + "/photon_energy_ev").ReadScalar(); - if (file.Exists("/metadata/pixelmask")) - file_mask = file.ReadVector("/metadata/pixelmask"); - } - - if (!(photon_energy_eV > 0) || !(distance_mm > 0)) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - files_[0] + " states no photon energy or no camera length"); - Build(panels, distance_mm, photon_energy_eV, file_mask); -} - -void JFJochMPCCDReader::ApplyCrystFELGeometry(const std::string &geom_file) { - if (!dataset_) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "No MPCCD files read"); - - std::ifstream in(geom_file); - if (!in) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Cannot open " + geom_file); - - std::map keys; - std::string line; - while (std::getline(in, line)) { - line = line.substr(0, line.find(';')); - const auto eq = line.find('='); - if (eq == std::string::npos) - continue; - auto trim = [](std::string s) { - s.erase(0, s.find_first_not_of(" \t")); - s.erase(s.find_last_not_of(" \t\r") + 1); - return s; - }; - keys[trim(line.substr(0, eq))] = trim(line.substr(eq + 1)); - } - - std::array panels; - double coffset = 0; - for (size_t i = 0; i < PANELS; i++) { - const std::string q = "q" + std::to_string(i + 1) + "/"; - if (!keys.contains(q + "corner_x") || !keys.contains(q + "fs") || !keys.contains(q + "ss")) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - geom_file + " does not describe panels q1..q8"); - panels[i].corner_x = std::stod(keys[q + "corner_x"]); - panels[i].corner_y = std::stod(keys[q + "corner_y"]); - ParseDirection(keys[q + "fs"], panels[i].fs_x, panels[i].fs_y); - ParseDirection(keys[q + "ss"], panels[i].ss_x, panels[i].ss_y); - if (keys.contains(q + "coffset")) - coffset = std::stod(keys[q + "coffset"]); - } - if (!keys.contains("clen")) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, geom_file + " states no clen"); - const double distance_mm = (std::stod(keys["clen"]) + coffset) * 1000.0; - - std::vector file_mask; - { - std::unique_lock ul(hdf5_mutex); - if (open_files_[0]->Exists("/metadata/pixelmask")) - file_mask = open_files_[0]->ReadVector("/metadata/pixelmask"); - } - Build(panels, distance_mm, dataset_->experiment.GetIncidentEnergy_keV() * 1000.0, file_mask); -} - -// CrystFEL's frame has y up and x to the left looking downstream; the flat image has columns to the -// right and rows down. The two differ by a half turn about the beam, which keeps the handedness, so -// a position (X, Y) lands on column beam_x - X and row beam_y - Y. -void JFJochMPCCDReader::Build(const std::array &panels, double distance_mm, - double photon_energy_eV, const std::vector &file_mask) { - auto position = [&](size_t p, double fs, double ss) { - return std::pair{panels[p].corner_x + fs * panels[p].fs_x + ss * panels[p].ss_x, - panels[p].corner_y + fs * panels[p].fs_y + ss * panels[p].ss_y}; - }; - double max_x = -1e9, min_x = 1e9, max_y = -1e9, min_y = 1e9; - for (size_t p = 0; p < PANELS; p++) - for (double fs: {0.0, double(PANEL_FS)}) - for (double ss: {0.0, double(PANEL_SS)}) { - const auto [x, y] = position(p, fs, ss); - max_x = std::max(max_x, x); - min_x = std::min(min_x, x); - max_y = std::max(max_y, y); - min_y = std::min(min_y, y); - } - const double beam_x = std::ceil(max_x) + 1; - const double beam_y = std::ceil(max_y) + 1; - width_ = static_cast(beam_x - std::floor(min_x)) + 2; - height_ = static_cast(beam_y - std::floor(min_y)) + 2; - - // Everything starts as a gap; each sensor's usable pixels are then placed on top. - std::vector mask(width_ * height_, 1u << PixelMask::ModuleGapPixelBit); - destination_.assign(STACK_PIXELS, -1); - for (size_t p = 0; p < PANELS; p++) { - for (size_t ss = 0; ss < PANEL_SS; ss++) { - for (size_t fs = 0; fs < PANEL_FS; fs++) { - const size_t src = (p * PANEL_SS + ss) * PANEL_FS + fs; - const auto [x, y] = position(p, fs + 0.5, ss + 0.5); - const auto col = std::lround(beam_x - x); - const auto row = std::lround(beam_y - y); - const size_t dst = static_cast(row) * width_ + static_cast(col); - if (fs < EDGE || fs >= PANEL_FS - EDGE || ss < EDGE || ss >= SHIELD_FIRST_SS) - continue; - destination_[src] = static_cast(dst); - mask[dst] = (src < file_mask.size() && file_mask[src] != 0) ? (1u << PixelMask::ErrorPixelBit) : 0; - } - } - } - - adu_per_photon_ = ADU_PER_EV * photon_energy_eV; - - dataset_ = std::make_shared(); - dataset_->experiment = default_experiment; - DetectorSetup detector = DetDECTRIS(static_cast(width_), static_cast(height_), "SACLA MPCCD", {}); - detector.PixelSize_um(50.0f); - // A CCD: nothing for a parallax or sensor-efficiency correction to work with is stated in the file. - detector.SensorThickness_um(0); - // The pipeline stores signed 16-bit ADU, so that is where a pixel stops counting. - detector.SaturationLimit(static_cast(std::floor(INT16_MAX / adu_per_photon_))); - detector.BitDepthImage(32); - detector.MinFrameTime(std::chrono::microseconds(0)); - detector.MinCountTime(std::chrono::microseconds(0)); - detector.ReadOutTime(std::chrono::nanoseconds(0)); - dataset_->experiment.Detector(detector); - dataset_->experiment.BeamX_pxl(static_cast(beam_x)); - dataset_->experiment.BeamY_pxl(static_cast(beam_y)); - dataset_->experiment.DetectorDistance_mm(static_cast(distance_mm)); - dataset_->experiment.IncidentEnergy_keV(static_cast(photon_energy_eV / 1000.0)); - dataset_->experiment.ImagesPerTrigger(static_cast(shots_.size())); - dataset_->pixel_mask = std::make_shared(mask); - - Logger("MPCCDReader").Info("{} shots in {} files; {} x {} image, beam at ({}, {}) px, {:.2f} mm, " - "{:.1f} eV, {:.2f} ADU per photon", shots_.size(), files_.size(), - width_, height_, beam_x, beam_y, distance_mm, photon_energy_eV, adu_per_photon_); - SetStartMessage(dataset_); -} - -uint64_t JFJochMPCCDReader::GetNumberOfImages() const { - return shots_.size(); -} - -void JFJochMPCCDReader::Close() { - std::unique_lock ul(hdf5_mutex); - open_files_.clear(); - files_.clear(); - shots_.clear(); - dataset_.reset(); -} - -// The stored chunk is the whole image, byte-shuffled and deflated. Only reading it needs the HDF5 lock; -// it is inflated, unshuffled and placed outside it, so the decoding runs on as many threads as read. -template -void JFJochMPCCDReader::DecodeInto(int64_t image_number, Buffer &buffer) const { - if (image_number < 0 || static_cast(image_number) >= shots_.size()) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Image number out of range"); - const auto &shot = shots_[image_number]; - - std::vector compressed; - { - std::unique_lock ul(hdf5_mutex); - HDF5DataSet dataset(*open_files_[shot.file], shot.group + "/data"); - dataset.ReadDirectChunk(compressed, {0, 0}); - } - - std::vector shuffled(STACK_PIXELS * sizeof(int16_t)); - uLongf out_size = shuffled.size(); - if (uncompress(shuffled.data(), &out_size, compressed.data(), compressed.size()) != Z_OK - || out_size != shuffled.size()) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - "Cannot inflate image " + shot.group + " of " + files_[shot.file]); - - buffer.assign(width_ * height_ * sizeof(int32_t), 0); - auto out = reinterpret_cast(buffer.data()); - for (size_t i = 0; i < STACK_PIXELS; i++) { - if (destination_[i] < 0) - continue; - const auto adu = static_cast(shuffled[i] | (shuffled[STACK_PIXELS + i] << 8)); - out[destination_[i]] = static_cast(std::lround(adu / adu_per_photon_)); - } -} - -bool JFJochMPCCDReader::LoadImage_i(std::shared_ptr &dataset, - DataMessage &message, - std::vector &buffer, - int64_t image_number, - bool update_dataset) { - (void) update_dataset; - if (!dataset || image_number < 0 || static_cast(image_number) >= shots_.size()) - return false; - DecodeInto(image_number, buffer); - message.image = CompressedImage(buffer.data(), buffer.size(), width_, height_, - CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION); - message.number = image_number; - return true; -} - -bool JFJochMPCCDReader::ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) { - if (image_number < 0 || static_cast(image_number) >= shots_.size()) - return false; - DecodeInto(image_number, image.image_buffer); - image.image = CompressedImage(image.image_buffer.data(), image.image_buffer.size(), width_, height_, - CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION); - return true; -} - -std::vector JFJochMPCCDReader::ReadSpots(int64_t) const { - return {}; -} diff --git a/reader/JFJochMPCCDReader.h b/reader/JFJochMPCCDReader.h deleted file mode 100644 index bce7419c2..000000000 --- a/reader/JFJochMPCCDReader.h +++ /dev/null @@ -1,78 +0,0 @@ -// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute -// SPDX-License-Identifier: GPL-3.0-only - -#pragma once - -#include -#include -#include -#include - -#include "JFJochReader.h" - -class HDF5ReadOnlyFile; - -// Reads serial (stills) data from the SACLA multi-port CCD (MPCCD) octal detector, as the facility's -// Cheetah pipeline writes it: one HDF5 file per run part, one /tag-/data group per hit, each -// an int16 8192 x 512 image of the eight 1024 x 512 sensors stacked along the slow axis. The pixels -// are dark-subtracted and gain-corrected ADU, so no pedestal is needed; they are converted to -// photons here. /metadata states each sensor's position and in-plane angle, from which the sensors -// are placed on one flat image with the gaps between them masked. -// -// A path is one such file, or a directory: every Cheetah-SACLA file in it is read as one run, in -// name order. -class JFJochMPCCDReader : public JFJochReader { -public: - // One sensor as CrystFEL describes a panel: the position of its first pixel's corner and the - // directions of its fast and slow axes, in pixels, in CrystFEL's laboratory frame. - struct Panel { - double corner_x = 0, corner_y = 0; - double fs_x = 0, fs_y = 0, ss_x = 0, ss_y = 0; - }; - -private: - struct Shot { - size_t file; - std::string group; - }; - - std::vector files_; - std::vector> open_files_; - std::vector shots_; - std::shared_ptr dataset_; - - size_t width_ = 0, height_ = 0; - // For every pixel of the 8192 x 512 stack, where it lands on the flat image (-1: not placed). - std::vector destination_; - double adu_per_photon_ = 1.0; - - void Build(const std::array &panels, double distance_mm, double photon_energy_eV, - const std::vector &file_mask); - - bool LoadImage_i(std::shared_ptr &dataset, - DataMessage &message, - std::vector &buffer, - int64_t image_number, - bool update_dataset) override; - - template - void DecodeInto(int64_t image_number, Buffer &buffer) const; - -public: - JFJochMPCCDReader(); - ~JFJochMPCCDReader() override; - - // True for a Cheetah-SACLA MPCCD file, or a directory holding at least one. - static bool CanRead(const std::string &path); - - void ReadFiles(const std::string &path); - - // Replaces the sensor placement and the camera length /metadata states with a refined CrystFEL - // geometry file for the same 512 x 8192 layout (panels q1..q8). - void ApplyCrystFELGeometry(const std::string &geom_file); - - [[nodiscard]] uint64_t GetNumberOfImages() const override; - void Close() override; - bool ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) override; - [[nodiscard]] std::vector ReadSpots(int64_t image) const override; -}; diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index dc46c49f6..956ff62a8 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -28,7 +28,7 @@ #include "../reader/JFJochCBFReader.h" #include "../reader/JFJochMarCCDReader.h" #include "../reader/JFJochSMVReader.h" -#include "../reader/JFJochMPCCDReader.h" +#include "../reader/JFJochCrystFELReader.h" #include "../reader/JFJochStartMessageReader.h" #include "BrokerFeed.h" #include "../writer/HDF5NXmx.h" @@ -149,7 +149,7 @@ void print_usage() { std::cout << std::endl; std::cout << " Detector geometry" << std::endl; - std::cout << " --crystfel-geometry SACLA MPCCD input only: place the eight sensors and set the camera length from this refined CrystFEL geometry file (panels q1..q8 of the 512x8192 layout) instead of the nominal placement the files' /metadata states" << std::endl; + std::cout << " --crystfel-geometry Read - an HDF5/CXI file, or a directory of them - as CrystFEL does with this geometry file: frames from its data path, panels placed at its corners, camera length, pixel size, gain, photon energy and bad regions from it. One wavelength and one camera length for the run; panel positions are taken as given" << std::endl; std::cout << std::endl; std::cout << " Detector mask" << std::endl; @@ -994,7 +994,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { float false_pixels_per_frame = SpotFindingSettings{}.false_pixels_per_frame; std::string ref_mtz; std::string ref_column; - std::string crystfel_geometry; // --crystfel-geometry: refined MPCCD sensor placement + std::string crystfel_geometry; // --crystfel-geometry: read the input as CrystFEL would with this geometry std::string model_pdb; // --model: PDB to validate merged intensities against (R-free + maps) std::string mr_model; // --mr: --model, placed by molecular replacement straight away bool export_unmerged = true; // on by default; --no-export-unmerged turns it off @@ -1729,20 +1729,25 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { SimulatedLive cbf_reader; SimulatedLive marccd_reader; SimulatedLive smv_reader; - SimulatedLive mpccd_reader; + SimulatedLive crystfel_reader; JFJochStartMessageReader live_reader; std::unique_ptr broker_feed; JFJochReader *reader_ptr = nullptr; // Which format this is. Asked INSIDE the try: these open and read the file - a directory whose // first plausible member cannot be read throws here - and an input the program cannot read has to // come out as the message below, not as an unhandled exception with nothing printed at all. - bool input_is_cbf = false, input_is_marccd = false, input_is_smv = false, input_is_mpccd = false; + bool input_is_cbf = false, input_is_marccd = false, input_is_smv = false, input_is_crystfel = false; try { - input_is_cbf = JFJochCBFReader::CanRead(input_file); + input_is_crystfel = !crystfel_geometry.empty(); + input_is_cbf = !input_is_crystfel && JFJochCBFReader::CanRead(input_file); input_is_marccd = !input_is_cbf && JFJochMarCCDReader::CanRead(input_file); input_is_smv = !input_is_cbf && !input_is_marccd && JFJochSMVReader::CanRead(input_file); - input_is_mpccd = !input_is_cbf && !input_is_marccd && !input_is_smv && JFJochMPCCDReader::CanRead(input_file); - if (input_is_cbf) { + if (input_is_crystfel) { + crystfel_reader.ReadFiles(input_file, crystfel_geometry); + reader_ptr = &crystfel_reader; + logger.Info("Read {} frames from {} with the CrystFEL geometry {}", + crystfel_reader.GetNumberOfImages(), input_file, crystfel_geometry); + } else if (input_is_cbf) { cbf_reader.ReadFiles(input_file); reader_ptr = &cbf_reader; logger.Info("Read {} CBF images of the sweep containing {}", @@ -1757,12 +1762,6 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { reader_ptr = &smv_reader; logger.Info("Read {} SMV images of the sweep containing {}", smv_reader.GetNumberOfImages(), input_file); - } else if (input_is_mpccd) { - mpccd_reader.ReadFiles(input_file); - if (!crystfel_geometry.empty()) - mpccd_reader.ApplyCrystFELGeometry(crystfel_geometry); - reader_ptr = &mpccd_reader; - logger.Info("Read {} SACLA MPCCD images from {}", mpccd_reader.GetNumberOfImages(), input_file); } else if (std::filesystem::is_directory(input_file)) { // A directory is never an HDF5 master, so handing it to the HDF5 reader only buries the // real answer - that no one-file-per-image sweep was found in it - under a page of HDF5 @@ -1937,7 +1936,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { "runs no space-group search. Use --mode mx, which makes the P1 merge the " "ledger is folded from."); // Re-scaling reads reflections a previous run integrated, which only a _process.h5 holds. - if (input_is_cbf || input_is_marccd || input_is_smv || input_is_mpccd) { + if (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel) { logger.Error("--mode scale needs the integrated reflections in a _process.h5; " "a raw image sweep holds none. Run --mode mx on it first."); exit(EXIT_FAILURE); @@ -3168,7 +3167,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { config.detect_beam_stop = detect_beam_stop; config.beam_center_check = beam_center_check; config.adaptive_integration_radius = adaptive_integration_radius; - config.counts_in_adu = input_is_smv || input_is_marccd; + config.counts_in_adu = input_is_smv || input_is_marccd || (input_is_crystfel && !crystfel_reader.CountsInPhotons()); config.rotation_postrefine_geometry = rotation_postrefine_geometry; // Measure the spot budget from the data unless the user pinned it. config.measure_spot_budget = !max_spot_count_override.has_value(); -- 2.54.0 From acf6b3604fe52119d65e474c88899068e2592de8 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 13:42:19 +0200 Subject: [PATCH 28/53] tests: CrystFEL geometry parser - old-syntax fixture and a multi-panel file with path-valued clen, directions, masks and bad regions Co-Authored-By: Claude Opus 5.5 --- tests/CMakeLists.txt | 2 + tests/CrystFELGeometryTest.cpp | 105 +++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+) create mode 100644 tests/CrystFELGeometryTest.cpp diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 48b37353b..b7eb6dae8 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -10,6 +10,7 @@ ADD_EXECUTABLE(jfjoch_portable_test EXCLUDE_FROM_ALL TestData.h ZSTDCompressorTest.cpp JFJochReaderTest.cpp + CrystFELGeometryTest.cpp SweepLayoutTest.cpp CBORTest.cpp ImageSpotFinderCPUTest.cpp @@ -99,6 +100,7 @@ ADD_EXECUTABLE(jfjoch_test ModuleSummationTest.cpp ZMQMetadataSocketTest.cpp JFJochReaderTest.cpp + CrystFELGeometryTest.cpp LiveCollectionTest.cpp RugnuxLiveStatusTest.cpp SweepLayoutTest.cpp diff --git a/tests/CrystFELGeometryTest.cpp b/tests/CrystFELGeometryTest.cpp new file mode 100644 index 000000000..9b5ef3f81 --- /dev/null +++ b/tests/CrystFELGeometryTest.cpp @@ -0,0 +1,105 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include +#include + +#include "../reader/CrystFELGeometry.h" + +namespace { + +std::string WriteGeom(const std::string &name, const std::string &text) { + const auto path = (std::filesystem::temp_directory_path() / name).string(); + std::ofstream(path) << text; + return path; +} + +} // namespace + +TEST_CASE("CrystFELGeometry_OldSyntax", "[CrystFELGeometry][portable]") { + // One panel and whole-frame bad strips, global keys only: the old syntax. + const auto geom = crystfel::Read("../../tests/crystfel/jf4m.geom"); + REQUIRE(geom.panels.size() == 1); + CHECK(geom.panels[0].res == Catch::Approx(13333.3)); + CHECK(geom.panels[0].clen == "0.075"); + CHECK(geom.panels[0].adu_per_photon.value() == Catch::Approx(1.0)); + CHECK(geom.photon_energy == "12400"); + CHECK(geom.data == "/entry/data/data"); + REQUIRE(!geom.bad.empty()); + CHECK(geom.bad[0].panel.empty()); + CHECK(!geom.bad[0].in_lab); + CHECK(geom.bad[0].min_fs == 1030); + CHECK(geom.bad[0].max_fs == 1037); + CHECK(geom.bad[0].max_ss == 2163); +} + +TEST_CASE("CrystFELGeometry_Panels", "[CrystFELGeometry][portable]") { + const auto path = WriteGeom("jfjoch_crystfel_geometry_test.geom", R"( +clen = /LCLS/detector_1/EncoderValue ; read from the file +res = 20000 +adu_per_eV = 0.001 ; per eV +data = /%/data +photon_energy = /%/photon_energy_ev +mask = /data/data +mask_file = mask.h5 +mask_good = 0x0 +mask_bad = 0x1 +q1/min_fs = 0 +q1/max_fs = 511 +q1/min_ss = 0 +q1/max_ss = 1023 +q1/fs = +0.000390x -1.000000y +q1/ss = +1.000000x +0.000390y +q1/corner_x = 32.9283 +q1/corner_y = 1033.68 +q1/coffset = 0.000067 +q2/min_ss = 1024 +q2/max_ss = 2047 +q2/max_fs = 511 +q2/fs = -x +q2/ss = y +q2/adu_per_photon = 9.5 +badq1/min_fs = 0 +badq1/max_fs = 4 +badq1/min_ss = 0 +badq1/max_ss = 1023 +badq1/panel = q1 +badbox/min_x = -10 +badbox/max_x = 10 +badbox/min_y = -5 +badbox/max_y = 5 +)"); + const auto geom = crystfel::Read(path); + REQUIRE(geom.panels.size() == 2); + const auto &q1 = geom.panels[0]; + CHECK(q1.name == "q1"); + CHECK(q1.clen == "/LCLS/detector_1/EncoderValue"); + CHECK(q1.coffset == Catch::Approx(0.000067)); + CHECK(q1.adu_per_eV.value() == Catch::Approx(0.001)); + CHECK(q1.fs_x == Catch::Approx(0.00039)); + CHECK(q1.fs_y == Catch::Approx(-1.0)); + CHECK(q1.ss_x == Catch::Approx(1.0)); + CHECK(q1.ss_y == Catch::Approx(0.00039)); + CHECK(q1.corner_y == Catch::Approx(1033.68)); + CHECK(q1.max_ss == 1023); + + const auto &q2 = geom.panels[1]; + CHECK(q2.fs_x == Catch::Approx(-1.0)); + CHECK(q2.fs_y == Catch::Approx(0.0)); + CHECK(q2.ss_y == Catch::Approx(1.0)); + CHECK(q2.min_ss == 1024); + CHECK(q2.adu_per_photon.value() == Catch::Approx(9.5)); + CHECK(q2.res == Catch::Approx(20000)); + + CHECK(geom.data == "/%/data"); + CHECK(geom.mask_file == "mask.h5"); + CHECK(geom.mask_bad == 1); + REQUIRE(geom.bad.size() == 2); + CHECK(geom.bad[0].panel == "q1"); + CHECK(geom.bad[0].max_fs == 4); + CHECK(geom.bad[1].in_lab); + CHECK(geom.bad[1].min_x == Catch::Approx(-10)); + CHECK(geom.bad[1].max_y == Catch::Approx(5)); + std::filesystem::remove(path); +} -- 2.54.0 From ae4786d14fb2e838d1c19ff31c9da09fd0b030d1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 11:57:54 +0200 Subject: [PATCH 29/53] WIP: LCLS-I XTC reader for single-panel camera stills (Rayonix), geometry from a DIALS .expt Not yet compiled or tested: work stopped after the feasibility study. Co-Authored-By: Claude Opus 5.5 --- reader/CMakeLists.txt | 4 + reader/JFJochXTCReader.cpp | 186 +++++++++++++++++++++++++++++++++++++ reader/JFJochXTCReader.h | 53 +++++++++++ reader/XTC.cpp | 149 +++++++++++++++++++++++++++++ reader/XTC.h | 50 ++++++++++ rugnux/rugnux_cli.cpp | 16 +++- 6 files changed, 455 insertions(+), 3 deletions(-) create mode 100644 reader/JFJochXTCReader.cpp create mode 100644 reader/JFJochXTCReader.h create mode 100644 reader/XTC.cpp create mode 100644 reader/XTC.h diff --git a/reader/CMakeLists.txt b/reader/CMakeLists.txt index fe0f0d7ec..1a07866cc 100644 --- a/reader/CMakeLists.txt +++ b/reader/CMakeLists.txt @@ -20,6 +20,10 @@ ADD_LIBRARY(JFJochReader STATIC SMV.h JFJochSMVReader.cpp JFJochSMVReader.h + XTC.cpp + XTC.h + JFJochXTCReader.cpp + JFJochXTCReader.h HDF5ImageLocator.cpp JFJochStartMessageReader.cpp JFJochStartMessageReader.h diff --git a/reader/JFJochXTCReader.cpp b/reader/JFJochXTCReader.cpp new file mode 100644 index 000000000..baea3d85d --- /dev/null +++ b/reader/JFJochXTCReader.cpp @@ -0,0 +1,186 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "JFJochXTCReader.h" + +#include +#include +#include + +#include + +#include "../common/JFJochException.h" +#include "../common/Logger.h" + +namespace { + +using Vec = std::array; + +double Dot(const Vec &a, const Vec &b) { return a[0] * b[0] + a[1] * b[1] + a[2] * b[2]; } + +Vec Cross(const Vec &a, const Vec &b) { + return {a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]}; +} + +Vec ReadVec(const nlohmann::json &j) { return {j.at(0).get(), j.at(1).get(), j.at(2).get()}; } + +// A DIALS laboratory vector in the internal frame. DIALS uses the imgCIF laboratory frame (z from +// the sample to the source, y up), which differs from the internal one by a half turn about x. +Coord DialsToInternal(const Vec &v) { + return {static_cast(v[0]), static_cast(-v[1]), static_cast(-v[2])}; +} + +// The DIALS experiment list beside the streams, or in the directory above them. +std::string FindExpt(const std::string &stream) { + const auto dir = std::filesystem::absolute(stream).parent_path(); + for (const auto &d: {dir, dir.parent_path()}) { + std::error_code ec; + for (const auto &e: std::filesystem::directory_iterator(d, ec)) + if (e.is_regular_file() && e.path().extension() == ".expt") + return e.path().string(); + } + return {}; +} + +} // namespace + +bool JFJochXTCReader::CanRead(const std::string &path) { + return xtc::CanRead(path); +} + +void JFJochXTCReader::ReadGeometry(const std::string &expt) { + std::ifstream in(expt); + const auto j = nlohmann::json::parse(in); + const auto &beam = j.at("beam").at(0); + const auto &panels = j.at("detector").at(0).at("panels"); + if (panels.size() != 1) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + expt + " describes a detector of " + std::to_string(panels.size()) + + " panels; only a single-panel detector is read from XTC"); + const auto &panel = panels.at(0); + + const double wavelength_A = beam.at("wavelength").get(); + // DIALS states the direction from the sample to the source; the beam travels the other way. + const Vec direction = ReadVec(beam.at("direction")); + const Vec b = {-direction[0], -direction[1], -direction[2]}; + const Vec fast = ReadVec(panel.at("fast_axis")), slow = ReadVec(panel.at("slow_axis")); + const Vec origin = ReadVec(panel.at("origin")); // mm, corner of the first pixel + const double pixel_mm = panel.at("pixel_size").at(0).get(); + + // Where the beam meets the detector plane, in pixels along fast and slow, and how far along the + // detector normal the plane is. + const Vec normal = Cross(fast, slow); + const double t = Dot(origin, normal) / Dot(b, normal); + const Vec hit = {t * b[0] - origin[0], t * b[1] - origin[1], t * b[2] - origin[2]}; + if (std::fabs(Dot(b, normal)) < 0.9999) + Logger("XTCReader").Warning("{}: the detector is tilted by {:.2f} deg against the beam; it is read " + "as square-on", expt, std::acos(std::fabs(Dot(b, normal))) * 180.0 / M_PI); + + auto &experiment = dataset_->experiment; + const auto &f0 = frames_.front(); + DetectorSetup detector = DetDECTRIS(f0.width, f0.height, "XTC camera", {}); + detector.PixelSize_um(static_cast(pixel_mm * 1000.0)); + // A CCD: the phosphor converts at the surface, so there is no depth to correct parallax for. + detector.SensorThickness_um(0); + detector.SaturationLimit(65535); + detector.BitDepthImage(32); + const auto orientation = DetectorOrientation::Match(DialsToInternal(fast), DialsToInternal(slow)); + if (!orientation.has_value()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + expt + ": the panel's fast and slow axes are not a square-on layout this reader supports"); + detector.ImageOrientation(orientation.value()); + detector.MinFrameTime(std::chrono::microseconds(0)); + detector.MinCountTime(std::chrono::microseconds(0)); + detector.ReadOutTime(std::chrono::nanoseconds(0)); + experiment.Detector(detector); + + experiment.BeamX_pxl(static_cast(Dot(hit, fast) / pixel_mm)); + experiment.BeamY_pxl(static_cast(Dot(hit, slow) / pixel_mm)); + experiment.DetectorDistance_mm(static_cast(std::fabs(Dot(origin, normal)))); + experiment.IncidentEnergy_keV(WVL_1A_IN_KEV / static_cast(wavelength_A)); + + // The polarization degree is 2f - 1 of the fraction f DIALS states; positive where the plane + // contains the lab x axis (normal along y), as in the SMV reader. + if (beam.contains("polarization_fraction") && beam.contains("polarization_normal")) { + const Coord n = DialsToInternal(ReadVec(beam.at("polarization_normal"))); + const float degree = 2.0f * beam.at("polarization_fraction").get() - 1.0f; + experiment.PolarizationFactor(std::fabs(n.x) > std::fabs(n.y) ? -degree : degree); + } +} + +void JFJochXTCReader::ReadFiles(const std::string &path) { + frames_.clear(); + for (const auto &stream: xtc::CollectStreams(path)) { + auto f = xtc::IndexFrames(stream); + frames_.insert(frames_.end(), f.begin(), f.end()); + } + if (frames_.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "No camera images found in the XTC of " + path); + for (const auto &f: frames_) + if (f.width != frames_.front().width || f.height != frames_.front().height) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + f.file + ": the images are not all the same size"); + + const std::string expt = FindExpt(frames_.front().file); + if (expt.empty()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "XTC carries no detector geometry: put the DIALS experiment list (.expt) of " + "the run beside the XTC files, or in the directory above them"); + Logger("XTCReader").Info("{} images in XTC, geometry from {}", frames_.size(), expt); + + dataset_ = std::make_shared(); + dataset_->experiment = default_experiment; + ReadGeometry(expt); + dataset_->error_value = -1; + dataset_->experiment.ImagesPerTrigger(static_cast(frames_.size())); + dataset_->pixel_mask = std::make_shared(frames_.front().width, frames_.front().height); + + SetStartMessage(dataset_); +} + +uint64_t JFJochXTCReader::GetNumberOfImages() const { + return frames_.size(); +} + +void JFJochXTCReader::Close() { + frames_.clear(); + dataset_.reset(); +} + +template +CompressedImage JFJochXTCReader::DecodeInto(int64_t image_number, Buffer &buffer, + std::vector &scratch) const { + if (image_number < 0 || static_cast(image_number) >= frames_.size()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Image number out of range"); + const auto &f = frames_[image_number]; + buffer.resize(static_cast(f.width) * f.height * sizeof(int32_t)); + xtc::ReadFrame(f, reinterpret_cast(buffer.data()), scratch); + return CompressedImage(buffer.data(), buffer.size(), f.width, f.height, + CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION); +} + +bool JFJochXTCReader::LoadImage_i(std::shared_ptr &dataset, + DataMessage &message, + std::vector &buffer, + int64_t image_number, + bool update_dataset) { + (void) update_dataset; + if (!dataset || image_number < 0 || static_cast(image_number) >= frames_.size()) + return false; + std::vector scratch; + message.image = DecodeInto(image_number, buffer, scratch); + message.number = image_number; + return true; +} + +bool JFJochXTCReader::ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) { + if (image_number < 0 || static_cast(image_number) >= frames_.size()) + return false; + image.image = DecodeInto(image_number, image.image_buffer, image.read_buffer); + return true; +} + +std::vector JFJochXTCReader::ReadSpots(int64_t) const { + return {}; // raw XTC stores no analysis results +} diff --git a/reader/JFJochXTCReader.h b/reader/JFJochXTCReader.h new file mode 100644 index 000000000..6b22eb4ef --- /dev/null +++ b/reader/JFJochXTCReader.h @@ -0,0 +1,53 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include + +#include "JFJochReader.h" +#include "XTC.h" + +// Reads serial-crystallography stills straight from LCLS-I XTC files: one camera image per event, +// from a single-panel detector such as the Rayonix CCDs at MFX and XPP. A multi-panel detector +// (CSPAD, ePix, JUNGFRAU at LCLS) is stored as raw per-module data with its own calibration, and is +// not read here. +// +// XTC records the events and not the experiment: there is no distance, beam centre or wavelength +// in it that this reader can use. Those come from a DIALS experiment list (.expt) beside the XTC +// files, or in their parent directory - the geometry a deposition ships for cctbx.xfel, and what +// cctbx.xfel itself reads them with. Without one the run is refused rather than guessed. +// +// The images are stills: no goniometer. Each is decoded on demand. The pixels are passed on as +// stored - the CCD's constant offset of a few counts is left in, as it is in an SMV image, and the +// pre-scan finds the defective pixels. +class JFJochXTCReader : public JFJochReader { + std::vector frames_; + std::shared_ptr dataset_; + + bool LoadImage_i(std::shared_ptr &dataset, + DataMessage &message, + std::vector &buffer, + int64_t image_number, + bool update_dataset) override; + + template + CompressedImage DecodeInto(int64_t image_number, Buffer &buffer, std::vector &scratch) const; + + void ReadGeometry(const std::string &expt); + +public: + ~JFJochXTCReader() override = default; + + static bool CanRead(const std::string &path); + + // path is one XTC stream, or a directory whose streams are all read. + void ReadFiles(const std::string &path); + + [[nodiscard]] uint64_t GetNumberOfImages() const override; + void Close() override; + + bool ReadRawImage(int64_t image_number, JFJochReaderRawImage &image) override; + [[nodiscard]] std::vector ReadSpots(int64_t image) const override; +}; diff --git a/reader/XTC.cpp b/reader/XTC.cpp new file mode 100644 index 000000000..03dd0de4c --- /dev/null +++ b/reader/XTC.cpp @@ -0,0 +1,149 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "XTC.h" + +#include +#include +#include +#include +#include + +#include "../common/JFJochException.h" + +namespace { + +constexpr size_t DGRAM_HEADER = 40; // clock time 8, timestamp 8, env 4, Xtc header 20 +constexpr size_t XTC_HEADER = 20; +constexpr size_t FRAME_HEADER = 16; // Camera::FrameV1: width, height, depth, offset +constexpr uint32_t ID_XTC = 1; +constexpr uint32_t ID_FRAME = 2; +constexpr uint32_t TRANSITION_L1ACCEPT = 12; +constexpr uint32_t TRANSITION_LAST = 12; + +struct FileCloser { + void operator()(FILE *f) const { if (f) fclose(f); } +}; +using File = std::unique_ptr; + +File Open(const std::string &path) { + File f(fopen(path.c_str(), "rb")); + if (!f) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Cannot open " + path); + return f; +} + +uint32_t Word(const uint8_t *p, size_t i) { + uint32_t v; + memcpy(&v, p + 4 * i, 4); + return v; +} + +// The transition a datagram header names, from the top bits of its timestamp's first word. +uint32_t Transition(const uint8_t *dgram) { + return (Word(dgram, 2) >> 24) & 0xf; +} + +// A first datagram is a Configure with a container as its outermost Xtc, whose extent covers at +// least its own header. +bool PlausibleFirstDatagram(const uint8_t *h) { + return Transition(h) <= TRANSITION_LAST && (Word(h, 8) & 0xffff) == ID_XTC && Word(h, 9) >= XTC_HEADER; +} + +// Walks the Xtcs in [begin, end) of the file, looking for camera frames; the largest found is kept. +void FindFrame(FILE *f, uint64_t begin, uint64_t end, xtc::Frame &best) { + uint64_t pos = begin; + while (pos + XTC_HEADER <= end) { + uint8_t h[XTC_HEADER]; + if (fseeko(f, static_cast(pos), SEEK_SET) != 0 || fread(h, 1, XTC_HEADER, f) != XTC_HEADER) + return; + const uint32_t type = Word(h, 3) & 0xffff; + const uint32_t extent = Word(h, 4); + if (extent < XTC_HEADER || pos + extent > end) + return; + if (type == ID_XTC) + FindFrame(f, pos + XTC_HEADER, pos + extent, best); + else if (type == ID_FRAME && extent >= XTC_HEADER + FRAME_HEADER) { + uint8_t fh[FRAME_HEADER]; + if (fread(fh, 1, FRAME_HEADER, f) == FRAME_HEADER) { + const uint32_t width = Word(fh, 0), height = Word(fh, 1), depth = Word(fh, 2); + const uint64_t npixel = static_cast(width) * height; + if (depth > 8 && depth <= 16 && XTC_HEADER + FRAME_HEADER + 2 * npixel <= extent + && npixel > static_cast(best.width) * best.height) + best = {"", pos + XTC_HEADER + FRAME_HEADER, width, height}; + } + } + pos += extent; + } +} + +} // namespace + +namespace xtc { + +bool CanRead(const std::string &path) { + std::error_code ec; + if (std::filesystem::is_directory(path, ec)) + return !CollectStreams(path).empty(); + if (!std::filesystem::is_regular_file(path, ec) || !path.ends_with(".xtc") || path.ends_with(".smd.xtc")) + return false; + File f(fopen(path.c_str(), "rb")); + uint8_t h[DGRAM_HEADER]; + return f && fread(h, 1, DGRAM_HEADER, f.get()) == DGRAM_HEADER && PlausibleFirstDatagram(h); +} + +std::vector CollectStreams(const std::string &path) { + std::error_code ec; + if (!std::filesystem::is_directory(path, ec)) + return {path}; + std::vector out; + for (const auto &e: std::filesystem::directory_iterator(path, ec)) { + const std::string p = e.path().string(); + if (e.is_regular_file() && p.ends_with(".xtc") && !p.ends_with(".smd.xtc")) + out.push_back(p); + } + std::sort(out.begin(), out.end()); + return out; +} + +std::vector IndexFrames(const std::string &file) { + File f = Open(file); + std::vector out; + uint64_t pos = 0; + uint8_t h[DGRAM_HEADER]; + while (fseeko(f.get(), static_cast(pos), SEEK_SET) == 0 + && fread(h, 1, DGRAM_HEADER, f.get()) == DGRAM_HEADER) { + const uint32_t extent = Word(h, 9); + if (extent < XTC_HEADER) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + file + ": a datagram's extent is shorter than its own header"); + const uint64_t payload = pos + DGRAM_HEADER, end = pos + DGRAM_HEADER - XTC_HEADER + extent; + if (Transition(h) == TRANSITION_L1ACCEPT) { + Frame frame; + FindFrame(f.get(), payload, end, frame); + if (frame.width > 0) { + frame.file = file; + out.push_back(frame); + } + } + pos = end; + } + return out; +} + +void ReadFrame(const Frame &frame, int32_t *out, std::vector &scratch) { + const size_t npixel = static_cast(frame.width) * frame.height; + scratch.resize(2 * npixel); + File f = Open(frame.file); + if (fseeko(f.get(), static_cast(frame.offset), SEEK_SET) != 0 + || fread(scratch.data(), 1, scratch.size(), f.get()) != scratch.size()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + frame.file + ": an image ends past the end of the file"); + for (size_t i = 0; i < npixel; i++) { + uint16_t v; + memcpy(&v, scratch.data() + 2 * i, 2); + out[i] = v; + } +} + +} // namespace xtc diff --git a/reader/XTC.h b/reader/XTC.h new file mode 100644 index 000000000..8e2dcff6a --- /dev/null +++ b/reader/XTC.h @@ -0,0 +1,50 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include + +// The LCLS-I XTC container, as far as reading camera frames out of it needs: what the LCLS data +// system (pdsdata) wrote at LCLS until 2023, and what serial-crystallography depositions from that +// era (Rayonix at MFX and XPP) are archived as. +// +// A file is a sequence of datagrams. Each is a 40-byte header - the event's clock time and timestamp +// (16 bytes, whose top bits name the transition: Configure, BeginRun, ..., L1Accept for an event), +// an environment word, and the outermost Xtc header - followed by the Xtc's payload. An Xtc header is +// five 32-bit words: damage, the source (two words), the type and version, and the extent, which +// counts the header itself. A container (type Id_Xtc) holds further Xtcs one after another; anything +// else is a leaf holding one data object. A camera image is a leaf of type Id_Frame (Camera::FrameV1): +// width, height, bit depth and offset, 32 bits each, then the pixels row by row. Everything is +// little-endian and uncompressed. +// +// A run is written as several streams (-s00, -s01, ...) by different recorders, each holding a +// share of the events. Their order across streams is the event time, which this reader does not +// reconstruct: stills are independent, so the streams are read one after another. +namespace xtc { + +// One camera image: where its pixels start in which file. +struct Frame { + std::string file; + uint64_t offset = 0; // of the first pixel + uint32_t width = 0; + uint32_t height = 0; +}; + +// True if the path is an XTC file (or a directory holding one) - by its first datagram, not its name. +bool CanRead(const std::string &path); + +// The event streams of the path: the file itself, or every XTC file in a directory, by name. The +// "smalldata" copies (.smd.xtc) carry the events without their images and are left out. +std::vector CollectStreams(const std::string &path); + +// Every L1Accept of the file that holds a 16-bit camera image, in file order. Where an event holds +// images from more than one camera, the largest is taken - the diffraction detector. +std::vector IndexFrames(const std::string &file); + +// Reads one image as 32-bit pixels into out (width*height values), through scratch. +void ReadFrame(const Frame &frame, int32_t *out, std::vector &scratch); + +} // namespace xtc diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 956ff62a8..09d2d711d 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -29,6 +29,7 @@ #include "../reader/JFJochMarCCDReader.h" #include "../reader/JFJochSMVReader.h" #include "../reader/JFJochCrystFELReader.h" +#include "../reader/JFJochXTCReader.h" #include "../reader/JFJochStartMessageReader.h" #include "BrokerFeed.h" #include "../writer/HDF5NXmx.h" @@ -1730,6 +1731,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { SimulatedLive marccd_reader; SimulatedLive smv_reader; SimulatedLive crystfel_reader; + SimulatedLive xtc_reader; JFJochStartMessageReader live_reader; std::unique_ptr broker_feed; JFJochReader *reader_ptr = nullptr; @@ -1737,11 +1739,14 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { // first plausible member cannot be read throws here - and an input the program cannot read has to // come out as the message below, not as an unhandled exception with nothing printed at all. bool input_is_cbf = false, input_is_marccd = false, input_is_smv = false, input_is_crystfel = false; + bool input_is_xtc = false; try { input_is_crystfel = !crystfel_geometry.empty(); input_is_cbf = !input_is_crystfel && JFJochCBFReader::CanRead(input_file); input_is_marccd = !input_is_cbf && JFJochMarCCDReader::CanRead(input_file); input_is_smv = !input_is_cbf && !input_is_marccd && JFJochSMVReader::CanRead(input_file); + input_is_xtc = !input_is_crystfel && !input_is_cbf && !input_is_marccd && !input_is_smv + && JFJochXTCReader::CanRead(input_file); if (input_is_crystfel) { crystfel_reader.ReadFiles(input_file, crystfel_geometry); reader_ptr = &crystfel_reader; @@ -1762,11 +1767,15 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { reader_ptr = &smv_reader; logger.Info("Read {} SMV images of the sweep containing {}", smv_reader.GetNumberOfImages(), input_file); + } else if (input_is_xtc) { + xtc_reader.ReadFiles(input_file); + reader_ptr = &xtc_reader; + logger.Info("Read {} XTC still images from {}", xtc_reader.GetNumberOfImages(), input_file); } else if (std::filesystem::is_directory(input_file)) { // A directory is never an HDF5 master, so handing it to the HDF5 reader only buries the // real answer - that no one-file-per-image sweep was found in it - under a page of HDF5 // stack output. - logger.Error("{} holds no CBF, marCCD or SMV sweep this program can read (a directory is " + logger.Error("{} holds no CBF, marCCD, SMV or XTC data this program can read (a directory is " "read as the sweep of frames it holds)", input_file); exit(EXIT_FAILURE); } else if (!broker_url.empty() && !std::filesystem::exists(input_file)) { @@ -1936,7 +1945,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { "runs no space-group search. Use --mode mx, which makes the P1 merge the " "ledger is folded from."); // Re-scaling reads reflections a previous run integrated, which only a _process.h5 holds. - if (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel) { + if (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_xtc) { logger.Error("--mode scale needs the integrated reflections in a _process.h5; " "a raw image sweep holds none. Run --mode mx on it first."); exit(EXIT_FAILURE); @@ -3167,7 +3176,8 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { config.detect_beam_stop = detect_beam_stop; config.beam_center_check = beam_center_check; config.adaptive_integration_radius = adaptive_integration_radius; - config.counts_in_adu = input_is_smv || input_is_marccd || (input_is_crystfel && !crystfel_reader.CountsInPhotons()); + config.counts_in_adu = input_is_smv || input_is_marccd || input_is_xtc + || (input_is_crystfel && !crystfel_reader.CountsInPhotons()); config.rotation_postrefine_geometry = rotation_postrefine_geometry; // Measure the spot budget from the data unless the user pinned it. config.measure_spot_budget = !max_spot_count_override.has_value(); -- 2.54.0 From 2a45eeb83126588ee00049a23a3376703b0e7e9f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 13:23:24 +0200 Subject: [PATCH 30/53] JFJochXTCReader: unit test with a synthetic XTC stream and a one-panel DIALS .expt The test writes a Configure and two L1Accepts, the first with a second, smaller camera beside the diffraction detector, and checks that the reader takes the larger image, reads the geometry, wavelength and polarization from the .expt, and hands out the stored pixels. The reader itself (previous commit) now builds and has processed an LCLS MFX Rayonix stream end to end. Co-Authored-By: Claude Opus 5.5 --- tests/JFJochReaderTest.cpp | 89 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/tests/JFJochReaderTest.cpp b/tests/JFJochReaderTest.cpp index 0362cd493..a5d1aee8f 100644 --- a/tests/JFJochReaderTest.cpp +++ b/tests/JFJochReaderTest.cpp @@ -10,10 +10,12 @@ #include "../reader/JFJochCBFReader.h" #include "../reader/JFJochMarCCDReader.h" #include "../reader/JFJochSMVReader.h" +#include "../reader/JFJochXTCReader.h" #include "../reader/MiniCBF.h" #include "../compression/JFJochCompressor.h" #include +#include #include #include #include @@ -4832,3 +4834,90 @@ TEST_CASE("JFJochReader_DECTRISMultichannel", "[HDF5][Full]") { // No leftover HDF5 objects REQUIRE(H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0); } + +// LCLS-I XTC: datagrams of a 40-byte header and a tree of Xtcs; a camera image is a Camera::FrameV1 +// leaf. The fixture writes a Configure and two L1Accepts. The first carries a second, smaller camera +// beside the diffraction detector, which the reader must pass over; the geometry comes from a +// one-panel DIALS experiment list beside the stream. +namespace { + void PutWords(std::string &s, std::initializer_list words) { + for (uint32_t w: words) + s.append(reinterpret_cast(&w), 4); + } + + std::string XtcLeaf(uint32_t type, const std::string &payload) { + std::string s; + PutWords(s, {0, 0, 0, type, static_cast(20 + payload.size())}); + return s + payload; + } + + std::string XtcFrame(uint32_t width, uint32_t height, uint16_t base) { + std::string p; + PutWords(p, {width, height, 16, 0}); + for (uint32_t i = 0; i < width * height; i++) { + const uint16_t v = base + i; + p.append(reinterpret_cast(&v), 2); + } + return XtcLeaf(2 | (1u << 16), p); + } + + std::string XtcDatagram(uint32_t transition, const std::string &payload) { + std::string s; + PutWords(s, {0, 0, transition << 24, 0, 0}); + PutWords(s, {0, 0, 0, 1, static_cast(20 + payload.size())}); + return s + payload; + } +} + +TEST_CASE("JFJochXTCReader_Geometry", "[portable]") { + const uint32_t nx = 24, ny = 16; + std::filesystem::create_directories("xtc_test"); + { + std::ofstream f("xtc_test/run-s00-c00.xtc", std::ios::binary); + f << XtcDatagram(4, ""); + f << XtcDatagram(12, XtcLeaf(1, XtcLeaf(13, std::string(32, '\0')) + XtcFrame(4, 4, 7)) + + XtcLeaf(1, XtcFrame(nx, ny, 100))); + f << XtcDatagram(12, XtcLeaf(1, XtcFrame(nx, ny, 200))); + } + { + // The beam meets the panel 12 pixels along fast and 8 along slow from the first pixel's + // corner, 150 mm away; 0.1 mm pixels. + std::ofstream f("xtc_test/geometry.expt"); + f << R"({"beam": [{"direction": [0, 0, 1], "wavelength": 1.3, "polarization_normal": [0, 1, 0], + "polarization_fraction": 0.999}], + "detector": [{"panels": [{"fast_axis": [1, 0, 0], "slow_axis": [0, -1, 0], + "origin": [-1.2, 0.8, -150], "pixel_size": [0.1, 0.1], + "image_size": [24, 16]}]}]})"; + } + + REQUIRE(JFJochXTCReader::CanRead("xtc_test/run-s00-c00.xtc")); + REQUIRE(JFJochXTCReader::CanRead("xtc_test")); + CHECK_FALSE(JFJochXTCReader::CanRead("xtc_test/geometry.expt")); + + JFJochXTCReader reader; + REQUIRE_NOTHROW(reader.ReadFiles("xtc_test")); + CHECK(reader.GetNumberOfImages() == 2); + const auto x = reader.GetDataset()->experiment; + CHECK(x.GetXPixelsNum() == nx); + CHECK(x.GetYPixelsNum() == ny); + CHECK(x.GetDetectorDistance_mm() == Catch::Approx(150.0)); + CHECK(x.GetPixelSize_mm() == Catch::Approx(0.1)); + CHECK(x.GetWavelength_A() == Catch::Approx(1.3).epsilon(1e-4)); + CHECK(x.GetBeamX_pxl() == Catch::Approx(12.0)); + CHECK(x.GetBeamY_pxl() == Catch::Approx(8.0)); + CHECK_FALSE(x.GetGoniometer().has_value()); // stills + REQUIRE(x.GetPolarizationFactor().has_value()); + CHECK(x.GetPolarizationFactor().value() == Catch::Approx(0.998)); + + for (int i = 0; i < 2; i++) { + auto raw = reader.GetRawImage(i); + REQUIRE(raw); + CHECK(raw->image.GetWidth() == nx); + CHECK(raw->image.GetHeight() == ny); + const auto *pixels = reinterpret_cast(raw->image_buffer.data()); + CHECK(pixels[0] == 100 * (i + 1)); + CHECK(pixels[nx * ny - 1] == 100 * (i + 1) + nx * ny - 1); + } + reader.Close(); + std::filesystem::remove_all("xtc_test"); +} -- 2.54.0 From 6a1d859d499356a6c287ef46da98faafe5e4ddfd Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 14:52:07 +0200 Subject: [PATCH 31/53] Stills merge: R_split and a frame-split CC1/2, per shell and overall Serial crystallography quotes R_split and a CC1/2 whose half-sets keep each crystal whole (CrystFEL's partialator), and rugnux's stills report had neither, so its numbers could not be set beside CrystFEL's. MergeOnTheFly now also sums every frame's observations into the frame's own half (the existing per-image hash preference), and MergeStats reports from those two means R_split = (1/sqrt 2) sum|I1-I2| / (1/2 sum(I1+I2)) and their CC1/2. They are printed as a "Serial statistics" table under the shell table (also in the --report-resolution table) and as R_SPLIT / CC_HALF_SPLIT and their REFRES_ keys. The balanced per-reflection split behind CC_HALF is unchanged, and so is everything decided on it (resolution cut, space-group search, pass gates). Rotation output is unchanged: RotationScaleMerge does not fill the new fields and the table is not printed. On a 4000-frame jet subset over 30-1.5 A: CC1/2 0.830, R_split 31.8%, against 0.834 / 31.3% computed by hand from two separate even/odd runs and 0.842 / 30.5% from CrystFEL on the same frames. Co-Authored-By: Claude Opus 5.5 --- common/Reflection.h | 3 ++ docs/RUGNUX_REPORT.md | 9 +++++ image_analysis/scale_merge/Merge.cpp | 54 ++++++++++++++++++++++++++++ image_analysis/scale_merge/Merge.h | 16 +++++++++ rugnux/ResultReport.cpp | 9 +++++ tests/MergeScaleTest.cpp | 51 ++++++++++++++++++++++++++ 6 files changed, 142 insertions(+) diff --git a/common/Reflection.h b/common/Reflection.h index a6b5ad9c5..1b8b5caec 100644 --- a/common/Reflection.h +++ b/common/Reflection.h @@ -85,6 +85,9 @@ struct MergedReflection { float sigma = NAN; float I_half[2] = {NAN, NAN}; float sigma_half[2] = {NAN, NAN}; + // Half-set means with every frame's observations kept whole in one half (stills: R_split and the + // CrystFEL-style CC1/2). NaN where a half saw none, and on rotation. + float I_split[2] = {NAN, NAN}; // Weight of this reflection in a CC1/2: the precision its half-sets would have had with every // observation at the run's typical frame scale, over the precision they have. 1 on a sweep // without a weak stretch; small for a reflection measured only where the crystal barely diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 49736a946..bf18df563 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -67,6 +67,15 @@ the number with its noise; a large gap between the two says weak frames are kept absent where CC1/2 is not positive. All three are also columns of the shell tables (`R_merge`, `R_pim`, `CC*`) and are written per shell into the merged mmCIF. +**`R_SPLIT=`** and **`CC_HALF_SPLIT=`** (and their `REFRES_` forms) are written for stills only, with +a "Serial statistics" table beside the shell table. Every frame's observations go whole into one of two +half-sets - as CrystFEL's partialator deals out crystals - and R_split = (1/sqrt 2) sum|I_1 - I_2| / +(1/2 sum (I_1 + I_2)) (White et al., 2012) and CC1/2 are taken over the reflections both halves +measured, so both compare directly with CrystFEL's `compare_hkl`. `CC_HALF=` deals each reflection's +observations out evenly instead, as XDS and cctbx do, and is the one the processing decides on; the two +agree overall and can differ in a single shell. On stills `R_MEAS=` measures the scatter of partial +observations, not the error of the merged intensity, and is not the number to quote. + **`OBSERVATIONS_REJECTED=`** counts the observations the merge's outlier tests removed, and **`OBSERVATIONS_REJECTED_WILSON=`** the part of them the Wilson test took (each listed in the developer report). **`OBSERVATIONS_REJECTED_OVERLOAD=`** is separate from both: the rotation measurements left diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index 039924825..e72daef80 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -150,6 +150,8 @@ void MergeOnTheFly::Accumulate(const std::vector &contributions, i it->second.sum_wI_half[half] += c.wI; it->second.sum_w_half[half] += c.w; it->second.n_half[half]++; + it->second.sum_wI_split[prefer] += c.wI; + it->second.sum_w_split[prefer] += c.w; if (c.on_ice_ring) it->second.on_ice_ring = true; @@ -452,6 +454,10 @@ std::vector MergeOnTheFly::ExportIntensities() { } } + for (int i = 0; i < 2; ++i) + if (accum.sum_w_split[i] > 0.0) + mr.I_split[i] = static_cast(accum.sum_wI_split[i] / accum.sum_w_split[i]); + if (!std::isfinite(accum.d) || accum.d <= 0.0f) continue; @@ -488,6 +494,11 @@ struct ShellAccum { CorrelationCoefficient cc_half; CorrelationCoefficient cc_ref; + + CorrelationCoefficient cc_split; + double split_abs_diff = 0.0; + double split_sum = 0.0; + int split_pairs = 0; }; std::pair ImageReferenceCC(std::span reflections, @@ -663,6 +674,7 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m CorrelationCoefficient cc_half_overall; CorrelationCoefficient cc_ref_overall; + ShellAccum split_overall; for (const auto &m: merged) { const auto shell = shells.GetShell(m.d); @@ -690,6 +702,15 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m cc_half_overall.Add(m.I_half[0], m.I_half[1]); } + if (std::isfinite(m.I_split[0]) && std::isfinite(m.I_split[1])) { + for (auto *a: {&acc[s], &split_overall}) { + a->cc_split.Add(m.I_split[0], m.I_split[1]); + a->split_abs_diff += std::fabs(m.I_split[0] - m.I_split[1]); + a->split_sum += 0.5 * (m.I_split[0] + m.I_split[1]); + a->split_pairs++; + } + } + } } } @@ -773,6 +794,16 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m rmeas_den_all += ra.sum_I; } + // R_split as White et al. (2012) J. Appl. Cryst. 45, 335-341 define it (CrystFEL's compare_hkl). + auto set_split = [](const ShellAccum &a, MergeStatisticsShell &ss) { + ss.n_split_pairs = a.split_pairs; + if (a.split_pairs == 0) + return; + ss.cc_half_split = a.cc_split.GetCC(); + if (a.split_sum > 0.0) + ss.r_split = a.split_abs_diff / a.split_sum / std::sqrt(2.0); + }; + MergeStatistics out; out.shells.resize(n_shells); @@ -795,6 +826,7 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m ss.r_meas = rmeas_den[s] > 0.0 ? rmeas_num[s] / rmeas_den[s] : NAN; ss.r_merge = rmeas_den[s] > 0.0 ? rmerge_num[s] / rmeas_den[s] : NAN; ss.r_pim = rmeas_den[s] > 0.0 ? rpim_num[s] / rmeas_den[s] : NAN; + set_split(sa, ss); } auto &overall = out.overall; @@ -824,6 +856,7 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m overall.r_meas = rmeas_den_all > 0.0 ? rmeas_num_all / rmeas_den_all : NAN; overall.r_merge = rmeas_den_all > 0.0 ? rmerge_num_all / rmeas_den_all : NAN; overall.r_pim = rmeas_den_all > 0.0 ? rpim_num_all / rmeas_den_all : NAN; + set_split(split_overall, overall); return out; } @@ -923,6 +956,26 @@ std::string ShellTableRule() { "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""); } +std::string SerialShellTable(const MergeStatistics &in) { + if (in.overall.n_split_pairs == 0) + return ""; + std::string o = "\n Serial statistics: every frame whole in one half-set (as CrystFEL's partialator splits by crystal)\n"; + o += fmt::format(" {:>8s} {:>8s} {:>8s} {:>8s} {:>8s}\n", "d_min", "N_pairs", "CC1/2", "CC*", "R_split"); + o += fmt::format(" {:->8s} {:->8s} {:->8s} {:->8s} {:->8s}\n", "", "", "", "", ""); + auto row = [](const MergeStatisticsShell &sh) { + return fmt::format("{:8d} {:>8s} {:>8s} {:>8s}\n", sh.n_split_pairs, + Measured(sh.cc_half_split * 100.0, "%"), + Measured(CCStar(sh.cc_half_split) * 100.0, "%"), + Measured(sh.r_split * 100.0, "%")); + }; + for (const auto &sh: in.shells) + if (sh.n_split_pairs > 0) + o += fmt::format(" {:8.2f} ", sh.d_min) + row(sh); + o += fmt::format(" {:->8s} {:->8s} {:->8s} {:->8s} {:->8s}\n", "", "", "", "", ""); + o += fmt::format(" {:>8s} ", "Overall") + row(in.overall); + return o; +} + std::ostream &operator<<(std::ostream &output, const MergeStatistics &in) { output << std::endl; output << ShellTableHeader() << std::endl; @@ -939,6 +992,7 @@ std::ostream &operator<<(std::ostream &output, const MergeStatistics &in) { output << fmt::format(" {:>8s} ", "Overall"); output << in.overall; output << std::endl; + output << SerialShellTable(in); if (std::isfinite(in.wilson_b) && in.wilson_b > 0.0) output << fmt::format(" Wilson B-factor estimate: {:.2f} A^2 (correlation {:.3f}, fitted over {:.2f}-{:.2f} A)", in.wilson_b, in.wilson_b_correlation, in.wilson_b_d_low_A, in.wilson_b_d_high_A) diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index cfb5cd615..6d73040f0 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -58,6 +58,16 @@ struct MergeStatisticsShell { // it. NaN when no reflection has an anomalous split (the stills path), which is not the same // statement as zero - zero means measured and absent. double cc_anom = NAN; + + // The serial-crystallography pair (stills only, NaN on rotation): every frame's observations go + // whole to one half-set, as CrystFEL's partialator deals out crystals, and the two half-set means + // give CC1/2 and R_split = (1/sqrt 2) sum|I_1 - I_2| / (1/2 sum (I_1 + I_2)) (White et al. (2012) + // J. Appl. Cryst. 45, 335-341), over the reflections both halves measured (n_split_pairs). This + // CC1/2 is not cc_half above, which deals each reflection's observations out evenly and is the + // one the processing decides on. + double r_split = NAN; + double cc_half_split = NAN; + int n_split_pairs = 0; }; // Why a stretch of the sweep came out much weaker than the rest of the run. The codes are the field's @@ -227,6 +237,9 @@ std::ostream &operator<<(std::ostream &output, const MergeStatistics &in); // elsewhere (the report's reference-range table). std::string ShellTableHeader(); std::string ShellTableRule(); +// The serial table of R_split and the frame-split CC1/2, per shell and overall; empty where they were +// not computed (rotation). +std::string SerialShellTable(const MergeStatistics &in); struct MergeAccum { int32_t h = 0; @@ -239,6 +252,9 @@ struct MergeAccum { double sum_wI_half[2] = {0.0, 0.0}; double sum_w_half[2] = {0.0, 0.0}; size_t n_half[2] = {0, 0}; + // The same sums with each frame's observations kept whole in the frame's half (R_split). + double sum_wI_split[2] = {0.0, 0.0}; + double sum_w_split[2] = {0.0, 0.0}; }; // XDS's error-model convention (Diederichs, Acta Cryst. D66 (2010) 733) is sigma^2 = a*(sigma0^2 + diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index e1fc9a660..bc7bb3c64 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -330,6 +330,7 @@ namespace { } o << ShellTableRule() << "\n"; o << fmt::format(" {:>8s} ", "Overall") << rs.overall << "\n"; + o << SerialShellTable(rs); return o.str(); } @@ -1155,6 +1156,10 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, Add(s, KeyReal("CC_HALF", o.cc_half, "{:.4f}")); if (std::isfinite(CCStar(o.cc_half))) Add(s, KeyReal("CC_STAR", CCStar(o.cc_half), "{:.4f}")); + if (std::isfinite(o.r_split)) + Add(s, KeyReal("R_SPLIT", o.r_split, "{:.4f}")); + if (std::isfinite(o.cc_half_split)) + Add(s, KeyReal("CC_HALF_SPLIT", o.cc_half_split, "{:.4f}")); if (std::isfinite(o.abs_diff_over_sigma_anomalous)) Add(s, KeyReal("SIGANO", o.abs_diff_over_sigma_anomalous, "{:.3f}")); if (std::isfinite(o.cc_anom)) @@ -1301,6 +1306,10 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, Add(s, KeyReal("REFRES_R_MEAS_WEIGHTED", ro.r_meas_weighted, "{:.4f}")); if (std::isfinite(ro.cc_half)) Add(s, KeyReal("REFRES_CC_HALF", ro.cc_half, "{:.4f}")); + if (std::isfinite(ro.r_split)) + Add(s, KeyReal("REFRES_R_SPLIT", ro.r_split, "{:.4f}")); + if (std::isfinite(ro.cc_half_split)) + Add(s, KeyReal("REFRES_CC_HALF_SPLIT", ro.cc_half_split, "{:.4f}")); if (std::isfinite(ro.abs_diff_over_sigma_anomalous)) Add(s, KeyReal("REFRES_SIGANO", ro.abs_diff_over_sigma_anomalous, "{:.3f}")); if (std::isfinite(ro.cc_anom)) diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index 6bc6e0ff2..70f08b06b 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -863,3 +863,54 @@ TEST_CASE("SublatticeRelabel_SignalPicksTheAxisThatWasDoubled") { CHECK((ScatteringVector(outcomes[0].latt, r) - ScatteringVector(before.latt, b)).Length() < 1e-5f); } } + +// R_split's half-sets keep every frame whole: each frame measures the same reflections at its own +// intensity, so every reflection's two half-set means are the same two numbers. R_split is then +// (1/sqrt 2) |I_1 - I_2| / ((I_1 + I_2) / 2) in every shell, and the frame-split CC1/2 is undefined +// (the half-set means do not vary) - the per-reflection balanced split, which the processing decides on, +// is left as it was. +TEST_CASE("MergeOnTheFly_RsplitKeepsEveryFrameWhole") { + DiffractionExperiment x; + x.SetSpaceGroup(TestSpaceGroup()); + + std::vector outcomes(64); + for (size_t f = 0; f < outcomes.size(); ++f) { + for (int h = 1; h <= 4; ++h) + for (int k = 1; k <= 4; ++k) { + Reflection r{}; + r.h = h; + r.k = k; + r.l = 1; + r.d = 20.0f / std::sqrt(static_cast(h * h + k * k + 1)); + r.I = 100.0f + static_cast(f); + r.sigma = 10.0f; + r.prescaling_corr = 1.0f; + r.partiality = 1.0f; + r.image_scale_corr = 1.0f; + outcomes[f].reflections.push_back(r); + } + } + + MergeOnTheFly merge(x); + merge.AddImages(outcomes); + const auto merged = merge.ExportReflections(); + REQUIRE(merged.size() == 16); + + const float a = merged[0].I_split[0], b = merged[0].I_split[1]; + REQUIRE(std::isfinite(a)); + REQUIRE(std::isfinite(b)); + CHECK(a != b); + for (const auto &m: merged) { + CHECK(m.I_split[0] == a); + CHECK(m.I_split[1] == b); + } + + const auto stats = merge.MergeStats(merged, outcomes); + const double expected = std::fabs(a - b) / (0.5 * (a + b)) / std::sqrt(2.0); + CHECK(stats.overall.n_split_pairs == 16); + CHECK(stats.overall.r_split == Catch::Approx(expected).epsilon(1e-5)); + for (const auto &sh: stats.shells) + if (sh.n_split_pairs > 0) + CHECK(sh.r_split == Catch::Approx(expected).epsilon(1e-5)); + CHECK_FALSE(SerialShellTable(stats).empty()); +} -- 2.54.0 From 3f632995575cb9b76c7ddd5680c4a41baf7b55ef Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 15:56:01 +0200 Subject: [PATCH 32/53] rugnux stills: the beam centre (and, with -C, the distance) decided on the frames; Cheetah CXI read without a geometry file Stills never used the beam centre the pre-scan measures - the "second first pass indexes both centres" exists only in the rotation path - so a header centre a dozen pixels off indexed a few percent of the frames, or a wrong cell. DecideStillsGeometry runs after the pre-scan: spots are found once on 96 spread frames, and each hypothesis re-indexes those spots (IndexFrameOnly) at its geometry; the score is the number of found spots on the lattice of their frame. Centres: the file's, the background-isotropy measurement and the point-symmetry capture (kept beside it now; the two estimators fail differently - on a single-panel Rayonix with a tape shadow the isotropy fit reads 4 px off, the capture 1.4 px). With a reference cell the distance is a hypothesis too: a 2.5% grid over 60-160% of the file's on the 32 strongest frames, a 1% one about its best on all 96, then the centre decided again at the new distance. The file's value is kept unless another puts at least 1/0.9 times as many spots on the lattice. Where the geometry moved, the stills bundle adjustment (-C) runs again from there, since its bounds are +-8 px / +-8 mm about its start. Without a cell the distance is not searched: a wrong distance indexes as a cell of the wrong size. The CrystFEL reader also opens a Cheetah CXI file (or a directory of them) with no .geom: one flat panel, pixel size, photon energy and camera length from the file, centred on the detector. Measured (single-panel Rayonix XFEL stills, a P6122 crystal, 1316 frames, -C/-S, ice rings on): hand-built .geom (measured centre, 105 mm) 54.56% indexed, rugnux R-free 0.376 plain CXI + --detector-distance 105 54.71%, 0.382 (centre found: 733.2,717.8) plain CXI, file's wrong 89.4 mm kept 54.41%, 0.367 (distance found 105.5 -> 104.97 refined) A second single-panel Rayonix set, P6522, 1000 frames, -C/-S, centre moved off the .expt value: 0 px 59.26% R-free 0.286; 14 px 59.16% 0.298; 28 px 59.16% 0.285 (before: 5-10% indexed) Unchanged (bit-identical merge statistics and R-free): JUNGFRAU lysozyme stills, 4,000 frames (file centre kept, +1.6 s) and mpCCD 1339 frames (file centre kept, +2.5 s). The trial costs 1.6-2.5 s without a cell and 2-11 s with one. Co-Authored-By: Claude Opus 5.5 --- reader/JFJochCrystFELReader.cpp | 72 +++++++++-- reader/JFJochCrystFELReader.h | 7 ++ rugnux/Rugnux.cpp | 10 +- rugnux/Rugnux.h | 24 +++- rugnux/RugnuxPreScan.cpp | 5 +- rugnux/RugnuxStills.cpp | 204 ++++++++++++++++++++++++++++++++ rugnux/rugnux_cli.cpp | 17 ++- 7 files changed, 323 insertions(+), 16 deletions(-) diff --git a/reader/JFJochCrystFELReader.cpp b/reader/JFJochCrystFELReader.cpp index 77fb47f26..cb8d5e0f3 100644 --- a/reader/JFJochCrystFELReader.cpp +++ b/reader/JFJochCrystFELReader.cpp @@ -53,19 +53,75 @@ double JFJochCrystFELReader::MeanOverFirstFile(const std::string &path) { return std::accumulate(values.begin(), values.end(), 0.0) / static_cast(values.size()); } -void JFJochCrystFELReader::ReadFiles(const std::string &path, const std::string &geometry_file) { - Close(); - geom_ = crystfel::Read(geometry_file); - +std::vector JFJochCrystFELReader::ListFiles(const std::string &path) { + std::vector files; if (std::filesystem::is_directory(path)) { for (const auto &e: std::filesystem::directory_iterator(path)) if (e.path().extension() == ".h5" || e.path().extension() == ".cxi") - files_.push_back(e.path().string()); - std::sort(files_.begin(), files_.end()); + files.push_back(e.path().string()); + std::sort(files.begin(), files.end()); } else - files_.push_back(path); - if (files_.empty()) + files.push_back(path); + if (files.empty()) throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "No .h5 or .cxi files in " + path); + return files; +} + +void JFJochCrystFELReader::ReadFiles(const std::string &path, const std::string &geometry_file) { + Close(); + geom_ = crystfel::Read(geometry_file); + Open(path, geometry_file); +} + +// A Cheetah CXI file carries what one flat panel needs - the frames at /entry_1/data_1/data, the pixel +// size, the photon energy and a camera length, all in SI units - but not where the beam is. The panel is +// put with its middle on the beam, which is as good as no centre at all: the run measures it. +bool JFJochCrystFELReader::CanReadCXI(const std::string &path) { + try { + if (!std::filesystem::exists(path)) + return false; + const auto files = ListFiles(path); + std::unique_lock ul(hdf5_mutex); + HDF5ReadOnlyFile file(files[0]); + return file.Exists("/entry_1/data_1/data") && file.GetDimension("/entry_1/data_1/data").size() == 3 + && file.Exists("/entry_1/instrument_1/detector_1/x_pixel_size") + && file.Exists("/entry_1/instrument_1/source_1/energy"); + } catch (...) { + return false; + } +} + +void JFJochCrystFELReader::ReadCXI(const std::string &path) { + Close(); + const auto files = ListFiles(path); + { + std::unique_lock ul(hdf5_mutex); + HDF5ReadOnlyFile file(files[0]); + const auto mean = [&](const std::string &name) { + const auto v = file.ReadVector(name); + return std::accumulate(v.begin(), v.end(), 0.0) / static_cast(v.size()); + }; + const auto dim = file.GetDimension("/entry_1/data_1/data"); + constexpr double JOULE_PER_EV = 1.602176634e-19; + const std::string detector = "/entry_1/instrument_1/detector_1/"; + crystfel::Panel p; + p.name = "cxi"; + p.max_ss = static_cast(dim[1]) - 1; + p.max_fs = static_cast(dim[2]) - 1; + p.corner_x = -0.5 * static_cast(dim[2]); + p.corner_y = -0.5 * static_cast(dim[1]); + p.res = 1.0 / mean(detector + "x_pixel_size"); + p.clen = fmt::format("{}", file.Exists(detector + "distance") ? mean(detector + "distance") : 0.1); + geom_ = {}; + geom_.panels.push_back(p); + geom_.data = "/entry_1/data_1/data"; + geom_.photon_energy = fmt::format("{}", mean("/entry_1/instrument_1/source_1/energy") / JOULE_PER_EV); + } + Open(path, "the CXI metadata"); +} + +void JFJochCrystFELReader::Open(const std::string &path, const std::string &geometry_file) { + files_ = ListFiles(path); std::unique_lock ul(hdf5_mutex); diff --git a/reader/JFJochCrystFELReader.h b/reader/JFJochCrystFELReader.h index 4eb111382..a1fbac988 100644 --- a/reader/JFJochCrystFELReader.h +++ b/reader/JFJochCrystFELReader.h @@ -55,6 +55,9 @@ class JFJochCrystFELReader : public JFJochReader { void CheckDirectChunks(); double MeanOverFirstFile(const std::string &path); + static std::vector ListFiles(const std::string &path); + // The files at path, placed by geom_; geometry_file names where geom_ came from, for messages. + void Open(const std::string &path, const std::string &geometry_file); bool LoadImage_i(std::shared_ptr &dataset, DataMessage &message, @@ -70,6 +73,10 @@ public: ~JFJochCrystFELReader() override; void ReadFiles(const std::string &path, const std::string &geometry_file); + // A Cheetah CXI file (or a directory of them), with no geometry file: one flat panel, placed by what + // the file states, centred on the beam. + static bool CanReadCXI(const std::string &path); + void ReadCXI(const std::string &path); // False where the geometry states no gain, so the images are in detector units, not photons. [[nodiscard]] bool CountsInPhotons() const; diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 552a3c982..23e053bc3 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -646,7 +646,9 @@ void Rugnux::StartSpeculativeGeometryProbe(std::shared_ptr run, const st void Rugnux::JoinBeamCenterCapture() { if (!background_center_capture_.valid()) return; - background_center_ = background_center_capture_.get(); + const BackgroundCentres c = background_center_capture_.get(); + background_center_ = c.measured; + point_symmetry_center_ = c.point_symmetry; measured_beam_center_ = background_center_; background_center_capture_ = {}; } @@ -863,6 +865,12 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b std::min(config_.detect_beam_stop.value_or(BEAM_CENTER_PROJECTION_IMAGES), HotPixelFinder::MAX_SAMPLED_FRAMES), observer); + // The stills geometry refinement above ran from the file's centre and distance, within a few pixels and + // millimetres of them; where the frames have just moved either, it is run again from where they are. + if (full && !experiment_.IsRotationIndexing() && config_.beam_center_check && !config_.live + && DecideStillsGeometry(start_image, images_to_process) && config_.refine_geometry.has_value()) + RefineStillsGeometry(start_image, end_image, images_to_process, observer); + // The geometry pre-pass integrates at the radius the run started with; the width the pre-scan just // measured is applied to the canonical pass instead (RunAllPasses, between the two). See // bragg_adaptive_ for why - in short, the pre-pass keeps only the geometry it fits to the observed diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 776aa617c..49a9d1e26 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -50,6 +50,7 @@ class JFJochReader; class JFJochHDF5Reader; struct JFJochReaderRawImage; struct PipelineLocals; // RugnuxPipeline.h, private to the implementation +class IndexerThreadPool; // Offline reprocessing of a stored Jungfraujoch HDF5 dataset, shared by rugnux (all of its --mode // values that read images), the viewer and the broker. The full workflow lives here, not in the CLIs: @@ -987,7 +988,14 @@ class Rugnux { std::optional background_center_; // The pre-scan's capture of it while it is still running beside what follows; JoinBeamCenterCapture // waits for it and sets background_center_ and measured_beam_center_ from it, wherever they are read. - std::shared_future> background_center_capture_; + // The measurement, and the strongest point-symmetry centre the capture found on the way to it - a + // second reading of the same background, by a different estimator, which stills decide between. + struct BackgroundCentres { + std::optional measured; + std::optional> point_symmetry; + }; + std::shared_future background_center_capture_; + std::optional> point_symmetry_center_; void JoinBeamCenterCapture(); // The centre the background measured, kept when the two centres index the SAME cell but a // different metric symmetry off it and the disagreement is by itself large enough to have decided @@ -1357,6 +1365,20 @@ class Rugnux { void RefineStillsGeometry(int start_image, int end_image, int images_to_process, RugnuxObserver *observer); + // Stills: the beam centre and the detector distance as hypotheses, each indexed on the same sample + // of frames and decided on what that sample shows (see DecideStillsGeometry). + struct StillsTrial { + int frames = 0; // frames read + int indexed = 0; // frames that indexed + int64_t spots = 0; // spots found on the frames that indexed + int64_t on_lattice = 0; // ... of which on the lattice + }; + std::vector FindStillsSampleSpots(const std::vector &ordinals, int start_image); + StillsTrial IndexStillsSample(const DiffractionExperiment &x, IndexerThreadPool &pool, + const std::vector &frames); + // Returns whether the run's geometry was moved off the file's. + bool DecideStillsGeometry(int start_image, int images_to_process); + // One full run of the pipeline (index + integrate + optional scale/merge/space-group + output). // write_output == false makes no outputs (ProcessResult::outputs) and streams nothing to the image sink; // nor does the geometry pre-pass. diff --git a/rugnux/RugnuxPreScan.cpp b/rugnux/RugnuxPreScan.cpp index ce975589c..0ae59a893 100644 --- a/rugnux/RugnuxPreScan.cpp +++ b/rugnux/RugnuxPreScan.cpp @@ -1076,7 +1076,10 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru across ? "it is mostly along the spindle, where nothing announces itself" : "this is not a rotation sweep, so there is no spindle to resolve it on"); } - return background_center; + BackgroundCentres centres{background_center, std::nullopt}; + if (!capture.point.empty()) + centres.point_symmetry = std::make_pair(capture.point[0].beam_x_pxl, capture.point[0].beam_y_pxl); + return centres; }).share(); finish_spot_measurement(); diff --git a/rugnux/RugnuxStills.cpp b/rugnux/RugnuxStills.cpp index 4e7286ba8..d353fb020 100644 --- a/rugnux/RugnuxStills.cpp +++ b/rugnux/RugnuxStills.cpp @@ -264,3 +264,207 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_ experiment_.BeamX_pxl(r.beam_x_px).BeamY_pxl(r.beam_y_px).DetectorDistance_mm(r.distance_mm); experiment_.SetUnitCell(r.cell); } + +std::vector Rugnux::FindStillsSampleSpots(const std::vector &ordinals, int start_image) { + const auto dataset = reader_.GetDataset(); + AzimuthalIntegrationMapping mapping(experiment_, pixel_mask_); + IndexAndRefine indexer(experiment_, nullptr, /*retain_outcomes=*/false); + + auto settings = config_.spot_finding; + settings.enable = true; + settings.indexing = false; + settings.quick_integration = false; + + std::vector frames(ordinals.size()); + std::atomic next = 0; + auto worker = [&]() { + pin_gpu(); + JFJochReaderRawImage img; + MXAnalysisWithoutFPGA analysis(experiment_, mapping, pixel_mask_, indexer, /*enable_fused_adaptive_gpu=*/true); + AzimuthalIntegrationProfile profile(mapping); + for (size_t i = next.fetch_add(1); i < ordinals.size() && !cancelled_; i = next.fetch_add(1)) { + const int image_idx = start_image + ordinals[i] * config_.stride; + if (!reader_.ReadRawImage(image_idx, img)) + continue; + DataMessage msg{}; + msg.image = img.image; + msg.number = ordinals[i]; + msg.original_number = image_idx; + if (dataset->efficiency.size() > image_idx) + msg.image_collection_efficiency = dataset->efficiency[image_idx]; + analysis.Analyze(msg, profile, settings); + msg.image = {}; // the buffer is the worker's, and only the spots are kept + frames[i] = std::move(msg); + } + }; + std::vector> futures; + for (int i = 0; i < config_.nthreads; ++i) + futures.push_back(std::async(std::launch::async, worker)); + for (auto &f : futures) + f.get(); + return frames; +} + +Rugnux::StillsTrial Rugnux::IndexStillsSample(const DiffractionExperiment &x, IndexerThreadPool &pool, + const std::vector &frames) { + IndexAndRefine indexer(x, &pool, /*retain_outcomes=*/false); + const DiffractionGeometry geom = x.GetDiffractionGeometry(); + auto settings = config_.spot_finding; + settings.enable = true; + settings.indexing = true; + + std::mutex m; + StillsTrial total; + std::atomic next = 0; + auto worker = [&]() { + StillsTrial t; + for (size_t i = next.fetch_add(1); i < frames.size() && !cancelled_; i = next.fetch_add(1)) { + // The spots where the frame put them, at this trial's geometry. + DataMessage msg = frames[i]; + for (auto &s : msg.spots) { + s.d_A = geom.PxlToRes(s.x, s.y); + s.indexed = false; + s.lattice = -1; + } + t.frames++; + if (!indexer.IndexFrameOnly(msg, settings)) + continue; + t.indexed++; + t.spots += static_cast(msg.spots.size()); + t.on_lattice += msg.spot_count_indexed.value_or(0); + } + indexer.RethrowIndexerError(); + std::lock_guard lock(m); + total.frames += t.frames; + total.indexed += t.indexed; + total.spots += t.spots; + total.on_lattice += t.on_lattice; + }; + std::vector> futures; + for (int i = 0; i < config_.nthreads; ++i) + futures.push_back(std::async(std::launch::async, worker)); + for (auto &f : futures) + f.get(); + return total; +} + +// The beam centre of a stills run - and, where a reference cell is given, the detector distance - +// decided on the frames rather than taken from the file. Both are written into a header by hand often +// enough to be wrong, and a still, unlike a rotation sweep, has no first pass that would show it: every +// frame is indexed on its own, so a wrong centre shows only as frames that index on a distorted lattice, +// or not at all, and a merge that does not agree with itself. +// +// So each is a hypothesis, indexed on the same sample of frames, and the run keeps the one that puts the +// most of the found spots on the lattice of their frame. The file's value is kept unless another puts +// materially more there, not merely more. +bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) { + Logger logger("Rugnux"); + JoinBeamCenterCapture(); + TimingMark mark(logger, "stills geometry trial"); + + constexpr int SAMPLE_FRAMES = 96; + // The distance grid is scanned on the frames with the most spots only: a frame with few has little + // at high angle, where the distance shows. + constexpr int GRID_FRAMES = 32; + // Below this fraction of the best hypothesis' count the file's value is outside the peak. + constexpr double FILE_KEPT_ABOVE = 0.9; + // The distance grid: the file's value times STEP^k, over [MIN_FACTOR, MAX_FACTOR]. Wide enough for an + // encoder reading or a nominal camera length to be wrong, fine enough not to step over the peak. + constexpr float STEP = 1.025f, MIN_FACTOR = 0.6f, MAX_FACTOR = 1.6f; + + const std::vector sample = select_equally_spaced_image_ordinals( + images_to_process, std::min(images_to_process, SAMPLE_FRAMES)); + std::vector frames = FindStillsSampleSpots(sample, start_image); + std::stable_sort(frames.begin(), frames.end(), [](const DataMessage &a, const DataMessage &b) { + return a.spots.size() > b.spots.size(); + }); + const std::vector strongest(frames.begin(), + frames.begin() + std::min(frames.size(), GRID_FRAMES)); + IndexerThreadPool pool(experiment_.GetIndexingSettings(), IndexerConstruction::OnFirstUse, gpu_work_beside_); + + const float file_x = experiment_.GetBeamX_pxl(), file_y = experiment_.GetBeamY_pxl(); + const float file_distance = experiment_.GetDetectorDistance_mm(); + const auto trial = [&](float x, float y, float distance, const std::vector &on) { + DiffractionExperiment e = experiment_; + e.BeamX_pxl(x).BeamY_pxl(y).DetectorDistance_mm(distance); + const StillsTrial t = IndexStillsSample(e, pool, on); + logger.Debug("Stills geometry trial: centre ({:.2f},{:.2f}) distance {:.2f} mm - {}/{} frames indexed, " + "{} of their {} spots on the lattice", x, y, distance, t.indexed, t.frames, + t.on_lattice, t.spots); + return t.on_lattice; + }; + + // The centres: the file's, the measured one and the point-symmetry capture's - two estimators of the + // same background, which do not fail alike (a shadow across the background biases the isotropy fit, + // a flat background leaves the capture without a peak). + std::vector> candidates; + if (background_center_) + candidates.emplace_back(background_center_->beam_x_pxl, background_center_->beam_y_pxl); + if (point_symmetry_center_) + candidates.push_back(*point_symmetry_center_); + float x = file_x, y = file_y; + int64_t score_here = 0; + const auto decide_centre = [&](float distance) { + const int64_t file_score = trial(file_x, file_y, distance, frames); + int64_t best = file_score; + float best_x = file_x, best_y = file_y; + for (const auto &[cx, cy] : candidates) { + const int64_t score = trial(cx, cy, distance, frames); + logger.Info("Stills beam centre at {:.2f} mm: ({:.2f},{:.2f}) puts {} sampled spots on the lattice, " + "the file's ({:.2f},{:.2f}) {}", distance, cx, cy, score, file_x, file_y, file_score); + if (score > best) { + best = score; + best_x = cx; + best_y = cy; + } + } + const bool moved = file_score < FILE_KEPT_ABOVE * best; + x = moved ? best_x : file_x; + y = moved ? best_y : file_y; + score_here = moved ? best : file_score; + }; + decide_centre(file_distance); + + // The distance, where a reference cell is given: a grid at the centre just decided, on the strongest + // frames, then a finer one about its best on the whole sample, which also decides against the file's + // value. With the cell fixed each frame is matched rather than searched for, so the grid is cheap - + // and only then is the distance a hypothesis of its own: without a cell a wrong distance indexes as a + // cell of the wrong size, which only the curvature far from the beam tells apart. The stills geometry + // refinement takes it from there (RunPipeline). + float distance = file_distance; + if (experiment_.GetUnitCell().has_value()) { + float grid_best = file_distance; + int64_t grid_score = trial(x, y, file_distance, strongest); + for (float f = MIN_FACTOR; f <= MAX_FACTOR; f *= STEP) { + if (std::fabs(f - 1.0f) < 0.5f * (STEP - 1.0f)) + continue; // the file's own, scored above + const int64_t score = trial(x, y, file_distance * f, strongest); + if (score > grid_score) { + grid_score = score; + grid_best = file_distance * f; + } + } + float best_distance = file_distance; + int64_t best = score_here; + for (const float f : {0.98f, 0.99f, 1.0f, 1.01f, 1.02f}) { + const int64_t score = trial(x, y, grid_best * f, frames); + if (score > best) { + best = score; + best_distance = grid_best * f; + } + } + if (score_here < FILE_KEPT_ABOVE * best) + distance = best_distance; + logger.Info("Stills detector distance: the file's {:.2f} mm puts {} sampled spots on the lattice, the best " + "between {:.0f}% and {:.0f}% of it {} at {:.2f} mm - the run uses {:.2f} mm", + file_distance, score_here, 100 * MIN_FACTOR, 100 * MAX_FACTOR, best, best_distance, distance); + // A centre decided at a distance this far off had little indexing to go on; decided again here. + if (distance != file_distance) + decide_centre(distance); + } + + logger.Info("Stills beam centre: the run uses ({:.2f},{:.2f}){}", x, y, + x == file_x && y == file_y ? ", the file's" : ""); + experiment_.BeamX_pxl(x).BeamY_pxl(y).DetectorDistance_mm(distance); + return x != file_x || y != file_y || distance != file_distance; +} diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 09d2d711d..ade463023 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -151,11 +151,12 @@ void print_usage() { std::cout << " Detector geometry" << std::endl; std::cout << " --crystfel-geometry Read - an HDF5/CXI file, or a directory of them - as CrystFEL does with this geometry file: frames from its data path, panels placed at its corners, camera length, pixel size, gain, photon energy and bad regions from it. One wavelength and one camera length for the run; panel positions are taken as given" << std::endl; + std::cout << " (A Cheetah CXI file, or a directory of them, is read without one: one flat panel, with the pixel size, photon energy and camera length the file states and the beam centre measured; --detector-distance overrides a camera length the file has wrong)" << std::endl; std::cout << std::endl; std::cout << " Detector mask" << std::endl; std::cout << " --detect-beam-stop[=N|off] Find the beam stop and its holder in a projection of N images and add them to the pixel mask (bit 9), so nothing shadowed by them is integrated. ON by default (60 images); =off disables. Reflections behind the stop are attenuated but not flagged, so they are integrated low with a plausible sigma and no existing rejection catches them" << std::endl; - std::cout << " --beam-center-check[=off] Measure the beam centre from the isotropy of the scattered background on EVERY run, report how far the file's value is from it, and index a second first pass with it to see whether the two centres give the same lattice. The fit reads the projection --detect-beam-stop already builds, so it costs no extra frames. On a run that indexes, the measured centre is adopted where the file's centre indexes nothing and the measured one indexes a majority, and where the two centres return cells related by an integer volume factor, in either direction, and the measured centre's lattice carries materially more of the pooled spots against its own chance level - the axis harmonic a centre error along the spindle produces, which nothing downstream repairs. Where the indexing cannot tell the two centres apart - the same cell with a different metric symmetry, or the same lattice found at both only by reading less of each frame - the first pass is run at both and the run keeps the one that merges better (on the same lattice: the one that measures more reflections at I/sigma >= 2). ON by default; =off disables" << std::endl; + std::cout << " --beam-center-check[=off] Measure the beam centre from the isotropy of the scattered background on EVERY run, report how far the file's value is from it, and index a second first pass with it to see whether the two centres give the same lattice. The fit reads the projection --detect-beam-stop already builds, so it costs no extra frames. On a run that indexes, the measured centre is adopted where the file's centre indexes nothing and the measured one indexes a majority, and where the two centres return cells related by an integer volume factor, in either direction, and the measured centre's lattice carries materially more of the pooled spots against its own chance level - the axis harmonic a centre error along the spindle produces, which nothing downstream repairs. Where the indexing cannot tell the two centres apart - the same cell with a different metric symmetry, or the same lattice found at both only by reading less of each frame - the first pass is run at both and the run keeps the one that merges better (on the same lattice: the one that measures more reflections at I/sigma >= 2). ON by default; =off disables On stills, where there is no first pass, the file's centre, the measured one and the point-symmetry capture's are each indexed on a sample of frames and the run takes the one that puts materially more spots on the lattice of their frame; with a reference cell (-C) the detector distance is searched the same way, over 60-160% of the file's, before the stills geometry refinement" << std::endl; std::cout << std::endl; std::cout << " Spot finding" << std::endl; @@ -1740,18 +1741,24 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { // come out as the message below, not as an unhandled exception with nothing printed at all. bool input_is_cbf = false, input_is_marccd = false, input_is_smv = false, input_is_crystfel = false; bool input_is_xtc = false; + bool input_is_cxi = false; try { input_is_crystfel = !crystfel_geometry.empty(); - input_is_cbf = !input_is_crystfel && JFJochCBFReader::CanRead(input_file); + input_is_cxi = !input_is_crystfel && JFJochCrystFELReader::CanReadCXI(input_file); + input_is_cbf = !input_is_crystfel && !input_is_cxi && JFJochCBFReader::CanRead(input_file); input_is_marccd = !input_is_cbf && JFJochMarCCDReader::CanRead(input_file); input_is_smv = !input_is_cbf && !input_is_marccd && JFJochSMVReader::CanRead(input_file); - input_is_xtc = !input_is_crystfel && !input_is_cbf && !input_is_marccd && !input_is_smv + input_is_xtc = !input_is_crystfel && !input_is_cxi && !input_is_cbf && !input_is_marccd && !input_is_smv && JFJochXTCReader::CanRead(input_file); if (input_is_crystfel) { crystfel_reader.ReadFiles(input_file, crystfel_geometry); reader_ptr = &crystfel_reader; logger.Info("Read {} frames from {} with the CrystFEL geometry {}", crystfel_reader.GetNumberOfImages(), input_file, crystfel_geometry); + } else if (input_is_cxi) { + crystfel_reader.ReadCXI(input_file); + reader_ptr = &crystfel_reader; + logger.Info("Read {} frames from the CXI file {}", crystfel_reader.GetNumberOfImages(), input_file); } else if (input_is_cbf) { cbf_reader.ReadFiles(input_file); reader_ptr = &cbf_reader; @@ -1945,7 +1952,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { "runs no space-group search. Use --mode mx, which makes the P1 merge the " "ledger is folded from."); // Re-scaling reads reflections a previous run integrated, which only a _process.h5 holds. - if (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_xtc) { + if (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_cxi || input_is_xtc) { logger.Error("--mode scale needs the integrated reflections in a _process.h5; " "a raw image sweep holds none. Run --mode mx on it first."); exit(EXIT_FAILURE); @@ -3177,7 +3184,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { config.beam_center_check = beam_center_check; config.adaptive_integration_radius = adaptive_integration_radius; config.counts_in_adu = input_is_smv || input_is_marccd || input_is_xtc - || (input_is_crystfel && !crystfel_reader.CountsInPhotons()); + || ((input_is_crystfel || input_is_cxi) && !crystfel_reader.CountsInPhotons()); config.rotation_postrefine_geometry = rotation_postrefine_geometry; // Measure the spot budget from the data unless the user pinned it. config.measure_spot_budget = !max_spot_count_override.has_value(); -- 2.54.0 From e4cb3e1b09d5a023263c03044847373e286eb68e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 16:27:40 +0200 Subject: [PATCH 33/53] rugnux stills: the saturation level read off the frames; Cheetah CXI only for a square frame Foreign stills files state their saturation level wrongly or not at all: an MPCCD Cheetah file is dark-subtracted per pixel, so each pixel clips at the ADC limit less its own dark, and its CrystFEL geometry leaves max_adu commented out - the run reported "Overloads: 0" while the inner panels saturated in most frames. saturation_ceiling (image_preprocessing/SaturationCeiling) histograms pixel values on a log scale (64 bins per octave from 256) and returns the lower edge of the lowest bin holding over twice what any of the 8 bins below it held (and at least 50 values): a bright tail only falls, a clipped one piles up below where it stops, as a band where the level differs per pixel. DetectSaturation runs it on 40 frames spread over every stills run and logs it beside the file's limit; for any input but a Jungfraujoch-written file (ProcessConfig::detect_saturation) a level below the file's replaces it. The CrystFEL/CXI reader, where the geometry states no max_adu, now takes the stored integer type's maximum (an int16 Cheetah file clips at 32767). The geometry-file-free CXI path (previous commit) is limited to square frames: a tiled detector's Cheetah slab (CSPAD, AGIPD, MPCCD, ...) read as one flat panel would place every pixel wrongly without a word, and those slabs are not square. Measured: MPCCD stills, 1339 frames: pile-up from 1218 photons (by hand 12000 ADU ~ 1200 ph); 5.5% of reflections coarser than 4 A flagged; lowest-shell CC1/2 0.545 -> 0.674, overall 0.741 -> 0.793; same dimple refinement against the deposited model R-free 0.2586 -> 0.2556 single-panel Rayonix CXI, 1316 frames: no pile-up (20 pixels at the int16 limit in 40 frames); the int16 limit flags 123 reflections; CC1/2 0.737 -> 0.787, rugnux R-free 0.382 -> 0.358 JUNGFRAU stills (Jungfraujoch files: lysozyme and two private sets): no pile-up below the stated limit, results bit-identical Cost 0.5 s per run. Co-Authored-By: Claude Opus 5.5 --- .../image_preprocessing/CMakeLists.txt | 4 +- .../image_preprocessing/SaturationCeiling.cpp | 46 ++++++++++++ .../image_preprocessing/SaturationCeiling.h | 39 ++++++++++ reader/JFJochCrystFELReader.cpp | 16 +++- rugnux/Rugnux.cpp | 3 + rugnux/Rugnux.h | 5 ++ rugnux/RugnuxStills.cpp | 75 +++++++++++++++++++ rugnux/rugnux_cli.cpp | 3 +- tests/CMakeLists.txt | 1 + tests/SaturationCeilingTest.cpp | 52 +++++++++++++ 10 files changed, 241 insertions(+), 3 deletions(-) create mode 100644 image_analysis/image_preprocessing/SaturationCeiling.cpp create mode 100644 image_analysis/image_preprocessing/SaturationCeiling.h create mode 100644 tests/SaturationCeilingTest.cpp diff --git a/image_analysis/image_preprocessing/CMakeLists.txt b/image_analysis/image_preprocessing/CMakeLists.txt index fe6dd3ea9..7fa931935 100644 --- a/image_analysis/image_preprocessing/CMakeLists.txt +++ b/image_analysis/image_preprocessing/CMakeLists.txt @@ -2,7 +2,9 @@ ADD_LIBRARY(JFJochImagePreprocessing STATIC ImagePreprocessor.cpp ImagePreproces ImagePreprocessorCPU.cpp ImagePreprocessorCPU.h ImagePreprocessorBuffer.cpp - ImagePreprocessorBuffer.h) + ImagePreprocessorBuffer.h + SaturationCeiling.cpp + SaturationCeiling.h) TARGET_LINK_LIBRARIES(JFJochImagePreprocessing JFJochCommon) diff --git a/image_analysis/image_preprocessing/SaturationCeiling.cpp b/image_analysis/image_preprocessing/SaturationCeiling.cpp new file mode 100644 index 000000000..7d07dce25 --- /dev/null +++ b/image_analysis/image_preprocessing/SaturationCeiling.cpp @@ -0,0 +1,46 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "SaturationCeiling.h" + +#include +#include + +namespace saturation_ceiling { + +namespace { + +int Bin(int64_t value) { + return static_cast(std::floor(std::log2(static_cast(value) / FLOOR) * BINS_PER_OCTAVE)); +} + +int64_t LowerEdge(int bin) { + return static_cast(std::ceil(FLOOR * std::exp2(static_cast(bin) / BINS_PER_OCTAVE))); +} + +} // namespace + +void Add(Histogram &h, int64_t value) { + if (value < FLOOR) + return; + const int b = Bin(value); + if (b < N_BINS) + h[b]++; +} + +void Add(Histogram &h, const std::vector &values, const std::vector &mask) { + for (size_t i = 0; i < values.size(); i++) + if (i >= mask.size() || mask[i] == 0) + Add(h, values[i]); +} + +std::optional Ceiling(const Histogram &h) { + for (int b = WINDOW; b < N_BINS; b++) { + const uint64_t below = *std::max_element(h.begin() + b - WINDOW, h.begin() + b); + if (h[b] >= MIN_PIXELS && static_cast(h[b]) > RISE * static_cast(below)) + return LowerEdge(b); + } + return std::nullopt; +} + +} // namespace saturation_ceiling diff --git a/image_analysis/image_preprocessing/SaturationCeiling.h b/image_analysis/image_preprocessing/SaturationCeiling.h new file mode 100644 index 000000000..f61ecbb25 --- /dev/null +++ b/image_analysis/image_preprocessing/SaturationCeiling.h @@ -0,0 +1,39 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include +#include + +// Where a detector's pixel values stop, read off the values themselves. The bright tail of a frame - +// the cores of Bragg peaks - falls off steadily with value; a detector that saturates clips everything +// beyond its ceiling down to it, so the values pile up just below where they stop. Where each pixel has +// its own ceiling (an ADC limit less that pixel's dark), the pile-up is a band rather than one value, +// and its lower edge is where clipping starts. A file's own statement of the limit is often missing +// or wrong for detectors outside the facility that wrote it. +namespace saturation_ceiling { + +// Values are binned on a log scale, BINS_PER_OCTAVE to a factor of two, from FLOOR up: a bin of the +// tail is then about 1% wide, narrow enough to place the edge, while below FLOOR a bin is narrower +// than one integer and the counts jump between neighbouring bins for no reason in the data. +constexpr int BINS_PER_OCTAVE = 64; +constexpr int64_t FLOOR = 256; +constexpr int N_BINS = BINS_PER_OCTAVE * 23; // up to 2^31 +// A pile-up bin holds more than RISE times the most any bin in the WINDOW below it held - a tail that +// only falls never does that - and at least MIN_PIXELS pixel values. +constexpr double RISE = 2.0; +constexpr int WINDOW = 8; +constexpr uint64_t MIN_PIXELS = 50; + +using Histogram = std::array; + +void Add(Histogram &h, const std::vector &values, const std::vector &mask); +void Add(Histogram &h, int64_t value); + +// The lower edge of the lowest bin that rises above the tail below it, if any does. +std::optional Ceiling(const Histogram &h); + +} // namespace saturation_ceiling diff --git a/reader/JFJochCrystFELReader.cpp b/reader/JFJochCrystFELReader.cpp index cb8d5e0f3..62fcf5d7c 100644 --- a/reader/JFJochCrystFELReader.cpp +++ b/reader/JFJochCrystFELReader.cpp @@ -76,6 +76,11 @@ void JFJochCrystFELReader::ReadFiles(const std::string &path, const std::string // A Cheetah CXI file carries what one flat panel needs - the frames at /entry_1/data_1/data, the pixel // size, the photon energy and a camera length, all in SI units - but not where the beam is. The panel is // put with its middle on the beam, which is as good as no centre at all: the run measures it. +// +// Only a square frame is taken this way. A Cheetah file from a tiled detector (CSPAD, AGIPD, ePix, +// JUNGFRAU, MPCCD) holds its panels stacked in a slab, which read as one flat panel would place every +// pixel wrongly and say nothing; those slabs are not square, the single-panel CCDs are, and a tiled +// detector needs its geometry file (--crystfel-geometry) anyway. bool JFJochCrystFELReader::CanReadCXI(const std::string &path) { try { if (!std::filesystem::exists(path)) @@ -83,7 +88,10 @@ bool JFJochCrystFELReader::CanReadCXI(const std::string &path) { const auto files = ListFiles(path); std::unique_lock ul(hdf5_mutex); HDF5ReadOnlyFile file(files[0]); - return file.Exists("/entry_1/data_1/data") && file.GetDimension("/entry_1/data_1/data").size() == 3 + if (!file.Exists("/entry_1/data_1/data")) + return false; + const auto dim = file.GetDimension("/entry_1/data_1/data"); + return dim.size() == 3 && dim[1] == dim[2] && file.Exists("/entry_1/instrument_1/detector_1/x_pixel_size") && file.Exists("/entry_1/instrument_1/source_1/energy"); } catch (...) { @@ -260,6 +268,12 @@ void JFJochCrystFELReader::Open(const std::string &path, const std::string &geom detector.PixelSize_um(static_cast(1e6 / p0.res)); // Nothing a parallax or sensor-efficiency correction needs is stated in a geometry file. detector.SensorThickness_um(0); + // Where the geometry states no limit, the stored integer type still has one - a Cheetah file keeps + // its frames as int16, and a value clipped at 32767 is no measurement either. + if (!saturation.has_value() && !stored_float_ && stored_size_ > 0) { + const double type_max = std::exp2(8.0 * static_cast(stored_size_) - (stored_signed_ ? 1 : 0)) - 1; + saturation = type_max / *std::max_element(adu_per_photon_.begin(), adu_per_photon_.end()); + } if (saturation.has_value()) detector.SaturationLimit(static_cast(std::floor(saturation.value()))); detector.BitDepthImage(32); diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 23e053bc3..c81c21744 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -858,6 +858,9 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b const int prepass_images = (prepass_end - start_image + config_.stride - 1) / config_.stride; const int early_images = experiment_.IsRotationIndexing() ? prepass_images : images_to_process; + if (full && !experiment_.IsRotationIndexing() && !config_.live) + DetectSaturation(start_image, images_to_process); + // After any geometry refinement, so the shadow is found about the beam centre actually used, // and before the azimuthal mapping and the output mask below, which both read pixel_mask_. if (config_.detect_beam_stop.has_value() || config_.adaptive_integration_radius) diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 49a9d1e26..4cac16c57 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -164,6 +164,9 @@ struct ProcessConfig { // two answers - the one comparison an on-failure trigger cannot make, because a centre wrong // along the spindle does not fail. bool beam_center_check = true; + // Whether a saturation level the frames show below the file's limit replaces it (DetectSaturation): + // for every input but a Jungfraujoch-written file, whose stated limit is trusted. + bool detect_saturation = false; // Adaptive integration radius (--adaptive-integration-radius). When set, the pre-scan measures how // wide the recorded spots are - r80, the radius holding 80 % of a spot's flux, over a fixed @@ -1378,6 +1381,8 @@ class Rugnux { const std::vector &frames); // Returns whether the run's geometry was moved off the file's. bool DecideStillsGeometry(int start_image, int images_to_process); + // Stills: where the frames' pixel values pile up, against the file's saturation limit (see there). + void DetectSaturation(int start_image, int images_to_process); // One full run of the pipeline (index + integrate + optional scale/merge/space-group + output). // write_output == false makes no outputs (ProcessResult::outputs) and streams nothing to the image sink; diff --git a/rugnux/RugnuxStills.cpp b/rugnux/RugnuxStills.cpp index d353fb020..440ddbcec 100644 --- a/rugnux/RugnuxStills.cpp +++ b/rugnux/RugnuxStills.cpp @@ -52,6 +52,7 @@ #include "../image_analysis/azint/AzIntEngineCPU.h" #include "../image_analysis/image_preprocessing/ImagePreprocessorCPU.h" #include "../image_analysis/image_preprocessing/ImagePreprocessorBuffer.h" +#include "../image_analysis/image_preprocessing/SaturationCeiling.h" #ifdef JFJOCH_USE_CUDA #include "../image_analysis/image_preprocessing/ImagePreprocessorGPU.h" #include "../image_analysis/image_preprocessing/ImagePreprocessorBufferGPU.h" @@ -468,3 +469,77 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) { experiment_.BeamX_pxl(x).BeamY_pxl(y).DetectorDistance_mm(distance); return x != file_x || y != file_y || distance != file_distance; } + +// Where this run's pixel values stop, read off the frames (saturation_ceiling): a sample spread over +// the run, every unmasked pixel of it. Measured on every stills run and reported beside what the file +// states. It replaces the file's limit only where the frames pile up below it and the file is not one +// Jungfraujoch wrote (config_.detect_saturation) - a file that states no limit, or the ADC's full +// range where each pixel's dark has already been taken off, otherwise integrates clipped peaks as if +// they were measured, and the strongest reflections at low resolution are the ones it spoils. +void Rugnux::DetectSaturation(int start_image, int images_to_process) { + Logger logger("Rugnux"); + TimingMark mark(logger, "saturation ceiling"); + constexpr int SAMPLE_FRAMES = 40; + const std::vector sample = select_equally_spaced_image_ordinals( + images_to_process, std::min(images_to_process, SAMPLE_FRAMES)); + const std::vector &mask = pixel_mask_.GetMask(); + + std::mutex m; + saturation_ceiling::Histogram total{}; + std::atomic next = 0; + auto worker = [&]() { + saturation_ceiling::Histogram h{}; + JFJochReaderRawImage img; + std::vector buffer; + std::vector values; + for (size_t i = next.fetch_add(1); i < sample.size() && !cancelled_; i = next.fetch_add(1)) { + if (!reader_.ReadRawImage(start_image + sample[i] * config_.stride, img)) + continue; + const size_t n = img.image.GetWidth() * img.image.GetHeight(); + const uint8_t *ptr = img.image.GetUncompressedPtr(buffer); + values.resize(n); + switch (img.image.GetMode()) { + case CompressedImageMode::Int32: + std::memcpy(values.data(), ptr, n * sizeof(int32_t)); + break; + case CompressedImageMode::Uint32: + for (size_t j = 0; j < n; j++) + values[j] = static_cast(std::min( + reinterpret_cast(ptr)[j], INT32_MAX)); + break; + case CompressedImageMode::Uint16: + std::copy_n(reinterpret_cast(ptr), n, values.begin()); + break; + case CompressedImageMode::Int16: + std::copy_n(reinterpret_cast(ptr), n, values.begin()); + break; + default: + continue; // a float or colour image has no integer ceiling to find + } + saturation_ceiling::Add(h, values, mask); + } + std::lock_guard lock(m); + for (size_t b = 0; b < h.size(); b++) + total[b] += h[b]; + }; + std::vector> futures; + for (int i = 0; i < config_.nthreads; ++i) + futures.push_back(std::async(std::launch::async, worker)); + for (auto &f : futures) + f.get(); + + const int64_t stated = experiment_.GetSaturationLimit(); + const std::optional ceiling = saturation_ceiling::Ceiling(total); + if (!ceiling) { + logger.Info("Saturation: the values of {} sampled frames fall off without piling up below any level - " + "nothing clips them under the file's limit of {}", sample.size(), stated); + return; + } + const bool adopt = config_.detect_saturation && *ceiling < stated; + logger.Info("Saturation: the values of {} sampled frames pile up from {} - the file's limit is {}{}", + sample.size(), *ceiling, stated, + adopt ? ", so pixels from there on are treated as overloaded" + : *ceiling < stated ? "; the file's is kept (written by Jungfraujoch)" : ""); + if (adopt) + experiment_.Detector().SaturationLimit(*ceiling); +} diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index ade463023..98bf3feac 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -151,7 +151,7 @@ void print_usage() { std::cout << " Detector geometry" << std::endl; std::cout << " --crystfel-geometry Read - an HDF5/CXI file, or a directory of them - as CrystFEL does with this geometry file: frames from its data path, panels placed at its corners, camera length, pixel size, gain, photon energy and bad regions from it. One wavelength and one camera length for the run; panel positions are taken as given" << std::endl; - std::cout << " (A Cheetah CXI file, or a directory of them, is read without one: one flat panel, with the pixel size, photon energy and camera length the file states and the beam centre measured; --detector-distance overrides a camera length the file has wrong)" << std::endl; + std::cout << " (A Cheetah CXI file with a square frame, or a directory of them, is read without one: one flat panel, with the pixel size, photon energy and camera length the file states and the beam centre measured; --detector-distance overrides a camera length the file has wrong)" << std::endl; std::cout << std::endl; std::cout << " Detector mask" << std::endl; @@ -3182,6 +3182,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { config.two_pass_rotation = two_pass_rotation; config.detect_beam_stop = detect_beam_stop; config.beam_center_check = beam_center_check; + config.detect_saturation = input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_cxi; config.adaptive_integration_radius = adaptive_integration_radius; config.counts_in_adu = input_is_smv || input_is_marccd || input_is_xtc || ((input_is_crystfel || input_is_cxi) && !crystfel_reader.CountsInPhotons()); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index b7eb6dae8..b440124ba 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -11,6 +11,7 @@ ADD_EXECUTABLE(jfjoch_portable_test EXCLUDE_FROM_ALL ZSTDCompressorTest.cpp JFJochReaderTest.cpp CrystFELGeometryTest.cpp + SaturationCeilingTest.cpp SweepLayoutTest.cpp CBORTest.cpp ImageSpotFinderCPUTest.cpp diff --git a/tests/SaturationCeilingTest.cpp b/tests/SaturationCeilingTest.cpp new file mode 100644 index 000000000..1a5f16cab --- /dev/null +++ b/tests/SaturationCeilingTest.cpp @@ -0,0 +1,52 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include +#include + +#include "../image_analysis/image_preprocessing/SaturationCeiling.h" + +// A bright tail, P(> v) ~ 1/v, as the cores of Bragg peaks give. +static saturation_ceiling::Histogram Tail(int64_t clip_from, int64_t clip_to) { + saturation_ceiling::Histogram h{}; + std::mt19937 rng(1); + std::uniform_real_distribution u(0.0, 1.0); + for (int i = 0; i < 2000000; i++) { + auto v = static_cast(std::min(1e9, 100.0 / (1.0 - u(rng)))); // P(> v) ~ 1/v above 100 + if (v >= clip_from) { + // Clipped at a ceiling of its own, somewhere in [clip_from, clip_to]. + v = clip_from + static_cast(u(rng) * static_cast(clip_to - clip_from)); + } + saturation_ceiling::Add(h, v); + } + return h; +} + +TEST_CASE("SaturationCeiling_NoClipping", "[portable]") { + CHECK(!saturation_ceiling::Ceiling(Tail(INT32_MAX, INT32_MAX)).has_value()); +} + +TEST_CASE("SaturationCeiling_OneLevel", "[portable]") { + const auto c = saturation_ceiling::Ceiling(Tail(12000, 12000)); + REQUIRE(c.has_value()); + CHECK(*c <= 12000); + CHECK(*c > 12000 * 0.98); +} + +TEST_CASE("SaturationCeiling_BandPerPixel", "[portable]") { + // Each pixel stops at its own level - the ADC's limit less that pixel's dark. + const auto c = saturation_ceiling::Ceiling(Tail(12400, 15600)); + REQUIRE(c.has_value()); + CHECK(*c <= 12400); + CHECK(*c > 12400 * 0.98); +} + +TEST_CASE("SaturationCeiling_MaskedPixelsLeftOut", "[portable]") { + saturation_ceiling::Histogram h{}; + const std::vector values(1000, 5000); + saturation_ceiling::Add(h, values, std::vector(1000, 1)); + CHECK(!saturation_ceiling::Ceiling(h).has_value()); + saturation_ceiling::Add(h, values, {}); + CHECK(saturation_ceiling::Ceiling(h).has_value()); +} -- 2.54.0 From 3643b890b0b0ebfad41a57bc6a157643190cd853 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 16:31:20 +0200 Subject: [PATCH 34/53] rugnux: --pixel-size, --gain and --saturation for what a foreign file states wrongly or not at all Beside --beam-x/--beam-y/--detector-distance/--wavelength, the remaining metadata a foreign XFEL file can get wrong are given on the command line rather than through a hand-built geometry file: --pixel-size (um); --saturation (the pixel value from which a pixel is overloaded, in the units the frames are read in), which also replaces the level stills read off the frames; and --gain (detector units per photon), for CXI and --crystfel-geometry input, whose reader converts the frames to photons with it - refused for other inputs, which are read in the units they state. Single-panel Rayonix CXI, 1316 frames, -C/-S, --detector-distance 105: --gain 0.37 (the pre-scan's background measurement) gives rugnux R-free 0.344 against 0.358 in detector units. Co-Authored-By: Claude Opus 5.5 --- reader/JFJochCrystFELReader.cpp | 10 ++++++++-- reader/JFJochCrystFELReader.h | 7 +++++-- rugnux/rugnux_cli.cpp | 27 ++++++++++++++++++++++++--- 3 files changed, 37 insertions(+), 7 deletions(-) diff --git a/reader/JFJochCrystFELReader.cpp b/reader/JFJochCrystFELReader.cpp index 62fcf5d7c..7392a855b 100644 --- a/reader/JFJochCrystFELReader.cpp +++ b/reader/JFJochCrystFELReader.cpp @@ -67,9 +67,13 @@ std::vector JFJochCrystFELReader::ListFiles(const std::string &path return files; } -void JFJochCrystFELReader::ReadFiles(const std::string &path, const std::string &geometry_file) { +void JFJochCrystFELReader::ReadFiles(const std::string &path, const std::string &geometry_file, + std::optional gain) { Close(); geom_ = crystfel::Read(geometry_file); + if (gain) + for (auto &p: geom_.panels) + p.adu_per_photon = *gain; Open(path, geometry_file); } @@ -99,7 +103,7 @@ bool JFJochCrystFELReader::CanReadCXI(const std::string &path) { } } -void JFJochCrystFELReader::ReadCXI(const std::string &path) { +void JFJochCrystFELReader::ReadCXI(const std::string &path, std::optional gain) { Close(); const auto files = ListFiles(path); { @@ -120,6 +124,8 @@ void JFJochCrystFELReader::ReadCXI(const std::string &path) { p.corner_y = -0.5 * static_cast(dim[1]); p.res = 1.0 / mean(detector + "x_pixel_size"); p.clen = fmt::format("{}", file.Exists(detector + "distance") ? mean(detector + "distance") : 0.1); + if (gain) + p.adu_per_photon = *gain; geom_ = {}; geom_.panels.push_back(p); geom_.data = "/entry_1/data_1/data"; diff --git a/reader/JFJochCrystFELReader.h b/reader/JFJochCrystFELReader.h index a1fbac988..3a034c11c 100644 --- a/reader/JFJochCrystFELReader.h +++ b/reader/JFJochCrystFELReader.h @@ -4,6 +4,7 @@ #pragma once #include +#include #include #include @@ -72,11 +73,13 @@ public: JFJochCrystFELReader(); ~JFJochCrystFELReader() override; - void ReadFiles(const std::string &path, const std::string &geometry_file); + // gain, where given, is the detector units per photon of every panel, whatever the geometry says. + void ReadFiles(const std::string &path, const std::string &geometry_file, + std::optional gain = std::nullopt); // A Cheetah CXI file (or a directory of them), with no geometry file: one flat panel, placed by what // the file states, centred on the beam. static bool CanReadCXI(const std::string &path); - void ReadCXI(const std::string &path); + void ReadCXI(const std::string &path, std::optional gain = std::nullopt); // False where the geometry states no gain, so the images are in detector units, not photons. [[nodiscard]] bool CountsInPhotons() const; diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 98bf3feac..4f385d706 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -259,6 +259,9 @@ void print_usage() { std::cout << " --beam-y Beam center Y (pixel)" << std::endl; std::cout << " --detector-distance Detector distance (mm)" << std::endl; std::cout << " --wavelength Wavelength (A)" << std::endl; + std::cout << " --pixel-size Pixel size (um)" << std::endl; + std::cout << " --gain Detector units per photon, for a file that states none (CXI and --crystfel-geometry input; the frames are converted to photons with it)" << std::endl; + std::cout << " --saturation Pixel value from which a pixel is overloaded, in the units the frames are read in (photons where they are converted). Replaces the file's limit and the one stills read off the frames" << std::endl; std::cout << " --rot1 PONI rotation 1 (rad)" << std::endl; std::cout << " --rot2 PONI rotation 2 (rad)" << std::endl; std::cout << " --rot3 PONI rotation 3, about the beam (rad)" << std::endl; @@ -353,6 +356,9 @@ enum { OPT_BEAM_Y, OPT_DETECTOR_DISTANCE, OPT_WAVELENGTH, + OPT_PIXEL_SIZE, + OPT_GAIN, + OPT_SATURATION, OPT_ROT1, OPT_ROT2, OPT_ROT3, @@ -418,6 +424,9 @@ static option long_options[] = { {"beam-y", required_argument, nullptr, OPT_BEAM_Y}, {"detector-distance", required_argument, nullptr, OPT_DETECTOR_DISTANCE}, {"wavelength", required_argument, nullptr, OPT_WAVELENGTH}, + {"pixel-size", required_argument, nullptr, OPT_PIXEL_SIZE}, + {"gain", required_argument, nullptr, OPT_GAIN}, + {"saturation", required_argument, nullptr, OPT_SATURATION}, {"rot1", required_argument, nullptr, OPT_ROT1}, {"rot2", required_argument, nullptr, OPT_ROT2}, {"rot3", required_argument, nullptr, OPT_ROT3}, @@ -979,6 +988,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { // Geometry overrides (default: keep the value stored in the input file) std::optional beam_x, beam_y, detector_distance_mm, wavelength_A, rot1_rad, rot2_rad, rot3_rad, polarization_factor; + std::optional pixel_size_um, gain_per_photon, saturation_value; bool detector_mirror_y = false; int64_t detector_quarter_turns = 0; std::optional smooth_g_deg_arg; // --smooth-g[=deg]; default 5 deg for rot3d, 0 (off) otherwise @@ -1573,6 +1583,9 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { case OPT_BEAM_X: beam_x = parse_float_arg(optarg, "--beam-x", logger); break; case OPT_BEAM_Y: beam_y = parse_float_arg(optarg, "--beam-y", logger); break; case OPT_DETECTOR_DISTANCE: detector_distance_mm = parse_float_arg(optarg, "--detector-distance", logger); break; + case OPT_PIXEL_SIZE: pixel_size_um = parse_float_arg(optarg, "--pixel-size", logger); break; + case OPT_GAIN: gain_per_photon = parse_float_arg(optarg, "--gain", logger); break; + case OPT_SATURATION: saturation_value = parse_float_arg(optarg, "--saturation", logger); break; case OPT_WAVELENGTH: { // Guard > 0: wavelength is used as a divisor (WVL_1A_IN_KEV / wavelength) below, and a 0 // would produce a non-finite incident energy that throws unguarded and aborts the process. @@ -1744,6 +1757,11 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { bool input_is_cxi = false; try { input_is_crystfel = !crystfel_geometry.empty(); + if (gain_per_photon && !input_is_crystfel && !JFJochCrystFELReader::CanReadCXI(input_file)) { + logger.Error("--gain is for a CXI file or --crystfel-geometry input; {} is read in the units it states", + input_file); + exit(EXIT_FAILURE); + } input_is_cxi = !input_is_crystfel && JFJochCrystFELReader::CanReadCXI(input_file); input_is_cbf = !input_is_crystfel && !input_is_cxi && JFJochCBFReader::CanRead(input_file); input_is_marccd = !input_is_cbf && JFJochMarCCDReader::CanRead(input_file); @@ -1751,12 +1769,12 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { input_is_xtc = !input_is_crystfel && !input_is_cxi && !input_is_cbf && !input_is_marccd && !input_is_smv && JFJochXTCReader::CanRead(input_file); if (input_is_crystfel) { - crystfel_reader.ReadFiles(input_file, crystfel_geometry); + crystfel_reader.ReadFiles(input_file, crystfel_geometry, gain_per_photon); reader_ptr = &crystfel_reader; logger.Info("Read {} frames from {} with the CrystFEL geometry {}", crystfel_reader.GetNumberOfImages(), input_file, crystfel_geometry); } else if (input_is_cxi) { - crystfel_reader.ReadCXI(input_file); + crystfel_reader.ReadCXI(input_file, gain_per_photon); reader_ptr = &crystfel_reader; logger.Info("Read {} frames from the CXI file {}", crystfel_reader.GetNumberOfImages(), input_file); } else if (input_is_cbf) { @@ -2532,6 +2550,8 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { if (beam_y) experiment.BeamY_pxl(beam_y.value()); if (detector_distance_mm) experiment.DetectorDistance_mm(detector_distance_mm.value()); if (wavelength_A) experiment.IncidentEnergy_keV(WVL_1A_IN_KEV / wavelength_A.value()); + if (pixel_size_um) experiment.Detector().PixelSize_um(pixel_size_um.value()); + if (saturation_value) experiment.Detector().SaturationLimit(static_cast(std::ceil(saturation_value.value()))); if (rot1_rad) experiment.PoniRot1_rad(rot1_rad.value()); if (rot2_rad) experiment.PoniRot2_rad(rot2_rad.value()); if (rot3_rad) experiment.PoniRot3_rad(rot3_rad.value()); @@ -3182,7 +3202,8 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { config.two_pass_rotation = two_pass_rotation; config.detect_beam_stop = detect_beam_stop; config.beam_center_check = beam_center_check; - config.detect_saturation = input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_cxi; + config.detect_saturation = !saturation_value + && (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_cxi); config.adaptive_integration_radius = adaptive_integration_radius; config.counts_in_adu = input_is_smv || input_is_marccd || input_is_xtc || ((input_is_crystfel || input_is_cxi) && !crystfel_reader.CountsInPhotons()); -- 2.54.0 From 63595c0513f5f94905287fd9efb287d371c261af Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 16:37:58 +0200 Subject: [PATCH 35/53] rugnux: an LCLS XTC stream is read without a .expt when --pixel-size, --detector-distance and --wavelength are given Brings the stills-geom-auto-lcls merge resolution (b10a8ce00) onto the rc178-based stills branch. Co-Authored-By: Claude Opus 5.5 --- reader/JFJochXTCReader.cpp | 28 +++++++++++++++++++++------- reader/JFJochXTCReader.h | 11 ++++++++--- rugnux/rugnux_cli.cpp | 8 +++++++- 3 files changed, 36 insertions(+), 11 deletions(-) diff --git a/reader/JFJochXTCReader.cpp b/reader/JFJochXTCReader.cpp index baea3d85d..0ad5f1ba4 100644 --- a/reader/JFJochXTCReader.cpp +++ b/reader/JFJochXTCReader.cpp @@ -123,15 +123,29 @@ void JFJochXTCReader::ReadFiles(const std::string &path) { f.file + ": the images are not all the same size"); const std::string expt = FindExpt(frames_.front().file); - if (expt.empty()) - throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, - "XTC carries no detector geometry: put the DIALS experiment list (.expt) of " - "the run beside the XTC files, or in the directory above them"); - Logger("XTCReader").Info("{} images in XTC, geometry from {}", frames_.size(), expt); - dataset_ = std::make_shared(); dataset_->experiment = default_experiment; - ReadGeometry(expt); + has_geometry_ = !expt.empty(); + if (has_geometry_) { + Logger("XTCReader").Info("{} images in XTC, geometry from {}", frames_.size(), expt); + ReadGeometry(expt); + } else { + // XTC states no geometry. One flat panel, square-on, with its middle on the beam - the run + // measures the centre - and the pixel size, distance and wavelength to be given by the caller. + Logger("XTCReader").Info("{} images in XTC, no .expt beside them: one flat panel centred on the beam", + frames_.size()); + const auto &f0 = frames_.front(); + DetectorSetup detector = DetDECTRIS(f0.width, f0.height, "XTC camera", {}); + detector.SensorThickness_um(0); + detector.SaturationLimit(65535); + detector.BitDepthImage(32); + detector.MinFrameTime(std::chrono::microseconds(0)); + detector.MinCountTime(std::chrono::microseconds(0)); + detector.ReadOutTime(std::chrono::nanoseconds(0)); + dataset_->experiment.Detector(detector); + dataset_->experiment.BeamX_pxl(0.5f * static_cast(f0.width)); + dataset_->experiment.BeamY_pxl(0.5f * static_cast(f0.height)); + } dataset_->error_value = -1; dataset_->experiment.ImagesPerTrigger(static_cast(frames_.size())); dataset_->pixel_mask = std::make_shared(frames_.front().width, frames_.front().height); diff --git a/reader/JFJochXTCReader.h b/reader/JFJochXTCReader.h index 6b22eb4ef..17fe30f69 100644 --- a/reader/JFJochXTCReader.h +++ b/reader/JFJochXTCReader.h @@ -15,9 +15,10 @@ // not read here. // // XTC records the events and not the experiment: there is no distance, beam centre or wavelength -// in it that this reader can use. Those come from a DIALS experiment list (.expt) beside the XTC -// files, or in their parent directory - the geometry a deposition ships for cctbx.xfel, and what -// cctbx.xfel itself reads them with. Without one the run is refused rather than guessed. +// in it that this reader can use. Where a DIALS experiment list (.expt) lies beside the XTC files, or +// in their parent directory - the geometry a deposition ships for cctbx.xfel - it is read; otherwise +// the panel is placed square-on with its middle on the beam, the run measures the centre, and the +// pixel size, distance and wavelength are given on the command line. // // The images are stills: no goniometer. Each is decoded on demand. The pixels are passed on as // stored - the CCD's constant offset of a few counts is left in, as it is in an SMV image, and the @@ -25,6 +26,7 @@ class JFJochXTCReader : public JFJochReader { std::vector frames_; std::shared_ptr dataset_; + bool has_geometry_ = false; bool LoadImage_i(std::shared_ptr &dataset, DataMessage &message, @@ -44,6 +46,9 @@ public: // path is one XTC stream, or a directory whose streams are all read. void ReadFiles(const std::string &path); + // Whether a DIALS .expt beside the streams gave the geometry; without one the pixel size, the + // distance and the wavelength are the caller's to give. + [[nodiscard]] bool HasGeometry() const { return has_geometry_; } [[nodiscard]] uint64_t GetNumberOfImages() const override; void Close() override; diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 4f385d706..071cc599a 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -1794,6 +1794,11 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { smv_reader.GetNumberOfImages(), input_file); } else if (input_is_xtc) { xtc_reader.ReadFiles(input_file); + if (!xtc_reader.HasGeometry() && (!pixel_size_um || !detector_distance_mm || !wavelength_A)) { + logger.Error("XTC states no geometry and no DIALS .expt lies beside it: give --pixel-size, " + "--detector-distance and --wavelength (the beam centre is measured)"); + exit(EXIT_FAILURE); + } reader_ptr = &xtc_reader; logger.Info("Read {} XTC still images from {}", xtc_reader.GetNumberOfImages(), input_file); } else if (std::filesystem::is_directory(input_file)) { @@ -3203,7 +3208,8 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { config.detect_beam_stop = detect_beam_stop; config.beam_center_check = beam_center_check; config.detect_saturation = !saturation_value - && (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_cxi); + && (input_is_cbf || input_is_marccd || input_is_smv + || input_is_crystfel || input_is_cxi || input_is_xtc); config.adaptive_integration_radius = adaptive_integration_radius; config.counts_in_adu = input_is_smv || input_is_marccd || input_is_xtc || ((input_is_crystfel || input_is_cxi) && !crystfel_reader.CountsInPhotons()); -- 2.54.0 From 750579bfe42c15cf80af8a79fca99686533a48b5 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 14:52:09 +0200 Subject: [PATCH 36/53] _process.h5 carries the offline per-image scale; -e documented as exclusive The writer's MX plugin decided the image-scale arrays were present whenever its own vector was non-empty, which is always, so every data file wrote them as NaN - and the master's SaveVectorIfMissing then left the end message's values out. It now marks them present only when an image message carries a scale. rugnux, whose scale exists only after the image messages went to the file, puts the per-image scale, its CC and mosaicity into the end message, so imageScaleFactor/imageScaleCC/imageScaleMosaicity hold the run's values (NaN only for frames without a scale). The usage line of -e now says the end image is exclusive, which is what the code does. Co-Authored-By: Claude Opus 5.5 --- rugnux/Rugnux.cpp | 9 +++++++++ rugnux/rugnux_cli.cpp | 2 +- writer/HDF5DataFilePluginMX.cpp | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index c81c21744..ed5a51d6d 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -1252,6 +1252,15 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b result.outputs->plot = std::move(plot); } + // The per-image scale exists only now, after scaling - long after the image messages went to the + // file - so it reaches the _process.h5 through the end message. + if (indexer && full && !cancelled_ && !superseded) { + const ScalingResult scaling(indexer->GetIntegrationOutcome()); + end_msg.image_scale_factor = scaling.image_scale_g; + end_msg.image_scale_cc = scaling.image_cc; + end_msg.image_scale_mosaicity = scaling.mosaicity_deg; + } + if (sink) sink->End(end_msg); result.end_message = end_msg; diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 071cc599a..1dcf98c1c 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -129,7 +129,7 @@ void print_usage() { std::cout << " --no-gpu Run on the CPU only, as on a machine without a GPU: CUDA is not " "touched and every GPU step takes its CPU path (-X Auto and FFT index with FFTW)" << std::endl; std::cout << " -s, --start-image Start image number (default: 0)" << std::endl; - std::cout << " -e, --end-image End image number (default: all)" << std::endl; + std::cout << " -e, --end-image End image number, exclusive: -s 0 -e 100 processes images 0-99 (default: all)" << std::endl; std::cout << " -t, --stride Image stride (default: 1)" << std::endl; std::cout << " -v, --verbose Show the whole log on the terminal in place of the steps " "(the log file is written either way)" << std::endl; diff --git a/writer/HDF5DataFilePluginMX.cpp b/writer/HDF5DataFilePluginMX.cpp index ea4c7d81a..3acf351ab 100644 --- a/writer/HDF5DataFilePluginMX.cpp +++ b/writer/HDF5DataFilePluginMX.cpp @@ -195,7 +195,7 @@ void HDF5DataFilePluginMX::Write(const DataMessage &msg, uint64_t image_number) niggli_class[image_number] = 0; } - if (!image_scale_factor.empty()) + if (msg.image_scale_factor.has_value()) image_scale_present = true; image_scale_factor[image_number] = msg.image_scale_factor.value_or(NAN); -- 2.54.0 From ae7b68dd68fed12f4ce0e193aec0a4f14ae550f5 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 16:39:44 +0200 Subject: [PATCH 37/53] rugnux stills: a given cell is where the geometry refinement starts, not what it fits the distance to On stills a cell and a detector distance are degenerate: a spot's radius fixes only their ratio. The geometry refinement that -C switches on anchored the cell and refined the distance, so a deposited cell a percent off this crystal moved a correct distance by as much (MPCCD: 50.47 -> 50.14 mm, R-free 0.259 -> 0.318). Now the distance is held and the cell refined from the spots, starting from the given one. Only where the frames have rejected the file's distance (DecideStillsGeometry's grid, which is what -C recovers a bad header distance with) is the cell anchored and the distance refined from it. The grid itself asks the given cell, not one refined at the file's distance. Co-Authored-By: Claude Opus 5.5 --- .../geom_refinement/GeometryRefiner.cpp | 2 ++ .../geom_refinement/GeometryRefiner.h | 2 ++ rugnux/Rugnux.h | 5 +++++ rugnux/RugnuxStills.cpp | 22 +++++++++++++++++-- 4 files changed, 29 insertions(+), 2 deletions(-) diff --git a/image_analysis/geom_refinement/GeometryRefiner.cpp b/image_analysis/geom_refinement/GeometryRefiner.cpp index 45630b291..ded2b3dd5 100644 --- a/image_analysis/geom_refinement/GeometryRefiner.cpp +++ b/image_analysis/geom_refinement/GeometryRefiner.cpp @@ -258,6 +258,8 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge problem.SetParameterUpperBound(cell_len, j, 1.3 * cell_len0[j]); } + if (!settings.refine_distance) + problem.SetParameterBlockConstant(&distance_mm); problem.SetParameterBlockConstant(detector_rot); // tilt not refined problem.SetParameterBlockConstant(rot_vec); problem.SetParameterBlockConstant(cell_ang); // angles not refined in v1 diff --git a/image_analysis/geom_refinement/GeometryRefiner.h b/image_analysis/geom_refinement/GeometryRefiner.h index 5dce6c73a..ccd18cd4e 100644 --- a/image_analysis/geom_refinement/GeometryRefiner.h +++ b/image_analysis/geom_refinement/GeometryRefiner.h @@ -36,6 +36,8 @@ struct GeometryRefinerSettings { // Cell-scale regularizer strength (anchors the cell lengths to the input cell so the otherwise // degenerate distance is determined). Large => cell effectively fixed (the validated safe regime). double cell_reg_coeff = 30.0; + // With the distance held, the cell is no longer degenerate with it and is refined from the spots. + bool refine_distance = true; int rounds = 3; // hkl-reassignment / tolerance-tightening rounds double beam_range_px = 8.0; // beam bound around the nominal value double distance_range_mm = 8.0; // distance bound around the nominal value diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 4cac16c57..23da36c49 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -1381,6 +1381,11 @@ class Rugnux { const std::vector &frames); // Returns whether the run's geometry was moved off the file's. bool DecideStillsGeometry(int start_image, int images_to_process); + // Stills: the cell the user gave (-C / reference), kept apart from the one the geometry refinement + // writes into experiment_, and whether the detector distance has been taken from it - only once the + // frames have rejected the file's distance (DecideStillsGeometry). See RefineStillsGeometry. + std::optional stills_user_cell_; + bool stills_distance_from_cell_ = false; // Stills: where the frames' pixel values pile up, against the file's saturation limit (see there). void DetectSaturation(int start_image, int images_to_process); diff --git a/rugnux/RugnuxStills.cpp b/rugnux/RugnuxStills.cpp index 440ddbcec..02d516729 100644 --- a/rugnux/RugnuxStills.cpp +++ b/rugnux/RugnuxStills.cpp @@ -95,7 +95,9 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_ return; } // The bundle anchors a known cell; without one there is nothing to refine against. - const auto cell = experiment_.GetUnitCell(); + if (!stills_user_cell_) + stills_user_cell_ = experiment_.GetUnitCell(); + const auto cell = stills_user_cell_; if (!cell.has_value()) { logger.Warning("--refine-geometry needs a reference unit cell (-C / -S / reference MTZ). Skipping."); return; @@ -245,6 +247,16 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_ GeometryRefinerSettings gr_settings; gr_settings.crystal_system = experiment_.GetCrystalSystem(); gr_settings.num_threads = config_.nthreads; + // A cell and a distance are degenerate on stills: a spot's radius fixes only their ratio. Which one + // to believe is a question of which is the better measurement. A deposited or textbook cell is + // routinely a percent off this crystal (another temperature, another batch), and a detector distance + // a facility states is usually better than that - so the distance is held and the cell refined, with + // the user's cell only the starting point. Only where the frames have already shown the file's + // distance to be wrong (DecideStillsGeometry, which is what -C buys on a header with a bad distance) + // is the cell the better of the two, and then it is anchored and the distance refined from it. + gr_settings.refine_distance = stills_distance_from_cell_; + if (!stills_distance_from_cell_) + gr_settings.cell_reg_coeff = 0.0; const GeometryRefinerResult r = RefineGlobalGeometry( experiment_.GetDiffractionGeometry(), *cell, strong, gr_settings); @@ -388,6 +400,8 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) { const auto trial = [&](float x, float y, float distance, const std::vector &on) { DiffractionExperiment e = experiment_; e.BeamX_pxl(x).BeamY_pxl(y).DetectorDistance_mm(distance); + if (stills_user_cell_) + e.SetUnitCell(*stills_user_cell_); // the grid asks the user's cell, not the one refined at the file's distance const StillsTrial t = IndexStillsSample(e, pool, on); logger.Debug("Stills geometry trial: centre ({:.2f},{:.2f}) distance {:.2f} mm - {}/{} frames indexed, " "{} of their {} spots on the lattice", x, y, distance, t.indexed, t.frames, @@ -454,8 +468,12 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) { best_distance = grid_best * f; } } - if (score_here < FILE_KEPT_ABOVE * best) + if (score_here < FILE_KEPT_ABOVE * best) { distance = best_distance; + stills_distance_from_cell_ = true; + if (stills_user_cell_) + experiment_.SetUnitCell(*stills_user_cell_); + } logger.Info("Stills detector distance: the file's {:.2f} mm puts {} sampled spots on the lattice, the best " "between {:.0f}% and {:.0f}% of it {} at {:.2f} mm - the run uses {:.2f} mm", file_distance, score_here, 100 * MIN_FACTOR, 100 * MAX_FACTOR, best, best_distance, distance); -- 2.54.0 From 380cdf3dc7c6fa4ba095d99eefeb0b6ff6bba74d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 18:02:10 +0200 Subject: [PATCH 38/53] rugnux stills with a given cell: one crystal form merged, distance and centre searched together, a stated distance kept - A given cell keeps only the frames whose refined cell belongs to its population (GivenCellPopulation): the peak of the refined-volume histogram nearest the given cell, out to the valley that separates it from another population, with the bin set by the data's own spread. A sample of two crystal forms indexed against one form's cell otherwise merges both; a run with one population keeps every frame. - The distance grid is run from every centre hypothesis, not only the one decided at the file's distance, and a local grid of centres follows at the distance decided. A wrong header distance and a centre several pixels off (both background estimators misread it on a Rayonix set) otherwise leave each other flat; now the cell alone finds both. - A distance given with --detector-distance is not searched. Co-Authored-By: Claude Opus 5.5 --- image_analysis/scale_merge/CMakeLists.txt | 2 + image_analysis/scale_merge/CellPopulation.cpp | 77 +++++++++++++++++++ image_analysis/scale_merge/CellPopulation.h | 19 +++++ rugnux/Rugnux.cpp | 2 + rugnux/Rugnux.h | 2 + rugnux/RugnuxScaleMerge.cpp | 26 +++++++ rugnux/RugnuxStills.cpp | 68 ++++++++++++---- rugnux/rugnux_cli.cpp | 1 + tests/CMakeLists.txt | 1 + tests/CellPopulationTest.cpp | 50 ++++++++++++ 10 files changed, 233 insertions(+), 15 deletions(-) create mode 100644 image_analysis/scale_merge/CellPopulation.cpp create mode 100644 image_analysis/scale_merge/CellPopulation.h create mode 100644 tests/CellPopulationTest.cpp diff --git a/image_analysis/scale_merge/CMakeLists.txt b/image_analysis/scale_merge/CMakeLists.txt index bae06d3d2..0fd093890 100644 --- a/image_analysis/scale_merge/CMakeLists.txt +++ b/image_analysis/scale_merge/CMakeLists.txt @@ -11,6 +11,8 @@ ADD_LIBRARY(JFJochScaleMerge Merge.h ScaleOnTheFly.cpp ScaleOnTheFly.h + CellPopulation.cpp + CellPopulation.h StillsPartialityRefine.cpp StillsPartialityRefine.h RotationScaleMerge.cpp diff --git a/image_analysis/scale_merge/CellPopulation.cpp b/image_analysis/scale_merge/CellPopulation.cpp new file mode 100644 index 000000000..ff8f7b145 --- /dev/null +++ b/image_analysis/scale_merge/CellPopulation.cpp @@ -0,0 +1,77 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "CellPopulation.h" + +#include +#include +#include + +std::pair GivenCellPopulation(const std::vector &relative_volume) { + constexpr double everything = std::numeric_limits::infinity(); + // Below this many frames a histogram has no valleys to speak of. + constexpr size_t MIN_FRAMES = 50; + // A valley separates two populations only where it dips below this fraction of the kept peak; a + // shallower dip is noise in one population's tail. + constexpr double VALLEY_FRACTION = 0.5; + + const size_t n = relative_volume.size(); + if (n < MIN_FRAMES) + return {-everything, everything}; + + std::vector v = relative_volume; + std::sort(v.begin(), v.end()); + // The bin: twice a robust standard deviation over n^(1/3) (Freedman-Diaconis with the median + // absolute deviation in place of the interquartile range, which a second population inflates). + const double median = v[n / 2]; + std::vector dev(n); + for (size_t i = 0; i < n; i++) + dev[i] = std::fabs(v[i] - median); + std::nth_element(dev.begin(), dev.begin() + n / 2, dev.end()); + const double bin = 2.0 * 1.4826 * dev[n / 2] / std::cbrt(static_cast(n)); + if (!(bin > 0)) + return {-everything, everything}; + + const double lo = v.front(); + const int nbins = static_cast((v.back() - lo) / bin) + 1; + std::vector count(nbins, 0.0); + for (double x : v) + count[std::min(nbins - 1, static_cast((x - lo) / bin))] += 1.0; + // Three-bin running mean, so a single empty or full bin is not a valley or a peak. + std::vector smooth(nbins, 0.0); + for (int i = 0; i < nbins; i++) { + const int a = std::max(0, i - 1), b = std::min(nbins - 1, i + 1); + double s = 0; + for (int j = a; j <= b; j++) + s += count[j]; + smooth[i] = s / (b - a + 1); + } + + // The peak nearest the given cell: from its bin, uphill. + int peak = std::clamp(static_cast(std::floor(-lo / bin)), 0, nbins - 1); + while (true) { + if (peak + 1 < nbins && smooth[peak + 1] > smooth[peak]) peak++; + else if (peak > 0 && smooth[peak - 1] > smooth[peak]) peak--; + else break; + } + + // Out from the peak on each side: the valley is the lowest point, below VALLEY_FRACTION of the peak, + // before the histogram rises again to twice its height by more than counting noise - the next + // population, however small. None: the population runs out to the end of the data. + const auto valley = [&](int step) { + int lowest = peak; + for (int i = peak + step; i >= 0 && i < nbins; i += step) { + if (smooth[i] < smooth[lowest]) + lowest = i; + const double low = smooth[lowest], here = smooth[i]; + // A three-bin mean of Poisson counts has a variance of a third of its value. + const bool rises = here >= 2.0 * low && here - low > 3.0 * std::sqrt((here + low) / 3.0); + if (low < VALLEY_FRACTION * smooth[peak] && rises) + return lowest; + } + return -1; + }; + const int left = valley(-1), right = valley(+1); + return {left < 0 ? -everything : lo + (left + 0.5) * bin, + right < 0 ? everything : lo + (right + 0.5) * bin}; +} diff --git a/image_analysis/scale_merge/CellPopulation.h b/image_analysis/scale_merge/CellPopulation.h new file mode 100644 index 000000000..09f326375 --- /dev/null +++ b/image_analysis/scale_merge/CellPopulation.h @@ -0,0 +1,19 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include + +// Serial stills given a cell: which frames belong to that crystal. A sample can hold two crystal forms +// (a polymorph, a different hydration state), and a known-cell indexer fits both to the given cell's +// orientation search - the refined cells then fall into two volume populations, and merging both mixes +// two structures. The frames kept are the population the given cell sits in: the peak of the histogram +// of v / v_given - 1 nearest to 0, out to the valley on either side that separates it from another +// population. The histogram bin follows the spread of the data, so neither the +// width of a population nor its distance to the next is a fixed number. A run with one population has +// no such valley and keeps every frame. +// +// relative_volume: v / v_given - 1 per indexed frame. Returns the [low, high] range of it to keep. +std::pair GivenCellPopulation(const std::vector &relative_volume); diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index ed5a51d6d..6f1db8cd2 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -849,6 +849,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // sample of strongly-indexed frames and apply it to experiment_, so everything below (azimuthal // mapping, the main index+integrate pass, scaling) uses the refined geometry. Rotation is skipped // inside (it has its own two-pass). + if (!experiment_.IsRotationIndexing()) + stills_user_cell_ = experiment_.GetUnitCell(); if (full && config_.refine_geometry.has_value()) RefineStillsGeometry(start_image, end_image, images_to_process, observer); diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 23da36c49..14c37f309 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -164,6 +164,8 @@ struct ProcessConfig { // two answers - the one comparison an on-failure trigger cannot make, because a centre wrong // along the spindle does not fail. bool beam_center_check = true; + // The detector distance was given on the command line: stills do not search it (DecideStillsGeometry). + bool detector_distance_given = false; // Whether a saturation level the frames show below the file's limit replaces it (DetectSaturation): // for every input but a Jungfraujoch-written file, whose stated limit is trusted. bool detect_saturation = false; diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 661941763..49dd2d3b4 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -83,6 +83,7 @@ #include #include #include "RugnuxPipeline.h" +#include "../image_analysis/scale_merge/CellPopulation.h" using namespace rugnux_internal; @@ -512,6 +513,31 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { logger.Info("Sweep ends: left out {} frame(s) at the start and {} at the end that recorded no " "diffraction - a shutter or beam that was not on them", at_start, at_end); } + // Stills given a cell: only the frames whose refined cell belongs to its population are merged + // (see GivenCellPopulation) - a second crystal form, fitted to the same cell, is left out. + if (!experiment_.IsRotationIndexing() && stills_user_cell_) { + auto &outcomes = indexer->GetIntegrationOutcome(); + const double given = CrystalLattice(*stills_user_cell_).CalcVolume(); + std::vector rel; + for (const auto &o : outcomes) + if (!o.reflections.empty()) + rel.push_back(o.latt.CalcVolume() / given - 1.0); + const auto [low, high] = GivenCellPopulation(rel); + size_t left_out = 0; + for (auto &o : outcomes) { + if (o.reflections.empty()) + continue; + const double r = o.latt.CalcVolume() / given - 1.0; + if (r < low || r > high) { + o.reflections.clear(); + left_out++; + } + } + if (left_out > 0) + logger.Info("Given cell: {} of {} indexed frames refined to a cell outside its population " + "({:+.1f}% to {:+.1f}% in volume) and are left out of the merge", + left_out, rel.size(), 100 * low, 100 * high); + } size_t ice_flagged = 0, ice_total = 0; { auto &outcomes = indexer->GetIntegrationOutcome(); diff --git a/rugnux/RugnuxStills.cpp b/rugnux/RugnuxStills.cpp index 02d516729..a7a5c3ced 100644 --- a/rugnux/RugnuxStills.cpp +++ b/rugnux/RugnuxStills.cpp @@ -95,8 +95,6 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_ return; } // The bundle anchors a known cell; without one there is nothing to refine against. - if (!stills_user_cell_) - stills_user_cell_ = experiment_.GetUnitCell(); const auto cell = stills_user_cell_; if (!cell.has_value()) { logger.Warning("--refine-geometry needs a reference unit cell (-C / -S / reference MTZ). Skipping."); @@ -446,30 +444,41 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) { // and only then is the distance a hypothesis of its own: without a cell a wrong distance indexes as a // cell of the wrong size, which only the curvature far from the beam tells apart. The stills geometry // refinement takes it from there (RunPipeline). + // A distance the user gave on the command line is taken as given: a sample of mixed crystal forms + // can put more spots on one given cell at a distance that is not the detector's. float distance = file_distance; - if (experiment_.GetUnitCell().has_value()) { - float grid_best = file_distance; - int64_t grid_score = trial(x, y, file_distance, strongest); - for (float f = MIN_FACTOR; f <= MAX_FACTOR; f *= STEP) { - if (std::fabs(f - 1.0f) < 0.5f * (STEP - 1.0f)) - continue; // the file's own, scored above - const int64_t score = trial(x, y, file_distance * f, strongest); - if (score > grid_score) { - grid_score = score; - grid_best = file_distance * f; + if (experiment_.GetUnitCell().has_value() && !config_.detector_distance_given) { + // Every centre hypothesis gets its own grid: where the file's distance is wrong, the centre + // decided at it had little indexing to go on, and a wrong centre in turn leaves the grid flat. + std::vector> centres{{x, y}, {file_x, file_y}}; + centres.insert(centres.end(), candidates.begin(), candidates.end()); + float grid_best = file_distance, grid_x = x, grid_y = y; + int64_t grid_score = -1; + for (const auto &[cx, cy] : centres) + for (float f = MIN_FACTOR; f <= MAX_FACTOR; f *= STEP) { + const int64_t score = trial(cx, cy, file_distance * f, strongest); + if (score > grid_score) { + grid_score = score; + grid_best = file_distance * f; + grid_x = cx; + grid_y = cy; + } } - } - float best_distance = file_distance; + float best_distance = file_distance, best_x = x, best_y = y; int64_t best = score_here; for (const float f : {0.98f, 0.99f, 1.0f, 1.01f, 1.02f}) { - const int64_t score = trial(x, y, grid_best * f, frames); + const int64_t score = trial(grid_x, grid_y, grid_best * f, frames); if (score > best) { best = score; best_distance = grid_best * f; + best_x = grid_x; + best_y = grid_y; } } if (score_here < FILE_KEPT_ABOVE * best) { distance = best_distance; + x = best_x; + y = best_y; stills_distance_from_cell_ = true; if (stills_user_cell_) experiment_.SetUnitCell(*stills_user_cell_); @@ -482,6 +491,35 @@ bool Rugnux::DecideStillsGeometry(int start_image, int images_to_process) { decide_centre(distance); } + // With a cell, also a local grid of centres about the one decided, at the distance decided: the two + // background estimators can both be several pixels off (a shadowed or uneven background), and the + // geometry refinement only moves the centre a few pixels. Coarse, then fine about the coarse best. + if (experiment_.GetUnitCell().has_value()) { + float gx = x, gy = y; + int64_t g_best = trial(x, y, distance, strongest); + for (const auto &[step, reach] : {std::pair{3.0f, 4}, std::pair{1.0f, 2}}) { + const float cx0 = gx, cy0 = gy; + for (int i = -reach; i <= reach; i++) + for (int j = -reach; j <= reach; j++) { + if (i == 0 && j == 0) + continue; + const int64_t score = trial(cx0 + step * i, cy0 + step * j, distance, strongest); + if (score > g_best) { + g_best = score; + gx = cx0 + step * i; + gy = cy0 + step * j; + } + } + } + const int64_t here = trial(x, y, distance, frames), there = trial(gx, gy, distance, frames); + logger.Info("Stills beam centre grid at {:.2f} mm: ({:.2f},{:.2f}) puts {} sampled spots on the lattice, " + "({:.2f},{:.2f}) {}", distance, gx, gy, there, x, y, here); + if (here < FILE_KEPT_ABOVE * there) { + x = gx; + y = gy; + } + } + logger.Info("Stills beam centre: the run uses ({:.2f},{:.2f}){}", x, y, x == file_x && y == file_y ? ", the file's" : ""); experiment_.BeamX_pxl(x).BeamY_pxl(y).DetectorDistance_mm(distance); diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 1dcf98c1c..ff8363290 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -3207,6 +3207,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) { config.two_pass_rotation = two_pass_rotation; config.detect_beam_stop = detect_beam_stop; config.beam_center_check = beam_center_check; + config.detector_distance_given = detector_distance_mm.has_value(); config.detect_saturation = !saturation_value && (input_is_cbf || input_is_marccd || input_is_smv || input_is_crystfel || input_is_cxi || input_is_xtc); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index b440124ba..9d1805532 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -12,6 +12,7 @@ ADD_EXECUTABLE(jfjoch_portable_test EXCLUDE_FROM_ALL JFJochReaderTest.cpp CrystFELGeometryTest.cpp SaturationCeilingTest.cpp + CellPopulationTest.cpp SweepLayoutTest.cpp CBORTest.cpp ImageSpotFinderCPUTest.cpp diff --git a/tests/CellPopulationTest.cpp b/tests/CellPopulationTest.cpp new file mode 100644 index 000000000..e7fc0f248 --- /dev/null +++ b/tests/CellPopulationTest.cpp @@ -0,0 +1,50 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include + +#include + +#include "../image_analysis/scale_merge/CellPopulation.h" + +TEST_CASE("CellPopulation_OnePopulationKeepsEverything", "[CellPopulation][portable]") { + std::mt19937 rng(7); + std::normal_distribution d(0.004, 0.01); + std::vector r(2000); + for (auto &x : r) x = d(rng); + const auto [low, high] = GivenCellPopulation(r); + for (double x : r) + CHECK((x >= low && x <= high)); +} + +TEST_CASE("CellPopulation_SecondFormLeftOut", "[CellPopulation][portable]") { + // Two crystal forms 9% apart in volume, the given cell one of them; whichever is given, its own + // population is kept and the other left out. + std::mt19937 rng(11); + std::normal_distribution small(0.0, 0.012), large(0.09, 0.008); + std::vector r; + for (int i = 0; i < 450; i++) r.push_back(small(rng)); + for (int i = 0; i < 150; i++) r.push_back(large(rng)); + + // Kept frames of each form, against either cell. + const auto kept = [](const std::vector &x, size_t from, size_t to) { + const auto [low, high] = GivenCellPopulation(x); + size_t n = 0; + for (size_t i = from; i < to; i++) + n += (x[i] >= low && x[i] <= high); + return n; + }; + CHECK(kept(r, 0, 450) > 440); + CHECK(kept(r, 450, 600) < 3); + + std::vector r2; + for (double x : r) r2.push_back((1.0 + x) / 1.09 - 1.0); + CHECK(kept(r2, 450, 600) > 145); + CHECK(kept(r2, 0, 450) < 5); +} + +TEST_CASE("CellPopulation_TooFewFrames", "[CellPopulation][portable]") { + const auto [low, high] = GivenCellPopulation({0.0, 0.1, 0.2}); + CHECK(std::isinf(low)); + CHECK(std::isinf(high)); +} -- 2.54.0 From 990bbaf9a73bd42eaaa1adbdd0abbf1378d74a92 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 18:39:41 +0200 Subject: [PATCH 39/53] GivenCellPopulation: the population nearest the given cell among those holding a real share of the frames The peak nearest the given cell could be a single stray frame when the given cell is a few percent off the crystal (a cryo cell given for room-temperature lysozyme stills): the run then merged one frame. Every population is now found from its own peak and valleys, and of those holding at least a tenth of the frames the one centred nearest the given cell is kept. Co-Authored-By: Claude Opus 5.5 --- image_analysis/scale_merge/CellPopulation.cpp | 45 +++++++++++++------ tests/CellPopulationTest.cpp | 15 +++++++ 2 files changed, 47 insertions(+), 13 deletions(-) diff --git a/image_analysis/scale_merge/CellPopulation.cpp b/image_analysis/scale_merge/CellPopulation.cpp index ff8f7b145..601708b4e 100644 --- a/image_analysis/scale_merge/CellPopulation.cpp +++ b/image_analysis/scale_merge/CellPopulation.cpp @@ -14,6 +14,8 @@ std::pair GivenCellPopulation(const std::vector &relativ // A valley separates two populations only where it dips below this fraction of the kept peak; a // shallower dip is noise in one population's tail. constexpr double VALLEY_FRACTION = 0.5; + // A population holding less than this share of the frames is a stray group, not a crystal form. + constexpr double MIN_SHARE = 0.1; const size_t n = relative_volume.size(); if (n < MIN_FRAMES) @@ -47,18 +49,10 @@ std::pair GivenCellPopulation(const std::vector &relativ smooth[i] = s / (b - a + 1); } - // The peak nearest the given cell: from its bin, uphill. - int peak = std::clamp(static_cast(std::floor(-lo / bin)), 0, nbins - 1); - while (true) { - if (peak + 1 < nbins && smooth[peak + 1] > smooth[peak]) peak++; - else if (peak > 0 && smooth[peak - 1] > smooth[peak]) peak--; - else break; - } - - // Out from the peak on each side: the valley is the lowest point, below VALLEY_FRACTION of the peak, + // Out from a peak on each side: the valley is the lowest point, below VALLEY_FRACTION of the peak, // before the histogram rises again to twice its height by more than counting noise - the next // population, however small. None: the population runs out to the end of the data. - const auto valley = [&](int step) { + const auto valley = [&](int peak, int step) { int lowest = peak; for (int i = peak + step; i >= 0 && i < nbins; i += step) { if (smooth[i] < smooth[lowest]) @@ -71,7 +65,32 @@ std::pair GivenCellPopulation(const std::vector &relativ } return -1; }; - const int left = valley(-1), right = valley(+1); - return {left < 0 ? -everything : lo + (left + 0.5) * bin, - right < 0 ? everything : lo + (right + 0.5) * bin}; + + // Every population: a local maximum of the histogram and the valleys about it. Of those that hold + // a real share of the frames, the one centred nearest the given cell is kept - not merely the peak + // nearest it, which can be a few stray frames when the given cell is a percent off the crystal. + double best_low = -everything, best_high = everything, best_distance = everything; + for (int i = 0; i < nbins; i++) { + if (smooth[i] <= 0 || (i > 0 && smooth[i - 1] > smooth[i]) || (i + 1 < nbins && smooth[i + 1] >= smooth[i])) + continue; + const int left = valley(i, -1), right = valley(i, +1); + const double low = left < 0 ? -everything : lo + (left + 0.5) * bin; + const double high = right < 0 ? everything : lo + (right + 0.5) * bin; + size_t count = 0; + double sum = 0; + for (double x : v) + if (x >= low && x <= high) { + count++; + sum += x; + } + if (count < MIN_SHARE * static_cast(n)) + continue; + const double distance = std::fabs(sum / static_cast(count)); + if (distance < best_distance) { + best_distance = distance; + best_low = low; + best_high = high; + } + } + return {best_low, best_high}; } diff --git a/tests/CellPopulationTest.cpp b/tests/CellPopulationTest.cpp index e7fc0f248..31335a5e1 100644 --- a/tests/CellPopulationTest.cpp +++ b/tests/CellPopulationTest.cpp @@ -43,6 +43,21 @@ TEST_CASE("CellPopulation_SecondFormLeftOut", "[CellPopulation][portable]") { CHECK(kept(r2, 0, 450) < 5); } +TEST_CASE("CellPopulation_GivenCellOffTheCrystal", "[CellPopulation][portable]") { + // A given cell 3.5% smaller than the crystal's, and a single stray frame right at it: the crystal's + // population is kept, not the stray frame. + std::mt19937 rng(3); + std::normal_distribution d(0.035, 0.003); + std::vector r(2000); + for (auto &x : r) x = d(rng); + r.push_back(0.0); + const auto [low, high] = GivenCellPopulation(r); + size_t kept = 0; + for (double x : r) + kept += (x >= low && x <= high); + CHECK(kept >= 2000); +} + TEST_CASE("CellPopulation_TooFewFrames", "[CellPopulation][portable]") { const auto [low, high] = GivenCellPopulation({0.0, 0.1, 0.2}); CHECK(std::isinf(low)); -- 2.54.0 From ace890e300606085ec01c8ed840674e3d7e4bd93 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 19:45:55 +0200 Subject: [PATCH 40/53] rugnux stills: the given-cell filter says how many frames it left out on either side of the population Co-Authored-By: Claude Opus 5.5 --- rugnux/RugnuxScaleMerge.cpp | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 49dd2d3b4..474251b37 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -523,20 +523,20 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { if (!o.reflections.empty()) rel.push_back(o.latt.CalcVolume() / given - 1.0); const auto [low, high] = GivenCellPopulation(rel); - size_t left_out = 0; + size_t below = 0, above = 0; for (auto &o : outcomes) { if (o.reflections.empty()) continue; const double r = o.latt.CalcVolume() / given - 1.0; if (r < low || r > high) { o.reflections.clear(); - left_out++; + (r < low ? below : above)++; } } - if (left_out > 0) + if (below + above > 0) logger.Info("Given cell: {} of {} indexed frames refined to a cell outside its population " - "({:+.1f}% to {:+.1f}% in volume) and are left out of the merge", - left_out, rel.size(), 100 * low, 100 * high); + "({:+.1f}% to {:+.1f}% in volume) and are left out of the merge - {} smaller, {} larger", + below + above, rel.size(), 100 * low, 100 * high, below, above); } size_t ice_flagged = 0, ice_total = 0; { -- 2.54.0 From f483033383762b2361063d120e0344230293e37f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 19:53:22 +0200 Subject: [PATCH 41/53] rugnux stills: the given-cell filter logs the refined volumes it decided on (debug) Co-Authored-By: Claude Opus 5.5 --- rugnux/RugnuxScaleMerge.cpp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 474251b37..b4d19bc2b 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -523,6 +523,10 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { if (!o.reflections.empty()) rel.push_back(o.latt.CalcVolume() / given - 1.0); const auto [low, high] = GivenCellPopulation(rel); + std::string all; + for (double r : rel) + all += fmt::format(" {:.4f}", r); + logger.Debug("Given cell: refined volumes relative to it:{}", all); size_t below = 0, above = 0; for (auto &o : outcomes) { if (o.reflections.empty()) -- 2.54.0 From 093069be290cada9c4aad16be49d4d4d183e80d5 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 20:56:07 +0200 Subject: [PATCH 42/53] rugnux stills: a given cell is matched in any setting, and per-image reindexing moves the lattice with the indices Three defects that stopped known-cell stills runs: - The consensus cell compared every frame's cell to -C axis by axis. A run whose frames index in another setting of the same lattice (axes in another order, another monoclinic choice) had no frame accepted, no consensus cell and no output ("No crystal lattice was determined"), although it merged fine. A frame whose lattice contains the given cell now counts on the given cell's axes (CellOnReference, the same FindSublatticeBases the merge-stage relabel uses). - The per-image ambiguity resolver relabelled a frame's reflections but left its lattice on the old axes. Everything after it predicts from index and lattice: the sublattice relabel put such a frame through a second transformation, and on a monoclinic cell with a near-symmetry (a+c nearly as long as a) the merge came out at indices finer than anything measured, shells over 100% complete. The lattice now follows the indices (ReindexLattice, B' = R^-1 B). - With -C/-S the stills path replaced the known-cell indexer's lattice, already on the given axes, by LatticeSearch's conventional choice, which on that near-symmetric cell is the other description on half the frames. The indexer's lattice is now kept when it is close to the given cell. Measured (cell + space group given, dimple against the deposited model and free set): - monoclinic MbCO-type still set: was over-complete/broken with --model; now 2.06 A, R/Rfree 0.233/0.314 at 2.11 A (P1 merge before: 0.275/0.306). - C-centred orthorhombic set: no MTZ before; now 3.27 A, R/Rfree 0.298/0.403 at 3.68 A (de novo 0.336/0.448). - monoclinic two-form set, one form: cell P 1 21 1 with beta (was alpha); Rfree 0.310 -> 0.260, 3.05 -> 2.72 A. - No change on seven other stills sets, nor on two rotation battery sets (5reo, lyso_x10sa_90deg_1). Co-Authored-By: Claude Opus 5.5 --- image_analysis/IndexAndRefine.cpp | 21 +++++- .../scale_merge/ReindexAmbiguity.cpp | 20 +++++- image_analysis/scale_merge/ReindexAmbiguity.h | 9 ++- .../scale_merge/SublatticeRelabel.cpp | 31 ++++++--- .../scale_merge/SublatticeRelabel.h | 6 ++ tests/ReindexAmbiguityTest.cpp | 67 +++++++++++++++++++ 6 files changed, 141 insertions(+), 13 deletions(-) diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index fbde4042b..f56aa0b26 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -16,6 +16,7 @@ #include "lattice_search/LatticeSearch.h" #include "scale_merge/ReindexAmbiguity.h" #include "scale_merge/ScaleOnTheFly.h" +#include "scale_merge/SublatticeRelabel.h" namespace { // Iterations the offline geometry refinement is allowed, standing in for the 40 ms the online path @@ -339,8 +340,17 @@ IndexAndRefine::IndexingOutcome IndexAndRefine::DetermineLatticeAndSymmetry(Data // niggli_class is left unassigned (0): it needs the primitive cell incl. // centering, which LatticeSearch cannot recover from a (possibly centred, e.g. // C2) user cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred. + // A lattice FFBIDX returned on the given cell is already on its axes; LatticeSearch's + // conventional choice is only one of the descriptions of it, and on a cell with a + // near-symmetry (a monoclinic one whose a+c is nearly as long as a) it picks the other + // one on half the frames, which the merge then has to tell apart again. + const auto settings = experiment.GetIndexingSettings(); auto sym_result = LatticeSearch(latt); - if (sym_result.system == sg->crystal_system()) + if (!de_novo && latt.GetUnitCell().is_close(*experiment.GetUnitCell(), + settings.GetUnitCellDistTolerance(), + settings.GetUnitCellAngleTolerance_deg())) + outcome.lattice_candidate = latt; + else if (sym_result.system == sg->crystal_system()) outcome.lattice_candidate = sym_result.conventional; else if (!de_novo) outcome.lattice_candidate = latt; @@ -938,7 +948,7 @@ void IndexAndRefine::ScaleImage(DataMessage &msg, IntegrationOutcome& outcome) { return; auto scaling_start_time = std::chrono::steady_clock::now(); - reindex_resolver->Resolve(outcome.reflections); + reindex_resolver->Resolve(outcome.reflections, outcome.latt); auto scaling_end_time = std::chrono::steady_clock::now(); msg.image_scale_time_s = std::chrono::duration(scaling_end_time - scaling_start_time).count(); } @@ -990,9 +1000,16 @@ std::optional IndexAndRefine::GetConsensusUnitCell() const { std::vector accepted; accepted.reserve(cells.size()); + // A frame whose lattice is the given cell in another setting (axes in another order, another + // monoclinic or centred choice) counts on the given cell's axes - the merge puts it there + // (RelabelOnSublattice) - rather than being left out. Compared axis by axis only, a run whose + // frames all index in another setting than the one typed after -C has no consensus at all. for (const auto &cell: cells) { if (cell.is_close(*experiment.GetUnitCell(), dist_tolerance, angle_tolerance)) accepted.emplace_back(cell); + else if (const auto on_ref = CellOnReference(cell, *experiment.GetUnitCell(), + experiment.GetIndexingSettings())) + accepted.emplace_back(*on_ref); } return MeanUnitCell(accepted); diff --git a/image_analysis/scale_merge/ReindexAmbiguity.cpp b/image_analysis/scale_merge/ReindexAmbiguity.cpp index ef46588e5..61783520c 100644 --- a/image_analysis/scale_merge/ReindexAmbiguity.cpp +++ b/image_analysis/scale_merge/ReindexAmbiguity.cpp @@ -50,6 +50,18 @@ std::vector ReindexAmbiguityOperators(const UnitCell &cell, const gem /*all_ops=*/false); } +CrystalLattice ReindexLattice(const CrystalLattice &latt, const gemmi::Op &op) { + const gemmi::Op inv = op.inverse(); + const Coord old_star[3] = {latt.Astar(), latt.Bstar(), latt.Cstar()}; + Coord rs[3]; + for (int i = 0; i < 3; ++i) + rs[i] = old_star[0] * (static_cast(inv.rot[i][0]) / gemmi::Op::DEN) + + old_star[1] * (static_cast(inv.rot[i][1]) / gemmi::Op::DEN) + + old_star[2] * (static_cast(inv.rot[i][2]) / gemmi::Op::DEN); + const float v_star = rs[0] * (rs[1] % rs[2]); + return {(rs[1] % rs[2]) / v_star, (rs[2] % rs[0]) / v_star, (rs[0] % rs[1]) / v_star}; +} + std::vector ReindexReflections(const std::vector &merged, const gemmi::Op &op) { std::vector out = merged; @@ -254,7 +266,7 @@ double ReindexAmbiguityResolver::ReferenceCC(std::span reflect // Serial stills index each crystal in one of the merohedrally-equivalent hands at random; pick, for this // image alone, the reindexing whose intensities correlate best with the external reference and apply it. -void ReindexAmbiguityResolver::Resolve(std::span reflections) const { +void ReindexAmbiguityResolver::Resolve(std::span reflections, CrystalLattice &latt) const { if (ops.empty()) return; @@ -268,4 +280,10 @@ void ReindexAmbiguityResolver::Resolve(std::span reflections) const const gemmi::Op::Miller h = best.op.apply_to_hkl({{r.h, r.k, r.l}}); r.h = h[0]; r.k = h[1]; r.l = h[2]; } + // Everything after this - the sublattice relabel, the partiality post-refinement - predicts a spot + // from its index and the lattice, so the lattice has to describe the new indices. Left on the old + // axes, a frame relabelled by an operator that is only a near-symmetry of the lattice was put + // through the sublattice relabel a second time, and its reflections came out at indices finer + // than anything measured. + latt = ReindexLattice(latt, best.op); } diff --git a/image_analysis/scale_merge/ReindexAmbiguity.h b/image_analysis/scale_merge/ReindexAmbiguity.h index 55282e232..773011342 100644 --- a/image_analysis/scale_merge/ReindexAmbiguity.h +++ b/image_analysis/scale_merge/ReindexAmbiguity.h @@ -16,6 +16,7 @@ #include "../../common/DiffractionExperiment.h" #include "../../common/Reflection.h" #include "../../common/UnitCell.h" +#include "../../common/CrystalLattice.h" // Resolving the indexing ("alternative indexing" / reindexing) ambiguity: when the lattice symmetry // is higher than the crystal Laue symmetry (merohedral / pseudo-merohedral), a crystal can be validly @@ -34,6 +35,10 @@ std::vector ReindexAmbiguityOperators(const UnitCell &cell, const gem // A copy of `merged` with each Miller index transformed by `op` (the intensities are unchanged). // The indices are left where `op` puts them, which is what the scoring callers want - they reduce to // the ASU themselves. +// The lattice an image has once its indices are moved by `op` (h' = h R): the reciprocal axes follow as +// R^-1 times the old ones, so h' (a*, b*, c*)' is the same reciprocal vector as h (a*, b*, c*). +CrystalLattice ReindexLattice(const CrystalLattice &latt, const gemmi::Op &op); + std::vector ReindexReflections(const std::vector &merged, const gemmi::Op &op); @@ -130,5 +135,7 @@ public: // Reindex this image's reflections in place into the hand best-correlated with the reference (done // once per image, for good). No-op when there is no ambiguity. - void Resolve(std::span reflections) const; + // Relabels the image's reflections and moves its lattice onto the same axes, so that a spot's new + // index still predicts it where it was measured. + void Resolve(std::span reflections, CrystalLattice &latt) const; }; diff --git a/image_analysis/scale_merge/SublatticeRelabel.cpp b/image_analysis/scale_merge/SublatticeRelabel.cpp index ed9b3e0e3..299df94e5 100644 --- a/image_analysis/scale_merge/SublatticeRelabel.cpp +++ b/image_analysis/scale_merge/SublatticeRelabel.cpp @@ -68,6 +68,17 @@ std::vector, 3>> FindSublatticeBases(const Crystal } namespace { + // The reference basis in this frame's orientation: the reciprocal axes are rows of M times the + // frame's, and the real axes their reciprocal. + CrystalLattice OnBasis(const CrystalLattice &frame, const std::array, 3> &M) { + const Coord fs[3] = {frame.Astar(), frame.Bstar(), frame.Cstar()}; + Coord rs[3]; + for (int i = 0; i < 3; ++i) + rs[i] = fs[0] * M[i][0] + fs[1] * M[i][1] + fs[2] * M[i][2]; + const float v_star = rs[0] * (rs[1] % rs[2]); + return {(rs[1] % rs[2]) / v_star, (rs[2] % rs[0]) / v_star, (rs[0] % rs[1]) / v_star}; + } + // The frame's reflections at nodes of the lattice M picks out, re-indexed on it: a frame index h' // is h = h' M^-1 = h' adj(M) / det(M) there, a node only where all three are integers. ReflectionVector NodesOf(const ReflectionVector &reflections, const std::array, 3> &M, @@ -155,16 +166,18 @@ SublatticeRelabelResult RelabelOnSublattice(std::vector &out result.reflections_kept += best_nodes.size(); o.reflections = std::move(best_nodes); - // The reference basis in this frame's orientation: the reciprocal axes are rows of M times the - // frame's, and the real axes their reciprocal. - const auto &M = *best; - const Coord fs[3] = {o.latt.Astar(), o.latt.Bstar(), o.latt.Cstar()}; - Coord rs[3]; - for (int i = 0; i < 3; ++i) - rs[i] = fs[0] * M[i][0] + fs[1] * M[i][1] + fs[2] * M[i][2]; - const float v_star = rs[0] * (rs[1] % rs[2]); - o.latt = CrystalLattice((rs[1] % rs[2]) / v_star, (rs[2] % rs[0]) / v_star, (rs[0] % rs[1]) / v_star); + o.latt = OnBasis(o.latt, *best); ++result.frames; } return result; } + +std::optional CellOnReference(const UnitCell &cell, const UnitCell &reference, + const IndexingSettings &settings) { + auto bases = FindSublatticeBases(CrystalLattice(cell), reference, settings); + if (bases.empty()) + return {}; + // The reference in another setting (index 1) before any axis multiple of it. + std::sort(bases.begin(), bases.end(), [](const auto &x, const auto &y) { return Det(x) < Det(y); }); + return OnBasis(CrystalLattice(cell), bases.front()).GetUnitCell(); +} diff --git a/image_analysis/scale_merge/SublatticeRelabel.h b/image_analysis/scale_merge/SublatticeRelabel.h index 72b3c7895..1dbe0c49b 100644 --- a/image_analysis/scale_merge/SublatticeRelabel.h +++ b/image_analysis/scale_merge/SublatticeRelabel.h @@ -25,6 +25,12 @@ std::vector, 3>> FindSublatticeBases(const Crystal const UnitCell &reference, const IndexingSettings &settings); +// A frame's cell described on the reference's axes, when its lattice contains the reference (the +// reference in another setting, or an axis multiple of it): the cell the frame has once it is put on the +// reference. Empty where the lattice does not contain it. +std::optional CellOnReference(const UnitCell &cell, const UnitCell &reference, + const IndexingSettings &settings); + struct SublatticeRelabelResult { size_t frames = 0; // frames put on the reference cell size_t reflections_kept = 0; // reflections of those frames at nodes of the reference lattice diff --git a/tests/ReindexAmbiguityTest.cpp b/tests/ReindexAmbiguityTest.cpp index e092c464d..1f821295f 100644 --- a/tests/ReindexAmbiguityTest.cpp +++ b/tests/ReindexAmbiguityTest.cpp @@ -11,6 +11,7 @@ #include "../image_analysis/scale_merge/CrystalSetting.h" #include "../image_analysis/scale_merge/HKLKey.h" #include "../image_analysis/scale_merge/ReindexAmbiguity.h" +#include "../image_analysis/scale_merge/SublatticeRelabel.h" namespace { const gemmi::SpaceGroup &SG(int number) { return *gemmi::find_spacegroup_by_number(number); } @@ -239,3 +240,69 @@ TEST_CASE("Reindex: a near-symmetry law moves the cell with the indices", "[rein CHECK(moved); } } + +namespace { + // A P2(1) cell with a near-symmetry: a+c is within 1% of a, so the lattice is metrically close to + // a C-centred orthorhombic one (obliquity ~0.7 deg) without being one. + UnitCell PseudoMonoclinic() { return UnitCell{64.0f, 29.0f, 36.0f, 90.0f, 107.0f, 90.0f}; } + + Coord Recip(const CrystalLattice &latt, int h, int k, int l) { + return latt.Astar() * static_cast(h) + latt.Bstar() * static_cast(k) + + latt.Cstar() * static_cast(l); + } +} + +TEST_CASE("Reindex: an ambiguity operator of a pseudo-symmetric monoclinic cell keeps every reflection's resolution", + "[reindex]") { + const UnitCell cell = PseudoMonoclinic(); + const auto laws = ReindexAmbiguityOperators(cell, SG(4)); + REQUIRE_FALSE(laws.empty()); + const CrystalLattice latt(cell); + for (const auto &op : laws) + for (int h = -6; h <= 6; ++h) + for (int l = -6; l <= 6; ++l) { + if (h == 0 && l == 0) continue; + // As the resolver applies it: an operator that is a near-symmetry of the metric moves an + // index to one of nearly the same length, never to a finer one. + const auto m = op.apply_to_hkl({{h, 3, l}}); + const float d0 = 1.0f / Recip(latt, h, 3, l).Length(); + const float d1 = 1.0f / Recip(latt, m[0], m[1], m[2]).Length(); + CHECK(std::fabs(d1 - d0) / d0 < 0.03f); + } +} + +TEST_CASE("Reindex: the reindexed lattice predicts every spot where it was", "[reindex]") { + const UnitCell cell = PseudoMonoclinic(); + const auto laws = ReindexAmbiguityOperators(cell, SG(4)); + REQUIRE_FALSE(laws.empty()); + const CrystalLattice latt(cell); + for (const auto &op : laws) { + const CrystalLattice moved = ReindexLattice(latt, op); + for (int h = -4; h <= 4; ++h) + for (int k = -4; k <= 4; ++k) + for (int l = -4; l <= 4; ++l) { + const auto m = op.apply_to_hkl({{h, k, l}}); + const Coord before = Recip(latt, h, k, l); + const Coord after = Recip(moved, m[0], m[1], m[2]); + CHECK((after - before).Length() < 1e-5f); + } + } +} + +TEST_CASE("Sublattice: a frame on the given cell in another setting is described on the given cell's axes", + "[reindex]") { + // The given C222(1)-style cell, and the same lattice with b and c swapped (and a flipped to keep the + // basis right-handed): not close axis by axis, but the same lattice. + const UnitCell given{40.0f, 180.0f, 143.0f, 90.0f, 90.0f, 90.0f}; + const UnitCell frame{40.2f, 143.4f, 180.5f, 90.0f, 90.0f, 90.0f}; + IndexingSettings settings; + REQUIRE_FALSE(frame.is_close(given, settings.GetUnitCellDistTolerance(), settings.GetUnitCellAngleTolerance_deg())); + const auto on_given = CellOnReference(frame, given, settings); + REQUIRE(on_given.has_value()); + CHECK(on_given->is_close(given, settings.GetUnitCellDistTolerance(), settings.GetUnitCellAngleTolerance_deg())); + CHECK(on_given->b == Catch::Approx(180.5f).epsilon(0.001)); + CHECK(on_given->c == Catch::Approx(143.4f).epsilon(0.001)); + + // An unrelated cell is not put on it. + CHECK_FALSE(CellOnReference(UnitCell{55.0f, 66.0f, 77.0f, 90.0f, 90.0f, 90.0f}, given, settings).has_value()); +} -- 2.54.0 From 02dbdb0b9dd58ff9815611f6383ded2852e391e6 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 23:00:10 +0200 Subject: [PATCH 43/53] v1.0.0-rc.178: version bump VERSION and the strings update_version.sh rewrites (API spec, python and TypeScript clients, C++ model, Redoc page, docs/conf.py, FPGA HDL and PCIe driver). The regenerated clients and Redoc page also pick up rc178's API changes made since the last regeneration (rugnux live settings and model without a bearer token, the live status fields). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu --- VERSION | 2 +- broker/gen/model/Azim_int_settings.cpp | 2 +- broker/gen/model/Azim_int_settings.h | 2 +- .../gen/model/Bragg_integration_settings.cpp | 2 +- broker/gen/model/Bragg_integration_settings.h | 2 +- broker/gen/model/Broker_status.cpp | 2 +- broker/gen/model/Broker_status.h | 2 +- .../model/Calibration_statistics_inner.cpp | 2 +- .../gen/model/Calibration_statistics_inner.h | 2 +- broker/gen/model/Dark_mask_settings.cpp | 2 +- broker/gen/model/Dark_mask_settings.h | 2 +- broker/gen/model/Dataset_settings.cpp | 2 +- broker/gen/model/Dataset_settings.h | 2 +- broker/gen/model/Dataset_settings_smargon.cpp | 2 +- broker/gen/model/Dataset_settings_smargon.h | 2 +- ...et_settings_xray_fluorescence_spectrum.cpp | 2 +- ...aset_settings_xray_fluorescence_spectrum.h | 2 +- broker/gen/model/Detector.cpp | 2 +- broker/gen/model/Detector.h | 2 +- broker/gen/model/Detector_list.cpp | 2 +- broker/gen/model/Detector_list.h | 2 +- broker/gen/model/Detector_list_element.cpp | 2 +- broker/gen/model/Detector_list_element.h | 2 +- broker/gen/model/Detector_module.cpp | 2 +- broker/gen/model/Detector_module.h | 2 +- .../gen/model/Detector_module_direction.cpp | 2 +- broker/gen/model/Detector_module_direction.h | 2 +- broker/gen/model/Detector_power_state.cpp | 2 +- broker/gen/model/Detector_power_state.h | 2 +- broker/gen/model/Detector_selection.cpp | 2 +- broker/gen/model/Detector_selection.h | 2 +- broker/gen/model/Detector_settings.cpp | 2 +- broker/gen/model/Detector_settings.h | 2 +- broker/gen/model/Detector_state.cpp | 2 +- broker/gen/model/Detector_state.h | 2 +- broker/gen/model/Detector_status.cpp | 2 +- broker/gen/model/Detector_status.h | 2 +- broker/gen/model/Detector_timing.cpp | 2 +- broker/gen/model/Detector_timing.h | 2 +- broker/gen/model/Detector_type.cpp | 2 +- broker/gen/model/Detector_type.h | 2 +- broker/gen/model/Error_message.cpp | 2 +- broker/gen/model/Error_message.h | 2 +- broker/gen/model/File_writer_format.cpp | 2 +- broker/gen/model/File_writer_format.h | 2 +- broker/gen/model/File_writer_settings.cpp | 2 +- broker/gen/model/File_writer_settings.h | 2 +- broker/gen/model/Fpga_status_inner.cpp | 2 +- broker/gen/model/Fpga_status_inner.h | 2 +- .../gen/model/Geom_refinement_algorithm.cpp | 2 +- broker/gen/model/Geom_refinement_algorithm.h | 2 +- broker/gen/model/Grid_scan.cpp | 2 +- broker/gen/model/Grid_scan.h | 2 +- broker/gen/model/Helpers.cpp | 2 +- broker/gen/model/Helpers.h | 2 +- broker/gen/model/Image_buffer_status.cpp | 2 +- broker/gen/model/Image_buffer_status.h | 2 +- broker/gen/model/Image_format_settings.cpp | 2 +- broker/gen/model/Image_format_settings.h | 2 +- broker/gen/model/Image_pusher_status.cpp | 2 +- broker/gen/model/Image_pusher_status.h | 2 +- broker/gen/model/Image_pusher_type.cpp | 2 +- broker/gen/model/Image_pusher_type.h | 2 +- broker/gen/model/Indexing_algorithm.cpp | 2 +- broker/gen/model/Indexing_algorithm.h | 2 +- broker/gen/model/Indexing_settings.cpp | 2 +- broker/gen/model/Indexing_settings.h | 2 +- broker/gen/model/Instrument_metadata.cpp | 2 +- broker/gen/model/Instrument_metadata.h | 2 +- broker/gen/model/Integration_model.cpp | 2 +- broker/gen/model/Integration_model.h | 2 +- broker/gen/model/Jfjoch_settings.cpp | 2 +- broker/gen/model/Jfjoch_settings.h | 2 +- broker/gen/model/Jfjoch_statistics.cpp | 2 +- broker/gen/model/Jfjoch_statistics.h | 2 +- broker/gen/model/Measurement_statistics.cpp | 2 +- broker/gen/model/Measurement_statistics.h | 2 +- broker/gen/model/Pcie_devices_inner.cpp | 2 +- broker/gen/model/Pcie_devices_inner.h | 2 +- broker/gen/model/Pixel_mask_statistics.cpp | 2 +- broker/gen/model/Pixel_mask_statistics.h | 2 +- broker/gen/model/Plot.cpp | 2 +- broker/gen/model/Plot.h | 2 +- broker/gen/model/Plot_unit_x.cpp | 2 +- broker/gen/model/Plot_unit_x.h | 2 +- broker/gen/model/Plots.cpp | 2 +- broker/gen/model/Plots.h | 2 +- .../model/Powder_calibration_fit_sigma.cpp | 2 +- .../gen/model/Powder_calibration_fit_sigma.h | 2 +- .../gen/model/Powder_calibration_output.cpp | 2 +- broker/gen/model/Powder_calibration_output.h | 2 +- .../gen/model/Powder_calibration_quality.cpp | 2 +- broker/gen/model/Powder_calibration_quality.h | 2 +- .../model/Powder_calibration_spot_check.cpp | 2 +- .../gen/model/Powder_calibration_spot_check.h | 2 +- broker/gen/model/Roi_azim_list.cpp | 2 +- broker/gen/model/Roi_azim_list.h | 2 +- broker/gen/model/Roi_azimuthal.cpp | 2 +- broker/gen/model/Roi_azimuthal.h | 2 +- broker/gen/model/Roi_box.cpp | 2 +- broker/gen/model/Roi_box.h | 2 +- broker/gen/model/Roi_box_list.cpp | 2 +- broker/gen/model/Roi_box_list.h | 2 +- broker/gen/model/Roi_circle.cpp | 2 +- broker/gen/model/Roi_circle.h | 2 +- broker/gen/model/Roi_circle_list.cpp | 2 +- broker/gen/model/Roi_circle_list.h | 2 +- broker/gen/model/Roi_definitions.cpp | 2 +- broker/gen/model/Roi_definitions.h | 2 +- broker/gen/model/Rotation_axis.cpp | 2 +- broker/gen/model/Rotation_axis.h | 2 +- broker/gen/model/Rugnux_live_status.cpp | 2 +- broker/gen/model/Rugnux_live_status.h | 2 +- broker/gen/model/Rugnux_live_step.cpp | 2 +- broker/gen/model/Rugnux_live_step.h | 2 +- broker/gen/model/Rugnux_settings.cpp | 2 +- broker/gen/model/Rugnux_settings.h | 2 +- broker/gen/model/Rugnux_status.cpp | 2 +- broker/gen/model/Rugnux_status.h | 2 +- broker/gen/model/Rugnux_summary_row.cpp | 2 +- broker/gen/model/Rugnux_summary_row.h | 2 +- broker/gen/model/Scan_result.cpp | 2 +- broker/gen/model/Scan_result.h | 2 +- broker/gen/model/Scan_result_images_inner.cpp | 2 +- broker/gen/model/Scan_result_images_inner.h | 2 +- broker/gen/model/Spot_finding_settings.cpp | 2 +- broker/gen/model/Spot_finding_settings.h | 2 +- .../gen/model/Standard_detector_geometry.cpp | 2 +- broker/gen/model/Standard_detector_geometry.h | 2 +- broker/gen/model/Tcp_settings.cpp | 2 +- broker/gen/model/Tcp_settings.h | 2 +- broker/gen/model/Unit_cell.cpp | 2 +- broker/gen/model/Unit_cell.h | 2 +- broker/gen/model/Zeromq_metadata_settings.cpp | 2 +- broker/gen/model/Zeromq_metadata_settings.h | 2 +- broker/gen/model/Zeromq_preview_settings.cpp | 2 +- broker/gen/model/Zeromq_preview_settings.h | 2 +- broker/gen/model/Zeromq_settings.cpp | 2 +- broker/gen/model/Zeromq_settings.h | 2 +- broker/jfjoch_api.yaml | 2 +- broker/redoc-static.html | 22 ++++--- docs/conf.py | 2 +- docs/python_client/README.md | 4 +- docs/python_client/docs/DefaultApi.md | 60 +++---------------- docs/python_client/docs/RugnuxLiveStatus.md | 10 ++-- docs/python_client/docs/RugnuxLiveStep.md | 2 +- fpga/hdl/action_config.v | 2 +- fpga/pcie_driver/dkms.conf | 2 +- fpga/pcie_driver/install_dkms.sh | 2 +- fpga/pcie_driver/jfjoch_drv.c | 2 +- fpga/pcie_driver/postinstall.sh | 2 +- fpga/pcie_driver/preuninstall.sh | 2 +- frontend/package-lock.json | 4 +- frontend/package.json | 2 +- .../src/client/@tanstack/react-query.gen.ts | 5 +- frontend/src/client/index.ts | 2 +- frontend/src/client/sdk.gen.ts | 13 ++-- frontend/src/client/types.gen.ts | 21 ------- frontend/src/version.ts | 2 +- 159 files changed, 187 insertions(+), 254 deletions(-) diff --git a/VERSION b/VERSION index 65c128cce..16e4f91ce 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.0.0-rc.177 +1.0.0-rc.178 diff --git a/broker/gen/model/Azim_int_settings.cpp b/broker/gen/model/Azim_int_settings.cpp index 5d1dd800a..4f0ef0b87 100644 --- a/broker/gen/model/Azim_int_settings.cpp +++ b/broker/gen/model/Azim_int_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Azim_int_settings.h b/broker/gen/model/Azim_int_settings.h index f34830f23..bf12ee889 100644 --- a/broker/gen/model/Azim_int_settings.h +++ b/broker/gen/model/Azim_int_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Bragg_integration_settings.cpp b/broker/gen/model/Bragg_integration_settings.cpp index 4a7118b19..7a3885f7d 100644 --- a/broker/gen/model/Bragg_integration_settings.cpp +++ b/broker/gen/model/Bragg_integration_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Bragg_integration_settings.h b/broker/gen/model/Bragg_integration_settings.h index 19e1b5356..0106f34f6 100644 --- a/broker/gen/model/Bragg_integration_settings.h +++ b/broker/gen/model/Bragg_integration_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Broker_status.cpp b/broker/gen/model/Broker_status.cpp index f96e03889..857b5d141 100644 --- a/broker/gen/model/Broker_status.cpp +++ b/broker/gen/model/Broker_status.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Broker_status.h b/broker/gen/model/Broker_status.h index 65195e619..ada8830ae 100644 --- a/broker/gen/model/Broker_status.h +++ b/broker/gen/model/Broker_status.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Calibration_statistics_inner.cpp b/broker/gen/model/Calibration_statistics_inner.cpp index 867780888..a6f68a5ce 100644 --- a/broker/gen/model/Calibration_statistics_inner.cpp +++ b/broker/gen/model/Calibration_statistics_inner.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Calibration_statistics_inner.h b/broker/gen/model/Calibration_statistics_inner.h index aed822f5b..8e43010b3 100644 --- a/broker/gen/model/Calibration_statistics_inner.h +++ b/broker/gen/model/Calibration_statistics_inner.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dark_mask_settings.cpp b/broker/gen/model/Dark_mask_settings.cpp index 54e95ab67..09ee43ab6 100644 --- a/broker/gen/model/Dark_mask_settings.cpp +++ b/broker/gen/model/Dark_mask_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dark_mask_settings.h b/broker/gen/model/Dark_mask_settings.h index 324d7b7c0..a704c5795 100644 --- a/broker/gen/model/Dark_mask_settings.h +++ b/broker/gen/model/Dark_mask_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dataset_settings.cpp b/broker/gen/model/Dataset_settings.cpp index 8e75e8403..64e69b905 100644 --- a/broker/gen/model/Dataset_settings.cpp +++ b/broker/gen/model/Dataset_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dataset_settings.h b/broker/gen/model/Dataset_settings.h index dee0f494f..47bfed1c7 100644 --- a/broker/gen/model/Dataset_settings.h +++ b/broker/gen/model/Dataset_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dataset_settings_smargon.cpp b/broker/gen/model/Dataset_settings_smargon.cpp index df2c966fd..3ba726788 100644 --- a/broker/gen/model/Dataset_settings_smargon.cpp +++ b/broker/gen/model/Dataset_settings_smargon.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dataset_settings_smargon.h b/broker/gen/model/Dataset_settings_smargon.h index 121c2f83f..e811b150b 100644 --- a/broker/gen/model/Dataset_settings_smargon.h +++ b/broker/gen/model/Dataset_settings_smargon.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.cpp b/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.cpp index 2641dc5a2..4837e5649 100644 --- a/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.cpp +++ b/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.h b/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.h index 2eb8d0418..e483dad45 100644 --- a/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.h +++ b/broker/gen/model/Dataset_settings_xray_fluorescence_spectrum.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector.cpp b/broker/gen/model/Detector.cpp index de74c1348..683903152 100644 --- a/broker/gen/model/Detector.cpp +++ b/broker/gen/model/Detector.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector.h b/broker/gen/model/Detector.h index df9a95878..9772eadc2 100644 --- a/broker/gen/model/Detector.h +++ b/broker/gen/model/Detector.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_list.cpp b/broker/gen/model/Detector_list.cpp index 3158c9b20..48d0a2753 100644 --- a/broker/gen/model/Detector_list.cpp +++ b/broker/gen/model/Detector_list.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_list.h b/broker/gen/model/Detector_list.h index 4edcd4bb3..8aab7aeca 100644 --- a/broker/gen/model/Detector_list.h +++ b/broker/gen/model/Detector_list.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_list_element.cpp b/broker/gen/model/Detector_list_element.cpp index bc8cac242..ce4833bbf 100644 --- a/broker/gen/model/Detector_list_element.cpp +++ b/broker/gen/model/Detector_list_element.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_list_element.h b/broker/gen/model/Detector_list_element.h index 7e2e972df..3bf3a5d07 100644 --- a/broker/gen/model/Detector_list_element.h +++ b/broker/gen/model/Detector_list_element.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_module.cpp b/broker/gen/model/Detector_module.cpp index 669c8e07f..c3ec5ae7f 100644 --- a/broker/gen/model/Detector_module.cpp +++ b/broker/gen/model/Detector_module.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_module.h b/broker/gen/model/Detector_module.h index cdc04e06d..25c8a026b 100644 --- a/broker/gen/model/Detector_module.h +++ b/broker/gen/model/Detector_module.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_module_direction.cpp b/broker/gen/model/Detector_module_direction.cpp index b804c7c75..479884a5a 100644 --- a/broker/gen/model/Detector_module_direction.cpp +++ b/broker/gen/model/Detector_module_direction.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_module_direction.h b/broker/gen/model/Detector_module_direction.h index 3819fcf54..e01ab0171 100644 --- a/broker/gen/model/Detector_module_direction.h +++ b/broker/gen/model/Detector_module_direction.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_power_state.cpp b/broker/gen/model/Detector_power_state.cpp index 281bbfddd..48d4c962d 100644 --- a/broker/gen/model/Detector_power_state.cpp +++ b/broker/gen/model/Detector_power_state.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_power_state.h b/broker/gen/model/Detector_power_state.h index c95fc293a..e93f0d83d 100644 --- a/broker/gen/model/Detector_power_state.h +++ b/broker/gen/model/Detector_power_state.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_selection.cpp b/broker/gen/model/Detector_selection.cpp index 90be0a2f0..1c8f55db7 100644 --- a/broker/gen/model/Detector_selection.cpp +++ b/broker/gen/model/Detector_selection.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_selection.h b/broker/gen/model/Detector_selection.h index ac3e18efa..c9eee8f5d 100644 --- a/broker/gen/model/Detector_selection.h +++ b/broker/gen/model/Detector_selection.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_settings.cpp b/broker/gen/model/Detector_settings.cpp index 8673861cc..f828216d0 100644 --- a/broker/gen/model/Detector_settings.cpp +++ b/broker/gen/model/Detector_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_settings.h b/broker/gen/model/Detector_settings.h index 85fce0418..5daad6677 100644 --- a/broker/gen/model/Detector_settings.h +++ b/broker/gen/model/Detector_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_state.cpp b/broker/gen/model/Detector_state.cpp index 5ec8c0eaa..66045a235 100644 --- a/broker/gen/model/Detector_state.cpp +++ b/broker/gen/model/Detector_state.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_state.h b/broker/gen/model/Detector_state.h index ea92330d7..718a79109 100644 --- a/broker/gen/model/Detector_state.h +++ b/broker/gen/model/Detector_state.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_status.cpp b/broker/gen/model/Detector_status.cpp index 5ad8a3c88..19637232c 100644 --- a/broker/gen/model/Detector_status.cpp +++ b/broker/gen/model/Detector_status.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_status.h b/broker/gen/model/Detector_status.h index cca994b60..70f404203 100644 --- a/broker/gen/model/Detector_status.h +++ b/broker/gen/model/Detector_status.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_timing.cpp b/broker/gen/model/Detector_timing.cpp index d2ddd264b..bb7910041 100644 --- a/broker/gen/model/Detector_timing.cpp +++ b/broker/gen/model/Detector_timing.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_timing.h b/broker/gen/model/Detector_timing.h index 98714bcf3..5034d7291 100644 --- a/broker/gen/model/Detector_timing.h +++ b/broker/gen/model/Detector_timing.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_type.cpp b/broker/gen/model/Detector_type.cpp index 715897e6d..5f83aed2a 100644 --- a/broker/gen/model/Detector_type.cpp +++ b/broker/gen/model/Detector_type.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Detector_type.h b/broker/gen/model/Detector_type.h index bb9de3a56..f4ccc5cf1 100644 --- a/broker/gen/model/Detector_type.h +++ b/broker/gen/model/Detector_type.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Error_message.cpp b/broker/gen/model/Error_message.cpp index 56a8e6572..bce25c6ac 100644 --- a/broker/gen/model/Error_message.cpp +++ b/broker/gen/model/Error_message.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Error_message.h b/broker/gen/model/Error_message.h index 89b3dc31f..f5c874def 100644 --- a/broker/gen/model/Error_message.h +++ b/broker/gen/model/Error_message.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/File_writer_format.cpp b/broker/gen/model/File_writer_format.cpp index 06f2eabf5..5dbf40638 100644 --- a/broker/gen/model/File_writer_format.cpp +++ b/broker/gen/model/File_writer_format.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/File_writer_format.h b/broker/gen/model/File_writer_format.h index bd0ea9c62..cef161fe5 100644 --- a/broker/gen/model/File_writer_format.h +++ b/broker/gen/model/File_writer_format.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/File_writer_settings.cpp b/broker/gen/model/File_writer_settings.cpp index bbfac01c9..ef3bc19bb 100644 --- a/broker/gen/model/File_writer_settings.cpp +++ b/broker/gen/model/File_writer_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/File_writer_settings.h b/broker/gen/model/File_writer_settings.h index ac67820e7..595382a15 100644 --- a/broker/gen/model/File_writer_settings.h +++ b/broker/gen/model/File_writer_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Fpga_status_inner.cpp b/broker/gen/model/Fpga_status_inner.cpp index ea5d0f9b6..a15217237 100644 --- a/broker/gen/model/Fpga_status_inner.cpp +++ b/broker/gen/model/Fpga_status_inner.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Fpga_status_inner.h b/broker/gen/model/Fpga_status_inner.h index aeae1d903..8e17bf6f5 100644 --- a/broker/gen/model/Fpga_status_inner.h +++ b/broker/gen/model/Fpga_status_inner.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Geom_refinement_algorithm.cpp b/broker/gen/model/Geom_refinement_algorithm.cpp index 311fe29ae..dbb6f897e 100644 --- a/broker/gen/model/Geom_refinement_algorithm.cpp +++ b/broker/gen/model/Geom_refinement_algorithm.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Geom_refinement_algorithm.h b/broker/gen/model/Geom_refinement_algorithm.h index d2b663a85..fb655021b 100644 --- a/broker/gen/model/Geom_refinement_algorithm.h +++ b/broker/gen/model/Geom_refinement_algorithm.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Grid_scan.cpp b/broker/gen/model/Grid_scan.cpp index ca7f8b406..c4d945918 100644 --- a/broker/gen/model/Grid_scan.cpp +++ b/broker/gen/model/Grid_scan.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Grid_scan.h b/broker/gen/model/Grid_scan.h index 4d9dd60e4..a0eea9cd2 100644 --- a/broker/gen/model/Grid_scan.h +++ b/broker/gen/model/Grid_scan.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Helpers.cpp b/broker/gen/model/Helpers.cpp index f5ea0dc97..8403620e0 100644 --- a/broker/gen/model/Helpers.cpp +++ b/broker/gen/model/Helpers.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Helpers.h b/broker/gen/model/Helpers.h index 70fb4c41a..4e22d3e8c 100644 --- a/broker/gen/model/Helpers.h +++ b/broker/gen/model/Helpers.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_buffer_status.cpp b/broker/gen/model/Image_buffer_status.cpp index a38327a1b..196a19935 100644 --- a/broker/gen/model/Image_buffer_status.cpp +++ b/broker/gen/model/Image_buffer_status.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_buffer_status.h b/broker/gen/model/Image_buffer_status.h index 07341c888..80ef00933 100644 --- a/broker/gen/model/Image_buffer_status.h +++ b/broker/gen/model/Image_buffer_status.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_format_settings.cpp b/broker/gen/model/Image_format_settings.cpp index 9d5769495..655da7901 100644 --- a/broker/gen/model/Image_format_settings.cpp +++ b/broker/gen/model/Image_format_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_format_settings.h b/broker/gen/model/Image_format_settings.h index 0f5917105..27bb17ab6 100644 --- a/broker/gen/model/Image_format_settings.h +++ b/broker/gen/model/Image_format_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_pusher_status.cpp b/broker/gen/model/Image_pusher_status.cpp index 174516578..fc83ba15d 100644 --- a/broker/gen/model/Image_pusher_status.cpp +++ b/broker/gen/model/Image_pusher_status.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_pusher_status.h b/broker/gen/model/Image_pusher_status.h index d096cdfef..1e8cbba20 100644 --- a/broker/gen/model/Image_pusher_status.h +++ b/broker/gen/model/Image_pusher_status.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_pusher_type.cpp b/broker/gen/model/Image_pusher_type.cpp index f58fc6306..24636ebb9 100644 --- a/broker/gen/model/Image_pusher_type.cpp +++ b/broker/gen/model/Image_pusher_type.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Image_pusher_type.h b/broker/gen/model/Image_pusher_type.h index 56a195f11..ef809f98a 100644 --- a/broker/gen/model/Image_pusher_type.h +++ b/broker/gen/model/Image_pusher_type.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Indexing_algorithm.cpp b/broker/gen/model/Indexing_algorithm.cpp index 69c3a94b6..fd631b62d 100644 --- a/broker/gen/model/Indexing_algorithm.cpp +++ b/broker/gen/model/Indexing_algorithm.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Indexing_algorithm.h b/broker/gen/model/Indexing_algorithm.h index 85f60c5d6..06e802350 100644 --- a/broker/gen/model/Indexing_algorithm.h +++ b/broker/gen/model/Indexing_algorithm.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Indexing_settings.cpp b/broker/gen/model/Indexing_settings.cpp index 364ac06de..85a9fab23 100644 --- a/broker/gen/model/Indexing_settings.cpp +++ b/broker/gen/model/Indexing_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Indexing_settings.h b/broker/gen/model/Indexing_settings.h index 6c1688b81..c233e35a3 100644 --- a/broker/gen/model/Indexing_settings.h +++ b/broker/gen/model/Indexing_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Instrument_metadata.cpp b/broker/gen/model/Instrument_metadata.cpp index 4687ed3db..c06601d4d 100644 --- a/broker/gen/model/Instrument_metadata.cpp +++ b/broker/gen/model/Instrument_metadata.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Instrument_metadata.h b/broker/gen/model/Instrument_metadata.h index f1a22ddba..30a585ae5 100644 --- a/broker/gen/model/Instrument_metadata.h +++ b/broker/gen/model/Instrument_metadata.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Integration_model.cpp b/broker/gen/model/Integration_model.cpp index 9960735f8..865ebf38b 100644 --- a/broker/gen/model/Integration_model.cpp +++ b/broker/gen/model/Integration_model.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Integration_model.h b/broker/gen/model/Integration_model.h index 447fbb0c8..1c58d0844 100644 --- a/broker/gen/model/Integration_model.h +++ b/broker/gen/model/Integration_model.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Jfjoch_settings.cpp b/broker/gen/model/Jfjoch_settings.cpp index 64a5f9227..f4ad889e4 100644 --- a/broker/gen/model/Jfjoch_settings.cpp +++ b/broker/gen/model/Jfjoch_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Jfjoch_settings.h b/broker/gen/model/Jfjoch_settings.h index 08a1239c5..7edb12b3e 100644 --- a/broker/gen/model/Jfjoch_settings.h +++ b/broker/gen/model/Jfjoch_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Jfjoch_statistics.cpp b/broker/gen/model/Jfjoch_statistics.cpp index c5df961c0..b0db94a1d 100644 --- a/broker/gen/model/Jfjoch_statistics.cpp +++ b/broker/gen/model/Jfjoch_statistics.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Jfjoch_statistics.h b/broker/gen/model/Jfjoch_statistics.h index 9d5956e5b..3d24112d8 100644 --- a/broker/gen/model/Jfjoch_statistics.h +++ b/broker/gen/model/Jfjoch_statistics.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Measurement_statistics.cpp b/broker/gen/model/Measurement_statistics.cpp index 7342c4949..a51a7696d 100644 --- a/broker/gen/model/Measurement_statistics.cpp +++ b/broker/gen/model/Measurement_statistics.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Measurement_statistics.h b/broker/gen/model/Measurement_statistics.h index a4979ea3e..66a9f9d92 100644 --- a/broker/gen/model/Measurement_statistics.h +++ b/broker/gen/model/Measurement_statistics.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Pcie_devices_inner.cpp b/broker/gen/model/Pcie_devices_inner.cpp index 3021b5f6b..46f502360 100644 --- a/broker/gen/model/Pcie_devices_inner.cpp +++ b/broker/gen/model/Pcie_devices_inner.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Pcie_devices_inner.h b/broker/gen/model/Pcie_devices_inner.h index 8e6a4dad1..dd41c7801 100644 --- a/broker/gen/model/Pcie_devices_inner.h +++ b/broker/gen/model/Pcie_devices_inner.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Pixel_mask_statistics.cpp b/broker/gen/model/Pixel_mask_statistics.cpp index 8a1719fb4..5c66f5dda 100644 --- a/broker/gen/model/Pixel_mask_statistics.cpp +++ b/broker/gen/model/Pixel_mask_statistics.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Pixel_mask_statistics.h b/broker/gen/model/Pixel_mask_statistics.h index 7ecbfabae..e8a462f93 100644 --- a/broker/gen/model/Pixel_mask_statistics.h +++ b/broker/gen/model/Pixel_mask_statistics.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Plot.cpp b/broker/gen/model/Plot.cpp index f488b3c14..66b181d4b 100644 --- a/broker/gen/model/Plot.cpp +++ b/broker/gen/model/Plot.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Plot.h b/broker/gen/model/Plot.h index bf8281ce4..75622ac9a 100644 --- a/broker/gen/model/Plot.h +++ b/broker/gen/model/Plot.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Plot_unit_x.cpp b/broker/gen/model/Plot_unit_x.cpp index 141572802..c63b8ce10 100644 --- a/broker/gen/model/Plot_unit_x.cpp +++ b/broker/gen/model/Plot_unit_x.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Plot_unit_x.h b/broker/gen/model/Plot_unit_x.h index f10b95edb..189cc31fa 100644 --- a/broker/gen/model/Plot_unit_x.h +++ b/broker/gen/model/Plot_unit_x.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Plots.cpp b/broker/gen/model/Plots.cpp index 4601b53bc..1ba538b81 100644 --- a/broker/gen/model/Plots.cpp +++ b/broker/gen/model/Plots.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Plots.h b/broker/gen/model/Plots.h index 4a756b43b..865ab094f 100644 --- a/broker/gen/model/Plots.h +++ b/broker/gen/model/Plots.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_fit_sigma.cpp b/broker/gen/model/Powder_calibration_fit_sigma.cpp index c7f42401d..54e1d83a6 100644 --- a/broker/gen/model/Powder_calibration_fit_sigma.cpp +++ b/broker/gen/model/Powder_calibration_fit_sigma.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_fit_sigma.h b/broker/gen/model/Powder_calibration_fit_sigma.h index 6066d51a5..c88c27f55 100644 --- a/broker/gen/model/Powder_calibration_fit_sigma.h +++ b/broker/gen/model/Powder_calibration_fit_sigma.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_output.cpp b/broker/gen/model/Powder_calibration_output.cpp index 9d97e3fc5..773d67e29 100644 --- a/broker/gen/model/Powder_calibration_output.cpp +++ b/broker/gen/model/Powder_calibration_output.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_output.h b/broker/gen/model/Powder_calibration_output.h index aa58980f0..eac47b9cb 100644 --- a/broker/gen/model/Powder_calibration_output.h +++ b/broker/gen/model/Powder_calibration_output.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_quality.cpp b/broker/gen/model/Powder_calibration_quality.cpp index 11b1522d4..b776766b0 100644 --- a/broker/gen/model/Powder_calibration_quality.cpp +++ b/broker/gen/model/Powder_calibration_quality.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_quality.h b/broker/gen/model/Powder_calibration_quality.h index c1708df97..5d99817e0 100644 --- a/broker/gen/model/Powder_calibration_quality.h +++ b/broker/gen/model/Powder_calibration_quality.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_spot_check.cpp b/broker/gen/model/Powder_calibration_spot_check.cpp index 0d8da0b33..6ff29f5a2 100644 --- a/broker/gen/model/Powder_calibration_spot_check.cpp +++ b/broker/gen/model/Powder_calibration_spot_check.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Powder_calibration_spot_check.h b/broker/gen/model/Powder_calibration_spot_check.h index ac7515d79..86cb68a6a 100644 --- a/broker/gen/model/Powder_calibration_spot_check.h +++ b/broker/gen/model/Powder_calibration_spot_check.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_azim_list.cpp b/broker/gen/model/Roi_azim_list.cpp index 4d249090c..3242303e9 100644 --- a/broker/gen/model/Roi_azim_list.cpp +++ b/broker/gen/model/Roi_azim_list.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_azim_list.h b/broker/gen/model/Roi_azim_list.h index e7c3fc0b7..cb3ae211f 100644 --- a/broker/gen/model/Roi_azim_list.h +++ b/broker/gen/model/Roi_azim_list.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_azimuthal.cpp b/broker/gen/model/Roi_azimuthal.cpp index b67c2d9f6..930cbcf7b 100644 --- a/broker/gen/model/Roi_azimuthal.cpp +++ b/broker/gen/model/Roi_azimuthal.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_azimuthal.h b/broker/gen/model/Roi_azimuthal.h index 73f18eb68..48d734300 100644 --- a/broker/gen/model/Roi_azimuthal.h +++ b/broker/gen/model/Roi_azimuthal.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_box.cpp b/broker/gen/model/Roi_box.cpp index 286f3c292..a2597164c 100644 --- a/broker/gen/model/Roi_box.cpp +++ b/broker/gen/model/Roi_box.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_box.h b/broker/gen/model/Roi_box.h index 63d2ecbf1..4c0683670 100644 --- a/broker/gen/model/Roi_box.h +++ b/broker/gen/model/Roi_box.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_box_list.cpp b/broker/gen/model/Roi_box_list.cpp index 1009a564b..f949cbb8a 100644 --- a/broker/gen/model/Roi_box_list.cpp +++ b/broker/gen/model/Roi_box_list.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_box_list.h b/broker/gen/model/Roi_box_list.h index 3c8c1a970..1443be76d 100644 --- a/broker/gen/model/Roi_box_list.h +++ b/broker/gen/model/Roi_box_list.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_circle.cpp b/broker/gen/model/Roi_circle.cpp index 580a0c0ee..7c879e706 100644 --- a/broker/gen/model/Roi_circle.cpp +++ b/broker/gen/model/Roi_circle.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_circle.h b/broker/gen/model/Roi_circle.h index ced9ff2d9..b885060ea 100644 --- a/broker/gen/model/Roi_circle.h +++ b/broker/gen/model/Roi_circle.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_circle_list.cpp b/broker/gen/model/Roi_circle_list.cpp index 087458fdb..3b754a8b8 100644 --- a/broker/gen/model/Roi_circle_list.cpp +++ b/broker/gen/model/Roi_circle_list.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_circle_list.h b/broker/gen/model/Roi_circle_list.h index dca6b1dad..4cf341f27 100644 --- a/broker/gen/model/Roi_circle_list.h +++ b/broker/gen/model/Roi_circle_list.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_definitions.cpp b/broker/gen/model/Roi_definitions.cpp index dbd5acfa8..1d8bb7c93 100644 --- a/broker/gen/model/Roi_definitions.cpp +++ b/broker/gen/model/Roi_definitions.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Roi_definitions.h b/broker/gen/model/Roi_definitions.h index 21ac3bbde..f755e9ff1 100644 --- a/broker/gen/model/Roi_definitions.h +++ b/broker/gen/model/Roi_definitions.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rotation_axis.cpp b/broker/gen/model/Rotation_axis.cpp index 8c8203771..39eb89b8b 100644 --- a/broker/gen/model/Rotation_axis.cpp +++ b/broker/gen/model/Rotation_axis.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rotation_axis.h b/broker/gen/model/Rotation_axis.h index 5393ecdc0..91a02cfaa 100644 --- a/broker/gen/model/Rotation_axis.h +++ b/broker/gen/model/Rotation_axis.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_live_status.cpp b/broker/gen/model/Rugnux_live_status.cpp index 21b3c0b01..df6c689da 100644 --- a/broker/gen/model/Rugnux_live_status.cpp +++ b/broker/gen/model/Rugnux_live_status.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_live_status.h b/broker/gen/model/Rugnux_live_status.h index cab0e75e8..e70814cee 100644 --- a/broker/gen/model/Rugnux_live_status.h +++ b/broker/gen/model/Rugnux_live_status.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_live_step.cpp b/broker/gen/model/Rugnux_live_step.cpp index 427bb3ce6..d0c27765a 100644 --- a/broker/gen/model/Rugnux_live_step.cpp +++ b/broker/gen/model/Rugnux_live_step.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_live_step.h b/broker/gen/model/Rugnux_live_step.h index e99a86926..e277ac7e3 100644 --- a/broker/gen/model/Rugnux_live_step.h +++ b/broker/gen/model/Rugnux_live_step.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_settings.cpp b/broker/gen/model/Rugnux_settings.cpp index 85a22ddda..e2c6df52f 100644 --- a/broker/gen/model/Rugnux_settings.cpp +++ b/broker/gen/model/Rugnux_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_settings.h b/broker/gen/model/Rugnux_settings.h index ab3c39f9b..3fe955872 100644 --- a/broker/gen/model/Rugnux_settings.h +++ b/broker/gen/model/Rugnux_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_status.cpp b/broker/gen/model/Rugnux_status.cpp index f96408895..9cb4ee8bc 100644 --- a/broker/gen/model/Rugnux_status.cpp +++ b/broker/gen/model/Rugnux_status.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_status.h b/broker/gen/model/Rugnux_status.h index b23c9bc3d..4a30b1a4a 100644 --- a/broker/gen/model/Rugnux_status.h +++ b/broker/gen/model/Rugnux_status.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_summary_row.cpp b/broker/gen/model/Rugnux_summary_row.cpp index 8991ca597..82867a488 100644 --- a/broker/gen/model/Rugnux_summary_row.cpp +++ b/broker/gen/model/Rugnux_summary_row.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Rugnux_summary_row.h b/broker/gen/model/Rugnux_summary_row.h index db5715111..d38ae894f 100644 --- a/broker/gen/model/Rugnux_summary_row.h +++ b/broker/gen/model/Rugnux_summary_row.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Scan_result.cpp b/broker/gen/model/Scan_result.cpp index 7dd997f99..56e5e647c 100644 --- a/broker/gen/model/Scan_result.cpp +++ b/broker/gen/model/Scan_result.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Scan_result.h b/broker/gen/model/Scan_result.h index aa214917b..7a822ab5b 100644 --- a/broker/gen/model/Scan_result.h +++ b/broker/gen/model/Scan_result.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Scan_result_images_inner.cpp b/broker/gen/model/Scan_result_images_inner.cpp index b20a6cfbb..cedd70d08 100644 --- a/broker/gen/model/Scan_result_images_inner.cpp +++ b/broker/gen/model/Scan_result_images_inner.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Scan_result_images_inner.h b/broker/gen/model/Scan_result_images_inner.h index 2252a7010..fc9d26f8d 100644 --- a/broker/gen/model/Scan_result_images_inner.h +++ b/broker/gen/model/Scan_result_images_inner.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Spot_finding_settings.cpp b/broker/gen/model/Spot_finding_settings.cpp index ab7cdb319..eb18cda4d 100644 --- a/broker/gen/model/Spot_finding_settings.cpp +++ b/broker/gen/model/Spot_finding_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Spot_finding_settings.h b/broker/gen/model/Spot_finding_settings.h index 28d09e728..01a13ccc3 100644 --- a/broker/gen/model/Spot_finding_settings.h +++ b/broker/gen/model/Spot_finding_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Standard_detector_geometry.cpp b/broker/gen/model/Standard_detector_geometry.cpp index a1667cbfb..4998edc7f 100644 --- a/broker/gen/model/Standard_detector_geometry.cpp +++ b/broker/gen/model/Standard_detector_geometry.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Standard_detector_geometry.h b/broker/gen/model/Standard_detector_geometry.h index adda0a526..b2f8039a6 100644 --- a/broker/gen/model/Standard_detector_geometry.h +++ b/broker/gen/model/Standard_detector_geometry.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Tcp_settings.cpp b/broker/gen/model/Tcp_settings.cpp index 0115d4b8c..47bc1888d 100644 --- a/broker/gen/model/Tcp_settings.cpp +++ b/broker/gen/model/Tcp_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Tcp_settings.h b/broker/gen/model/Tcp_settings.h index b9946c04d..c58ffb2f7 100644 --- a/broker/gen/model/Tcp_settings.h +++ b/broker/gen/model/Tcp_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Unit_cell.cpp b/broker/gen/model/Unit_cell.cpp index a20831c00..e77e49054 100644 --- a/broker/gen/model/Unit_cell.cpp +++ b/broker/gen/model/Unit_cell.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Unit_cell.h b/broker/gen/model/Unit_cell.h index a1310ec51..3a35eebc8 100644 --- a/broker/gen/model/Unit_cell.h +++ b/broker/gen/model/Unit_cell.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Zeromq_metadata_settings.cpp b/broker/gen/model/Zeromq_metadata_settings.cpp index 0fc8846fa..c2427d59d 100644 --- a/broker/gen/model/Zeromq_metadata_settings.cpp +++ b/broker/gen/model/Zeromq_metadata_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Zeromq_metadata_settings.h b/broker/gen/model/Zeromq_metadata_settings.h index ae2d4d584..d51642477 100644 --- a/broker/gen/model/Zeromq_metadata_settings.h +++ b/broker/gen/model/Zeromq_metadata_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Zeromq_preview_settings.cpp b/broker/gen/model/Zeromq_preview_settings.cpp index cde7db0b0..0adb76a9b 100644 --- a/broker/gen/model/Zeromq_preview_settings.cpp +++ b/broker/gen/model/Zeromq_preview_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Zeromq_preview_settings.h b/broker/gen/model/Zeromq_preview_settings.h index 8f64b5a99..14a3ed787 100644 --- a/broker/gen/model/Zeromq_preview_settings.h +++ b/broker/gen/model/Zeromq_preview_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Zeromq_settings.cpp b/broker/gen/model/Zeromq_settings.cpp index 1241f1d7e..fde851d5b 100644 --- a/broker/gen/model/Zeromq_settings.cpp +++ b/broker/gen/model/Zeromq_settings.cpp @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/gen/model/Zeromq_settings.h b/broker/gen/model/Zeromq_settings.h index 0bb0ae670..5a0fa6727 100644 --- a/broker/gen/model/Zeromq_settings.h +++ b/broker/gen/model/Zeromq_settings.h @@ -2,7 +2,7 @@ * Jungfraujoch * API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. * -* The version of the OpenAPI document: 1.0.0-rc.177 +* The version of the OpenAPI document: 1.0.0-rc.178 * Contact: filip.leonarski@psi.ch * * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech). diff --git a/broker/jfjoch_api.yaml b/broker/jfjoch_api.yaml index 866d1a2e4..4f45de5e6 100644 --- a/broker/jfjoch_api.yaml +++ b/broker/jfjoch_api.yaml @@ -22,7 +22,7 @@ info: requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms. - version: 1.0.0-rc.177 + version: 1.0.0-rc.178 contact: name: Filip Leonarski (Paul Scherrer Institute) email: filip.leonarski@psi.ch diff --git a/broker/redoc-static.html b/broker/redoc-static.html index 0687fa0d0..bb3abf9d2 100644 --- a/broker/redoc-static.html +++ b/broker/redoc-static.html @@ -426,7 +426,7 @@ This format doesn't transmit information about X-axis, only values, so it i 55.627 l 55.6165,55.627 -231.245496,231.24803 c -127.185,127.1864 -231.5279,231.248 -231.873,231.248 -0.3451,0 -104.688, -104.0616 -231.873,-231.248 z - " fill="currentColor">

Jungfraujoch (1.0.0-rc.177)

Download OpenAPI specification:

Filip Leonarski (Paul Scherrer Institute): filip.leonarski@psi.ch License: GPL-3.0

API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). + " fill="currentColor">

Jungfraujoch (1.0.0-rc.178)

Download OpenAPI specification:

Filip Leonarski (Paul Scherrer Institute): filip.leonarski@psi.ch License: GPL-3.0

API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates.

License Clarification

While this API definition is licensed under GPL-3.0, the GPL copyleft provisions do not apply @@ -447,7 +447,9 @@ communicate through network calls or other mechanisms.

X-ray shutter must be closed. This operation will reconfigure network interface of the detector. During operation of the detector it is recommended to use the POST /pedestal operation instead. -If storage cells are used, the execution time might be few minutes. +If storage cells are used, the execution time might be few minutes. +It also clears the image buffer, the previous collection's plots and live rugnux status, disables live rugnux analysis and removes the rugnux model. +It drops the previous dataset's bearer tokens and resets its settings, so nothing of that dataset stays readable.

This is async function - one needs to use POST /wait_till_done to ensure operation is done.

Responses

Responses

Response samples

Content type
application/json
{
  • "detectors": [
    ],
  • "current_id": 0
}

Change rugnux live mode settings

This can only be done when detector is Idle, Error or Inactive states.

-
Authorizations:
bearerAuth
Request Body schema: application/json
enable
boolean
Default: false

Process DECTRIS rotation collections with rugnux live

+
Request Body schema: application/json
enable
boolean
Default: false

Process DECTRIS rotation collections with rugnux live

reserved_cpus
integer <int64> >= 0
Default: 4

CPU threads left to the receiver and the image pusher while rugnux runs

deadline_s
number <double> >= 0
Default: 60

Fixed cutoff: this long after the last image the analysis is cut, and the collection ends with what it has established (as POST /cancel after the last image does). 0 means no cutoff.

Responses

Request samples

Content type
application/json
{
  • "enable": false,
  • "reserved_cpus": 4,
  • "deadline_s": 60
}

Response samples

Content type
application/json
{
  • "msg": "Detector in wrong state",
  • "reason": "WrongDAQState"
}

Get rugnux live mode settings and model status

Can be done anytime

-
Authorizations:
bearerAuth

Responses

Responses

Response samples

Content type
application/json
{
  • "enable": true,
  • "reserved_cpus": 0,
  • "deadline_s": 0.1,
  • "model_loaded": true,
  • "model_size_bytes": 0
}

Upload the model used by rugnux live mode

This can only be done when detector is Idle, Error or Inactive states. The model (PDB or mmCIF, optionally gzipped; at most 64 MiB) is used for molecular replacement and validation of every following rugnux live collection, until it is removed with DELETE /config/rugnux/model or replaced by another upload.

-
Authorizations:
bearerAuth
Request Body schema: application/octet-stream
string <binary>

Responses

Request Body schema: application/octet-stream
string <binary>

Responses

Response samples

Content type
application/json
{
  • "msg": "Detector in wrong state",
  • "reason": "WrongDAQState"
}

Remove the model used by rugnux live mode

This can only be done when detector is Idle, Error or Inactive states.

-
Authorizations:
bearerAuth

Responses

Responses

Response samples

Content type
application/json
{
  • "msg": "Detector in wrong state",
  • "reason": "WrongDAQState"
}

Get live rugnux processing status of the last collection

Can be done anytime

Authorizations:
bearerAuth

Responses

Response samples

Content type
application/json
{
  • "state": "Idle",
  • "probe": "string",
  • "probe_s": 0.1,
  • "message": "string",
  • "images_expected": 0,
  • "images_received": 0,
  • "elapsed_s": 0.1,
  • "step_progress": 1,
  • "steps": [
    ],
  • "last_image_s": 0.1,
  • "cut": "string",
  • "cut_s": 0.1,
  • "last_image_to_written_s": 0.1,
  • "verdict": "string",
  • "verdict_text": "string",
  • "summary": [
    ],
  • "warnings": [
    ]
}

Set ZeroMQ preview settings

Jungfraujoch can generate preview message stream on ZeroMQ SUB socket. +

http://localhost:5232/rugnux/status

Response samples

Content type
application/json
{
  • "state": "Idle",
  • "probe": "string",
  • "probe_s": 0.1,
  • "start_to_first_image_s": 0.1,
  • "message": "string",
  • "images_expected": 0,
  • "expected_collection_s": 0.1,
  • "images_received": 0,
  • "elapsed_s": 0.1,
  • "step_progress": 1,
  • "steps": [
    ],
  • "last_image_s": 0.1,
  • "cut": "string",
  • "cut_s": 0.1,
  • "last_image_to_written_s": 0.1,
  • "verdict": "string",
  • "verdict_text": "string",
  • "summary": [
    ],
  • "warnings": [
    ]
}

Set ZeroMQ preview settings

Jungfraujoch can generate preview message stream on ZeroMQ SUB socket. Here settings of the socket can be adjusted. While the data structure contains also socket_address, this cannot be changed via HTTP and is ignore in PUT request. Options set with this PUT request have no effect on HTTP based preview.

@@ -1098,7 +1096,7 @@ then image might be replaced in the buffer between calling /images and /image.cb