mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 67 Wiki Activity
Files
a6be35ccdb1646c5aee0ef2bf4f14a183d9b9018
Jungfraujoch/rugnux/rugnux_cli.cpp
T
leonarski_fandClaude Opus 5 a6be35ccdb Azimuthal integration: optional sigma clipping of the reported profile 
The profile is the MEAN of each bin, so a few strong reflections landing in a bin
lift it exactly as a smooth powder ring does. That is the wrong quantity whenever
the profile is wanted as a background rather than as a measurement of what is in
the bin - the ice score being the case in point, where reading a plain profile
INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals
had no ice at all.

The adaptive spot finder already computes the right thing, a sigma-clipped
per-resolution-ring background, as a byproduct of its own threshold. Where it
runs, the ice score uses that. Where it does not - --no-adaptive-spots,
--azint-only, and anything reading the profile the broker wrote - there was no way
to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip),
0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian
bin and biases the estimate low rather than removing outliers.

Two clip passes follow the plain one, matching the finder's recipe - the first
pass's standard deviation is itself inflated by the peaks being removed, so one
pass leaves a threshold that is still too generous. A bin with fewer than eight
pixels is left alone: at the detector edge and behind the beam stop there is no
spread to clip on.

Both engines do it. On the GPU the accept range is computed by a small kernel and
stays resident, so a clip pass is one more read of the same pixels and no round
trip; the two accumulation kernels take the range as a pointer that is null on the
plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal
integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict,
against a 0.34 ms per-image total.

Note what the result IS: the smooth background under the peaks, not the bin mean.
It should not be switched on where a ring's integrated intensity is wanted - the
powder-ring geometry fit reads ring peaks, and those are what a clip is designed
to remove. Off by default, so nothing changes unless it is asked for.

Not exposed over the REST API - that needs the generated model regenerated, which
is a separate step.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 00:18:52 +02:00

1852 lines
104 KiB
C++
Raw Blame History

// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cmath>
#include <csignal>
#include <optional>
#include <getopt.h>
#include <iostream>
#include <limits>
#include <sstream>
#include <string>
#include <thread>
#include <type_traits>
#include <vector>
#include "../reader/JFJochHDF5Reader.h"
#include "../common/Logger.h"
#include "../common/Definitions.h"
#include "../common/DiffractionExperiment.h"
#include "../common/PixelMask.h"
#include "../common/print_license.h"
#include "../image_analysis/LoadFCalcFromMtz.h"
#include "../image_analysis/UpdateReflectionResolution.h"
#include "../image_analysis/WriteReflections.h"
#include "../image_analysis/scale_merge/Merge.h"
#include "../image_analysis/scale_merge/RfreeFlags.h"
#include "../image_analysis/scale_merge/ScaleOnTheFly.h"
#include "../image_analysis/scale_merge/StillsPartialityRefine.h"
#include "../image_analysis/scale_merge/RotationScaleMerge.h"
#include "../image_analysis/scale_merge/ResolutionCutoff.h"
#include "../image_analysis/scale_merge/TwinningAnalysis.h"
#include "../image_analysis/scale_merge/SearchSpaceGroup.h"
#include "Rugnux.h"
#include "RugnuxDefaults.h"
#include "ModelValidation.h"
// Spots kept per image (the strongest ones) and handed to indexing. Offline reprocessing is not
// bound by the online spot budget, so this is rugnux's own default rather than the 250 the
// DatasetSettings constructor uses for the detector. Measured over the 37-crystal rotation battery,
// lifting it to 1000 leaves low-resolution R_meas better or equal on every crystal (16 better, 0
// worse, the rest untouched because their frames never reach the cap), with R_meas 14/4 and ISa 14/6
// in its favour and no measurable cost in wall clock. It is also what jfjoch_viewer already sends.
constexpr int64_t RUGNUX_MAX_SPOT_COUNT = 1000;
// Default rot3d per-frame scale-G smoothing range (XDS DELPHI-like), in degrees of rotation.
constexpr double SMOOTH_G_DEFAULT_DEG = 5.0;
// Default rot3d per-batch relative-B batch width (bare --relative-b), in degrees of rotation.
constexpr double RELATIVE_B_DEFAULT_DEG = 10.0;
void print_usage() {
std::cout << "Usage rugnux {<options>} <input.h5>" << std::endl;
std::cout << "Options:" << std::endl;
std::cout << " -o, --output-prefix <txt> Output file prefix (default: output)" << std::endl;
std::cout << " -N, --threads <num> Number of threads (default: all hardware threads)" << std::endl;
std::cout << " -s, --start-image <num> Start image number (default: 0)" << std::endl;
std::cout << " -e, --end-image <num> End image number (default: all)" << std::endl;
std::cout << " -t, --stride <num> Image stride (default: 1)" << std::endl;
std::cout << " -v, --verbose Verbose output" << std::endl;
std::cout << std::endl;
std::cout << " Modes (default: full analysis - spot finding, indexing, integration and merging)" << std::endl;
std::cout << " --azint-only Only run azimuthal integration (no spot finding/indexing); writes <prefix>_process.h5" << std::endl;
std::cout << " --scale Only re-scale/merge the already-integrated reflections in <input> (no re-integration)" << std::endl;
std::cout << std::endl;
std::cout << " Spot finding" << std::endl;
std::cout << " --spot-sigma <num> Noise sigma level for spot finding (default: 4.0)" << std::endl;
std::cout << " --spot-threshold <num> Photon count threshold for spot finding (default: 10)" << std::endl;
std::cout << " --min-pix-per-spot <num> Minimum connected strong pixels per spot. If omitted, min-pix is chosen PER IMAGE (stills indexing): the frame is indexed at min-pix 3/2/1 and the one maximising indexed count x indexed fraction is kept. Give an explicit value to force a fixed min-pix instead." << std::endl;
std::cout << " --adaptive-spots Self-calibrating detection (DEFAULT): the strong-pixel threshold comes from each image's own per-resolution-ring noise instead of the fixed --spot-threshold, so one setting adapts across datasets (no per-dataset --spot-threshold/--spot-sigma tuning)." << std::endl;
std::cout << " --no-adaptive-spots Turn adaptive detection off and use the fixed --spot-threshold / --spot-sigma finder instead" << std::endl;
std::cout << " --spot-false-pixels <num> Adaptive detection operating point: expected noise pixels tolerated per frame (default: 100; implies --adaptive-spots)" << std::endl;
std::cout << " --spot-high-resolution <num> High resolution limit for spot finding. If omitted (or 0), spot finding is not clipped in resolution and extends as far as the detector reaches" << std::endl;
std::cout << " --spot-low-resolution <num> Low resolution limit for spot finding, in A (default: 50; lower it, e.g. 24, to exclude the direct-beam halo on weakly-diffracting serial data)" << std::endl;
std::cout << " --max-spots <num> Max spot count per image, the strongest ones, handed to indexing (default: 1000)" << std::endl;
std::cout << " --detect-ice-rings[=on|off] Flag ice-ring spots (de-prioritised in indexing) and exclude ice-ring reflections from scaling. Default: the master file's setting, or - where the file says nothing - on for rotation and off for stills" << std::endl;
std::cout << std::endl;
std::cout << " Indexing" << std::endl;
std::cout << " (A dataset with a rotation goniometer axis is processed as rotation data by default; use --force-still to override)" << std::endl;
std::cout << " --force-still Process a rotation (goniometer) dataset as independent stills (still indexing + per-image ScaleOnTheFly) instead of rotation" << std::endl;
std::cout << " -R, --two-pass-rotation[=num] Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default: 100)" << std::endl;
std::cout << " --single-pass-rotation[=num] Use online-like single-pass rotation indexing (optional: min angular range deg)" << std::endl;
std::cout << " --force-rotation-lattice <vec> Force rotation indexer with external lattice (in Angstrom) : \"a0x,a0y,a0z,a1x,a1y,a1z,a2x,a2y,a2z\" (9 floats, skips first pass)" << std::endl;
std::cout << " --rotation-no-postrefine Disable the (default-on) two-pass rotation post-refine (post-refine detector distance/beam + cell/axis, then re-integrate; the refined pass is the canonical <prefix>_* output, the header-geometry pass is kept as <prefix>_01_*)" << std::endl;
std::cout << " -X, --indexing-algorithm <txt> Indexing algorithm (FFBIDX|FFT|FFTW|Auto|None)" << std::endl;
std::cout << " -S, --space-group <num|symbol> Space group number (92) or symbol (P43212) - for indexing and scaling" << std::endl;
std::cout << " -C, --unit-cell <cell> Fix reference unit cell: \"a,b,c,alpha,beta,gamma\"" << std::endl;
std::cout << " -r, --refine <txt> Geometry refinement algorithm (none|orientation|beam_and_lattice|flex); flex tries all three per image and keeps whichever indexes the most spots (alias: multi)" << std::endl;
std::cout << " --refine-geometry[=N|off] Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default: 200) then re-indexes (lifts weak-stills indexing). Default ON for stills when a reference cell is given (-C / reference MTZ); =off disables" << std::endl;
std::cout << " --index-ice-rings[=on|off] Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: off; no effect without --detect-ice-rings)" << std::endl;
std::cout << std::endl;
std::cout << " Scaling and merging (on by default)" << std::endl;
std::cout << " --no-merge Skip scaling and merging; write only the per-image _process.h5" << std::endl;
std::cout << " --scale-fulls rot3d: after the 3D combine, refit a per-frame scale on the fulls (XDS order, Unity model). Default ON for rot3d" << std::endl;
std::cout << " --no-scale-fulls Disable the rot3d scale-fulls refit (it is on by default for rot3d)" << std::endl;
std::cout << " --write-process-h5 Also write the (large) _process.h5 when merging (default: only .mtz/.cif when merging)" << std::endl;
std::cout << " --smooth-g[=deg] rot3d: smooth per-frame scale G over a deg-degree rotation range (XDS DELPHI-like) before the combine (default: 5 for rot3d; 0 = off)" << std::endl;
std::cout << " --relative-b[=deg] rot3d: fit a per-batch relative-B (beyond the single decay slope) over deg-degree batches; cross-validated (default: 10 deg when bare; off otherwise)" << std::endl;
std::cout << " --no-scaling-corrections rot3d: disable the (default-on) decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls" << std::endl;
std::cout << " --no-expected-variance-merge stills: disable the default expected-variance merge weighting (which rebuilds each weak observation's signal variance at the reflection mean to de-bias the inverse-variance merge); restores observed-sigma weighting" << std::endl;
std::cout << " -A, --anomalous Anomalous mode (don't merge Friedel pairs)" << std::endl;
std::cout << " --scaling-high-resolution <num> High resolution limit for scaling/merging (manual override; default: no limit)" << std::endl;
std::cout << " --resolution-cutoff <txt> Automatic high-resolution cutoff for the written reflections + reported shells: cc-logistic|off (default: cc-logistic; ignored when --scaling-high-resolution is set)" << std::endl;
std::cout << " --resolution-cc-target <num> CC1/2 target defining the cc-logistic fall-off (default: 0.30)" << std::endl;
std::cout << " --resolution-shells <num> Number of resolution shells in the reported statistics table (default: 10)" << std::endl;
std::cout << " --ice-min-score <num> Ice-presence gate: measured ice score (1 = no ice) a run must reach before ANY ice handling is applied - the flagging and the exclusion from scaling (default: 1.5). The eleven hexagonal bands cover 16-26% of the unique reflections whether or not the crystal has ice, so handling ice on a clean crystal is a pure loss. 0 = no gate (always handle ice)" << std::endl;
std::cout << " --ice-min-spot-ratio <num> Second ice-presence channel: found spots on the hexagonal rings over the same q width of ice-free flanks beside them (1 = spots spread evenly). Ice in large crystallites diffracts as discrete spots and leaves the radial profile flat, so --ice-min-score alone is blind to it. Default 2.0; 0 disables this channel" << std::endl;
std::cout << " --min-partiality <num> Minimum partiality to accept reflection (default: 0.02)" << std::endl;
std::cout << " --capture-uncertainty <num> rot3d: systematic sigma ~num*(1-captured_fraction)*I on under-captured fulls (default: 1.0 for rot3d, 0 otherwise)" << std::endl;
std::cout << " --min-captured-fraction <num> rot3d: drop a combined full whose rocking curve was captured below this fraction (edge-of-sweep truncated fulls) (default: 0.7 for rotation, 0 otherwise; 0 = off)" << std::endl;
std::cout << " --mosaicity <num> Diagnostic: fix the scaling mosaicity (deg) instead of the per-image seed" << std::endl;
std::cout << " --reject-outliers <num> Per-observation merge outlier rejection, N sigma from the per-reflection median (default: 6 for rot3d, XDS/DIALS-style; 0 = off)" << std::endl;
std::cout << " --search-min-zeta <num> De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry |zeta| reaches this, and keep whichever search found MORE symmetry (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law" << std::endl;
std::cout << " --min-image-cc <num> Per-image CC limit in percent (default: no limit)" << std::endl;
std::cout << " --scaling-iterations <num> Number of scaling iterations with no reference data (default: 3)" << std::endl;
std::cout << " -z, --reference-mtz <file> Reference MTZ file" << std::endl;
std::cout << " --reference-column <label> Reference MTZ column to use (default: auto - F-model, else IMEAN/I, else FP/FOBS/F)" << std::endl;
std::cout << " --model <file.pdb> After merging, validate vs this model: R-free + 2Fo-Fc/Fo-Fc maps" << std::endl;
std::cout << std::endl;
std::cout << " Integration" << std::endl;
std::cout << " --bandwidth <num> Relative X-ray bandwidth FWHM (e.g. 0.01 for 1% DMM); default from file or 0" << std::endl;
std::cout << " --integration-radius <r> Signal-box radius r1, or r1,r2,r3 (px). One value => r2=r1+2, r3=r1+4" << std::endl;
std::cout << " --integration-high-resolution <num> High resolution limit for prediction/integration. If omitted (or 0), integration extends as far as the detector reaches" << std::endl;
std::cout << " --max-hkl <n> Predict reflections with |h|,|k|,|l| <= n. Default: derived per crystal from the refined cell (ceil(longest axis / d_min) + 1), which is the exact bound - set it only to override that" << std::endl;
std::cout << " --background-clip <n> Monochromatic (rotation + still): high-side clip of the background ring at mean + n*sqrt(mean) (default 4; 0 = off). This is the default background estimator - it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Broadband data always clip, at 3 sigma; ignored by --integrator boxsum" << std::endl;
std::cout << " --background-radial[=on|off|auto] Correct the background ring for the CURVATURE of the radial background (default auto). The signal disk and the background ring are concentric, so a background linear in position cancels between them and only curvature survives - which on a smooth ice ring reaches +26 counts on a single reflection. Auto applies it per image where that image's ice score shows a smooth powder ring, which is where a radius-only background model holds; on ice made of discrete crystallite spots there is no such ring and the correction makes the bias worse. Costs one short dot product per reflection and no extra pixel reads" << std::endl;
std::cout << " --background-trim <f> Use the old symmetric trimmed mean for the background ring instead of the clip (0<=f<0.5; 0.10 was the former default). Switches --background-clip off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; 0 = plain ring mean" << std::endl;
std::cout << " --integrator <txt> Spot integrator boxsum|gaussian|empirical (default: gaussian profile-fit; boxsum is the classical fallback)" << std::endl;
std::cout << " --simple-stills stills: treat every reflection as a full (p=1, single-pass scale/merge); disables the default physical partiality post-refinement" << std::endl;
std::cout << " -q, --azim-q-spacing <num> Azimuthal-integration Q bin spacing (1/A) (default: 0.01)" << std::endl;
std::cout << " --azim-min-q <num> Azimuthal-integration minimum Q (1/A)" << std::endl;
std::cout << " --azim-max-q <num> Azimuthal-integration maximum Q (1/A). If omitted, integration extends to the highest Q the detector reaches." << std::endl;
std::cout << " --azim-phi-bins <num> Number of azimuthal (phi) bins (default: 1)" << std::endl;
std::cout << " --azim-sigma-clip <num> Sigma-clip the azimuthal profile: repeat the integration twice more, each time rejecting pixels further than <num> sigma from their own bin's mean (default: 0 = off, plain per-bin mean; must be >= 2). A powder ring is azimuthally smooth and survives, Bragg peaks do not, so the result is the smooth background UNDER the peaks - do not use it where a ring's integrated intensity is wanted" << std::endl;
std::cout << " --polarization-correction <on|off> Enable/disable azimuthal polarization correction" << std::endl;
std::cout << " --solid-angle-correction <on|off> Enable/disable azimuthal solid angle correction" << std::endl;
std::cout << std::endl;
std::cout << " Geometry overrides (defaults taken from the input file)" << std::endl;
std::cout << " --beam-x <num> Beam center X (pixel)" << std::endl;
std::cout << " --beam-y <num> Beam center Y (pixel)" << std::endl;
std::cout << " --detector-distance <num> Detector distance (mm)" << std::endl;
std::cout << " --wavelength <num> Wavelength (A)" << std::endl;
std::cout << " --rot1 <num> PONI rotation 1 (rad)" << std::endl;
std::cout << " --rot2 <num> PONI rotation 2 (rad)" << std::endl;
std::cout << " --polarization <num> Polarization factor" << std::endl;
}
enum {
OPT_SPOT_SIGMA = 1000,
OPT_SPOT_THRESHOLD,
OPT_MIN_PIX_PER_SPOT,
OPT_ADAPTIVE_SPOTS,
OPT_NO_ADAPTIVE_SPOTS,
OPT_SPOT_FALSE_PIXELS,
OPT_SPOT_RESOLUTION,
OPT_SPOT_LOW_RESOLUTION,
OPT_MAX_SPOTS,
OPT_MIN_PARTIALITY,
OPT_MIN_IMAGE_CC,
OPT_SEARCH_MIN_ZETA,
OPT_SCALING_ITERATIONS,
OPT_SCALING_HIGH_RESOLUTION,
OPT_RESOLUTION_CUTOFF,
OPT_RESOLUTION_CC_TARGET,
OPT_RESOLUTION_SHELLS,
OPT_SINGLE_PASS_ROTATION,
OPT_FORCE_ROTATION_LATTICE,
OPT_ROTATION_NO_POSTREFINE,
OPT_BACKGROUND_CLIP,
OPT_BACKGROUND_RADIAL,
OPT_REFINE_GEOMETRY,
OPT_BANDWIDTH,
OPT_INTEGRATION_RADIUS,
OPT_BACKGROUND_TRIM,
OPT_MAX_HKL,
OPT_INTEGRATION_HIGH_RES,
OPT_REJECT_OUTLIERS,
OPT_REFERENCE_COLUMN,
OPT_MODEL,
OPT_DUMP_OBSERVATIONS,
OPT_INTEGRATOR,
OPT_SIMPLE_STILLS,
OPT_SCALE_FULLS,
OPT_CAPTURE_UNCERTAINTY,
OPT_MIN_CAPTURED_FRACTION,
OPT_MOSAICITY,
OPT_SMOOTH_G,
OPT_RELATIVE_B,
OPT_NO_SCALING_CORRECTIONS,
OPT_NO_EXPECTED_VARIANCE_MERGE,
OPT_DETECT_ICE_RINGS,
OPT_INDEX_ICE_RINGS,
OPT_ICE_MIN_SCORE,
OPT_ICE_MIN_SPOT_RATIO,
OPT_NO_SCALE_FULLS,
OPT_WRITE_PROCESS_H5,
OPT_FORCE_STILL,
OPT_AZIM_MIN_Q,
OPT_AZIM_MAX_Q,
OPT_AZIM_PHI_BINS,
OPT_AZIM_SIGMA_CLIP,
OPT_AZINT_ONLY,
OPT_SCALE,
OPT_NO_MERGE,
OPT_POLARIZATION_CORRECTION,
OPT_SOLID_ANGLE_CORRECTION,
OPT_BEAM_X,
OPT_BEAM_Y,
OPT_DETECTOR_DISTANCE,
OPT_WAVELENGTH,
OPT_ROT1,
OPT_ROT2,
OPT_POLARIZATION
};
static option long_options[] = {
{"verbose", no_argument, nullptr, 'v'},
{"output-prefix", required_argument, nullptr, 'o'},
{"threads", required_argument, nullptr, 'N'},
{"start-image", required_argument, nullptr, 's'},
{"end-image", required_argument, nullptr, 'e'},
{"stride", required_argument, nullptr, 't'},
{"indexing-algorithm", required_argument, nullptr, 'X'},
{"unit-cell", required_argument, nullptr, 'C'},
{"reference-mtz", required_argument, nullptr, 'z'},
{"reference-column", required_argument, nullptr, OPT_REFERENCE_COLUMN},
{"model", required_argument, nullptr, OPT_MODEL},
{"dump-observations", required_argument, nullptr, OPT_DUMP_OBSERVATIONS},
{"space-group", required_argument, nullptr, 'S'},
{"anomalous", no_argument, nullptr, 'A'},
{"azint-only", no_argument, nullptr, OPT_AZINT_ONLY},
{"scale", no_argument, nullptr, OPT_SCALE},
{"no-merge", no_argument, nullptr, OPT_NO_MERGE},
{"scale-fulls", no_argument, nullptr, OPT_SCALE_FULLS},
{"no-scale-fulls", no_argument, nullptr, OPT_NO_SCALE_FULLS},
{"write-process-h5", no_argument, nullptr, OPT_WRITE_PROCESS_H5},
{"smooth-g", optional_argument, nullptr, OPT_SMOOTH_G},
{"relative-b", optional_argument, nullptr, OPT_RELATIVE_B},
{"no-scaling-corrections", no_argument, nullptr, OPT_NO_SCALING_CORRECTIONS},
{"no-expected-variance-merge", no_argument, nullptr, OPT_NO_EXPECTED_VARIANCE_MERGE},
{"refine", required_argument, nullptr, 'r'},
{"two-pass-rotation", optional_argument, nullptr, 'R'},
{"single-pass-rotation", optional_argument, nullptr, OPT_SINGLE_PASS_ROTATION},
{"force-still", no_argument, nullptr, OPT_FORCE_STILL},
{"azim-q-spacing", required_argument, nullptr, 'q'},
{"azim-min-q", required_argument, nullptr, OPT_AZIM_MIN_Q},
{"azim-max-q", required_argument, nullptr, OPT_AZIM_MAX_Q},
{"azim-phi-bins", required_argument, nullptr, OPT_AZIM_PHI_BINS},
{"azim-sigma-clip", required_argument, nullptr, OPT_AZIM_SIGMA_CLIP},
{"polarization-correction", required_argument, nullptr, OPT_POLARIZATION_CORRECTION},
{"solid-angle-correction", required_argument, nullptr, OPT_SOLID_ANGLE_CORRECTION},
{"beam-x", required_argument, nullptr, OPT_BEAM_X},
{"beam-y", required_argument, nullptr, OPT_BEAM_Y},
{"detector-distance", required_argument, nullptr, OPT_DETECTOR_DISTANCE},
{"wavelength", required_argument, nullptr, OPT_WAVELENGTH},
{"rot1", required_argument, nullptr, OPT_ROT1},
{"rot2", required_argument, nullptr, OPT_ROT2},
{"polarization", required_argument, nullptr, OPT_POLARIZATION},
{"force-rotation-lattice", required_argument, nullptr, OPT_FORCE_ROTATION_LATTICE},
{"rotation-no-postrefine", no_argument, nullptr, OPT_ROTATION_NO_POSTREFINE},
{"refine-geometry", optional_argument, nullptr, OPT_REFINE_GEOMETRY},
{"spot-sigma", required_argument, nullptr, OPT_SPOT_SIGMA},
{"spot-threshold", required_argument, nullptr, OPT_SPOT_THRESHOLD},
{"min-pix-per-spot", required_argument, nullptr, OPT_MIN_PIX_PER_SPOT},
{"adaptive-spots", no_argument, nullptr, OPT_ADAPTIVE_SPOTS},
{"no-adaptive-spots", no_argument, nullptr, OPT_NO_ADAPTIVE_SPOTS},
{"spot-false-pixels", required_argument, nullptr, OPT_SPOT_FALSE_PIXELS},
{"spot-high-resolution", required_argument, nullptr, OPT_SPOT_RESOLUTION},
{"spot-low-resolution", required_argument, nullptr, OPT_SPOT_LOW_RESOLUTION},
{"max-spots", required_argument, nullptr, OPT_MAX_SPOTS},
{"min-partiality", required_argument, nullptr, OPT_MIN_PARTIALITY},
{"capture-uncertainty", required_argument, nullptr, OPT_CAPTURE_UNCERTAINTY},
{"min-captured-fraction", required_argument, nullptr, OPT_MIN_CAPTURED_FRACTION},
{"mosaicity", required_argument, nullptr, OPT_MOSAICITY},
{"min-image-cc", required_argument, nullptr, OPT_MIN_IMAGE_CC},
{"search-min-zeta", required_argument, nullptr, OPT_SEARCH_MIN_ZETA},
{"scaling-iterations", required_argument, nullptr, OPT_SCALING_ITERATIONS},
{"scaling-high-resolution", required_argument, nullptr, OPT_SCALING_HIGH_RESOLUTION},
{"background-clip", required_argument, nullptr, OPT_BACKGROUND_CLIP},
{"background-radial", optional_argument, nullptr, OPT_BACKGROUND_RADIAL},
{"resolution-cutoff", required_argument, nullptr, OPT_RESOLUTION_CUTOFF},
{"resolution-cc-target", required_argument, nullptr, OPT_RESOLUTION_CC_TARGET},
{"resolution-shells", required_argument, nullptr, OPT_RESOLUTION_SHELLS},
{"bandwidth", required_argument, nullptr, OPT_BANDWIDTH},
{"integration-radius", required_argument, nullptr, OPT_INTEGRATION_RADIUS},
{"background-trim", required_argument, nullptr, OPT_BACKGROUND_TRIM},
{"max-hkl", required_argument, nullptr, OPT_MAX_HKL},
{"integration-high-resolution", required_argument, nullptr, OPT_INTEGRATION_HIGH_RES},
{"integrator", required_argument, nullptr, OPT_INTEGRATOR},
{"simple-stills", no_argument, nullptr, OPT_SIMPLE_STILLS},
{"detect-ice-rings", optional_argument, nullptr, OPT_DETECT_ICE_RINGS},
{"index-ice-rings", optional_argument, nullptr, OPT_INDEX_ICE_RINGS},
{"ice-min-score", required_argument, nullptr, OPT_ICE_MIN_SCORE},
{"ice-min-spot-ratio", required_argument, nullptr, OPT_ICE_MIN_SPOT_RATIO},
{"reject-outliers", required_argument, nullptr, OPT_REJECT_OUTLIERS},
{nullptr, 0, nullptr, 0}
};
void trim_in_place(std::string &t) {
size_t b = 0;
while (b < t.size() && std::isspace(static_cast<unsigned char>(t[b]))) b++;
size_t e = t.size();
while (e > b && std::isspace(static_cast<unsigned char>(t[e - 1]))) e--;
t = t.substr(b, e - b);
};
bool parse_float_strict(const std::string &t, float &out) {
try {
size_t idx = 0;
out = std::stof(t, &idx);
return idx == t.size();
} catch (...) {
return false;
}
};
// Parse a required numeric option argument, optionally bounded to [min_value, max_value], or print a
// clear error and exit. getopt hands option arguments over as raw C strings; atoi()/atof() silently
// return 0 on non-numeric input (so a typo like "--min-pix-per-spot 2O" becomes 2 or 0) and std::sto*
// throws, which would terminate the program. This rejects non-numeric input, trailing garbage
// ("1.5foo"), integer overflow, and out-of-range values. T may be integral or floating-point; the
// bounds default to the full representable range (i.e. unbounded).
template <typename T>
T parse_number_arg(const char *arg, const char *option_name, Logger &logger,
T min_value = std::numeric_limits<T>::lowest(),
T max_value = std::numeric_limits<T>::max()) {
std::string s = arg ? arg : "";
trim_in_place(s);
T value{};
bool parsed = false;
if (!s.empty()) {
try {
size_t idx = 0;
if constexpr (std::is_integral_v<T>) {
const long long v = std::stoll(s, &idx);
value = static_cast<T>(v);
parsed = (idx == s.size()) && (static_cast<long long>(value) == v); // no overflow
} else {
value = static_cast<T>(std::stod(s, &idx));
parsed = (idx == s.size());
}
} catch (...) {}
}
if (!parsed) {
logger.Error("Invalid numeric value for {}: '{}'", option_name, arg ? arg : "<null>");
print_usage();
exit(EXIT_FAILURE);
}
if (value < min_value || value > max_value) {
logger.Error("Value for {} out of range: {} (expected {} to {})",
option_name, value, min_value, max_value);
print_usage();
exit(EXIT_FAILURE);
}
return value;
}
// Thin unbounded wrappers for the existing floating-point call sites.
double parse_double_arg(const char *arg, const char *option_name, Logger &logger) {
return parse_number_arg<double>(arg, option_name, logger);
}
float parse_float_arg(const char *arg, const char *option_name, Logger &logger) {
return parse_number_arg<float>(arg, option_name, logger);
}
bool parse_on_off(const char *arg, bool &out) {
std::string s = arg ? arg : "";
std::transform(s.begin(), s.end(), s.begin(),
[](unsigned char c) { return static_cast<char>(std::tolower(c)); });
if (s == "on" || s == "1" || s == "true" || s == "yes") {
out = true;
return true;
}
if (s == "off" || s == "0" || s == "false" || s == "no") {
out = false;
return true;
}
return false;
}
std::optional<UnitCell> parse_unit_cell_arg(const char *arg) {
if (!arg)
return std::nullopt;
std::string s(arg);
trim_in_place(s);
if (s.size() >= 2 && ((s.front() == '"' && s.back() == '"') || (s.front() == '\'' && s.back() == '\''))) {
s = s.substr(1, s.size() - 2);
trim_in_place(s);
}
std::vector<std::string> parts;
parts.reserve(6);
size_t start = 0;
while (true) {
size_t pos = s.find(',', start);
if (pos == std::string::npos) {
parts.push_back(s.substr(start));
break;
}
parts.push_back(s.substr(start, pos - start));
start = pos + 1;
}
if (parts.size() != 6)
return std::nullopt;
UnitCell uc{};
if (!parse_float_strict(parts[0], uc.a)) return std::nullopt;
if (!parse_float_strict(parts[1], uc.b)) return std::nullopt;
if (!parse_float_strict(parts[2], uc.c)) return std::nullopt;
if (!parse_float_strict(parts[3], uc.alpha)) return std::nullopt;
if (!parse_float_strict(parts[4], uc.beta)) return std::nullopt;
if (!parse_float_strict(parts[5], uc.gamma)) return std::nullopt;
return uc;
}
std::optional<CrystalLattice> parse_lattice_arg(const char *arg) {
if (!arg)
return std::nullopt;
std::string s(arg);
trim_in_place(s);
if (s.size() >= 2 && ((s.front() == '"' && s.back() == '"') || (s.front() == '\'' && s.back() == '\''))) {
s = s.substr(1, s.size() - 2);
trim_in_place(s);
}
std::vector<std::string> parts;
parts.reserve(9);
size_t start = 0;
while (true) {
size_t pos = s.find(',', start);
if (pos == std::string::npos) {
parts.push_back(s.substr(start));
break;
}
parts.push_back(s.substr(start, pos - start));
start = pos + 1;
}
if (parts.size() != 9)
return std::nullopt;
std::vector<float> vals(9);
for (int i = 0; i < 9; i++) {
if (!parse_float_strict(parts[i], vals[i]))
return std::nullopt;
}
return CrystalLattice(vals);
}
// Shared offline-output settings for the _process.h5 / reflection writer, used by both the --scale
// path and the full-analysis path (each then sets its own space group and images-per-trigger).
void configure_offline_output(DiffractionExperiment &experiment, const std::string &output_prefix) {
ApplyRugnuxExperimentDefaults(experiment); // analysis policy shared with the viewer
experiment.BitDepthImage(32).Compression(CompressionAlgorithm::BSHUF_LZ4);
// Offline CLI: the operator chose the output path, so allow an absolute -o (the multi-user guard
// that FilePrefix() applies is only for remotely-supplied prefixes in the broker/writer).
experiment.FilePrefixTrusted(output_prefix);
experiment.Mode(DetectorMode::Standard); // full image analysis
experiment.PixelSigned(true);
experiment.OverwriteExistingFiles(true);
experiment.SetFileWriterFormat(FileWriterFormat::NXmxLegacy);
experiment.NumTriggers(1);
}
namespace {
std::atomic<Rugnux *> g_active_process{nullptr};
void handle_sigint(int) {
if (auto *p = g_active_process.load())
p->Cancel();
}
}
// The body of main. Settings setters and the pipeline itself throw JFJochException on input the parser
// cannot reject on its own (a resolution limit of zero, a distance of zero, a polarisation above one,
// an unreadable file), so main wraps this and reports rather than letting the exception terminate the
// process with no diagnostic and exit code 134.
static int RunRugnux(int argc, char **argv) {
for (int i = 0; i < argc; i++) {
std::cout << argv[i] << " ";
}
std::cout << std::endl << std::endl;
RegisterHDF5Filter();
print_license("rugnux");
Logger logger("rugnux");
std::string input_file;
std::string output_prefix = "output";
int nthreads = 0; // 0 = auto: resolved to all hardware threads after parsing (see below)
int start_image = 0;
int end_image = -1; // -1 indicates process until end
int image_stride = 1;
bool verbose = false;
bool azint_only = false; // --azint-only: azimuthal integration only (no spot finding/indexing)
bool scale_only = false; // --scale: re-scale/merge stored reflections only (no re-integration)
bool rotation_indexing = false;
bool force_still = false; // --force-still: process a rotation dataset as stills (indexing + scaling)
bool two_pass_rotation = true;
// Set by any spot-finding option. The two-pass rotation first pass reuses the spots stored in the
// file when it has them, so those options would otherwise not reach the pass that determines the
// lattice - the setting would appear to do nothing at all on rotation data.
bool rotation_postrefine_geometry = true; // default on; --rotation-no-postrefine disables it
int rotation_indexing_image_count = 100;
std::optional<float> rotation_indexing_range;
bool run_scaling = true; // merge is on by default; --no-merge turns it off
std::optional<bool> scale_fulls_arg; // --scale-fulls / --no-scale-fulls; default on for rot3d
bool write_process_h5_flag = false; // --write-process-h5; also write _process.h5 when merging
std::optional<bool> detect_ice_rings; // --detect-ice-rings[=on|off]; unset => use the dataset (file) value
bool index_ice_rings = false; // --index-ice-rings[=on|off]; index on the ice-band spots too
std::optional<double> ice_min_score_arg; // --ice-min-score: ice-presence gate on the measured score
std::optional<double> ice_min_spot_ratio_arg; // --ice-min-spot-ratio: the same gate on the spot channel
std::optional<float> min_q, max_q, q_spacing; // azimuthal integration range / -q spacing (1/A)
std::optional<int32_t> azimuthal_bins; // --azimuthal-bins
std::optional<double> azim_sigma_clip; // --azim-sigma-clip: 0 = off
std::optional<bool> polarization_correction; // --polarization-correction (azimuthal integration)
std::optional<bool> solid_angle_correction; // --solid-angle-correction (azimuthal integration)
// Geometry overrides (default: keep the value stored in the input file)
std::optional<float> beam_x, beam_y, detector_distance_mm, wavelength_A, rot1_rad, rot2_rad, polarization_factor;
std::optional<double> smooth_g_deg_arg; // --smooth-g[=deg]; default 5 deg for rot3d, 0 (off) otherwise
std::optional<double> relative_b_deg_arg; // --relative-b[=deg]; per-batch relative-B width, 0 (off) unless given
bool no_scaling_corrections = false; // --no-scaling-corrections: disable rot3d decay+absorption+modulation surfaces
bool no_expected_variance_merge = false; // --no-expected-variance-merge: restore observed-sigma stills merge weighting
bool anomalous_mode = false;
std::optional<int64_t> space_group_number;
std::optional<UnitCell> fixed_reference_unit_cell;
std::optional<int64_t> max_spot_count_override;
float sigma_spot_finding = 4.0;
int64_t photon_count_threshold_spot_finding = 10;
std::optional<int64_t> min_pix_per_spot; // unset -> adaptive per image; a value -> fixed min-pix
std::optional<bool> adaptive_spots; // unset -> on, for both workflows
float false_pixels_per_frame = 100.0f;
std::string ref_mtz;
std::string ref_column;
std::string model_pdb; // --model: PDB to validate merged intensities against (R-free + maps)
std::string dump_observations; // diagnostic: dump unmerged -P rot3d fulls to this path
double min_partiality = 0.02;
std::optional<double> min_captured_fraction_arg; // explicit --min-captured-fraction; default depends on rotation
std::optional<double> capture_uncertainty_arg; // explicit --capture-uncertainty; default depends on rot3d
std::optional<double> forced_mosaicity_arg; // diagnostic: fix the scaling mosaicity (deg) instead of the per-image seed
double min_image_cc = 0.0;
std::optional<double> search_min_zeta_arg; // --search-min-zeta; rotation default below
int64_t scaling_iter = 3;
std::optional<CrystalLattice> forced_rotation_lattice;
std::optional<double> background_clip_arg; // --background-clip: background-ring high-side sigma clip
bool background_radial_given = false; // --background-radial seen at all (unset => auto)
std::optional<bool> background_radial_arg; // when given: set = force on/off, unset = auto
std::optional<int> refine_geometry; // --refine-geometry[=N]: stills global geometry-refinement pass
bool refine_geometry_disabled = false; // --refine-geometry=off: opt out of the stills default-on
std::optional<float> bandwidth_fwhm; // relative FWHM of dlambda/lambda
IndexingAlgorithmEnum indexing_algorithm = IndexingAlgorithmEnum::Auto;
GeomRefinementAlgorithmEnum refinement_algorithm = GeomRefinementAlgorithmEnum::BeamCenter;
std::optional<float> d_min_spot_finding; // unset -> as far as the detector reaches
float d_max_spot_finding = 0; // 0 = keep the SpotFindingSettings default (50 A)
std::optional<float> d_min_scale_merge;
std::optional<ResolutionCutoffMethod> resolution_cutoff_method; // --resolution-cutoff cc-logistic|off
std::optional<double> resolution_cc_target; // --resolution-cc-target
std::optional<int> report_shell_count; // --resolution-shells
std::optional<std::string> integration_radius_arg; // "r1" or "r1,r2,r3"
std::optional<double> background_trim_arg; // --background-trim: background-ring trimmed-mean fraction
std::optional<int64_t> max_hkl_arg; // --max-hkl: half-width of the predicted hkl box
std::optional<double> integration_d_min_arg; // --integration-high-resolution; unset = detector reach
std::optional<IntegratorMode> integrator_mode; // --integrator boxsum|gaussian|empirical
bool simple_stills_flag = false; // --simple-stills: disable the default stills partiality post-refinement
std::optional<double> outlier_reject_nsigma; // merge per-observation outlier rejection
if (argc == 1) {
print_usage();
exit(EXIT_FAILURE);
}
int opt;
int option_index = 0;
const char *short_opts = "vo:N:s:e:t:R::X:C:z:FAS:r:q:";
while ((opt = getopt_long(argc, argv, short_opts, long_options, &option_index)) != -1) {
switch (opt) {
case 'o':
output_prefix = optarg;
break;
case 'v':
verbose = true;
break;
case 'N':
nthreads = atoi(optarg);
break;
case 's':
start_image = atoi(optarg);
break;
case 'e':
end_image = atoi(optarg);
break;
case 't':
image_stride = atoi(optarg);
break;
case 'R':
if (rotation_indexing) {
logger.Error("Rotation indexing already enabled");
exit(EXIT_FAILURE);
}
rotation_indexing = true;
two_pass_rotation = true;
if (optarg)
rotation_indexing_image_count = atoi(optarg);
break;
case OPT_SINGLE_PASS_ROTATION:
if (rotation_indexing) {
logger.Error("Rotation indexing already enabled");
exit(EXIT_FAILURE);
}
rotation_indexing = true;
two_pass_rotation = false;
if (optarg)
rotation_indexing_range = atof(optarg);
break;
case OPT_ROTATION_NO_POSTREFINE:
rotation_postrefine_geometry = false;
break;
case OPT_REFINE_GEOMETRY: {
if (optarg && std::string(optarg) == "off") {
refine_geometry = std::nullopt;
refine_geometry_disabled = true; // opt out of the stills default-on
break;
}
// Positive frame count fed to the bundle adjust; upper-bounded so refine_frames * 50 in
// RefineStillsGeometry cannot overflow int (1e6 is already far past any real dataset).
refine_geometry = optarg
? parse_number_arg<int>(optarg, "--refine-geometry", logger, 1, 1000000)
: 200;
break;
}
case OPT_FORCE_ROTATION_LATTICE: {
if (rotation_indexing) {
logger.Error("Rotation indexing already enabled");
exit(EXIT_FAILURE);
}
rotation_indexing = true;
auto latt = parse_lattice_arg(optarg);
if (!latt.has_value()) {
logger.Error(
"Invalid rotation lattice. Expected: \"a0x,a0y,a0z,a1x,a1y,a1z,a2x,a2y,a2z\" (9 floats, comma-separated). Got: {}",
optarg ? optarg : "<null>");
print_usage();
exit(EXIT_FAILURE);
}
forced_rotation_lattice = latt;
auto uc = latt->GetUnitCell();
logger.Info(
"Forced rotation lattice set: a={:.3f} b={:.3f} c={:.3f} alpha={:.3f} beta={:.3f} gamma={:.3f}",
uc.a, uc.b, uc.c, uc.alpha, uc.beta, uc.gamma);
break;
}
case 'X': {
std::string alg = optarg ? optarg : "";
std::transform(alg.begin(), alg.end(), alg.begin(),
[](unsigned char c) { return static_cast<char>(std::tolower(c)); });
if (alg == "ffbidx")
indexing_algorithm = IndexingAlgorithmEnum::FFBIDX;
else if (alg == "fft")
indexing_algorithm = IndexingAlgorithmEnum::FFT;
else if (alg == "fftw")
indexing_algorithm = IndexingAlgorithmEnum::FFTW;
else if (alg == "auto")
indexing_algorithm = IndexingAlgorithmEnum::Auto;
else if (alg == "none")
indexing_algorithm = IndexingAlgorithmEnum::None;
else {
logger.Error("Invalid indexing algorithm: {}", alg);
print_usage();
exit(EXIT_FAILURE);
}
break;
}
case 'r': {
std::string alg = optarg ? optarg : "";
std::transform(alg.begin(), alg.end(), alg.begin(),
[](unsigned char c) { return static_cast<char>(std::tolower(c)); });
if (alg == "none")
refinement_algorithm = GeomRefinementAlgorithmEnum::None;
else if (alg == "beam_and_lattice")
refinement_algorithm = GeomRefinementAlgorithmEnum::BeamCenter;
else if (alg == "orientation")
refinement_algorithm = GeomRefinementAlgorithmEnum::OrientationOnly;
else if (alg == "flex" || alg == "multi") // "multi" kept as a back-compat alias
refinement_algorithm = GeomRefinementAlgorithmEnum::Flex;
else {
logger.Error("Invalid geom refinement algorithm: {}", alg);
print_usage();
exit(EXIT_FAILURE);
}
break;
}
case 'C': {
auto uc = parse_unit_cell_arg(optarg);
if (!uc.has_value()) {
logger.Error(
"Invalid unit cell. Expected: \"a,b,c,alpha,beta,gamma\" (6 floats, comma-separated, no spaces). Got: {}",
optarg ? optarg : "<null>");
print_usage();
exit(EXIT_FAILURE);
}
fixed_reference_unit_cell = uc;
logger.Info(
"Fixed reference unit cell set: a={:.3f} b={:.3f} c={:.3f} alpha={:.3f} beta={:.3f} gamma={:.3f}",
uc->a, uc->b, uc->c, uc->alpha, uc->beta, uc->gamma);
break;
}
case 'z':
ref_mtz = optarg;
break;
case OPT_REFERENCE_COLUMN:
ref_column = optarg;
break;
case OPT_MODEL:
model_pdb = optarg;
break;
case OPT_DUMP_OBSERVATIONS:
dump_observations = optarg;
break;
case 'F':
indexing_algorithm = IndexingAlgorithmEnum::FFT;
break;
case 'A':
anomalous_mode = true;
break;
case 'S': {
// Accept a space-group number ("92") or a Hermann-Mauguin symbol ("P43212", "P 43 21 2").
char *end = nullptr;
const long as_number = strtol(optarg, &end, 10);
if (end != optarg && *end == '\0') {
space_group_number = as_number;
} else if (const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name(optarg)) {
space_group_number = sg->number;
} else {
logger.Error("Unknown space group '{}' (use a number like 92 or a symbol like P43212)", optarg);
exit(EXIT_FAILURE);
}
break;
}
case OPT_SPOT_SIGMA:
sigma_spot_finding = parse_number_arg<float>(optarg, "--spot-sigma", logger, 1.0f);
logger.Info("Noise threshold level for spot finding set to {:.2f} sigma", sigma_spot_finding);
break;
case OPT_SPOT_THRESHOLD:
photon_count_threshold_spot_finding = parse_number_arg<int64_t>(optarg, "--spot-threshold", logger, 0);
logger.Info("Photon-count threshold level for spot finding set to {:d}",
photon_count_threshold_spot_finding);
break;
case OPT_MIN_PIX_PER_SPOT:
// Giving an explicit min-pix opts out of the per-image adaptive selection.
min_pix_per_spot = parse_number_arg<int64_t>(optarg, "--min-pix-per-spot", logger, 1);
logger.Info("Minimum pixels per spot fixed at {:d} (adaptive per-image min-pix off)", *min_pix_per_spot);
break;
case OPT_ADAPTIVE_SPOTS:
adaptive_spots = true;
logger.Info("Adaptive (self-calibrating) spot detection enabled");
break;
case OPT_NO_ADAPTIVE_SPOTS:
adaptive_spots = false;
logger.Info("Adaptive spot detection off: using the fixed --spot-threshold / --spot-sigma finder");
break;
case OPT_SPOT_FALSE_PIXELS:
false_pixels_per_frame = parse_number_arg<float>(optarg, "--spot-false-pixels", logger, 1.0f);
adaptive_spots = true;
logger.Info("Adaptive spot detection: expected false pixels/frame set to {:.0f}", false_pixels_per_frame);
break;
case OPT_SPOT_LOW_RESOLUTION:
d_max_spot_finding = parse_number_arg<float>(optarg, "--spot-low-resolution", logger, 0.0f);
logger.Info("Low resolution limit for spot finding set to {:.1f} A", d_max_spot_finding);
break;
case OPT_SPOT_RESOLUTION: {
// 0 has always meant "no limit" for this setting; keep that, but express it as the unset
// optional the rest of the code understands. Passing the 0 through instead reached
// ResolutionShells (via the spot plot), which rejects a zero d_min and threw away every image.
const auto d_min = parse_number_arg<float>(optarg, "--spot-high-resolution", logger, 0.0f);
if (d_min > 0.0f) {
d_min_spot_finding = d_min;
logger.Info("High resolution limit for spot finding set to {:.2f} A", d_min);
} else {
d_min_spot_finding.reset();
logger.Info("No high resolution limit for spot finding: as far as the detector reaches");
}
break;
}
case OPT_MAX_SPOTS:
max_spot_count_override = parse_number_arg<int64_t>(optarg, "--max-spots", logger, 1);
break;
case OPT_AZINT_ONLY:
azint_only = true;
break;
case OPT_SCALE:
scale_only = true;
break;
case OPT_NO_MERGE:
run_scaling = false;
break;
case OPT_SCALE_FULLS:
scale_fulls_arg = true;
break;
case OPT_NO_SCALE_FULLS:
scale_fulls_arg = false;
break;
case OPT_DETECT_ICE_RINGS:
if (optarg == nullptr || strcmp(optarg, "on") == 0)
detect_ice_rings = true;
else if (strcmp(optarg, "off") == 0)
detect_ice_rings = false;
else {
logger.Error("Invalid --detect-ice-rings value: {} (expected on|off)", optarg);
print_usage();
exit(EXIT_FAILURE);
}
break;
case OPT_INDEX_ICE_RINGS:
if (optarg == nullptr || strcmp(optarg, "on") == 0)
index_ice_rings = true;
else if (strcmp(optarg, "off") == 0)
index_ice_rings = false;
else {
logger.Error("Invalid --index-ice-rings value: {} (expected on|off)", optarg);
print_usage();
exit(EXIT_FAILURE);
}
break;
case OPT_ICE_MIN_SCORE:
ice_min_score_arg = parse_double_arg(optarg, "--ice-min-score", logger);
break;
case OPT_ICE_MIN_SPOT_RATIO:
ice_min_spot_ratio_arg = parse_double_arg(optarg, "--ice-min-spot-ratio", logger);
break;
case OPT_WRITE_PROCESS_H5:
write_process_h5_flag = true;
break;
case OPT_SMOOTH_G:
smooth_g_deg_arg = optarg ? parse_double_arg(optarg, "--smooth-g", logger) : SMOOTH_G_DEFAULT_DEG;
break;
case OPT_RELATIVE_B:
relative_b_deg_arg = optarg ? parse_double_arg(optarg, "--relative-b", logger) : RELATIVE_B_DEFAULT_DEG;
break;
case OPT_NO_EXPECTED_VARIANCE_MERGE:
no_expected_variance_merge = true;
break;
case OPT_NO_SCALING_CORRECTIONS:
no_scaling_corrections = true;
break;
case OPT_MIN_PARTIALITY:
min_partiality = parse_double_arg(optarg, "--min-partiality", logger);
break;
case OPT_CAPTURE_UNCERTAINTY:
capture_uncertainty_arg = parse_double_arg(optarg, "--capture-uncertainty", logger);
break;
case OPT_MIN_CAPTURED_FRACTION:
min_captured_fraction_arg = parse_double_arg(optarg, "--min-captured-fraction", logger);
break;
case OPT_MOSAICITY:
forced_mosaicity_arg = parse_double_arg(optarg, "--mosaicity", logger);
break;
case OPT_INTEGRATION_RADIUS:
integration_radius_arg = optarg;
break;
case OPT_BACKGROUND_TRIM:
background_trim_arg = parse_double_arg(optarg, "--background-trim", logger);
break;
case OPT_MAX_HKL:
max_hkl_arg = parse_number_arg<int64_t>(optarg, "--max-hkl", logger, 1, 511);
break;
case OPT_INTEGRATION_HIGH_RES:
integration_d_min_arg = parse_double_arg(optarg, "--integration-high-resolution", logger);
break;
case OPT_BACKGROUND_RADIAL:
background_radial_given = true;
if (optarg == nullptr || strcmp(optarg, "on") == 0)
background_radial_arg = true;
else if (strcmp(optarg, "off") == 0)
background_radial_arg = false;
else if (strcmp(optarg, "auto") == 0)
background_radial_arg = std::nullopt;
else {
logger.Error("Invalid --background-radial value: {} (expected on|off|auto)", optarg);
return 1;
}
break;
case OPT_BACKGROUND_CLIP:
background_clip_arg = parse_double_arg(optarg, "--background-clip", logger);
break;
case OPT_INTEGRATOR:
if (strcmp(optarg, "boxsum") == 0) integrator_mode = IntegratorMode::BoxSum;
else if (strcmp(optarg, "gaussian") == 0) integrator_mode = IntegratorMode::ProfileGaussian;
else if (strcmp(optarg, "empirical") == 0) integrator_mode = IntegratorMode::ProfileEmpirical;
else { logger.Error("--integrator expects boxsum|gaussian|empirical"); return 1; }
break;
case OPT_SIMPLE_STILLS:
simple_stills_flag = true;
break;
case OPT_REJECT_OUTLIERS:
outlier_reject_nsigma = parse_double_arg(optarg, "--reject-outliers", logger);
break;
case OPT_MIN_IMAGE_CC:
min_image_cc = parse_double_arg(optarg, "--min-image-cc", logger);
break;
case OPT_SEARCH_MIN_ZETA:
search_min_zeta_arg = parse_double_arg(optarg, "--search-min-zeta", logger);
break;
case OPT_SCALING_HIGH_RESOLUTION:
d_min_scale_merge = parse_number_arg<float>(optarg, "--scaling-high-resolution", logger,
0.1f, 1000.0f);
break;
case OPT_RESOLUTION_CUTOFF:
if (strcmp(optarg, "cc-logistic") == 0)
resolution_cutoff_method = ResolutionCutoffMethod::CCHalfLogistic;
else if (strcmp(optarg, "off") == 0)
resolution_cutoff_method = ResolutionCutoffMethod::Off;
else {
logger.Error("Invalid --resolution-cutoff value: {} (expected cc-logistic|off)", optarg);
print_usage();
exit(EXIT_FAILURE);
}
break;
case OPT_RESOLUTION_CC_TARGET:
resolution_cc_target = parse_double_arg(optarg, "--resolution-cc-target", logger);
break;
case OPT_RESOLUTION_SHELLS:
report_shell_count = atoi(optarg);
if (report_shell_count.value() < 1) {
logger.Error("Invalid --resolution-shells value: {} (must be >= 1)", report_shell_count.value());
exit(EXIT_FAILURE);
}
break;
case OPT_FORCE_STILL:
force_still = true;
break;
case 'q':
q_spacing = atof(optarg);
break;
case OPT_AZIM_MIN_Q:
min_q = atof(optarg);
break;
case OPT_AZIM_MAX_Q:
max_q = atof(optarg);
break;
case OPT_AZIM_SIGMA_CLIP:
azim_sigma_clip = parse_double_arg(optarg, "--azim-sigma-clip", logger);
break;
case OPT_AZIM_PHI_BINS:
azimuthal_bins = atoi(optarg);
break;
case OPT_POLARIZATION_CORRECTION: {
bool value;
if (!parse_on_off(optarg, value)) {
logger.Error("Invalid polarization correction value (expected on|off): {}", optarg);
exit(EXIT_FAILURE);
}
polarization_correction = value;
break;
}
case OPT_SOLID_ANGLE_CORRECTION: {
bool value;
if (!parse_on_off(optarg, value)) {
logger.Error("Invalid solid angle correction value (expected on|off): {}", optarg);
exit(EXIT_FAILURE);
}
solid_angle_correction = value;
break;
}
case OPT_BEAM_X: beam_x = parse_float_arg(optarg, "--beam-x", logger); break;
case OPT_BEAM_Y: beam_y = parse_float_arg(optarg, "--beam-y", logger); break;
case OPT_DETECTOR_DISTANCE: detector_distance_mm = parse_float_arg(optarg, "--detector-distance", logger); break;
case OPT_WAVELENGTH: {
// Guard > 0: wavelength is used as a divisor (WVL_1A_IN_KEV / wavelength) below, and a 0
// would produce a non-finite incident energy that throws unguarded and aborts the process.
float w = parse_float_arg(optarg, "--wavelength", logger);
if (!(w > 0.0f)) {
logger.Error("Invalid wavelength (must be > 0 A): {}", optarg);
exit(EXIT_FAILURE);
}
wavelength_A = w;
break;
}
case OPT_ROT1: rot1_rad = parse_float_arg(optarg, "--rot1", logger); break;
case OPT_ROT2: rot2_rad = parse_float_arg(optarg, "--rot2", logger); break;
case OPT_POLARIZATION: polarization_factor = parse_float_arg(optarg, "--polarization", logger); break;
case OPT_SCALING_ITERATIONS:
scaling_iter = atoi(optarg);
if (scaling_iter <= 0) {
logger.Error("Invalid scaling iteration count: {}", scaling_iter);
exit(EXIT_FAILURE);
}
break;
case OPT_BANDWIDTH:
bandwidth_fwhm = atof(optarg);
if (!(bandwidth_fwhm.value() >= 0.0f)) {
logger.Error("Invalid bandwidth: {}", optarg);
exit(EXIT_FAILURE);
}
break;
default:
print_usage();
exit(EXIT_FAILURE);
}
}
if (optind != argc - 1) {
logger.Error("Input file not specified");
print_usage();
exit(EXIT_FAILURE);
}
input_file = argv[optind];
logger.Verbose(verbose);
// -N defaults to 0 = "use all hardware threads"; resolve it to a concrete count here so every mode
// behaves the same. The scale/merge engines expand 0 on their own, but the per-image processing
// loop (Rugnux) spawns exactly nthreads workers, so passing 0 there would spawn none and process
// nothing - hence resolving it centrally rather than relying on each consumer.
if (nthreads <= 0) {
unsigned int hw = std::thread::hardware_concurrency();
nthreads = hw > 0 ? static_cast<int>(hw) : 1;
}
if (azint_only && scale_only) {
logger.Error("--azint-only and --scale are mutually exclusive");
exit(EXIT_FAILURE);
}
// Validate space group number early
const gemmi::SpaceGroup *space_group = nullptr;
if (space_group_number.has_value()) {
space_group = gemmi::find_spacegroup_by_number(space_group_number.value());
if (!space_group) {
logger.Error("Unknown space group number {}", space_group_number.value());
exit(EXIT_FAILURE);
}
logger.Info("Using space group {} (number {})", space_group->hm, space_group_number.value());
}
// 1. Read Input File
JFJochHDF5Reader reader;
try {
reader.ReadFile(input_file);
} catch (const std::exception &e) {
logger.Error("Error reading input file: {}", e.what());
exit(EXIT_FAILURE);
}
const auto dataset = reader.GetDataset();
if (!dataset) {
logger.Error("No experiment dataset found in the input file");
exit(EXIT_FAILURE);
}
if (rotation_indexing_image_count <= 0) {
logger.Error("Invalid number of rotation indexing images: {}", rotation_indexing_image_count);
exit(EXIT_FAILURE);
}
logger.Info("Loaded dataset from {}", input_file);
std::vector<MergedReflection> reference_data;
bool reference_has_free_flags = false;
if (!ref_mtz.empty()) {
try {
const auto reference = LoadReferenceMtz(
ref_mtz, ref_column.empty() ? std::nullopt : std::optional<std::string>(ref_column));
reference_data = reference.reflections;
reference_has_free_flags = reference.has_free_flags;
logger.Info("Loaded {} reference reflections from {} (column {}{}{})",
reference_data.size(), ref_mtz, reference.used_column,
reference.squared ? ", squared to intensity" : "",
reference.default_column ? ", auto-selected" : ", user-specified");
if (reference.has_free_flags)
logger.Info("Reference carries R-free flags (column {}): {} of {} free; the merged "
"reflections will inherit this test set",
reference.free_column, reference.n_free, reference_data.size());
if (reference.d_max > 0.0)
logger.Info("Reference resolution range {:.2f} - {:.2f} A", reference.d_max, reference.d_min);
if (reference.cell.has_value())
logger.Info("Reference unit cell: a={:.3f} b={:.3f} c={:.3f} alpha={:.2f} beta={:.2f} gamma={:.2f}",
reference.cell->a, reference.cell->b, reference.cell->c,
reference.cell->alpha, reference.cell->beta, reference.cell->gamma);
if (!reference.space_group_name.empty())
logger.Info("Reference space group: {} (number {})",
reference.space_group_name, reference.space_group_number.value_or(0));
// Check the reference against the cell that will actually drive the merge. --scale merges
// in the cell stored in the input file (as the former jfjoch_scale did); the -C override
// only takes effect on the full-analysis path, which otherwise determines its cell later by
// indexing (unknown here, so nothing can be checked yet).
const std::optional<UnitCell> data_cell =
scale_only ? dataset->experiment.GetUnitCell() : fixed_reference_unit_cell;
const auto warning = ReferenceConsistencyWarning(
reference, data_cell,
space_group_number.has_value() ? std::optional<int>(static_cast<int>(*space_group_number))
: std::nullopt);
if (!warning.empty())
logger.Warning("{}", warning);
// A reference MTZ fixes the space group and unit cell, unless -S / -C override them.
// (-S with the wrong enantiomorph, or -C with a different cell, is allowed - the explicit
// flag always wins.) The cell is a soft reference: indexing may drift within tolerance.
if (!space_group_number.has_value() && reference.space_group_number.has_value()) {
space_group_number = static_cast<int64_t>(*reference.space_group_number);
logger.Info("Fixing space group from reference MTZ: {} ({})",
reference.space_group_name, *space_group_number);
}
if (!fixed_reference_unit_cell.has_value() && reference.cell.has_value()) {
fixed_reference_unit_cell = reference.cell;
logger.Info("Fixing reference unit cell from reference MTZ (indexing may drift within tolerance)");
}
} catch (const std::exception &e) {
logger.Error("Error reading reference MTZ {}: {}", ref_mtz, e.what());
exit(EXIT_FAILURE);
}
}
// --scale: re-scale and merge the already-integrated reflections stored in the input file,
// without re-running spot finding or integration (folded in from the former rugnux_scale tool).
if (scale_only) {
const auto total_images = static_cast<int>(reader.GetNumberOfImages());
const int last_image = (end_image < 0 || end_image >= total_images) ? total_images - 1 : end_image;
auto reflections = reader.ReadReflections(start_image, last_image);
DiffractionExperiment experiment(dataset->experiment);
configure_offline_output(experiment, output_prefix);
// The reflections in the file are already indexed, so the cell and space group they were
// integrated in are the file's to supply here - but an explicit -S / -C still wins.
if (space_group_number.has_value())
experiment.SpaceGroupNumber(space_group_number);
if (fixed_reference_unit_cell.has_value())
experiment.SetUnitCell(fixed_reference_unit_cell);
// A rotation (goniometer) dataset uses RotationScaleMerge unless --force-still asks for stills scaling.
IndexingSettings indexing_settings;
indexing_settings.RotationIndexing(experiment.GetGoniometer().has_value() && !force_still);
// --detect-ice-rings, applied here as well as on the full path below: this block returns
// before that one runs, so without it the flag is silently ignored by --scale. Same
// precedence as there - command line, then the file, then the geometry's default.
if (detect_ice_rings.has_value())
experiment.DetectIceRings(detect_ice_rings.value());
else if (!dataset->file_detect_ice_rings.has_value())
experiment.DetectIceRings(indexing_settings.GetRotationIndexing());
experiment.ImportIndexingSettings(indexing_settings);
// Start from the same defaults the full pipeline uses, so --scale reproduces the merge that wrote
// the _process.h5 rather than a weaker model of its own. Every CLI override below then applies on
// top, exactly as in the full-analysis path.
const bool rot = experiment.GetGoniometer().has_value() && !force_still;
ScalingSettings scaling_settings = RugnuxDefaultScalingSettings(rot);
if (d_min_scale_merge)
scaling_settings.HighResolutionLimit_A(d_min_scale_merge.value());
if (resolution_cutoff_method) scaling_settings.ResolutionCutoff(*resolution_cutoff_method);
if (resolution_cc_target) scaling_settings.ResolutionCCTarget(*resolution_cc_target);
if (report_shell_count) scaling_settings.ReportShellCount(*report_shell_count);
scaling_settings.MergeFriedel(!anomalous_mode);
scaling_settings.MinPartiality(min_partiality);
scaling_settings.MinCapturedFraction(
min_captured_fraction_arg.value_or(scaling_settings.GetMinCapturedFraction()));
scaling_settings.CaptureUncertaintyCoeff(
capture_uncertainty_arg.value_or(scaling_settings.GetCaptureUncertaintyCoeff()));
scaling_settings.ForcedMosaicity(forced_mosaicity_arg);
scaling_settings.MinCCForImage(min_image_cc / 100.0); // --min-image-cc is percent; the setting is a fraction
scaling_settings.StillsPartialityRefine(!simple_stills_flag);
scaling_settings.ExpectedVarianceMerge(!no_expected_variance_merge);
if (ice_min_score_arg)
scaling_settings.IceMinScore(static_cast<float>(*ice_min_score_arg));
if (ice_min_spot_ratio_arg)
scaling_settings.IceMinSpotRatio(static_cast<float>(*ice_min_spot_ratio_arg));
scaling_settings.OutlierRejectNsigma(
outlier_reject_nsigma.value_or(scaling_settings.GetOutlierRejectNsigma()));
scaling_settings.ScaleFulls(scale_fulls_arg.value_or(scaling_settings.GetScaleFulls()));
scaling_settings.SmoothGDegrees(smooth_g_deg_arg.value_or(scaling_settings.GetSmoothGDegrees()));
scaling_settings.RelativeBDegrees(relative_b_deg_arg.value_or(0.0)); // opt-in only; default off
if (no_scaling_corrections)
scaling_settings.CorrectionSurfaces(false);
experiment.ImportScalingSettings(scaling_settings);
if (!experiment.GetUnitCell()) {
logger.Error("Experiment unit cell not found, cannot update reflection resolution");
exit(EXIT_FAILURE);
}
auto refl_stats = UpdateReflectionResolution(experiment.GetUnitCell().value(), reflections);
logger.Info("Read {} reflections from {} images", refl_stats.n_reflections, refl_stats.n_images);
experiment.ImagesPerTrigger(refl_stats.n_images);
// Ice-ring handling, as the full pipeline does it (Rugnux.cpp): flag reflections on a
// hexagonal-ice powder ring so scaling skips them while the merge keeps them. The flag is not
// stored per reflection, so it has to be recomputed here from the resolution just assigned -
// otherwise --scale re-scales a dataset the writing run had scaled without those reflections,
// and the per-image scales come out of a different fit than the ones in the file.
const float ice_width = SpotFindingSettings().ice_ring_width_Q_recipA;
// ...and gated the same way, on the per-image ice score the writing run stored in the file, so
// --scale reaches the same verdict on the same data as the pipeline that produced it.
double ice_sum = 0.0;
size_t ice_n = 0;
for (const float s : dataset->ice_ring_score)
if (std::isfinite(s)) {
ice_sum += s;
++ice_n;
}
const float ice_min_score = experiment.GetScalingSettings().GetIceMinScore();
// ...and the spot channel, pooled over the run exactly as the full pipeline pools it.
double ring_sum = 0.0, ctrl_sum = 0.0;
for (const float v : dataset->spot_count_ice_rings)
if (std::isfinite(v)) ring_sum += v;
for (const float v : dataset->spot_count_ice_control)
if (std::isfinite(v)) ctrl_sum += v;
// Empty control + spots on the rings = the strongest ice evidence, not its absence.
const double ice_spot_ratio = ctrl_sum > 0.0 ? ring_sum / ctrl_sum : (ring_sum > 0.0 ? 1.0e3 : 0.0);
const float ice_min_spot_ratio = experiment.GetScalingSettings().GetIceMinSpotRatio();
const bool ice_present = (ice_n == 0 || ice_sum / static_cast<double>(ice_n) >= ice_min_score)
|| (ice_min_spot_ratio > 0.0f && ice_spot_ratio >= ice_min_spot_ratio);
if (experiment.IsDetectIceRings() && !ice_present) {
logger.Info("Ice-ring handling: measured ice score {:.2f} and spot ratio {:.2f} below the "
"gates ({:.2f} / {:.2f}), no ice detected - ice-ring handling skipped entirely",
ice_sum / static_cast<double>(ice_n), ice_spot_ratio, ice_min_score,
ice_min_spot_ratio);
} else if (experiment.IsDetectIceRings()) {
size_t total = 0, flagged = 0;
for (auto &outcome : reflections) {
for (auto &r : outcome.reflections) {
++total;
r.on_ice_ring = IsOnIceRing(r.d, ice_width);
if (r.on_ice_ring)
++flagged;
}
}
logger.Info("Ice-ring handling: flagged {} of {} reflections on ice rings (half-width {:.3f} A^-1); "
"excluded from scaling, kept for merging", flagged, total, ice_width);
}
const auto scale_start = std::chrono::steady_clock::now();
std::vector<MergedReflection> merged_reflections;
MergeStatistics merged_statistics;
double error_model_isa = 0.0;
// Rotation (rot3d): the dedicated RotationScaleMerge does the whole self-scale -> 3D combine ->
// merge, including the default-on decay + absorption correction surfaces. It does not support
// external-reference scaling or wedge refinement.
// Everything else (stills, reference scaling) uses ScaleOnTheFly + MergeOnTheFly.
const bool is_rotation = experiment.IsRotationIndexing();
if (is_rotation) {
if (!reference_data.empty()
|| experiment.GetRefineRotationWedgeInScaling()
|| experiment.GetScalingSettings().GetRotationWedgeForScaling().has_value())
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Rotation scaling/merging (RotationScaleMerge) does not support reference "
"scaling or wedge refinement");
RotationScaleMerge rsm(experiment, reflections, experiment.GetUnitCell(),
scaling_iter, nthreads, logger);
rsm.Ingest();
auto r = rsm.Run(false);
merged_reflections = std::move(r.merged);
merged_statistics = std::move(r.statistics);
error_model_isa = r.isa;
} else {
// Scaling self-references: the reference MTZ (if any) fixes the cell/space group, reports
// CCref and provides the R-free test set, but is NOT a scale anchor - scaling each image
// against a foreign dataset injects cross-dataset systematics and is a worse reference than
// the data's own merge. The per-image scale G is the exact one-pass solution, so one pass
// (iterating a self-rebuilt reference only re-fits the freshly-scaled noise on weak stills).
ScaleOnTheFly(experiment, MergeAll(experiment, reflections)).Scale(reflections, nthreads);
// Physical partiality post-refinement (default on; --simple-stills disables): refine a per-crystal
// orientation tilt against the merge and recompute each reflection's partiality + scale correction
// (no re-integration), then merge with the improved corrections.
if (experiment.GetScalingSettings().GetStillsPartialityRefine()) {
StillsPartialityRefine refiner(experiment);
const double mean_tilt = refiner.Run(reflections, nthreads);
logger.Info("Stills partiality post-refine: mean |dpsi| = {:.3f} deg", mean_tilt);
}
MergeOnTheFly merge_engine(experiment);
merge_engine.ReferenceCell(experiment.GetUnitCell());
// --min-image-cc has to hold for the merge itself, not only for the reported statistics.
merge_engine.FilterByImageCC(experiment.GetScalingSettings().GetMinCCForImage() > 0.0);
// Fit the (a, b) error model from symmetry-mate scatter before merging, exactly as the full
// pipeline does (Rugnux.cpp). Without this the offline --scale merge would use the identity
// model and produce much worse stills intensities (no (b*I)^2 systematic term, no sigma floor).
merge_engine.RefineErrorModel(reflections);
if (merge_engine.ErrorModelActive())
logger.Info("Error model: a={:.3f} b={:.3f} ISa={:.1f} chi2={:.2f}", merge_engine.ErrorModelA(),
merge_engine.ErrorModelB(),
merge_engine.ErrorModelB() > 0 ? 1.0 / merge_engine.ErrorModelB() : 0.0,
merge_engine.ErrorModelChi2());
for (size_t i = 0; i < reflections.size(); ++i)
merge_engine.AddImage(reflections[i], static_cast<int64_t>(i));
merged_reflections = merge_engine.ExportReflections();
// Automatic high-resolution cutoff (post-merge), matching the full-analysis path: a manual
// --scaling-high-resolution wins, otherwise trim the written reflections + reported shells
// to the CC1/2 fall-off. (Rotation is cut inside RotationScaleMerge above.)
const auto &cut_ss = experiment.GetScalingSettings();
// The offline --scale path re-scales a stored _process.h5 and is never a P1 search merge.
const std::optional<double> effective_d_min = ApplyResolutionCutoff(
merged_reflections, cut_ss.GetHighResolutionLimit_A(), cut_ss.GetResolutionCutoff(),
cut_ss.GetResolutionCCTarget(), /*for_search=*/false, logger);
merged_statistics = merge_engine.MergeStats(merged_reflections, reflections, reference_data,
effective_d_min);
error_model_isa = merge_engine.ErrorModelB() > 0 ? 1.0 / merge_engine.ErrorModelB() : 0.0;
}
logger.Info("Scale + merge completed in {:.2f} s ({} unique reflections)",
std::chrono::duration<double>(std::chrono::steady_clock::now() - scale_start).count(),
merged_reflections.size());
// Inherit the campaign's shared R-free test set from the reference MTZ (overriding the
// per-hkl hash the merge assigned), so every dataset flags the same free reflections.
if (reference_has_free_flags && !reference_data.empty() && !merged_reflections.empty()) {
const auto sg = experiment.GetSpaceGroupNumber().value_or(1);
const size_t matched = ApplyReferenceFreeFlags(merged_reflections, static_cast<int32_t>(sg),
reference_data);
logger.Info("R-free flags: inherited the reference test set ({} of {} merged reflections matched)",
matched, merged_reflections.size());
}
std::cout << merged_statistics;
// Space-group determination lives in the full rugnux pipeline; --scale only consumes a space
// group (from the file or -S) and merges in it.
const bool fixed_space_group = space_group || experiment.GetGemmiSpaceGroup().has_value();
if (!fixed_space_group)
logger.Warning("No space group in the input file or on the command line - merged in P1. "
"Re-run rugnux (which determines and stores the space group) or pass "
"-S to scale and merge in the correct symmetry.");
const auto twin_sg_number = experiment.GetSpaceGroupNumber();
const gemmi::SpaceGroup *twin_sg = twin_sg_number
? gemmi::find_spacegroup_by_number(twin_sg_number.value()) : nullptr;
const auto twinning = AnalyzeTwinning(merged_reflections, twin_sg);
std::cout << std::endl << TwinningAnalysisToText(twinning) << std::endl;
if (!output_prefix.empty())
WriteReflections(merged_reflections, *experiment.GetUnitCell(), experiment, merged_statistics,
error_model_isa > 0 ? fmt::format("{:.2f}", error_model_isa) : "?",
twinning, output_prefix);
if (!output_prefix.empty() && !model_pdb.empty()) {
const auto data_sg = experiment.GetSpaceGroupNumber();
// With a reference MTZ the merohedral indexing was already resolved (stills per-image
// scaling); only probe indexing by R-free when model-only, with no reference.
ValidateAgainstModel(merged_reflections, *experiment.GetUnitCell(), model_pdb,
output_prefix, logger,
data_sg ? std::optional<int>(static_cast<int>(*data_sg)) : std::nullopt,
/*probe_indexing_ambiguity=*/reference_data.empty());
}
return 0;
}
uint64_t total_images_in_file = reader.GetNumberOfImages();
if (end_image < 0 || end_image > total_images_in_file)
end_image = total_images_in_file;
if (image_stride < 0) {
logger.Error("Image stride cannot be negative");
exit(EXIT_FAILURE);
}
if (image_stride == 0) {
logger.Error("Image stride cannot be zero");
exit(EXIT_FAILURE);
}
int images_to_process = (end_image - start_image) / image_stride;
if (images_to_process <= 0) {
logger.Warning("No images to process (Start: {}, End: {} Stride: {}, Total: {})", start_image, end_image,
image_stride, total_images_in_file);
return 0;
}
// 2. Setup Experiment & Components
DiffractionExperiment experiment(dataset->experiment);
// Geometry overrides (default: keep the value stored in the input file). Applied before the
// azimuthal-integration settings are derived, which depend on the geometry.
if (beam_x) experiment.BeamX_pxl(beam_x.value());
if (beam_y) experiment.BeamY_pxl(beam_y.value());
if (detector_distance_mm) experiment.DetectorDistance_mm(detector_distance_mm.value());
if (wavelength_A) experiment.IncidentEnergy_keV(WVL_1A_IN_KEV / wavelength_A.value());
if (rot1_rad) experiment.PoniRot1_rad(rot1_rad.value());
if (rot2_rad) experiment.PoniRot2_rad(rot2_rad.value());
// --polarization is applied after configure_offline_output below, which sets the rugnux default.
// Azimuthal integration (default q-spacing 0.01 1/A, from AzimuthalIntegrationSettings): the profile
// resolves the narrow ice rings for the ice-ring score. Shared by --azint-only and full analysis.
// -q / --azim-* / correction flags override; defaults come from the input file.
{
AzimuthalIntegrationSettings azint_settings = experiment.GetAzimuthalIntegrationSettings();
if (min_q || max_q)
azint_settings.QRange_recipA(min_q.value_or(azint_settings.GetLowQ_recipA()),
max_q ? max_q : azint_settings.GetRequestedHighQ_recipA());
if (q_spacing)
azint_settings.QSpacing_recipA(q_spacing.value());
if (azimuthal_bins)
azint_settings.AzimuthalBinCount(azimuthal_bins.value());
if (azim_sigma_clip)
azint_settings.SigmaClip(static_cast<float>(azim_sigma_clip.value()));
if (polarization_correction)
azint_settings.PolarizationCorrection(polarization_correction.value());
if (solid_angle_correction)
azint_settings.SolidAngleCorrection(solid_angle_correction.value());
experiment.ImportAzimuthalIntegrationSettings(azint_settings);
logger.Info("Azimuthal integration: Q [{:.4f}, {:.4f}] 1/A, spacing {:.4f}, {} Q x {} azimuthal bins",
azint_settings.GetLowQ_recipA(), azint_settings.GetHighQ_recipA(),
azint_settings.GetQSpacing_recipA(), azint_settings.GetQBinCount(),
azint_settings.GetAzimuthalBinCount());
}
// --azint-only: azimuthal integration only (no spot finding / indexing / scaling). Rugnux reads
// the geometry and azimuthal-integration settings configured above off the experiment.
if (azint_only) {
ProcessConfig config;
config.mode = ProcessMode::AzimuthalIntegration;
config.start_image = start_image;
config.end_image = end_image;
config.stride = image_stride;
config.nthreads = nthreads;
config.output_prefix = output_prefix;
Rugnux process(reader, experiment, *dataset->pixel_mask, config);
g_active_process = &process;
std::signal(SIGINT, handle_sigint);
ProcessResult result;
try {
result = process.Run();
} catch (const std::exception &e) {
logger.Error("Processing failed: {}", e.what());
exit(EXIT_FAILURE);
}
g_active_process = nullptr;
std::cout << fmt::format("Processing time: {:.2f} s", result.processing_time_s) << std::endl;
std::cout << fmt::format("Frame rate: {:.2f} Hz", result.frame_rate_hz) << std::endl;
std::cout << fmt::format("Total throughput: {:.2f} MB/s", result.throughput_MBs) << std::endl;
if (result.cancelled)
logger.Warning("Processing was cancelled after {} images", result.images_processed);
return 0;
}
configure_offline_output(experiment, output_prefix);
// configure_offline_output applies the rugnux analysis defaults, one of which is the polarization
// factor, so an explicit --polarization has to land after it or it is silently overwritten.
if (polarization_factor) experiment.PolarizationFactor(polarization_factor.value());
ClearStoredCrystal(experiment); // shared with the viewer; -S / -C below override it
experiment.SpaceGroupNumber(space_group_number);
experiment.ImagesPerTrigger(images_to_process);
// Re-determine the unit cell from scratch: discard any cell stored in the input file so
// indexing is not biased by it. A stale or wrong stored cell otherwise resolves the indexing
// algorithm to FFBIDX and drives it to the wrong lattice (e.g. a non-cubic cell for a cubic
// crystal). A user-supplied -C cell still takes effect (clears to nullopt when absent).
experiment.SetUnitCell(fixed_reference_unit_cell);
// --refine-geometry defaults ON for stills whenever a reference cell is available (-C or a
// reference MTZ): that is exactly when the geometry bundle-adjust can act (it anchors on a known
// cell) and where it helps weak/sparse stills. It is a no-op for rotation (which has its own
// two-pass) and for de-novo stills (no cell yet), so auto-enabling it only where it does something
// avoids spurious "skipping" warnings. Explicit --refine-geometry[=N] still forces it on;
// --refine-geometry=off opts out.
if (!refine_geometry.has_value() && !refine_geometry_disabled) {
const bool is_stills = !(experiment.GetGoniometer().has_value() && !force_still);
if (is_stills && experiment.GetUnitCell().has_value())
refine_geometry = 200;
}
experiment.MaxSpotCount(max_spot_count_override.value_or(RUGNUX_MAX_SPOT_COUNT));
if (max_spot_count_override.has_value())
logger.Info("Max spot count overridden to {}", max_spot_count_override.value());
// X-ray bandwidth: CLI overrides the value carried in the dataset; otherwise
// keep whatever the dataset provided (0 / none -> monochromatic).
if (bandwidth_fwhm)
experiment.BandwidthFWHM(bandwidth_fwhm);
if (experiment.GetBandwidthFWHM())
logger.Info("X-ray bandwidth FWHM set to {:.4f}", experiment.GetBandwidthFWHM().value());
// Rotation vs stills. A dataset collected on a rotation goniometer is processed as rotation data
// (two-pass indexing) by default; --force-still forces per-frame stills. The rotation flags
// (-R / --single-pass-rotation / --force-rotation-lattice) still request rotation explicitly and
// choose the pass/lattice; at this point they show up as rotation_indexing already being set.
const bool has_goniometer = experiment.GetGoniometer().has_value();
if (force_still) {
if (rotation_indexing) {
logger.Error("--force-still conflicts with -R / --single-pass-rotation / --force-rotation-lattice");
exit(EXIT_FAILURE);
}
if (has_goniometer)
logger.Info("--force-still: treating the rotation dataset as independent stills");
} else if (!rotation_indexing && has_goniometer) {
rotation_indexing = true;
two_pass_rotation = true;
logger.Info("Dataset has a rotation goniometer axis: processing as rotation data (two-pass "
"indexing). Use --force-still to treat it as stills.");
}
// Scaling and merging are on by default (run_scaling initialised true); --no-merge turns them off
// for both rotation and stills, in which case only the per-image _process.h5 is written.
// Configure Indexing
IndexingSettings indexing_settings;
indexing_settings.Algorithm(indexing_algorithm);
indexing_settings.RotationIndexing(rotation_indexing);
if (rotation_indexing_range.has_value())
indexing_settings.RotationIndexingMinAngularRange_deg(rotation_indexing_range.value());
indexing_settings.GeomRefinementAlgorithm(refinement_algorithm);
indexing_settings.IndexIceRings(index_ice_rings);
experiment.ImportIndexingSettings(indexing_settings);
// --detect-ice-rings[=on|off] overrides the value carried in from the dataset (HDF5MetadataSource
// sets DetectIceRings from the master file's detect_ice_rings key); with no flag the dataset stands.
// Where the file says nothing at all, the default is the geometry's: on for rotation, off for
// stills. A rotation sweep sits on the same rings for the whole run, so ice there is a coherent
// systematic worth handling, and the ice-presence gate keeps it inert on a clean crystal; a serial
// stills run has too few spots per image to spend any of them on flagging.
if (detect_ice_rings.has_value())
experiment.DetectIceRings(detect_ice_rings.value());
else if (!dataset->file_detect_ice_rings.has_value())
experiment.DetectIceRings(rotation_indexing);
// Scale-fulls refits the per-frame scale on the rotation combined fulls; on by default for rotation
// data (where it lifts ISa substantially) and off for stills. --no-scale-fulls overrides.
const bool scale_fulls = scale_fulls_arg.value_or(rotation_indexing);
ScalingSettings scaling_settings = RugnuxDefaultScalingSettings(rotation_indexing);
scaling_settings.ScaleFulls(scale_fulls);
scaling_settings.SmoothGDegrees(smooth_g_deg_arg.value_or(scaling_settings.GetSmoothGDegrees()));
scaling_settings.RelativeBDegrees(relative_b_deg_arg.value_or(0.0)); // opt-in only; default off
if (no_scaling_corrections)
scaling_settings.CorrectionSurfaces(false);
scaling_settings.StillsPartialityRefine(!simple_stills_flag);
scaling_settings.ExpectedVarianceMerge(!no_expected_variance_merge);
if (ice_min_score_arg)
scaling_settings.IceMinScore(static_cast<float>(*ice_min_score_arg));
if (ice_min_spot_ratio_arg)
scaling_settings.IceMinSpotRatio(static_cast<float>(*ice_min_spot_ratio_arg));
if (d_min_scale_merge)
scaling_settings.HighResolutionLimit_A(d_min_scale_merge.value());
if (resolution_cutoff_method) scaling_settings.ResolutionCutoff(*resolution_cutoff_method);
if (resolution_cc_target) scaling_settings.ResolutionCCTarget(*resolution_cc_target);
if (report_shell_count) scaling_settings.ReportShellCount(*report_shell_count);
scaling_settings.MergeFriedel(!anomalous_mode);
scaling_settings.MinPartiality(min_partiality);
// Drop edge-of-sweep truncated fulls (rocking curve captured < this fraction) from the rot3d combine.
// Defaults ON (0.7) for rotation - removes the low-capture fulls that inflate low-res R-meas and
// slightly bias accuracy; off for non-rot3d (no combine). 0.7 (rather than 0.5) also strips the
// partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting
// crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor
// (e.g. a weakly-diffracting F-cubic or hexagonal crystal recovers its true space group instead of
// P1). An explicit --min-captured-fraction wins.
scaling_settings.MinCapturedFraction(min_captured_fraction_arg.value_or(scaling_settings.GetMinCapturedFraction()));
// Capture-aware systematic sigma defaults ON (1.0) for the rot3d combine - it down-weights the
// over-extrapolated under-captured fulls and, with the mosaicity fix, lifts rotation ISa/anomalous
// substantially. Off for non-rot3d (no combine). An explicit --capture-uncertainty always wins.
scaling_settings.CaptureUncertaintyCoeff(capture_uncertainty_arg.value_or(scaling_settings.GetCaptureUncertaintyCoeff()));
scaling_settings.ForcedMosaicity(forced_mosaicity_arg);
scaling_settings.MinCCForImage(min_image_cc / 100.0); // --min-image-cc is in percent; the setting is a fraction
// Rotation default: run the de-novo space-group search a second time on a merge of only the
// well-measured observations and keep whichever found more symmetry. It cannot lose symmetry - see
// Rugnux.cpp - so the cut being imperfect only means the second opinion contributes nothing.
scaling_settings.SearchMinZeta(search_min_zeta_arg.value_or(scaling_settings.GetSearchMinZeta()));
scaling_settings.OutlierRejectNsigma(
outlier_reject_nsigma.value_or(scaling_settings.GetOutlierRejectNsigma()));
experiment.ImportScalingSettings(scaling_settings);
// Integration radii: r1 (signal box), r2/r3 (background annulus).
if (integration_radius_arg) {
std::vector<float> rr;
std::stringstream ss(*integration_radius_arg);
std::string tok;
while (std::getline(ss, tok, ',')) {
trim_in_place(tok);
if (!tok.empty())
rr.push_back(parse_number_arg<float>(tok.c_str(), "--integration-radius", logger,
0.1f, 1000.0f));
}
float r1, r2, r3;
if (rr.size() == 1) { r1 = rr[0]; r2 = r1 + 2.0f; r3 = r1 + 4.0f; }
else if (rr.size() == 3) { r1 = rr[0]; r2 = rr[1]; r3 = rr[2]; }
else { logger.Error("--integration-radius expects r1 or r1,r2,r3"); return 1; }
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
bis.R1(r1).R2(r2).R3(r3);
experiment.ImportBraggIntegrationSettings(bis);
logger.Info("Integration radii set to r1={:.1f} r2={:.1f} r3={:.1f}", r1, r2, r3);
} else if (!rotation_indexing) {
// Stills spots span a range of crystal orientations captured in a single shot, so they land
// wider on the detector than the r1=4 monochromatic-rotation default assumes. A larger signal
// box lets the profile-fit integrator capture the whole spot while its profile weighting keeps
// the extra background from adding noise (a plain box-sum degrades with it). Measured R-free
// gains on serial stills. An explicit --integration-radius always wins.
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
bis.R1(6.0f).R2(8.0f).R3(12.0f);
experiment.ImportBraggIntegrationSettings(bis);
logger.Info("Stills integration radii default to r1=6.0 r2=8.0 r3=12.0 (override with --integration-radius)");
}
if (integrator_mode) {
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
bis.Integrator(*integrator_mode);
experiment.ImportBraggIntegrationSettings(bis);
logger.Info("Integrator set to {}", *integrator_mode == IntegratorMode::BoxSum ? "box-sum"
: *integrator_mode == IntegratorMode::ProfileGaussian ? "profile (gaussian)"
: "profile (empirical)");
}
if (integration_d_min_arg) {
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
// 0 spells "no limit" for the sibling resolution options, so it has to mean the same here.
bis.DMinLimit_A(*integration_d_min_arg > 0.0
? std::optional<float>(static_cast<float>(*integration_d_min_arg))
: std::nullopt);
experiment.ImportBraggIntegrationSettings(bis);
}
if (max_hkl_arg) {
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
bis.MaxHKL(static_cast<int>(*max_hkl_arg));
experiment.ImportBraggIntegrationSettings(bis);
logger.Info("Predicting reflections with |h|,|k|,|l| <= {} (overriding the per-crystal bound)", *max_hkl_arg);
}
if (background_trim_arg) {
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
bis.BackgroundTrimFraction(static_cast<float>(*background_trim_arg));
experiment.ImportBraggIntegrationSettings(bis);
logger.Info("Background ring: symmetric trimmed mean at {:.2f} instead of the default high-side clip "
"(monochromatic data; broadband always clips)", *background_trim_arg);
}
if (background_clip_arg) {
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
bis.BackgroundClipNSigma(static_cast<float>(*background_clip_arg));
experiment.ImportBraggIntegrationSettings(bis);
logger.Info("Background ring: high-side clip at {:.1f} sigma", *background_clip_arg);
}
if (background_radial_given) {
BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings();
bis.BackgroundRadialCorrection(background_radial_arg);
experiment.ImportBraggIntegrationSettings(bis);
logger.Info("Background ring: radial curvature correction {}",
background_radial_arg.has_value() ? (*background_radial_arg ? "on" : "off") : "auto");
}
SpotFindingSettings spot_settings;
spot_settings.enable = true;
spot_settings.indexing = true;
spot_settings.signal_to_noise_threshold = sigma_spot_finding;
spot_settings.photon_count_threshold = photon_count_threshold_spot_finding;
// Detection defaults differ by workflow; each is overridden by its flag, which always wins.
// - min-pix: choosing it per image (unset) only means something where each frame is indexed on its
// own. Rotation indexing builds ONE lattice from all frames, so it keeps the fixed value.
// - adaptive detection: on by default for both workflows.
// The high-resolution limit is NOT one of them: unset means "as far as the detector reaches" for
// rotation as well as stills. Rotation used to keep 1.5 A on the strength of an indexing-rate
// measurement, but over the 33-crystal battery that limit changes nothing on 29 crystals, changes
// no space-group decision at all, and on the crystals where it does bite it is the LIMIT that is
// worse: the one crystal that loses appreciable indexing rate without it (99.5 -> 94.2%) comes back
// with better R_meas, better high-resolution CC1/2 and better ISa. Fewer frames, better data.
spot_settings.min_pix_per_spot = min_pix_per_spot;
if (rotation_indexing && !spot_settings.min_pix_per_spot.has_value())
spot_settings.min_pix_per_spot = 2;
spot_settings.adaptive_threshold = adaptive_spots.value_or(true);
spot_settings.high_resolution_limit = d_min_spot_finding;
spot_settings.false_pixels_per_frame = false_pixels_per_frame;
if (d_max_spot_finding > 0.0f)
spot_settings.low_resolution_limit = d_max_spot_finding;
// Validate the assembled spot-finding settings the same way the online receivers do (broker and
// receiver call this same function). It enforces the cross-field constraints that per-argument
// bounds cannot express - in particular that the low-resolution limit is coarser than the
// high-resolution limit, so --spot-low-resolution below the high-res cut no longer silently
// rejects every pixel.
try {
DiffractionExperiment::CheckDataProcessingSettings(spot_settings);
} catch (const std::exception &e) {
logger.Error("Invalid spot-finding settings: {}", e.what());
return 1;
}
// Run the shared full-analysis workflow (rotation indexing + scaling/merging live in
// Rugnux; the experiment above carries all algorithm settings).
ProcessConfig config;
config.mode = ProcessMode::FullAnalysis;
config.start_image = start_image;
config.end_image = end_image;
config.stride = image_stride;
config.nthreads = nthreads;
config.output_prefix = output_prefix;
config.spot_finding = spot_settings;
config.rotation_indexing = rotation_indexing;
config.two_pass_rotation = two_pass_rotation;
config.rotation_postrefine_geometry = rotation_postrefine_geometry;
config.rotation_indexing_image_count = rotation_indexing_image_count;
config.forced_rotation_lattice = forced_rotation_lattice;
config.refine_geometry = refine_geometry;
config.run_scaling = run_scaling;
config.scaling_iter = scaling_iter;
config.reference_data = reference_data;
config.reference_has_free_flags = reference_has_free_flags;
config.observation_dump_path = dump_observations;
config.model_path = model_pdb;
// When merging, the merged reflections (.mtz/.cif) are the wanted output; skip the large
// _process.h5 unless explicitly requested. Without merging, the _process.h5 is the only output.
config.write_process_h5 = run_scaling ? write_process_h5_flag : true;
Rugnux process(reader, experiment, *dataset->pixel_mask, config);
g_active_process = &process;
std::signal(SIGINT, handle_sigint);
ProcessResult result;
try {
result = process.Run();
} catch (const std::exception &e) {
logger.Error("Processing failed: {}", e.what());
exit(EXIT_FAILURE);
}
g_active_process = nullptr;
// The space-group search is rendered here (not in the library) so the viewer does not emit it on
// stdout and the CLI owns the format.
if (result.space_group_search.has_value())
std::cout << std::endl << SearchSpaceGroupResultToText(*result.space_group_search) << std::endl;
if (!result.merge_statistics_text.empty())
std::cout << std::endl << result.merge_statistics_text << std::endl;
// Report statistics
std::cout << fmt::format("Processing time: {:.2f} s", result.processing_time_s) << std::endl;
std::cout << fmt::format("Frame rate: {:.2f} Hz", result.frame_rate_hz) << std::endl;
std::cout << fmt::format("Total throughput:{:.2f} MB/s", result.throughput_MBs) << std::endl;
if (result.indexing_rate.has_value())
std::cout << fmt::format("Indexing rate: {:.2f}%", result.indexing_rate.value() * 100.0) << std::endl;
// Final one-line summary of the adopted crystal (whether de-novo determined or fixed with -S),
// so it is not buried in the space-group-search block (which is de-novo only) or only in the mmCIF.
// Only when something actually indexed: with a zero indexing rate the cell is whatever the lattice
// search happened to return and no reflection was measured on it, so printing it as the run's answer
// states a result the data do not support.
const bool anything_indexed = result.indexing_rate.value_or(0.0f) > 0.0f;
if (result.space_group_number.has_value() && anything_indexed) {
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_number(
static_cast<int>(result.space_group_number.value()));
std::string line = fmt::format("Space group: {} (No. {})", sg ? sg->short_name() : "?",
result.space_group_number.value());
// Name every group the data cannot separate, not just the representative. Some pairs share
// their whole absence pattern - an enantiomorphic pair (P4_1 vs P4_3), or I23 vs I2_13 and
// I222 vs I2_12_12_1, where the screw condition h00: h=2n is already implied by the
// I-centering - so the choice among them is a convention, not a measurement. The search
// reports the representative as the lowest space-group number; saying so here keeps the
// summary from claiming a decision the diffraction did not make.
if (result.space_group_search.has_value())
for (const auto &alt : result.space_group_search->alternatives)
line += fmt::format(" or {} (No. {})", alt.short_name(), alt.number);
if (result.space_group_search.has_value() && !result.space_group_search->alternatives.empty())
line += " - indistinguishable from these data";
std::cout << line << std::endl;
}
if (result.consensus_cell.has_value() && anything_indexed) {
const auto &c = result.consensus_cell.value();
std::cout << fmt::format("Unit cell: a={:.2f} b={:.2f} c={:.2f} alpha={:.2f} beta={:.2f} gamma={:.2f}",
c.a, c.b, c.c, c.alpha, c.beta, c.gamma) << std::endl;
}
if (result.indexing_rate.has_value() && !anything_indexed)
std::cout << "No image indexed - no crystal lattice was determined from this dataset" << std::endl;
// Each stage timer measures wall time inside one worker, so it counts the time that worker spent
// BLOCKED on a contended resource - above all the single GPU - as well as its own work. With N
// workers those waits overlap, so the per-image cost is the worker mean divided by the worker
// count, not the mean itself: printed raw at 32 workers these numbers overstate the truth by more
// than an order of magnitude, which is exactly backwards for the one output people tune against.
// Dividing is a lower bound (a worker that is idle rather than blocked is not counted), so the
// remainder is shown against the loop's own wall time rather than hidden.
const auto &t = result.mean_processing_time;
const double per_worker = std::max(1, nthreads);
auto stage = [&](const char *name, float mean_s) {
// A stage that never ran has no mean at all - the per-image indexing and scaling timers are
// never fed on the two-pass rotation path, where the lattice is forced and the merge happens
// outside the image loop. Say nothing rather than printing nan.
return std::isfinite(mean_s)
? fmt::format(" {} {:.2f}", name, mean_s * 1e3 / per_worker) : std::string();
};
std::cout << fmt::format("Per-image cost (ms, {} workers):", nthreads)
<< stage("decompress", t.compression) << stage("preprocess", t.preprocessing)
<< stage("azint", t.azint) << stage("spot-finding", t.spot_finding)
<< stage("indexing", t.indexing) << stage("refinement", t.refinement)
<< stage("indexing-analysis", t.indexing_analysis) << stage("prediction", t.bragg_prediction)
<< stage("integration", t.integration) << stage("scaling", t.image_scale)
<< stage("total", t.processing) << std::endl;
// The stage timers only cover the per-image loop. On a rotation run the first-pass indexing and the
// scaling/merging sit outside it and can be a large share of the run, so report the loop against the
// whole run instead of leaving the difference unexplained. Both are the last pass only: a two-pass
// rotation run does all of this twice.
if (result.images_processed > 0 && result.image_loop_time_s > 0.0) {
const double loop_ms = result.image_loop_time_s * 1e3 / static_cast<double>(result.images_processed);
const double outside_s = result.processing_time_s - result.image_loop_time_s;
std::cout << fmt::format("Per-image wall: {:.2f} ms in the image loop ({:.2f} s); "
"{:.2f} s outside it (first-pass indexing, scaling/merging) [last pass]",
loop_ms, result.image_loop_time_s, std::max(0.0, outside_s)) << std::endl;
}
if (result.cancelled)
logger.Warning("Processing was cancelled after {} images", result.images_processed);
return EXIT_SUCCESS;
}
int main(int argc, char **argv) {
try {
return RunRugnux(argc, argv);
} catch (const std::exception &e) {
Logger("rugnux").Error("{}", e.what());
return EXIT_FAILURE;
}
}
Reference in New Issue View Git Blame Copy Permalink