From c188c9509e273c8f1866c727ae87cca6355a270e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 07:16:33 +0100 Subject: [PATCH 01/34] Stills: --force-still drops the goniometer, signed excitation error, --export-observations --force-still only switched off rotation indexing and left the goniometer in the experiment, so a rotation sweep processed as stills was not processed like serial data: prediction skipped the run's rocking window (it is applied only without a goniometer) and used twice each frame's profile radius, the per-frame mosaicity / spindle checks ran, and the unmerged export took the frames for a sweep, summing neighbouring stills into "rocking events" and dropping summed fractions below the stills partiality floor. The goniometer is now removed under --force-still. Results of forced-stills runs change (one 5 keV S-SAD sweep, 1800 frames: anomalous-map S peaks 4.71/3.08 -> 4.85/3.34 sigma mean/weakest, independent-half CC_anom 0.378 -> 0.399); real stills are unaffected. The stills prediction stores the excitation error with its sign, |S| - 1/lambda (CPU and GPU); the window and the truncation ranking test its magnitude, the partiality refinement squares it, and the --mode scale reset recomputes it signed. A real serial run (4000 frames) merges bit-identically. --export-observations (stills, off by default, processing-inert) writes _obs.mtz: every integrated observation of the written merge with its signed excitation error, lab q, partiality, run rocking width, raw I/sigma/background, corrections, scale, the sigma the merge weighted it with, the frame's A*/B*/C*, cell, G and image CC, and a bit for each merge test it fails (MergeOnTheFly::Verdicts, every test evaluated on its own), including observations the merge drops. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- common/Reflection.h | 2 +- image_analysis/WriteReflections.cpp | 107 ++++++++++++++++++ image_analysis/WriteReflections.h | 14 +++ .../bragg_prediction/BraggPrediction.cpp | 8 +- .../bragg_prediction/BraggPredictionGPU.cu | 4 +- image_analysis/scale_merge/Merge.cpp | 42 +++++++ image_analysis/scale_merge/Merge.h | 21 ++++ .../scale_merge/StillsPartialityRefine.h | 2 +- rugnux/Rugnux.cpp | 4 + rugnux/Rugnux.h | 8 ++ rugnux/RugnuxOutputFiles.cpp | 4 + rugnux/RugnuxScaleMerge.cpp | 4 + rugnux/rugnux_cli.cpp | 12 +- tests/CalcBraggPredictionTest.cpp | 20 ++-- 14 files changed, 234 insertions(+), 18 deletions(-) diff --git a/common/Reflection.h b/common/Reflection.h index 9f054599a..d93240c0c 100644 --- a/common/Reflection.h +++ b/common/Reflection.h @@ -24,7 +24,7 @@ struct Reflection { float bkg; float var_bkg; // non-signal (background) part of sigma^2, carried to the merge float sigma; - float dist_ewald; + float dist_ewald; // stills: signed excitation error |S| - 1/lambda (1/A), positive outside the sphere // The reciprocal Lorentz factor (rotation only - a still's Lorentz factor is one) times the // reciprocal polarization factor, and nothing else. This is what LP means everywhere in the // field, and it is what the CBOR key "rlp" and the HDF5 dataset "lp" store the reciprocal of. diff --git a/image_analysis/WriteReflections.cpp b/image_analysis/WriteReflections.cpp index 1896c10b8..92055a8dd 100644 --- a/image_analysis/WriteReflections.cpp +++ b/image_analysis/WriteReflections.cpp @@ -8,6 +8,7 @@ #include "scale_merge/TwinningAnalysis.h" #include "bragg_integration/SystematicAbsence.h" #include "../common/ParallelFor.h" +#include "../common/PanelBoxes.h" #include "../writer/WriteRugnuxFiles.h" #include @@ -1360,3 +1361,109 @@ void WriteReflections(const std::vector &reflections, WriteShelxIns(unitCell, nullptr, experiment, std::filesystem::path(filename).filename().string(), filename + ".ins"); } + +MtzTable ObservationMtzTable(const std::vector &outcomes, + const std::vector> &verdicts, + std::optional written_d_min, + const std::optional &run_width, + const UnitCell &unitCell, + const DiffractionExperiment &experiment) { + constexpr uint32_t OBS_BEYOND_DMIN = 4096, OBS_NO_VERDICT = 2048; + MtzTable mtz; + mtz.space_group_name = experiment.GetSpaceGroupOrP1().xhm(); + mtz.unit_cell = unitCell; + mtz.title = "Rugnux stills observations (diagnostic)"; + mtz.datasets = {{"HKL_base", "HKL_base", "HKL_base", 0.0f}, + {"obs", experiment.GetSampleName(), "obs", experiment.GetWavelength_A()}}; + if (run_width) { + std::string t = "RUNWIDTH gamma0 " + std::to_string(run_width->gamma0) + " gamma_e " + std::to_string(run_width->gamma_e) + + " bw " + std::to_string(run_width->bandwidth_sigma) + " radial_scale " + + std::to_string(run_width->radial_scale); + mtz.history.push_back(t); + std::string x = "RUNWIDTH_DSTAR", y = "RUNWIDTH_S"; + for (size_t i = 0; i < run_width->table_dstar.size(); ++i) { + x += " " + std::to_string(run_width->table_dstar[i]); + y += " " + std::to_string(run_width->table_s[i]); + } + mtz.history.push_back(x); + mtz.history.push_back(y); + } + if (written_d_min) + mtz.history.push_back("WRITTEN_DMIN " + std::to_string(*written_d_min)); + + const std::vector labels = { + "H", "K", "L", "FRAME", "IMG", "XPRED", "YPRED", "XOBS", "YOBS", "PANEL", "D", + "EPS", "EPSR", "EPSABS", "QX", "QY", "QZ", "PART", "ROCKSIG", "ZETA", "DPHI", + "I", "SIGI", "BKG", "VARBKG", "LP", "QE", "FLIGHT", "SCALECORR", "ISCALED", "SIGSCALED", "SIGMERGE", + "G", "CCIMG", "MOSAIC", "ASX", "ASY", "ASZ", "BSX", "BSY", "BSZ", "CSX", "CSY", "CSZ", + "CELLA", "CELLB", "CELLC", "CELLAL", "CELLBE", "CELLGA", "FLAGS", "USED"}; + for (const auto &l : labels) + mtz.columns.push_back({l, (l == "H" || l == "K" || l == "L") ? 'H' : (l == "FLAGS" || l == "USED" ? 'I' : 'R'), 0, {}}); + + const auto panels = PanelBoxes(experiment.GetModules(), experiment.GetXPixelsNum(), experiment.GetYPixelsNum()); + const auto panel_of = [&](float x, float y) { + for (const auto &b : panels) + if (x >= b.x0 && x < b.x1 && y >= b.y0 && y < b.y1) + return static_cast(b.panel); + return -2.0f; + }; + // The run's rocking width at d*, as StillsPartialityRefine evaluates it: its table, linear in d* and held + // beyond the end shells, else the three-term formula. + const auto rock_sigma = [&](double dstar, double qz) { + if (!run_width) + return NAN; + const auto &x = run_width->table_dstar; + const auto &y = run_width->table_s; + if (!x.empty()) { + if (dstar <= x.front()) return static_cast(y.front()); + if (dstar >= x.back()) return static_cast(y.back()); + const size_t i = std::upper_bound(x.begin(), x.end(), dstar) - x.begin(); + return static_cast(y[i - 1] + (dstar - x[i - 1]) / (x[i] - x[i - 1]) * (y[i] - y[i - 1])); + } + const double ang = run_width->gamma_e * dstar, bw = run_width->bandwidth_sigma * qz; + return static_cast(std::sqrt(run_width->gamma0 * run_width->gamma0 + ang * ang + bw * bw)); + }; + const double radial = run_width ? run_width->radial_scale : 1.0; + + std::vector row(labels.size()); + const auto push = [&]() { + for (size_t c = 0; c < row.size(); ++c) + mtz.columns[c].values.push_back(row[c]); + }; + for (size_t i = 0; i < outcomes.size(); ++i) { + const auto &o = outcomes[i]; + if (o.reflections.empty() || o.latt.CalcVolume() <= 1.0f) // an unindexed frame: no lattice + continue; + const bool have_verdicts = i < verdicts.size() && verdicts[i].size() == o.reflections.size(); + const Coord A = o.latt.Astar(), B = o.latt.Bstar(), C = o.latt.Cstar(); + const Coord S0 = o.geom.GetScatteringVector(); + const double inv_lambda = 1.0 / o.geom.GetWavelength_A(); + const UnitCell cell = o.latt.GetUnitCell(); + for (size_t j = 0; j < o.reflections.size(); ++j) { + const Reflection &r = o.reflections[j]; + const Coord q = A * static_cast(r.h) + B * static_cast(r.k) + C * static_cast(r.l); + const auto eps_at = [&](double s) { + const double x = s * q.x + S0.x, y = s * q.y + S0.y, z = s * q.z + S0.z; + return static_cast(std::sqrt(x * x + y * y + z * z) - inv_lambda); + }; + const double dstar = std::sqrt(q.x * q.x + q.y * q.y + q.z * q.z); + uint32_t flags = have_verdicts ? verdicts[i][j].flags : OBS_NO_VERDICT; + if (written_d_min && r.d < *written_d_min) + flags |= OBS_BEYOND_DMIN; + row = {static_cast(r.h), static_cast(r.k), static_cast(r.l), r.image_number, + static_cast(i), r.predicted_x, r.predicted_y, r.observed_x, r.observed_y, + panel_of(r.predicted_x, r.predicted_y), r.d, + eps_at(1.0), eps_at(radial), r.dist_ewald, q.x, q.y, q.z, r.partiality, + rock_sigma(radial * dstar, std::fabs(radial * q.z)), r.zeta, r.delta_phi_deg, + r.I, r.sigma, r.bkg, r.var_bkg, r.prescaling_corr, r.qe_corr, r.flight_corr, + r.image_scale_corr, r.I * r.image_scale_corr, r.sigma * r.image_scale_corr, + have_verdicts ? verdicts[i][j].sigma : NAN, + o.image_scale_g.value_or(NAN), o.image_scale_cc.value_or(NAN), o.mosaicity_deg.value_or(NAN), + A.x, A.y, A.z, B.x, B.y, B.z, C.x, C.y, C.z, + cell.a, cell.b, cell.c, cell.alpha, cell.beta, cell.gamma, + static_cast(flags), flags == 0 ? 1.0f : 0.0f}; + push(); + } + } + return mtz; +} diff --git a/image_analysis/WriteReflections.h b/image_analysis/WriteReflections.h index 6a81d5491..14b5be3c6 100644 --- a/image_analysis/WriteReflections.h +++ b/image_analysis/WriteReflections.h @@ -11,6 +11,8 @@ #include "../common/DiffractionExperiment.h" #include "../common/MtzTable.h" #include "IntegrationOutcome.h" +#include "scale_merge/Merge.h" +#include "../common/StillsRunWidth.h" struct MergeStatistics; struct TwinningAnalysisResult; @@ -127,6 +129,18 @@ MtzTable UnmergedMtzTable(const std::vector &outcomes, const std::vector *scaled_fulls = nullptr); void SetUnmergedMtzMosaicity(MtzTable &mtz, const std::vector &outcomes); +// Diagnostic (--export-observations, stills): every integrated observation of the merged run with what the +// merge made of it - one row per reflection of every outcome, the ones the merge left out included, with a +// FLAGS bit for each test it failed (ObservationFlag; BEYOND_DMIN 4096 = past the written resolution cut). +// verdicts: MergeOnTheFly::Verdicts of the written merge, in the outcomes' order. Columns are listed in the +// code and documented with the campaign's loader, not here; H K L are the indices as measured. +MtzTable ObservationMtzTable(const std::vector &outcomes, + const std::vector> &verdicts, + std::optional written_d_min, + const std::optional &run_width, + const UnitCell &unitCell, + const DiffractionExperiment &experiment); + void WriteReflections(const std::vector &reflections, const UnitCell &unitCell, const DiffractionExperiment &experiment, diff --git a/image_analysis/bragg_prediction/BraggPrediction.cpp b/image_analysis/bragg_prediction/BraggPrediction.cpp index e6f2d230a..20dd30fb0 100644 --- a/image_analysis/bragg_prediction/BraggPrediction.cpp +++ b/image_analysis/bragg_prediction/BraggPrediction.cpp @@ -45,7 +45,8 @@ int BraggPrediction::TruncateToOutput(int count) { reflections.begin() + count, [](const Reflection &a, const Reflection &b) { if (a.partiality != b.partiality) return a.partiality > b.partiality; - if (a.dist_ewald != b.dist_ewald) return a.dist_ewald < b.dist_ewald; + if (std::fabs(a.dist_ewald) != std::fabs(b.dist_ewald)) + return std::fabs(a.dist_ewald) < std::fabs(b.dist_ewald); if (a.h != b.h) return a.h < b.h; if (a.k != b.k) return a.k < b.k; return a.l < b.l; @@ -142,7 +143,8 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal float S_y = recip_y + S0.y; float S_z = recip_z + S0.z; float S_len = sqrtf(S_x * S_x + S_y * S_y + S_z * S_z); - float dist_ewald_sphere = std::fabs(S_len - one_over_wavelength); + // Signed excitation error, |S| - 1/lambda: positive outside the sphere. The window tests its size. + float dist_ewald_sphere = S_len - one_over_wavelength; // Energy bandwidth thickens the Ewald shell radially: at the // diffraction condition |S|-1/λ shifts by recip_z·(Δλ/λ), i.e. @@ -154,7 +156,7 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal const float bw_tol = kBandwidthCutoffSigmas * settings.bandwidth_sigma * std::fabs(recip_z); radial_cutoff = std::sqrt(radial_cutoff * radial_cutoff + bw_tol * bw_tol); } - float window_dist = dist_ewald_sphere; + float window_dist = std::fabs(dist_ewald_sphere); if (settings.window_c0 > 0.0f || settings.window_ang > 0.0f || settings.window_bw > 0.0f) { const float wb = settings.window_bw * std::fabs(recip_z); radial_cutoff = std::sqrt(settings.window_c0 * settings.window_c0 diff --git a/image_analysis/bragg_prediction/BraggPredictionGPU.cu b/image_analysis/bragg_prediction/BraggPredictionGPU.cu index 36871330b..b6b1954c8 100644 --- a/image_analysis/bragg_prediction/BraggPredictionGPU.cu +++ b/image_analysis/bragg_prediction/BraggPredictionGPU.cu @@ -106,7 +106,7 @@ namespace { float Sy = recip_y + C.S0.y; float Sz = recip_z + C.S0.z; float S_len = sqrtf(Sx * Sx + Sy * Sy + Sz * Sz); - float dist_ewald = fabsf(S_len - C.one_over_wavelength); + float dist_ewald = S_len - C.one_over_wavelength; // signed, as on the CPU path // Energy bandwidth thickens the Ewald shell radially: σ_bw = |recip_z|·(Δλ/λ) // (= bλ/2d²). Broaden the acceptance window in quadrature (see CPU path). float radial_cutoff = C.ewald_cutoff; @@ -114,7 +114,7 @@ namespace { const float bw_tol = kBandwidthCutoffSigmas * C.bandwidth_sigma * fabsf(recip_z); radial_cutoff = sqrtf(radial_cutoff * radial_cutoff + bw_tol * bw_tol); } - float window_dist = dist_ewald; + float window_dist = fabsf(dist_ewald); if (C.window_c0 > 0.0f || C.window_ang > 0.0f || C.window_bw > 0.0f) { const float wb = C.window_bw * fabsf(recip_z); radial_cutoff = sqrtf(C.window_c0 * C.window_c0 + C.window_ang * C.window_ang * recip_sq + wb * wb); diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index f422a2cb6..521bd212e 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -136,6 +136,48 @@ size_t MergeOnTheFly::Contributions(const IntegrationOutcome &outcome, std::vect return rejected; } +// The tests of Contributions and Mask, each on its own rather than up to the first that fails. +std::vector> MergeOnTheFly::Verdicts(const std::vector &outcomes) const { + std::vector> out(outcomes.size()); + for (size_t i = 0; i < outcomes.size(); ++i) { + const uint32_t image_flags = Mask(outcomes[i]) ? OBS_IMAGE_MASKED : 0; + for (const auto &r: outcomes[i].reflections) { + ObservationVerdict v{.flags = image_flags}; + if (generator.IsSystematicallyAbsent(r)) + v.flags |= OBS_ABSENT; + const bool scaled = r.image_scale_corr > 0.0 && std::isfinite(r.image_scale_corr); + if (!scaled) + v.flags |= OBS_NO_SCALE; + if (!AcceptReflection(r, high_resolution_limit, std::nullopt)) + v.flags |= OBS_RESOLUTION; + if (exclude_ice_rings && r.on_ice_ring) + v.flags |= OBS_ICE_EXCLUDED; + if (r.partiality < min_partiality) + v.flags |= OBS_LOW_PARTIALITY; + const float I_corr = r.I * r.image_scale_corr; + const float sigma_corr = r.sigma * r.image_scale_corr; + if (!scaled || !std::isfinite(I_corr) || !std::isfinite(sigma_corr) || sigma_corr <= 0.0f) { + if (scaled) + v.flags |= OBS_BAD_VALUE; + out[i].push_back(v); + continue; + } + const uint64_t hkl_key = generator(r).pack(); + v.sigma = CorrectedSigma(I_corr, sigma_corr, r.image_scale_corr, r.var_bkg, r.partiality, hkl_key); + if (reject_outliers && error_model_active) { + const auto mit = reject_median_I.find(hkl_key); + const auto bit = error_model_mean_I.find(hkl_key); + if (mit != reject_median_I.end() && bit != error_model_mean_I.end() + && OutsideOutlierBand(I_corr, mit->second, reject_nsigma, static_cast(v.sigma) * v.sigma, + bit->second, error_model_b * bit->second, reject_up, reject_down)) + v.flags |= OBS_OUTLIER; + } + out[i].push_back(v); + } + } + return out; +} + void MergeOnTheFly::Accumulate(const std::vector &contributions, int64_t image_id) { const int prefer = HalfForImage(image_id); diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index bea32cae1..3e1f22ddd 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -278,6 +278,23 @@ inline XdsErrorModel ToXdsErrorModel(double a, double b) { return {a, b * b / a, 1.0 / b}; } +// Why the stills merge took or left one observation: a bit for every test it fails, 0 = merged +// (MergeOnTheFly::Verdicts; a diagnostic, --export-observations). +enum ObservationFlag : uint32_t { + OBS_IMAGE_MASKED = 1, // the image is not merged: cell outside the run's population, or --min-image-cc + OBS_ABSENT = 2, // systematic absence of the merge's space group + OBS_NO_SCALE = 4, // no usable scale correction (unscaled image, collapsed scale, p * G = 0) + OBS_RESOLUTION = 8, // outside the merge's resolution limits, or not finite + OBS_ICE_EXCLUDED = 16, // on an ice ring, in a merge that leaves them out + OBS_LOW_PARTIALITY = 32, // partiality below --min-partiality + OBS_BAD_VALUE = 64, // corrected I or sigma not finite, or sigma <= 0 + OBS_OUTLIER = 128, // outside the outlier band about the reflection's median +}; +struct ObservationVerdict { + uint32_t flags = 0; + float sigma = NAN; // the sigma the merge weights it with (error model applied); NaN if not computable +}; + class MergeOnTheFly { mutable std::mutex merged_mutex; @@ -384,6 +401,10 @@ public: // per-observation work spread over the workers and the accumulation kept in image order. void AddImages(const std::vector &outcomes); + // Every test AddImages applies, each evaluated on its own, for every reflection of every outcome (in + // their order); call after AddImages. A diagnostic: the merge itself does not use it. + std::vector> Verdicts(const std::vector &outcomes) const; + // d_min_override, when set, is the effective high-resolution limit for the shell table (used for // the automatic resolution cutoff computed by the caller); otherwise the manual // ScalingSettings high-resolution limit stands. The number of shells is ScalingSettings::ReportShellCount. diff --git a/image_analysis/scale_merge/StillsPartialityRefine.h b/image_analysis/scale_merge/StillsPartialityRefine.h index 9ba83f1bd..5b99d6d36 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.h +++ b/image_analysis/scale_merge/StillsPartialityRefine.h @@ -25,7 +25,7 @@ // // Because the integrated intensity I_obs is fixed, refining the tilt only recomputes the partiality // analytically from the stored per-image lattice/geometry (q = A*.h + B*.k + C*.l, then -// dist_ewald = | |q + S0| - 1/lambda |) - NO pixel re-integration. The loop is: merge -> per-crystal +// dist_ewald = |q + S0| - 1/lambda, signed) - NO pixel re-integration. The loop is: merge -> per-crystal // refine tilt (G profiled out by the same robust IRLS ScaleOnTheFly uses) -> recompute p and // image_scale_corr -> re-merge, iterated a few times. Mutates each reflection's `partiality` and // `image_scale_corr` in place; the existing MergeOnTheFly then consumes the improved corrections. The diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index b1dc52b18..89efe8b2d 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -1320,6 +1320,10 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b SetUnmergedMtzMosaicity(mtz, indexer->GetIntegrationOutcome()); result.outputs->unmerged = std::move(mtz); } + if (config_.export_observations && !observation_verdicts_.empty()) + result.outputs->observations = ObservationMtzTable(indexer->GetIntegrationOutcome(), observation_verdicts_, + observation_d_min_, end_msg.stills_run_width, + *result.consensus_cell, experiment_); if (config_.export_unmerged_partials) { if (observer) observer->OnPhase("Unmerged reflections"); MtzTable mtz = UnmergedMtzTable(indexer->GetIntegrationOutcome(), *result.consensus_cell, diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 96b6d4062..1ee456478 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -264,6 +264,10 @@ struct ProcessConfig { // run can write both. bool export_unmerged_partials = false; + // Diagnostic, stills: every integrated observation with what the merge made of it, as _obs.mtz + // (ObservationMtzTable; --export-observations). Off by default; it does not change the processing. + bool export_observations = false; + // If set, after merging, validate the merged intensities against this atomic model: compute // R-free with an optimized bulk solvent and write 2Fo-Fc / Fo-Fc maps (--model). The caller parses // it (ParseRugnuxModel, writer/RugnuxModelFile.h). @@ -460,6 +464,7 @@ struct RugnuxOutputs { std::optional solvent_build; // and the water build, in it std::optional unmerged; // _unmerged.mtz std::optional unmerged_partials; // _unmerged_partials.mtz + std::optional observations; // _obs.mtz (diagnostic) std::string detector_jpeg; // _detector.jpg; empty: none drawn std::optional plot; // _plot.txt // A run that indexed nothing, with no per-image analysis streamed (ProcessConfig::image_sink): the @@ -1604,6 +1609,9 @@ class Rugnux { // Stills: the rocking width measured before the image pass, on a sample of its frames, which every // frame is then predicted through (ImagePass) and the partialities are computed from (ScaleMerge). std::optional stills_run_width_; + // --export-observations: the written stills merge's verdict on every observation, and its resolution cut. + std::vector> observation_verdicts_; + std::optional observation_d_min_; static constexpr double STILLS_WINDOW_SIGMAS = 2.0; // the window's half-width, in rocking widths static constexpr int STILLS_WINDOW_MIN_FRAMES = 2000; // the sample: a tenth of the run, at least this static constexpr int STILLS_WINDOW_FRACTION = 10; diff --git a/rugnux/RugnuxOutputFiles.cpp b/rugnux/RugnuxOutputFiles.cpp index baebba6bd..6b1a0e2e7 100644 --- a/rugnux/RugnuxOutputFiles.cpp +++ b/rugnux/RugnuxOutputFiles.cpp @@ -123,6 +123,10 @@ void WriteOutputs(const RugnuxOutputs &o, const std::string &prefix, const Outpu WriteMtz(*o.unmerged_partials, prefix + "_unmerged_partials.mtz"); logger.Info("Unmerged partials written to {}_unmerged_partials.mtz", prefix); } + if (o.observations) { + WriteMtz(*o.observations, prefix + "_obs.mtz"); + logger.Info("Observations written to {}_obs.mtz", prefix); + } if (!o.detector_jpeg.empty()) { const std::string path = prefix + "_detector.jpg"; std::ofstream f(path, std::ofstream::out | std::ofstream::trunc | std::ofstream::binary); diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index a83d85cfa..77bb852d4 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -1768,6 +1768,8 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { em.a, em.b, em.isa, merge_engine.ErrorModelChi2()); } merge_engine.AddImages(merge_input); + if (config_.export_observations && !for_search && write_output) + observation_verdicts_ = merge_engine.Verdicts(merge_input); ScaleMergeResult out; out.merged = merge_engine.ExportReflections(); @@ -1782,6 +1784,8 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { const std::optional effective_d_min = ApplyResolutionCutoff( out.merged, cut_ss.GetHighResolutionLimit_A(), cut_ss.GetResolutionCutoff(), cut_ss.GetResolutionCCTarget(), for_search, logger, &result.resolution_fit_A); + if (config_.export_observations && !for_search && write_output) + observation_d_min_ = effective_d_min; phase("Computing statistics"); TimingMark stats_mark(logger, "stills statistics " + label); diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index db36abeea..4af7c6ede 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -234,6 +234,7 @@ enum { OPT_EXPORT_UNMERGED, OPT_NO_EXPORT_UNMERGED, OPT_EXPORT_UNMERGED_PARTIALS, + OPT_EXPORT_OBSERVATIONS, OPT_DUMP_OBSERVATIONS, OPT_INTEGRATOR, OPT_SIMPLE_STILLS, @@ -317,6 +318,7 @@ static option long_options[] = { {"export-unmerged", no_argument, nullptr, OPT_EXPORT_UNMERGED}, {"no-export-unmerged", no_argument, nullptr, OPT_NO_EXPORT_UNMERGED}, {"export-unmerged-partials", no_argument, nullptr, OPT_EXPORT_UNMERGED_PARTIALS}, + {"export-observations", no_argument, nullptr, OPT_EXPORT_OBSERVATIONS}, {"dump-observations", required_argument, nullptr, OPT_DUMP_OBSERVATIONS}, {"space-group", required_argument, nullptr, 'S'}, {"anomalous", no_argument, nullptr, 'A'}, @@ -911,6 +913,7 @@ struct RugnuxCli { double refine_d_min = 0; // --refine-d-min bool export_unmerged = true; // on by default; --no-export-unmerged turns it off bool export_unmerged_partials = false; // --export-unmerged-partials: the same, one row per image + bool export_observations = false; // --export-observations: _obs.mtz, stills diagnostic std::string dump_observations; // diagnostic: dump unmerged -P rot3d fulls to this path std::optional min_partiality_arg; // --min-partiality; default per mode (RugnuxDefaultScalingSettings) std::optional min_captured_fraction_arg; // explicit --min-captured-fraction; default depends on rotation @@ -1261,6 +1264,9 @@ int RugnuxCli::Parse(int argc, char **argv) { case OPT_EXPORT_UNMERGED_PARTIALS: export_unmerged_partials = true; break; + case OPT_EXPORT_OBSERVATIONS: + export_observations = true; + break; case OPT_DUMP_OBSERVATIONS: dump_observations = optarg; break; @@ -2541,7 +2547,7 @@ int RugnuxInputRun::PrepareRugnuxInput() { for (auto &r: o.reflections) { const Coord S = o.latt.Astar() * static_cast(r.h) + o.latt.Bstar() * static_cast(r.k) + o.latt.Cstar() * static_cast(r.l) + S0; - r.dist_ewald = static_cast(std::fabs(std::sqrt(S.x * S.x + S.y * S.y + S.z * S.z) - inv_lambda)); + r.dist_ewald = static_cast(std::sqrt(S.x * S.x + S.y * S.y + S.z * S.z) - inv_lambda); r.partiality = 1.0f; } } @@ -3279,6 +3285,9 @@ int RugnuxInputRun::PrepareRugnuxInput() { } if (has_goniometer) logger.Info("--force-still: treating the rotation dataset as independent stills"); + // Without the axis, as on serial data: otherwise the prediction window, the mosaicity and the + // unmerged export still read the frames as a sweep (summing neighbouring stills into one event). + experiment.Goniometer(std::nullopt); } else if (!rotation_indexing && has_goniometer) { rotation_indexing = true; two_pass_rotation = true; @@ -3602,6 +3611,7 @@ int RugnuxInputRun::PrepareRugnuxInput() { std::ofstream(path) << text; }; config.export_unmerged = export_unmerged; + config.export_observations = export_observations; config.export_unmerged_partials = export_unmerged_partials; config.model = model; config.force_mr = !mr_model.empty(); diff --git a/tests/CalcBraggPredictionTest.cpp b/tests/CalcBraggPredictionTest.cpp index 22d65dc01..9c071c358 100644 --- a/tests/CalcBraggPredictionTest.cpp +++ b/tests/CalcBraggPredictionTest.cpp @@ -122,7 +122,7 @@ TEST_CASE("BraggPrediction_11keV", "[portable]") { REQUIRE(std::abs(r.l) < settings.max_l ); REQUIRE(r.d >= settings.high_res_A); REQUIRE(r.d == Catch::Approx(1/std::sqrt(recip * recip)).margin(0.01f)); - REQUIRE(r.dist_ewald == Catch::Approx(std::abs(geom.DistFromEwaldSphere(recip))).epsilon(1e-4)); + REQUIRE(r.dist_ewald == Catch::Approx(geom.DistFromEwaldSphere(recip)).epsilon(1e-4)); auto [x,y] = geom.RecipToDetector(recip); REQUIRE(r.predicted_x == Catch::Approx(x).margin(0.01)); REQUIRE(r.predicted_y == Catch::Approx(y).margin(0.01)); @@ -158,7 +158,7 @@ TEST_CASE("BraggPrediction_15keV") { REQUIRE(std::abs(r.l) < settings.max_l ); REQUIRE(r.d >= settings.high_res_A); REQUIRE(r.d == Catch::Approx(1/std::sqrt(recip * recip)).margin(0.01f)); - REQUIRE(r.dist_ewald == Catch::Approx(std::abs(geom.DistFromEwaldSphere(recip))).epsilon(1e-3)); + REQUIRE(r.dist_ewald == Catch::Approx(geom.DistFromEwaldSphere(recip)).epsilon(1e-3)); auto [x,y] = geom.RecipToDetector(recip); REQUIRE(r.predicted_x == Catch::Approx(x).margin(0.01)); REQUIRE(r.predicted_y == Catch::Approx(y).margin(0.01)); @@ -195,7 +195,7 @@ TEST_CASE("BraggPrediction_Rot1_Rot2", "[portable]") { REQUIRE(std::abs(r.l) < settings.max_l ); REQUIRE(r.d >= settings.high_res_A); REQUIRE(r.d == Catch::Approx(1/std::sqrt(recip * recip)).margin(0.01f)); - REQUIRE(r.dist_ewald == Catch::Approx(std::abs(geom.DistFromEwaldSphere(recip))).epsilon(1e-4)); + REQUIRE(r.dist_ewald == Catch::Approx(geom.DistFromEwaldSphere(recip)).epsilon(1e-4)); auto [x,y] = geom.RecipToDetector(recip); REQUIRE(r.predicted_x == Catch::Approx(x).margin(0.01)); REQUIRE(r.predicted_y == Catch::Approx(y).margin(0.01)); @@ -419,15 +419,15 @@ TEST_CASE("BraggPrediction_WindowFollowsResolution", "[portable]") { std::set> expected, got; for (int i = 0; i < n_wide; ++i) { const auto &r = pred_wide.GetReflections()[i]; - if (r.dist_ewald <= window(r)) + if (std::fabs(r.dist_ewald) <= window(r)) expected.insert({r.h, r.k, r.l}); } int beyond_cutoff = 0, low_res = 0; for (int i = 0; i < n_curve; ++i) { const auto &r = pred_curve.GetReflections()[i]; got.insert({r.h, r.k, r.l}); - CHECK(r.dist_ewald <= window(r)); - if (r.dist_ewald > curve.ewald_dist_cutoff) ++beyond_cutoff; + CHECK(std::fabs(r.dist_ewald) <= window(r)); + if (std::fabs(r.dist_ewald) > curve.ewald_dist_cutoff) ++beyond_cutoff; if (r.d > 6.0f) ++low_res; } CHECK(got == expected); @@ -492,7 +492,7 @@ TEST_CASE("BraggPrediction_GrowsPastStartingCapacity", "[portable]") { const auto &r = grown->GetReflections()[i]; kept.insert({r.h, r.k, r.l, r.delta_phi_deg}); worst_partiality = std::min(worst_partiality, r.partiality); - worst_dist_ewald = std::max(worst_dist_ewald, r.dist_ewald); + worst_dist_ewald = std::max(worst_dist_ewald, std::fabs(r.dist_ewald)); } for (const auto &r : all) { if (kept.contains({r.h, r.k, r.l, r.delta_phi_deg})) @@ -500,7 +500,7 @@ TEST_CASE("BraggPrediction_GrowsPastStartingCapacity", "[portable]") { if (rotation) REQUIRE(r.partiality <= worst_partiality); else - REQUIRE(r.dist_ewald >= worst_dist_ewald); + REQUIRE(std::fabs(r.dist_ewald) >= worst_dist_ewald); } } } @@ -628,7 +628,7 @@ TEST_CASE("BraggPredictionGPU") { REQUIRE(std::abs(r.l) < settings.max_l ); REQUIRE(r.d >= settings.high_res_A); REQUIRE(r.d == Catch::Approx(1/std::sqrt(recip * recip)).margin(0.01f)); - REQUIRE(r.dist_ewald == Catch::Approx(std::abs(geom.DistFromEwaldSphere(recip))).epsilon(2e-2)); + REQUIRE(r.dist_ewald == Catch::Approx(geom.DistFromEwaldSphere(recip)).epsilon(2e-2)); auto [x,y] = geom.RecipToDetector(recip); REQUIRE(r.predicted_x == Catch::Approx(x).margin(0.05)); REQUIRE(r.predicted_y == Catch::Approx(y).margin(0.05)); @@ -667,7 +667,7 @@ TEST_CASE("BraggPredictionGPU_Rot1_Rot2") { REQUIRE(std::abs(r.l) < settings.max_l ); REQUIRE(r.d >= settings.high_res_A); REQUIRE(r.d == Catch::Approx(1/std::sqrt(recip * recip)).margin(0.01f)); - REQUIRE(r.dist_ewald == Catch::Approx(std::abs(geom.DistFromEwaldSphere(recip))).epsilon(1e-3)); + REQUIRE(r.dist_ewald == Catch::Approx(geom.DistFromEwaldSphere(recip)).epsilon(1e-3)); auto [x,y] = geom.RecipToDetector(recip); REQUIRE(r.predicted_x == Catch::Approx(x).margin(0.05)); REQUIRE(r.predicted_y == Catch::Approx(y).margin(0.05)); From b6b83b2a22fa10349ba0b742cee60898cb60fb2d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 07:48:00 +0100 Subject: [PATCH 02/34] Stills: --stills-window-sigmas sets the prediction window's half-width The stills prediction window was a fixed two run rocking widths. --stills-window-sigmas K predicts, integrates and (with --export-observations) exports the rocking curve's tails out to K widths, which is what a partiality model fitted from the data needs to see; --min-partiality decides separately what the merge takes. The default stays 2 and the processing is unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- rugnux/Rugnux.h | 5 ++++- rugnux/RugnuxImagePass.cpp | 6 +++--- rugnux/rugnux_cli.cpp | 8 ++++++++ 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 1ee456478..f18016ad7 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -268,6 +268,10 @@ struct ProcessConfig { // (ObservationMtzTable; --export-observations). Off by default; it does not change the processing. bool export_observations = false; + // Stills: the prediction window's half-width, in run rocking widths (--stills-window-sigmas). Wider + // predicts, integrates and exports more of the rocking curve's tails. + double stills_window_sigmas = 2.0; + // If set, after merging, validate the merged intensities against this atomic model: compute // R-free with an optimized bulk solvent and write 2Fo-Fc / Fo-Fc maps (--model). The caller parses // it (ParseRugnuxModel, writer/RugnuxModelFile.h). @@ -1612,7 +1616,6 @@ class Rugnux { // --export-observations: the written stills merge's verdict on every observation, and its resolution cut. std::vector> observation_verdicts_; std::optional observation_d_min_; - static constexpr double STILLS_WINDOW_SIGMAS = 2.0; // the window's half-width, in rocking widths static constexpr int STILLS_WINDOW_MIN_FRAMES = 2000; // the sample: a tenth of the run, at least this static constexpr int STILLS_WINDOW_FRACTION = 10; // Stills: where the frames' pixel values pile up, against the file's saturation limit (see there). diff --git a/rugnux/RugnuxImagePass.cpp b/rugnux/RugnuxImagePass.cpp index 92edbe15f..f9509795e 100644 --- a/rugnux/RugnuxImagePass.cpp +++ b/rugnux/RugnuxImagePass.cpp @@ -260,7 +260,7 @@ void Rugnux::ImagePass(PipelineLocals &p) { // predicted five to eight widths at low resolution, mostly empty, and cut the curve at one or two at // high resolution, so the partials there were never integrated. So the run's width is measured first, // on a sample of the frames integrated the old way (MeasureRunWidth, as the merge measures it), and - // every frame is then predicted through STILLS_WINDOW_SIGMAS of it. + // every frame is then predicted through config_.stills_window_sigmas of it. std::vector window_frames; const bool stills_window = full && per_image_analysis && !calibration && !geometry_prepass && indexer && !experiment_.IsRotationIndexing() && config_.spot_finding.indexing @@ -452,13 +452,13 @@ void Rugnux::ImagePass(PipelineLocals &p) { for (const int o: window_frames) outcomes[o] = IntegrationOutcome{}; if (stills_run_width_) { - const auto k = static_cast(STILLS_WINDOW_SIGMAS); + const auto k = static_cast(config_.stills_window_sigmas); indexer->SetStillsWindow(StillsWindow{.c0 = k * static_cast(stills_run_width_->gamma0), .ang = k * static_cast(stills_run_width_->gamma_e), .bw = k * static_cast(stills_run_width_->bandwidth_sigma), .q_scale = static_cast(stills_run_width_->radial_scale)}); Note(fmt::format("prediction window {:.1f} x the rocking width measured on {} frames, centred at " - "radial scale {:+.2e}", STILLS_WINDOW_SIGMAS, window_frames.size(), radial_scale - 1.0)); + "radial scale {:+.2e}", config_.stills_window_sigmas, window_frames.size(), radial_scale - 1.0)); } } for (int i = 0; i < image_workers; ++i) diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 4af7c6ede..7cec8bc94 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -235,6 +235,7 @@ enum { OPT_NO_EXPORT_UNMERGED, OPT_EXPORT_UNMERGED_PARTIALS, OPT_EXPORT_OBSERVATIONS, + OPT_STILLS_WINDOW_SIGMAS, OPT_DUMP_OBSERVATIONS, OPT_INTEGRATOR, OPT_SIMPLE_STILLS, @@ -319,6 +320,7 @@ static option long_options[] = { {"no-export-unmerged", no_argument, nullptr, OPT_NO_EXPORT_UNMERGED}, {"export-unmerged-partials", no_argument, nullptr, OPT_EXPORT_UNMERGED_PARTIALS}, {"export-observations", no_argument, nullptr, OPT_EXPORT_OBSERVATIONS}, + {"stills-window-sigmas", required_argument, nullptr, OPT_STILLS_WINDOW_SIGMAS}, {"dump-observations", required_argument, nullptr, OPT_DUMP_OBSERVATIONS}, {"space-group", required_argument, nullptr, 'S'}, {"anomalous", no_argument, nullptr, 'A'}, @@ -914,6 +916,7 @@ struct RugnuxCli { bool export_unmerged = true; // on by default; --no-export-unmerged turns it off bool export_unmerged_partials = false; // --export-unmerged-partials: the same, one row per image bool export_observations = false; // --export-observations: _obs.mtz, stills diagnostic + std::optional stills_window_sigmas_arg; // --stills-window-sigmas std::string dump_observations; // diagnostic: dump unmerged -P rot3d fulls to this path std::optional min_partiality_arg; // --min-partiality; default per mode (RugnuxDefaultScalingSettings) std::optional min_captured_fraction_arg; // explicit --min-captured-fraction; default depends on rotation @@ -1267,6 +1270,9 @@ int RugnuxCli::Parse(int argc, char **argv) { case OPT_EXPORT_OBSERVATIONS: export_observations = true; break; + case OPT_STILLS_WINDOW_SIGMAS: + stills_window_sigmas_arg = parse_double_arg(optarg, "--stills-window-sigmas", logger); + break; case OPT_DUMP_OBSERVATIONS: dump_observations = optarg; break; @@ -3612,6 +3618,8 @@ int RugnuxInputRun::PrepareRugnuxInput() { }; config.export_unmerged = export_unmerged; config.export_observations = export_observations; + if (stills_window_sigmas_arg) + config.stills_window_sigmas = *stills_window_sigmas_arg; config.export_unmerged_partials = export_unmerged_partials; config.model = model; config.force_mr = !mr_model.empty(); From f7bf79a156d6565cca06e09d0315461214d447b9 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 08:14:44 +0100 Subject: [PATCH 03/34] rugnux --mode scale: --force-still drops the goniometer too Re-scaling a rotation run's _process.h5 with --force-still kept the axis, so the unmerged export read the stills as a sweep: neighbouring frames summed into "rocking events" (FRACTIONCALC above 1), the stills cell check skipped, ROT taken from the axis. As in --mode mx, the goniometer is now removed. On 300 frames of a 0.2 deg sweep the unmerged file goes from 13,611 summed rows to 154,656 rows, one per observation; the merged reflections are unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- rugnux/rugnux_cli.cpp | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 7cec8bc94..6d98e9a00 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -2307,6 +2307,11 @@ int RugnuxInputRun::PrepareRugnuxInput() { // A rotation (goniometer) dataset uses RotationScaleMerge unless --force-still asks for stills scaling. IndexingSettings indexing_settings; indexing_settings.RotationIndexing(experiment.GetGoniometer().has_value() && !force_still); + const bool rot = experiment.GetGoniometer().has_value() && !force_still; + // Without the axis under --force-still, as in --mode mx: the unmerged export would otherwise read + // the frames as a sweep. + if (force_still) + experiment.Goniometer(std::nullopt); // --detect-ice-rings, applied here as well as on the full path below: this block returns // before that one runs, so without it the flag is silently ignored by --scale. Same @@ -2320,7 +2325,6 @@ int RugnuxInputRun::PrepareRugnuxInput() { // Start from the same defaults the full pipeline uses, so --scale reproduces the merge that wrote // the _process.h5 rather than a weaker model of its own. Every CLI override below then applies on // top, exactly as in the full-analysis path. - const bool rot = experiment.GetGoniometer().has_value() && !force_still; ScalingSettings scaling_settings = RugnuxDefaultScalingSettings(rot); if (d_min_scale_merge) scaling_settings.HighResolutionLimit_A(d_min_scale_merge.value()); From 3f337da4d8dd494194adde584b93dfa5c389a9f4 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 13:22:33 +0100 Subject: [PATCH 04/34] Pre-scan: mask unstable pixels, which read far below zero on frame after frame No photon makes a count negative and a reflection only adds counts, so a pixel reading below -(10 x the Poisson noise of its ring's level + 10 photons) on at least a twentieth of the pre-scan frames (and on two) is defective, whatever it reads in between. A column of a JUNGFRAU 9M swung between -2500 and +1000 counts from frame to frame with a normal median: its negative frames dragged the background ring of every reflection near it far below zero, a disk summing to nothing was then integrated huge with a one-photon sigma, and on 1500 frames of a jet run such observations zeroed the 3.4-4.9 A shell (CCref there 0.30 -> 0.35, overall 0.12 -> 0.55 with the column masked). The rule sits beside the dead one, on the host and the device alike; real reflections cannot trigger it. On rotation and serial sets from EIGER, JUNGFRAU 4M and 16M detectors it masks nothing and the merges are bit-identical. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/hot_pixels/HotPixels.cpp | 17 ++++++++-- image_analysis/hot_pixels/HotPixels.h | 25 ++++++++++++-- image_analysis/hot_pixels/HotPixelsGPU.cu | 40 +++++++++++++++-------- image_analysis/hot_pixels/HotPixelsGPU.h | 8 +++-- rugnux/Rugnux.h | 2 +- rugnux/RugnuxPreScan.cpp | 9 ++--- tests/HotPixelFinderTest.cpp | 11 +++++-- 7 files changed, 85 insertions(+), 27 deletions(-) diff --git a/image_analysis/hot_pixels/HotPixels.cpp b/image_analysis/hot_pixels/HotPixels.cpp index 2a516db88..2f56213f1 100644 --- a/image_analysis/hot_pixels/HotPixels.cpp +++ b/image_analysis/hot_pixels/HotPixels.cpp @@ -83,6 +83,7 @@ HotPixelFinder::HotPixelFinder(const DiffractionExperiment &experiment, const Pi key(std::make_unique_for_overwrite(width * height)), n_lit(std::make_unique_for_overwrite(width * height)), n_error(std::make_unique_for_overwrite(width * height)), + n_neg(std::make_unique_for_overwrite(width * height)), sum_value(std::make_unique_for_overwrite(width * height)), n_error_ring_ok(std::make_unique_for_overwrite(width * height)), error_level_sum(std::make_unique_for_overwrite(width * height)) { @@ -91,7 +92,7 @@ HotPixelFinder::HotPixelFinder(const DiffractionExperiment &experiment, const Pi const float ring_rad = RING_WIDTH_PX * geometry.GetPixelSize_mm() / geometry.GetDetectorDistance_mm(); ParallelChunks(static_cast(height), nthreads, [&](int y0, int y1) { for (size_t i = y0 * width; i < y1 * width; i++) { - n_lit[i] = n_error[i] = n_error_ring_ok[i] = 0; + n_lit[i] = n_error[i] = n_neg[i] = n_error_ring_ok[i] = 0; sum_value[i] = error_level_sum[i] = 0; key[i] = -1; } @@ -217,6 +218,8 @@ void HotPixelFinder::AddImage(const int32_t *image, std::vector &scratc sum_value[i] += v; if (static_cast(v) > threshold[k]) n_lit[i]++; + if (static_cast(v) < NegativeLimit(level[k], gain)) + n_neg[i]++; } } } @@ -270,7 +273,9 @@ void HotPixelFinder::PrepareDevice() { set_gpu(c); return std::make_unique(key.get(), width * height, key_begin, nrings, SECTORS, HotPixelLevelRules{MIN_SECTOR_PIXELS, MIN_RING_PIXELS, - LIT_NSIGMA, LIT_OFFSET * gain, gain}); + LIT_NSIGMA, LIT_OFFSET * gain, gain, + NEG_NSIGMA, NEG_OFFSET * gain, + NEG_FRACTION_INV, NEG_MIN_FRAMES}); })); for (auto &b : built) gpus.push_back(b.get()); @@ -368,6 +373,7 @@ HotPixelFinder::Result HotPixelFinder::GetMask(double oscillation_deg, double sp const size_t i = c.index[j]; sums_known[i] = 1; n_lit[i] = c.n_lit[j]; + n_neg[i] = c.n_neg[j]; n_error[i] = c.n_error[j]; n_error_ring_ok[i] = c.n_error_ring_ok[j]; sum_value[i] = c.sum_value[j]; @@ -412,6 +418,12 @@ HotPixelFinder::Result HotPixelFinder::GetMask(double oscillation_deg, double sp ret.mask[i] = 2; continue; } + // Unstable: far below zero on a twentieth of its frames, which no count of photons does. + if (sums_known[i] && n_valid(i) >= min_valid && n_neg[i] >= NEG_MIN_FRAMES + && NEG_FRACTION_INV * n_neg[i] >= n_valid(i)) { + ret.mask[i] = 4; + continue; + } // Dead: the mirror of hot. A pixel reading at most half of its own sector's level, by // more than the bound a hot pixel's excess has to clear, is not recording the light that // reaches it, and every reflection whose disk covers it is integrated short by that @@ -440,6 +452,7 @@ HotPixelFinder::Result HotPixelFinder::GetMask(double oscillation_deg, double sp ret.hot += v == 1; ret.error += v == 2; ret.dead += v == 3; + ret.unstable += v == 4; } return ret; } diff --git a/image_analysis/hot_pixels/HotPixels.h b/image_analysis/hot_pixels/HotPixels.h index c2357c553..270a9c911 100644 --- a/image_analysis/hot_pixels/HotPixels.h +++ b/image_analysis/hot_pixels/HotPixels.h @@ -42,11 +42,20 @@ // covers it is integrated short by the deficit. The detector's outermost row and column get no rule of // their own - a dead edge row, or a hot pixel there, is caught like any other. // +// An UNSTABLE pixel reads far below zero - ten times the Poisson noise of its ring's level and ten photons +// more under it - on at least a twentieth of its frames (and on two). No photon can make a count +// negative, and a reflection only adds counts, so such a pixel is broken whatever it reads in between: +// measured, a column of a JUNGFRAU 9M swinging between -2500 and +1000 counts from frame to frame, +// with a normal median. Its negative frames drag the background ring of every reflection near it far +// below zero, and a reflection whose disk then sums to nothing is integrated huge with a one-photon sigma. +// // A CCD is read with an offset and is not Poisson, but neither test needs it to be: a dead CCD pixel // reads its offset, far below a ring with any background on it, and a hot one ten times its ring. // ============================================================================= #include +#include +#include #include #include #include @@ -82,14 +91,21 @@ public: // strength at which the pixels behind the merged outliers were found, and well clear of a pixel // that is merely a little over-responding. static constexpr double STRONG_RATIO = 10.0; + // Unstable: below -(NEG_NSIGMA * noise of the level + NEG_OFFSET photons) on at least one frame in + // NEG_FRACTION_INV, and on NEG_MIN_FRAMES. + static constexpr float NEG_NSIGMA = 10.0f; + static constexpr float NEG_OFFSET = 10.0f; + static constexpr int NEG_FRACTION_INV = 20; + static constexpr int NEG_MIN_FRAMES = 2; // The most frames a finder can be given: its per-pixel counters are 16-bit. static constexpr int MAX_SAMPLED_FRAMES = UINT16_MAX; struct Result { - std::vector mask; // non-zero = masked (1 hot, 2 error value, 3 dead), converted geometry + std::vector mask; // non-zero = masked (1 hot, 2 error value, 3 dead, 4 unstable), converted geometry size_t hot = 0; // persistent above the ring size_t error = 0; // holding the error value or the saturation limit on most frames size_t dead = 0; // reading at most half of its sector's level + size_t unstable = 0; // far below zero on a twentieth of its frames uint32_t frames = 0; }; @@ -139,7 +155,7 @@ private: std::atomic next_band{0}; mutable std::mutex m; uint32_t frames = 0; - std::unique_ptr n_lit, n_error; + std::unique_ptr n_lit, n_error, n_neg; std::unique_ptr sum_value; // A pixel's valid frames and the sum of its ring-sector level over them, kept per ring-sector - the // frames its ring had a level on, and the sum of the level over those - less what the pixel's own @@ -167,6 +183,11 @@ private: const std::vector &ring_spread, const std::vector &ring_ok, const std::vector &local_level, std::vector &level, std::vector &threshold); + // The count below which a pixel of level `level` is unstable on a frame - one rounding for the fma, as + // on the device (HotPixelsGPU.cu). + [[nodiscard]] static float NegativeLimit(int32_t level, float gain) { + return -std::fma(NEG_NSIGMA, std::sqrt(gain * std::max(static_cast(level), gain)), NEG_OFFSET * gain); + } [[nodiscard]] int n_valid(size_t i) const { return key_frames[key[i]] - n_error_ring_ok[i]; } [[nodiscard]] int64_t sum_level(size_t i) const { return key_level_sum[key[i]] - error_level_sum[i]; } }; diff --git a/image_analysis/hot_pixels/HotPixelsGPU.cu b/image_analysis/hot_pixels/HotPixelsGPU.cu index 8b6e3ea4b..672196f10 100644 --- a/image_analysis/hot_pixels/HotPixelsGPU.cu +++ b/image_analysis/hot_pixels/HotPixelsGPU.cu @@ -123,11 +123,11 @@ __global__ void ring_kernel(const int32_t *__restrict__ image, const uint32_t *_ // HotPixelFinder::AddImage's per-pixel loop, line for line. __global__ void accumulate_kernel(const int32_t *__restrict__ image, const int32_t *__restrict__ key, - size_t npixels, int sectors, const int32_t *__restrict__ level, + size_t npixels, int sectors, HotPixelLevelRules rules, const int32_t *__restrict__ level, const float *__restrict__ threshold, const char *__restrict__ ring_ok, uint16_t *__restrict__ n_lit, uint16_t *__restrict__ n_error, int64_t *__restrict__ sum_value, uint16_t *__restrict__ n_error_ring_ok, - int64_t *__restrict__ error_level_sum) { + int64_t *__restrict__ error_level_sum, uint16_t *__restrict__ n_neg) { for (size_t i = blockIdx.x * static_cast(blockDim.x) + threadIdx.x; i < npixels; i += static_cast(blockDim.x) * gridDim.x) { const int32_t k = key[i], v = image[i]; @@ -144,6 +144,12 @@ __global__ void accumulate_kernel(const int32_t *__restrict__ image, const int32 sum_value[i] += v; if (static_cast(v) > threshold[k]) n_lit[i]++; + // HotPixelFinder::NegativeLimit, with its roundings + const float neg = -__fmaf_rn(rules.neg_nsigma, + __fsqrt_rn(__fmul_rn(rules.gain, fmaxf(static_cast(level[k]), rules.gain))), + rules.neg_offset); + if (static_cast(v) < neg) + n_neg[i]++; } } @@ -202,12 +208,12 @@ __global__ void chance_kernel(size_t npixels, int sectors, const int32_t *__rest } // Writes the candidates' indices from `out` on when `out` is given, and counts them either way. -__global__ void candidate_kernel(size_t npixels, int sectors, uint32_t frames, int min_valid, bool spacing_ok, +__global__ void candidate_kernel(size_t npixels, int sectors, HotPixelLevelRules rules, uint32_t frames, int min_valid, bool spacing_ok, const int32_t *__restrict__ key, const uint32_t *__restrict__ key_frames, const uint16_t *__restrict__ n_lit, const uint16_t *__restrict__ n_error, const uint16_t *__restrict__ n_error_ring_ok, const int64_t *__restrict__ sum_value, const int64_t *__restrict__ key_local_level_sum, const int *__restrict__ k_chance, - uint32_t *__restrict__ count, uint32_t *__restrict__ out) { + const uint16_t *__restrict__ n_neg, uint32_t *__restrict__ count, uint32_t *__restrict__ out) { const size_t i = blockIdx.x * static_cast(blockDim.x) + threadIdx.x; if (i >= npixels || key[i] < 0) return; const int nv = valid_frames(key, key_frames, n_error_ring_ok, i); @@ -217,7 +223,10 @@ __global__ void candidate_kernel(size_t npixels, int sectors, uint32_t frames, i && lit >= min(max(2, k_chance[key[i] / sectors]), nv); // Can be dead (HotPixelFinder::GetMask): at most half of its sector's level, no error frames. const bool dark = n_error[i] == 0 && nv >= min_valid && 2 * sum_value[i] <= key_local_level_sum[key[i]]; - if (!error && !persistent && !dark) return; + // Can be unstable (HotPixelFinder::GetMask): far below zero on a twentieth of its valid frames. + const bool unstable = nv >= min_valid && n_neg[i] >= rules.neg_min_frames + && rules.neg_fraction_inv * static_cast(n_neg[i]) >= nv; + if (!error && !persistent && !dark && !unstable) return; const uint32_t slot = atomicAdd(count, 1u); if (out) out[slot] = static_cast(i); } @@ -256,7 +265,7 @@ HotPixelFinderGPU::HotPixelFinderGPU(const int32_t *host_key, size_t npixels, sectors(sectors), rules(rules), key(npixels, ALLOC), pixels_by_key(std::max(host_key_begin.back(), 1), ALLOC), key_begin(host_key_begin.size(), ALLOC), - n_lit(npixels, ALLOC), n_error(npixels, ALLOC), n_error_ring_ok(npixels, ALLOC), + n_lit(npixels, ALLOC), n_error(npixels, ALLOC), n_error_ring_ok(npixels, ALLOC), n_neg(npixels, ALLOC), sum_value(npixels, ALLOC), error_level_sum(npixels, ALLOC), key_frames(std::max(nkeys, 1), ALLOC), key_level_sum(std::max(nkeys, 1), ALLOC), key_local_level_sum(std::max(nkeys, 1), ALLOC) { @@ -269,6 +278,7 @@ HotPixelFinderGPU::HotPixelFinderGPU(const int32_t *host_key, size_t npixels, cuda_err(cudaMemsetAsync(n_lit, 0, npixels * sizeof(uint16_t), stream)); cuda_err(cudaMemsetAsync(n_error, 0, npixels * sizeof(uint16_t), stream)); cuda_err(cudaMemsetAsync(n_error_ring_ok, 0, npixels * sizeof(uint16_t), stream)); + cuda_err(cudaMemsetAsync(n_neg, 0, npixels * sizeof(uint16_t), stream)); cuda_err(cudaMemsetAsync(sum_value, 0, npixels * sizeof(int64_t), stream)); cuda_err(cudaMemsetAsync(error_level_sum, 0, npixels * sizeof(int64_t), stream)); cuda_err(cudaMemsetAsync(key_frames, 0, std::max(nkeys, 1) * sizeof(uint32_t), stream)); @@ -326,8 +336,8 @@ void HotPixelFinderGPU::Add(const int32_t *device_image, Frame &frame) { std::lock_guard lock(accumulate_mutex); cuda_err(cudaStreamWaitEvent(stream, last_accumulate, 0)); accumulate_kernel<<((npixels + THREADS - 1) / THREADS), THREADS, 0, stream>>>( - device_image, key, npixels, sectors, frame.level, frame.threshold, frame.ring_ok, - n_lit, n_error, sum_value, n_error_ring_ok, error_level_sum); + device_image, key, npixels, sectors, rules, frame.level, frame.threshold, frame.ring_ok, + n_lit, n_error, sum_value, n_error_ring_ok, error_level_sum, n_neg); cuda_err(cudaGetLastError()); cuda_err(cudaEventRecord(last_accumulate, stream)); } @@ -348,6 +358,7 @@ void HotPixelFinderGPU::Fold(HotPixelFinderGPU &other) { add(n_lit.get(), other.n_lit.get(), npixels); add(n_error.get(), other.n_error.get(), npixels); add(n_error_ring_ok.get(), other.n_error_ring_ok.get(), npixels); + add(n_neg.get(), other.n_neg.get(), npixels); add(sum_value.get(), other.sum_value.get(), npixels); add(error_level_sum.get(), other.error_level_sum.get(), npixels); add(key_frames.get(), other.key_frames.get(), nkeys); @@ -385,9 +396,9 @@ HotPixelFinderGPU::Candidates HotPixelFinderGPU::GetCandidates(uint32_t frames, cuda_err(cudaMemcpyAsync(d_k_chance, k_chance.data(), k_chance.size() * sizeof(int), cudaMemcpyHostToDevice, stream)); cuda_err(cudaMemsetAsync(d_count, 0, sizeof(uint32_t), stream)); - candidate_kernel<<>>(npixels, sectors, frames, min_valid, spacing_ok, key, key_frames, + candidate_kernel<<>>(npixels, sectors, rules, frames, min_valid, spacing_ok, key, key_frames, n_lit, n_error, n_error_ring_ok, sum_value, key_local_level_sum, - d_k_chance, d_count, nullptr); + d_k_chance, n_neg, d_count, nullptr); cuda_err(cudaGetLastError()); uint32_t n = 0; cuda_err(cudaMemcpyAsync(&n, d_count, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream)); @@ -407,17 +418,18 @@ HotPixelFinderGPU::Candidates HotPixelFinderGPU::GetCandidates(uint32_t frames, } if (n > 0) { CudaDevicePtr index(n, ALLOC); - CudaDevicePtr g_lit(n, ALLOC), g_error(n, ALLOC), g_error_ring_ok(n, ALLOC); + CudaDevicePtr g_lit(n, ALLOC), g_error(n, ALLOC), g_error_ring_ok(n, ALLOC), g_neg(n, ALLOC); CudaDevicePtr g_sum(n, ALLOC), g_error_level(n, ALLOC); cuda_err(cudaMemsetAsync(d_count, 0, sizeof(uint32_t), stream)); - candidate_kernel<<>>(npixels, sectors, frames, min_valid, spacing_ok, key, + candidate_kernel<<>>(npixels, sectors, rules, frames, min_valid, spacing_ok, key, key_frames, n_lit, n_error, n_error_ring_ok, sum_value, - key_local_level_sum, d_k_chance, d_count, index); + key_local_level_sum, d_k_chance, n_neg, d_count, index); cuda_err(cudaGetLastError()); const unsigned gb = (n + THREADS - 1) / THREADS; gather_kernel<<>>(n, index, n_lit.get(), g_lit.get()); gather_kernel<<>>(n, index, n_error.get(), g_error.get()); gather_kernel<<>>(n, index, n_error_ring_ok.get(), g_error_ring_ok.get()); + gather_kernel<<>>(n, index, n_neg.get(), g_neg.get()); gather_kernel<<>>(n, index, sum_value.get(), g_sum.get()); gather_kernel<<>>(n, index, error_level_sum.get(), g_error_level.get()); cuda_err(cudaGetLastError()); @@ -425,6 +437,7 @@ HotPixelFinderGPU::Candidates HotPixelFinderGPU::GetCandidates(uint32_t frames, c.n_lit.resize(n); c.n_error.resize(n); c.n_error_ring_ok.resize(n); + c.n_neg.resize(n); c.sum_value.resize(n); c.error_level_sum.resize(n); cuda_err(cudaMemcpyAsync(c.index.data(), index, n * sizeof(uint32_t), cudaMemcpyDeviceToHost, stream)); @@ -432,6 +445,7 @@ HotPixelFinderGPU::Candidates HotPixelFinderGPU::GetCandidates(uint32_t frames, cuda_err(cudaMemcpyAsync(c.n_error.data(), g_error, n * sizeof(uint16_t), cudaMemcpyDeviceToHost, stream)); cuda_err(cudaMemcpyAsync(c.n_error_ring_ok.data(), g_error_ring_ok, n * sizeof(uint16_t), cudaMemcpyDeviceToHost, stream)); + cuda_err(cudaMemcpyAsync(c.n_neg.data(), g_neg, n * sizeof(uint16_t), cudaMemcpyDeviceToHost, stream)); cuda_err(cudaMemcpyAsync(c.sum_value.data(), g_sum, n * sizeof(int64_t), cudaMemcpyDeviceToHost, stream)); cuda_err(cudaMemcpyAsync(c.error_level_sum.data(), g_error_level, n * sizeof(int64_t), cudaMemcpyDeviceToHost, stream)); diff --git a/image_analysis/hot_pixels/HotPixelsGPU.h b/image_analysis/hot_pixels/HotPixelsGPU.h index c490c9438..15d0d2e4b 100644 --- a/image_analysis/hot_pixels/HotPixelsGPU.h +++ b/image_analysis/hot_pixels/HotPixelsGPU.h @@ -18,6 +18,10 @@ struct HotPixelLevelRules { float lit_nsigma; float lit_offset; // in the frames' units float gain; // the frames' units per photon + float neg_nsigma; // HotPixelFinder::NegativeLimit + float neg_offset; // in the frames' units + int neg_fraction_inv; + int neg_min_frames; }; // The device half of HotPixelFinder, for frames already preprocessed on the GPU. Everything a frame adds @@ -41,7 +45,7 @@ class HotPixelFinderGPU { CudaDevicePtr key_begin; // where each key's pixels start in pixels_by_key // The per-pixel and per-key sums, exactly those of HotPixelFinder. - CudaDevicePtr n_lit, n_error, n_error_ring_ok; + CudaDevicePtr n_lit, n_error, n_error_ring_ok, n_neg; CudaDevicePtr sum_value, error_level_sum; CudaDevicePtr key_frames; CudaDevicePtr key_level_sum, key_local_level_sum; @@ -99,7 +103,7 @@ public: // per-pixel sums; and every key's sums. struct Candidates { std::vector index; - std::vector n_lit, n_error, n_error_ring_ok; + std::vector n_lit, n_error, n_error_ring_ok, n_neg; std::vector sum_value, error_level_sum; std::vector key_frames; std::vector key_level_sum, key_local_level_sum; diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index f18016ad7..9f0695f52 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -1501,7 +1501,7 @@ class Rugnux { struct Defects { size_t pixels = 0; std::vector> mask; // the mask's non-zero pixels - size_t hot = 0, error = 0, dead = 0; + size_t hot = 0, error = 0, dead = 0, unstable = 0; uint32_t frames = 0; }; struct Entry { diff --git a/rugnux/RugnuxPreScan.cpp b/rugnux/RugnuxPreScan.cpp index 4ad8c8be1..c43ca77ca 100644 --- a/rugnux/RugnuxPreScan.cpp +++ b/rugnux/RugnuxPreScan.cpp @@ -1266,10 +1266,11 @@ void Rugnux::MaskDefectivePixels(int start_image, const std::vector &sample for (const auto &[i, v] : defects->mask) mask[i] = v; pixel_mask_.LoadHotPixelMask(experiment_, mask); - const size_t masked = defects->hot + defects->error + defects->dead; + const size_t masked = defects->hot + defects->error + defects->dead + defects->unstable; logger.Info("Defective pixels: {} masked from {} pre-scan frames - {} lit above their ring on more " - "frames than one reflection or chance explains, {} holding the detector's error value or saturation limit, {} dead", - masked, defects->frames, defects->hot, defects->error, defects->dead); + "frames than one reflection or chance explains, {} holding the detector's error value or saturation limit, {} dead, " + "{} unstable (far below zero on a twentieth of the frames)", + masked, defects->frames, defects->hot, defects->error, defects->dead, defects->unstable); Note(fmt::format("{} defective pixel{} masked", masked, masked == 1 ? "" : "s")); }; if (defects) { @@ -1360,7 +1361,7 @@ void Rugnux::MaskDefectivePixels(int start_image, const std::vector &sample ? oscillation * (sample.back() - sample.front()) / static_cast(sample.size() - 1) : 0.0; const auto hot = finder.GetMask(oscillation, spacing, nthreads); - defects = PreScanMemo::Defects{hot.mask.size(), {}, hot.hot, hot.error, hot.dead, hot.frames}; + defects = PreScanMemo::Defects{hot.mask.size(), {}, hot.hot, hot.error, hot.dead, hot.unstable, hot.frames}; for (size_t i = 0; i < hot.mask.size(); i++) if (hot.mask[i] != 0) defects->mask.emplace_back(static_cast(i), hot.mask[i]); diff --git a/tests/HotPixelFinderTest.cpp b/tests/HotPixelFinderTest.cpp index da00f00cd..d22461ae5 100644 --- a/tests/HotPixelFinderTest.cpp +++ b/tests/HotPixelFinderTest.cpp @@ -31,7 +31,8 @@ namespace { // but fewer than the one-reflection bound at that zeta. Only the first, third and fourth are masked - and // a 3x3 patch reading high on every frame, which is stationary in the lab and so no reflection either. // Three dark pixels besides: one dead on the powder ring, which is masked, one on the ring reading 70% -// of it, and one dead on the 5-count background, whose deficit is too small to make an outlier. +// of it, and one dead on the 5-count background, whose deficit is too small to make an outlier. And an +// unstable pixel: normal, but at -2000 on every fourth frame, which no photons do - masked as unstable. TEST_CASE("HotPixelFinder_PersistentPixelNotBragg", "[HotPixelFinder]") { DiffractionExperiment x(DetDECTRIS(W, H, "Test detector", "")); x.IncidentEnergy_keV(WVL_1A_IN_KEV).DetectorDistance_mm(10.0f); @@ -42,7 +43,7 @@ TEST_CASE("HotPixelFinder_PersistentPixelNotBragg", "[HotPixelFinder]") { constexpr int HOT_X = 200, HOT_Y = 200, WARM_X = 190, WARM_Y = 60, ERR_X = 60, ERR_Y = 190; constexpr int BRAGG_X = 40, BRAGG_Y = 115, STUCK_X = 100, STUCK_Y = 30; - constexpr int PATCH_X = 150, PATCH_Y = 225; + constexpr int PATCH_X = 150, PATCH_Y = 225, UNSTABLE_X = 30, UNSTABLE_Y = 150; constexpr int DEAD_X = C + 82, DEAD_Y = C, DIM_X = C, DIM_Y = C + 82, DEAD_BKG_X = 20, DEAD_BKG_Y = 20; std::mt19937 rng(1); // The powder ring is a smooth radial profile, as a real one is: 45 counts over the background at @@ -67,12 +68,14 @@ TEST_CASE("HotPixelFinder_PersistentPixelNotBragg", "[HotPixelFinder]") { frame[I(DEAD_X, DEAD_Y)] = 0; frame[I(DIM_X, DIM_Y)] = static_cast(0.7 * frame[I(DIM_X, DIM_Y)]); frame[I(DEAD_BKG_X, DEAD_BKG_Y)] = 0; + if (f % 4 == 0) + frame[I(UNSTABLE_X, UNSTABLE_Y)] = -2000; finder.AddImage(frame.data(), scratch); } const auto result = finder.GetMask(OSC_DEG, SPACING_DEG, 4); CHECK(result.frames == NFRAMES); - CHECK(result.mask[I(HOT_X, HOT_Y)] == 1); // 1 = hot, 2 = error value, 3 = dead + CHECK(result.mask[I(HOT_X, HOT_Y)] == 1); // 1 = hot, 2 = error value, 3 = dead, 4 = unstable CHECK(result.mask[I(ERR_X, ERR_Y)] == 2); CHECK(result.mask[I(STUCK_X, STUCK_Y)] == 2); CHECK(result.mask[I(WARM_X, WARM_Y)] == 0); @@ -86,6 +89,8 @@ TEST_CASE("HotPixelFinder_PersistentPixelNotBragg", "[HotPixelFinder]") { CHECK(result.hot == 10); CHECK(result.error == 2); CHECK(result.dead == 1); + CHECK(result.mask[I(UNSTABLE_X, UNSTABLE_Y)] == 4); + CHECK(result.unstable == 1); } #ifdef JFJOCH_USE_CUDA From 2bae65c72e8d62044b0ff2c2a5a969e71a0180dc Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 12:54:02 +0100 Subject: [PATCH 05/34] Stills partiality: each crystal refines its radial scale and rocking width, held to the run's spread The post-refinement now fits, per crystal and beside the orientation tilt, a radial scale eta of the reciprocal lattice (q -> (1 + eta) q: the photon energy of an XFEL shot, or a cell the spot positions did not pin) and a scale of the run's rocking width (the crystal's own mosaicity and size). Each changes the partialities differently across resolution, so neither is degenerate with the scale G. Both are held by Gaussian priors about the run whose widths come from the data: a first cycle refines against loose priors, and the spread of those estimates over the crystals, less the uncertainty each carries from its own reflections, is the prior of the second. A run whose crystals agree (a synthetic test, or a stable beam) is held to the run's curve; one whose crystals differ is not. On the quick panel (one run per set, merged MTZ): XFEL serial 18k frames, S anomalous peaks mean 6.80 -> 7.35 (weakest site 2.64 -> 1.65), CCref last shell 0.657 -> 0.653; XFEL serial 1.5k frames CCref last shell 0.222 -> 0.252; monochromatic jet CCref last 0.405 -> 0.458 (overall 0.447 -> 0.396); pink jet 0.299 -> 0.316; rotation sweeps as stills: lysozyme 5 keV peaks 3.36 -> 3.13 (CCref last 0.674 -> 0.689), thaumatin unchanged, a Zn site sweep: peaks 18.8 -> 20.6. On the full XFEL run CC_anom between independent halves 0.334 -> 0.39 and R-free 0.243 -> 0.235. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- docs/ACKNOWLEDGEMENT.md | 10 + .../scale_merge/StillsPartialityRefine.cpp | 273 +++++++++++------- .../scale_merge/StillsPartialityRefine.h | 35 ++- 3 files changed, 209 insertions(+), 109 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 80e94cd33..9330bf040 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -78,6 +78,16 @@ high-throughput reduction and analysis of serial femtosecond X-ray diffraction d J. Appl. Cryst. 47, 1118-1131 [doi:10.1107/S1600576714007626](https://doi.org/10.1107/S1600576714007626). +**Per-crystal post-refinement of serial stills** - refining, for each still, the partiality +parameters the spot positions cannot pin (a small reorientation, the crystal's own rocking width and the +radial scale of its reciprocal lattice, which a shot's photon energy or a crystal's cell moves) against +the merged intensities follows the post-refinement of PRIME and the mosaic-block model of Sauter et al.: +M. Uervirojnangkoorn, O. B. Zeldin, A. Y. Lyubimov, J. Hattne, A. S. Brewster, N. K. Sauter, A. T. Brunger +and W. I. Weis, "Enabling X-ray free electron laser crystallography for challenging biological systems +from a limited number of crystals" (2015), eLife 4, e05421; N. K. Sauter, J. Hattne, A. S. Brewster, +N. Echols, P. H. Zwart and P. D. Adams, "Improved crystal orientation and physical properties from +single-shot XFEL stills" (2014), Acta Cryst. D70, 3299-3309 (DOIs not verified here, so not given). + Spot extraction groups strong pixels into spots with the sparse connected-component labelling of the ACTS traccc project: P. Gessinger, H. M. Gray, A. Krasznahorkay, C. Leggett, J. Niermann, A. Salzburger, S. N. Swatman and B. Yeo, "traccc: GPU track reconstruction library for HEP diff --git a/image_analysis/scale_merge/StillsPartialityRefine.cpp b/image_analysis/scale_merge/StillsPartialityRefine.cpp index 6096da7f4..89d5413db 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.cpp +++ b/image_analysis/scale_merge/StillsPartialityRefine.cpp @@ -25,6 +25,10 @@ namespace { // Model error of the per-crystal scale fit, as a fraction of the predicted intensity - ScaleOnTheFly's. constexpr double SCALE_MODEL_ERROR = 0.3; + // The smallest priors on a crystal's radial scale and log width scale (RunPrior). + constexpr double MIN_PRIOR_ETA = 1e-5; + constexpr double MIN_PRIOR_LN_WIDTH = 0.01; + double SafeInv(double x, double fallback) { if (!std::isfinite(x) || x == 0.0) return fallback; @@ -79,30 +83,6 @@ namespace { } }; - // Analytic partiality for a reflection whose base reciprocal vector is q, tilted by (psi_x, psi_y). - // Mirrors BraggPrediction: dist_ewald = |S| - 1/lambda with S = q_rot + S0, and - // p = exp(-dist_ewald^2 / 2 sigma^2), sigma^2 = gamma0^2 + (gamma_e*d*)^2 + (bw*|q_z|)^2 - three - // independent broadenings added in quadrature: the reciprocal-lattice point's own radius (gamma0, - // ~1/domain size, resolution-INdependent), the mosaic/divergence spread (gamma_e*d*, proportional to - // d*) and the bandwidth smear along the beam. In practice the fit below returns gamma_e ~ 0 and the - // width is essentially gamma0 - see there. - double ComputeP(double qx, double qy, double qz, - double psi_x, double psi_y, - double s0x, double s0y, double s0z, double inv_lambda, - const Width &width) { - double aa[3] = {psi_x, psi_y, 0.0}; - double q[3] = {qx, qy, qz}; - double qr[3]; - ceres::AngleAxisRotatePoint(aa, q, qr); - const double Sx = qr[0] + s0x, Sy = qr[1] + s0y, Sz = qr[2] + s0z; - const double de = std::sqrt(Sx * Sx + Sy * Sy + Sz * Sz) - inv_lambda; - const double dstar = std::sqrt(qr[0] * qr[0] + qr[1] * qr[1] + qr[2] * qr[2]); - const double sig2 = width.Sigma2(dstar, qr[2]); - if (!(sig2 > 0.0)) - return 1.0; - return std::exp(-0.5 * de * de / sig2); - } - // The prediction window cuts the rocking curve. Reflections fill a window of half-width W evenly, so // their plain moment of de is u = W^2 / 3, and the intensity-weighted moment m of a Gaussian of width // s cut at +-W is u * g(a), a = W / s, with g(a) = 3/a^2 (1 - 2 a phi(a) / erf(a / sqrt 2)): 1 for a @@ -170,79 +150,118 @@ namespace { -s * ky, s * kx, c}; } - // ComputeP with the tilt given as its rotation matrix. - struct TiltModel { + // A crystal's model of its partialities: the orientation tilt (psi_x, psi_y) about the two axes + // perpendicular to the beam, a radial scale eta of its reciprocal lattice (q -> (1 + eta) q: a + // wavelength or a cell a little off for this crystal - on an XFEL the photon energy of its shot) and a + // scale exp(ln_width) of the run's rocking width (a crystal's own mosaicity or domain size). + // Per-crystal mosaicity and lattice scale in post-refinement, following Uervirojnangkoorn et al. (2015) + // eLife 4, e05421 and Sauter et al. (2014) Acta Cryst. D70, 3299-3309. + struct CrystalModel { + double psi_x = 0.0, psi_y = 0.0, eta = 0.0, ln_width = 0.0; + }; + + // The partiality of a reflection whose base reciprocal vector is q under a crystal model. Mirrors + // BraggPrediction: dist_ewald = |S| - 1/lambda with S = q' + S0, q' = (1 + eta) R(psi) q, and + // p = exp(-dist_ewald^2 / 2 sigma^2), sigma = exp(ln_width) sigma_run(d*). + + struct PartialityModel { double s0x, s0y, s0z, inv_lambda; Width width; - double P(const std::array &R, const FitObs &o) const { - const double qx = R[0] * o.qx + R[1] * o.qy + R[2] * o.qz; - const double qy = R[3] * o.qx + R[4] * o.qy + R[5] * o.qz; - const double qz = R[6] * o.qx + R[7] * o.qy + R[8] * o.qz; + double At(const std::array &R, const CrystalModel &m, double qx0, double qy0, double qz0) const { + const double f = 1.0 + m.eta; + const double qx = f * (R[0] * qx0 + R[1] * qy0 + R[2] * qz0); + const double qy = f * (R[3] * qx0 + R[4] * qy0 + R[5] * qz0); + const double qz = f * (R[6] * qx0 + R[7] * qy0 + R[8] * qz0); const double de = VecLen(qx + s0x, qy + s0y, qz + s0z) - inv_lambda; - const double sig2 = width.Sigma2(VecLen(qx, qy, qz), qz); + const double sig2 = width.Sigma2(VecLen(qx, qy, qz), qz) * std::exp(2.0 * m.ln_width); if (!(sig2 > 0.0)) return 1.0; return std::exp(-0.5 * de * de / sig2); } + + double P(const std::array &R, const CrystalModel &m, const FitObs &o) const { + return At(R, m, o.qx, o.qy, o.qz); + } }; - // Refine the orientation tilt (psi_x, psi_y) holding the scale G fixed: minimise - // sum_i rho(r_i^2) / 2 + |psi / sigma_prior|^2 / 2, r_i = weight_i (G p_i(psi) Iref_i / corr_i - Iobs_i), - // with the Cauchy rho(s) = k^2 log(1 + s / k^2) - the intensity-space objective of ScaleOnTheFly, and - // the Gaussian prior a proper MAP term since r_i is already in sigma units. Levenberg-Marquardt on the - // normal equations reweighted by rho', the Jacobian by central differences, every trial tilt held inside - // the bound. Returns false when the cost cannot be evaluated, leaving psi as it came in. - bool RefineTilt(const std::vector &obs, const TiltModel &model, double G, double robust_k, - double inv_sigma_prior, double max_tilt, double psi[2]) { + // Refine the crystal model holding the scale G fixed: minimise + // sum_i rho(r_i^2) / 2 + sum_k (x_k / sigma_k)^2 / 2, r_i = weight_i (G p_i(x) Iref_i / corr_i - Iobs_i), + // over x = (psi_x, psi_y, eta, ln_width), with the Cauchy rho(s) = k^2 log(1 + s / k^2) - the intensity- + // space objective of ScaleOnTheFly - and Gaussian priors about the run (x = 0), proper MAP terms since + // r_i is already in sigma units. Levenberg-Marquardt on the normal equations reweighted by rho', the + // Jacobian by central differences, every trial held inside the bounds. Returns false when the cost + // cannot be evaluated, leaving the model as it came in. + bool RefineCrystal(const std::vector &obs, const PartialityModel &model, double G, double robust_k, + const std::array &inv_sigma_prior, const std::array &bound, + CrystalModel &crystal, std::array &variance) { constexpr int MAX_ITERATIONS = 25; - constexpr double STEP = 1e-7; // rad, for the central differences + constexpr std::array STEP = {1e-7, 1e-7, 1e-7, 1e-5}; // for the central differences const double k2 = robust_k * robust_k; - const double prior2 = inv_sigma_prior * inv_sigma_prior; + auto as_model = [](const Eigen::Vector4d &x) { return CrystalModel{x(0), x(1), x(2), x(3)}; }; - auto cost = [&](const double p[2]) { - const auto R = TiltMatrix(p[0], p[1]); - double c = 0.5 * prior2 * (p[0] * p[0] + p[1] * p[1]); + auto cost = [&](const Eigen::Vector4d &x) { + const auto R = TiltMatrix(x(0), x(1)); + const CrystalModel m = as_model(x); + double c = 0.0; + for (int k = 0; k < 4; ++k) + c += 0.5 * inv_sigma_prior[k] * inv_sigma_prior[k] * x(k) * x(k); for (const auto &o: obs) { - const double r = o.weight * (G * model.P(R, o) * o.inv_corr * o.Iref - o.Iobs); + const double r = o.weight * (G * model.P(R, m, o) * o.inv_corr * o.Iref - o.Iobs); c += 0.5 * k2 * std::log1p(r * r / k2); } return c; }; - double current = cost(psi); + Eigen::Vector4d x(crystal.psi_x, crystal.psi_y, crystal.eta, crystal.ln_width); + double current = cost(x); if (!std::isfinite(current)) return false; double lambda = 1e-4; + Eigen::Matrix4d data_A = Eigen::Matrix4d::Identity(); // the normal matrix of the data alone for (int iter = 0; iter < MAX_ITERATIONS; ++iter) { - const auto R = TiltMatrix(psi[0], psi[1]); - const auto Rxp = TiltMatrix(psi[0] + STEP, psi[1]), Rxm = TiltMatrix(psi[0] - STEP, psi[1]); - const auto Ryp = TiltMatrix(psi[0], psi[1] + STEP), Rym = TiltMatrix(psi[0], psi[1] - STEP); - double A00 = prior2, A01 = 0.0, A11 = prior2; - double b0 = prior2 * psi[0], b1 = prior2 * psi[1]; + Eigen::Matrix4d A = Eigen::Matrix4d::Zero(); + Eigen::Vector4d b = Eigen::Vector4d::Zero(); + for (int k = 0; k < 4; ++k) { + A(k, k) = inv_sigma_prior[k] * inv_sigma_prior[k]; + b(k) = A(k, k) * x(k); + } + // The model at x and at x +- one step in each parameter. + std::array, 9> R; + std::array m; + for (int j = 0; j < 9; ++j) { + Eigen::Vector4d xj = x; + if (j > 0) + xj((j - 1) / 2) += (j % 2 ? 1.0 : -1.0) * STEP[(j - 1) / 2]; + R[j] = TiltMatrix(xj(0), xj(1)); + m[j] = as_model(xj); + } for (const auto &o: obs) { const double a = o.weight * G * o.inv_corr * o.Iref; - const double r = a * model.P(R, o) - o.weight * o.Iobs; - const double jx = a * (model.P(Rxp, o) - model.P(Rxm, o)) / (2.0 * STEP); - const double jy = a * (model.P(Ryp, o) - model.P(Rym, o)) / (2.0 * STEP); + const double r = a * model.P(R[0], m[0], o) - o.weight * o.Iobs; + Eigen::Vector4d J; + for (int k = 0; k < 4; ++k) + J(k) = a * (model.P(R[2 * k + 1], m[2 * k + 1], o) - model.P(R[2 * k + 2], m[2 * k + 2], o)) + / (2.0 * STEP[k]); const double w = 1.0 / (1.0 + r * r / k2); // rho' - A00 += w * jx * jx; - A01 += w * jx * jy; - A11 += w * jy * jy; - b0 += w * jx * r; - b1 += w * jy * r; + A += w * J * J.transpose(); + b += w * r * J; } + data_A = A; + for (int k = 0; k < 4; ++k) + data_A(k, k) -= inv_sigma_prior[k] * inv_sigma_prior[k]; bool accepted = false; - double trial[2] = {psi[0], psi[1]}; + Eigen::Vector4d trial = x; double trial_cost = current; while (lambda < 1e10) { - const double M00 = A00 * (1.0 + lambda), M11 = A11 * (1.0 + lambda); - const double det = M00 * M11 - A01 * A01; - if (!(det > 0.0)) + Eigen::Matrix4d M = A; + M.diagonal() *= 1.0 + lambda; + const Eigen::Vector4d dx = M.ldlt().solve(b); + if (!dx.allFinite()) break; - trial[0] = std::clamp(psi[0] - (M11 * b0 - A01 * b1) / det, -max_tilt, max_tilt); - trial[1] = std::clamp(psi[1] - (M00 * b1 - A01 * b0) / det, -max_tilt, max_tilt); + for (int k = 0; k < 4; ++k) + trial(k) = std::clamp(x(k) - dx(k), -bound[k], bound[k]); trial_cost = cost(trial); if (std::isfinite(trial_cost) && trial_cost < current) { accepted = true; @@ -253,14 +272,17 @@ namespace { } if (!accepted) break; - const double step = std::hypot(trial[0] - psi[0], trial[1] - psi[1]); const double decrease = current - trial_cost; - psi[0] = trial[0]; - psi[1] = trial[1]; + const double step = (trial - x).norm(); + x = trial; current = trial_cost; if (decrease <= 1e-6 * current || step <= 1e-10) break; } + crystal = as_model(x); + const Eigen::Matrix4d cov = data_A.inverse(); + for (int k = 0; k < 4; ++k) + variance[k] = cov(k, k); return true; } } @@ -273,12 +295,12 @@ StillsPartialityRefine::StillsPartialityRefine(const DiffractionExperiment &x) min_partiality_(x.GetScalingSettings().GetMinPartiality()), bandwidth_sigma_(x.GetBandwidthFWHM().value_or(0.0f) / 2.3548f) {} -double StillsPartialityRefine::RefineOne(IntegrationOutcome &outcome, +StillsPartialityRefine::CrystalFit StillsPartialityRefine::RefineOne(IntegrationOutcome &outcome, const std::map &reference, const std::optional &run_width, - double radial_scale) const { + double radial_scale, const CrystalPrior &prior) const { if (outcome.reflections.empty()) - return 0.0; + return {}; const Coord Astar = outcome.latt.Astar(); const Coord Bstar = outcome.latt.Bstar(); @@ -337,7 +359,7 @@ double StillsPartialityRefine::RefineOne(IntegrationOutcome &outcome, } if (obs.size() < MIN_FIT_REFLECTIONS || n_de == 0) - return 0.0; + return {}; // Without the run's measured width (too few reflections in the run to measure it), the width comes // from this crystal's own reflections. Solve the 2x2 normal equations for de^2 = A + B d*^2. A degenerate spread in d* (all reflections in @@ -372,44 +394,47 @@ double StillsPartialityRefine::RefineOne(IntegrationOutcome &outcome, .table_dstar = run_width ? &run_width->table_dstar : nullptr, .table_s = run_width ? &run_width->table_s : nullptr}; - double psi[2] = {0.0, 0.0}; + CrystalModel crystal; double G = 1.0; - const TiltModel model{.s0x = S0.x, .s0y = S0.y, .s0z = S0.z, .inv_lambda = inv_lambda, - .width = width}; - const bool refine_tilt = obs.size() >= settings_.min_reflections; - const int inner = refine_tilt ? settings_.inner_iterations : 1; + const PartialityModel model{.s0x = S0.x, .s0y = S0.y, .s0z = S0.z, .inv_lambda = inv_lambda, .width = width}; + const bool refine_crystal = obs.size() >= settings_.min_reflections; + const int inner = refine_crystal ? settings_.inner_iterations : 1; + const std::array inv_sigma_prior = {kRadToDeg / settings_.prior_sigma_deg, + kRadToDeg / settings_.prior_sigma_deg, + 1.0 / prior.sigma_eta, 1.0 / prior.sigma_ln_width}; + std::array variance{NAN, NAN, NAN, NAN}; + const std::array bound = {settings_.max_tilt_rad, settings_.max_tilt_rad, + settings_.max_eta, settings_.max_ln_width}; for (int it = 0; it < inner; ++it) { - // (1) Solve G given the current partialities, and weight the tilt fit by the variances at that model. + // (1) Solve G given the current partialities, and weight the model fit by the variances at that model. + const auto R = TiltMatrix(crystal.psi_x, crystal.psi_y); std::vector coeff(obs.size()); - for (size_t j = 0; j < obs.size(); ++j) { - const double p = ComputeP(obs[j].qx, obs[j].qy, obs[j].qz, psi[0], psi[1], - S0.x, S0.y, S0.z, inv_lambda, width); - coeff[j] = p * obs[j].inv_corr * obs[j].Iref; - } + for (size_t j = 0; j < obs.size(); ++j) + coeff[j] = model.P(R, crystal, obs[j]) * obs[j].inv_corr * obs[j].Iref; G = SolveScaleIRLS(obs, coeff, settings_.robust_k); if (!(G > 0.0) || !std::isfinite(G)) - return 0.0; + return {}; for (size_t j = 0; j < obs.size(); ++j) obs[j].weight = 1.0 / std::sqrt(ModelVariance(obs[j], G * coeff[j])); - if (!refine_tilt) + if (!refine_crystal) break; - // (2) Refine the tilt holding G fixed. A cost that cannot be evaluated keeps the tilt this crystal - // came in with and stops refining it - G alone is still a usable model. - const double inv_sigma_prior = settings_.prior_sigma_deg > 0.0 ? kRadToDeg / settings_.prior_sigma_deg : 0.0; - if (!RefineTilt(obs, model, G, settings_.robust_k, inv_sigma_prior, settings_.max_tilt_rad, psi)) + // (2) Refine the crystal model holding G fixed. A cost that cannot be evaluated keeps the model this + // crystal came in with and stops refining it - G alone is still a usable model. + if (!RefineCrystal(obs, model, G, settings_.robust_k, inv_sigma_prior, bound, crystal, variance)) break; } // Write the refined partiality + scale correction onto every reflection of the crystal (not only the // fit subset), so the merge sees a consistent model. image_scale_corr = prescaling_corr * qe_corr * flight_corr / // (partiality * G), the same composition ScaleOnTheFly writes. + const auto R = TiltMatrix(crystal.psi_x, crystal.psi_y); for (auto &r: outcome.reflections) { const Coord q = base_q(r); - const double p = ComputeP(q.x, q.y, q.z, psi[0], psi[1], S0.x, S0.y, S0.z, inv_lambda, width); + const double p = model.At(R, crystal, q.x, q.y, q.z); r.partiality = static_cast(p); const float corr = r.prescaling_corr * r.qe_corr * r.flight_corr; const double denom = p * G; @@ -428,8 +453,8 @@ double StillsPartialityRefine::RefineOne(IntegrationOutcome &outcome, outcome.image_scale_cc = cc; outcome.image_scale_cc_n = cc_n; - const double tilt_deg = std::sqrt(psi[0] * psi[0] + psi[1] * psi[1]) * kRadToDeg; - return tilt_deg; + return {.refined = refine_crystal, .tilt_deg = std::hypot(crystal.psi_x, crystal.psi_y) * kRadToDeg, + .eta = crystal.eta, .ln_width = crystal.ln_width, .var_eta = variance[2], .var_ln_width = variance[3]}; } double StillsPartialityRefine::MeasureRadialScale(const std::vector &outcomes, @@ -718,6 +743,40 @@ std::optional StillsPartialityRefine::MeasureR return width; } +StillsPartialityRefine::CrystalPrior StillsPartialityRefine::RunPrior(const std::vector &fits) const { + // The spread of a parameter over the crystals: the variance of the estimates less the median variance + // they carry from their own data, floored at a small value (a crystal is never held exactly). + auto spread = [&](auto value, auto variance, double floor) { + std::vector x, v; + for (const auto &f: fits) + if (f.refined && std::isfinite(variance(f)) && variance(f) > 0.0) { + x.push_back(value(f)); + v.push_back(variance(f)); + } + if (x.size() < 10) + return floor; + double mean = 0.0; + for (double e: x) + mean += e; + mean /= static_cast(x.size()); + double var = 0.0; + for (double e: x) + var += (e - mean) * (e - mean); + var /= static_cast(x.size() - 1); + std::nth_element(v.begin(), v.begin() + v.size() / 2, v.end()); + return std::max(floor, std::sqrt(std::max(0.0, var - v[v.size() / 2]))); + }; + const CrystalPrior prior{ + .sigma_eta = spread([](const CrystalFit &f) { return f.eta; }, + [](const CrystalFit &f) { return f.var_eta; }, MIN_PRIOR_ETA), + .sigma_ln_width = spread([](const CrystalFit &f) { return f.ln_width; }, + [](const CrystalFit &f) { return f.var_ln_width; }, MIN_PRIOR_LN_WIDTH)}; + Logger("StillsPartialityRefine").Info("Stills crystals differ in radial scale by {:.1e} and in rocking width " + "by {:.0f}% (sd over the crystals, net of their own uncertainty)", + prior.sigma_eta, 100.0 * prior.sigma_ln_width); + return prior; +} + double StillsPartialityRefine::Run(std::vector &outcomes, size_t nthreads, const std::optional &reference_cell, const std::optional &given_width, @@ -732,7 +791,6 @@ double StillsPartialityRefine::Run(std::vector &outcomes, si nthreads = AvailableCpus(); nthreads = std::max(1, nthreads); - double last_mean_tilt = 0.0; // Where the frames were predicted through a curve measured before the image pass, the partialities use // that same curve. Measured again here, through windows that follow it, the width reads narrower at low // resolution (its constant term falls to about zero), and the merge it gave moved R-free against the @@ -750,6 +808,13 @@ double StillsPartialityRefine::Run(std::vector &outcomes, si if (used_width) *used_width = run_width; + // The priors on a crystal's radial scale and width are the run's own spread of them: the first cycle + // refines every crystal against loose priors, and the spread of those estimates less their own + // uncertainty (method of moments) is how much crystals really differ. On an XFEL each shot has its + // photon energy and the spread is large; on a monochromatic or a stable pink beam it comes out small, + // and the crystals are held to the run's curve. + CrystalPrior prior{.sigma_eta = settings_.max_eta, .sigma_ln_width = settings_.max_ln_width}; + std::vector fits(outcomes.size()); for (int outer = 0; outer < settings_.outer_iterations; ++outer) { // Reference full intensities from the current corrections. const std::vector merged = MergeAll(experiment_, outcomes, nthreads, reference_cell); @@ -757,19 +822,12 @@ double StillsPartialityRefine::Run(std::vector &outcomes, si for (const auto &m: merged) reference[hkl_key_generator_(m)] = m.I; - std::atomic tilt_sum{0.0}; - std::atomic tilt_n{0}; std::atomic next{0}; - auto worker = [&]() { size_t i = next.fetch_add(1); while (i < outcomes.size()) { - const double t = merged_crystal(outcomes[i]) ? RefineOne(outcomes[i], reference, run_width, radial_scale) : 0.0; - if (t > 0.0) { - double prev = tilt_sum.load(); - while (!tilt_sum.compare_exchange_weak(prev, prev + t)) {} - tilt_n.fetch_add(1); - } + fits[i] = merged_crystal(outcomes[i]) ? RefineOne(outcomes[i], reference, run_width, radial_scale, prior) + : CrystalFit{}; i = next.fetch_add(1); } }; @@ -786,8 +844,17 @@ double StillsPartialityRefine::Run(std::vector &outcomes, si f.get(); } - last_mean_tilt = tilt_n > 0 ? tilt_sum.load() / static_cast(tilt_n.load()) : 0.0; + if (outer == 0) + prior = RunPrior(fits); } + double tilt_sum = 0.0; + size_t tilt_n = 0; + for (const auto &f: fits) + if (f.refined) { + tilt_sum += f.tilt_deg; + ++tilt_n; + } + const double last_mean_tilt = tilt_n > 0 ? tilt_sum / static_cast(tilt_n) : 0.0; return last_mean_tilt; } diff --git a/image_analysis/scale_merge/StillsPartialityRefine.h b/image_analysis/scale_merge/StillsPartialityRefine.h index 5b99d6d36..6f644fb8c 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.h +++ b/image_analysis/scale_merge/StillsPartialityRefine.h @@ -23,10 +23,18 @@ // reshapes the SPATIAL pattern of partialities in a way a single G cannot mimic. That breaks the // degeneracy that killed the scalar-sigma fit. // +// Two more parameters per crystal, each held by a prior about the run: a radial scale eta of its +// reciprocal lattice (q -> (1 + eta) q, which moves the excitation error by about eta lambda d*^2 / 2 - +// on an XFEL the photon energy of the crystal's own shot, which spot positions cannot pin) and a scale of +// the run's rocking width (the crystal's own mosaicity and size). Both change p differently across +// resolution, so neither is degenerate with G. Their priors are not constants: the first cycle refines +// against loose ones, and the run's spread of the estimates, net of their own uncertainty, becomes the +// prior of the next (RunPrior) - large on SASE XFEL data, small on a monochromatic or stable pink beam. +// // Because the integrated intensity I_obs is fixed, refining the tilt only recomputes the partiality // analytically from the stored per-image lattice/geometry (q = A*.h + B*.k + C*.l, then // dist_ewald = |q + S0| - 1/lambda, signed) - NO pixel re-integration. The loop is: merge -> per-crystal -// refine tilt (G profiled out by the same robust IRLS ScaleOnTheFly uses) -> recompute p and +// refine tilt, eta and width (G profiled out by the same robust IRLS ScaleOnTheFly uses) -> recompute p and // image_scale_corr -> re-merge, iterated a few times. Mutates each reflection's `partiality` and // `image_scale_corr` in place; the existing MergeOnTheFly then consumes the improved corrections. The // width of the Gaussian partiality is the run's, measured once from the intensity-weighted excitation @@ -38,8 +46,10 @@ public: int inner_iterations = 3; // (solve G) <-> (refine tilt) alternations per crystal size_t min_reflections = 40; // skip tilt refinement below this (anti-overfit on sparse crystals) double max_tilt_rad = 0.0175; // hard bound on |dpsi| (~1 deg); indexing already refined orientation - double prior_sigma_deg = 0.02; // soft prior pulling dpsi toward 0 (0 = off); tames weak-data overfit, + double prior_sigma_deg = 0.02; // soft prior pulling dpsi toward 0; tames weak-data overfit, // inert on strong data (well-supported tilts overcome it) + double max_eta = 5e-3; // bound on a crystal's radial scale |eta|, and its prior in the first cycle + double max_ln_width = 1.1; // the same for the log of its width scale (a factor of 3) double robust_k = 3.0; // Cauchy loss scale (sigma units) double gamma_e = 0.0; // angular width sigma(d*) = gamma_e*d* (0 = measure from data) }; @@ -84,8 +94,21 @@ private: const double min_partiality_; const float bandwidth_sigma_; - // Refine one crystal against the reference map; returns |dpsi| in degrees (0 if skipped). - double RefineOne(IntegrationOutcome &outcome, - const std::map &reference, - const std::optional &run_width, double radial_scale) const; + // The priors of a crystal's radial scale eta and log width scale (Gaussian, about 0). + struct CrystalPrior { + double sigma_eta, sigma_ln_width; + }; + // One crystal's refined model: |dpsi| in degrees, eta, ln width and the variances of the last two from + // its own data (no prior). refined = false where only G (or nothing) was fitted. + struct CrystalFit { + bool refined = false; + double tilt_deg = 0.0, eta = 0.0, ln_width = 0.0, var_eta = NAN, var_ln_width = NAN; + }; + + // Refine one crystal against the reference map. + CrystalFit RefineOne(IntegrationOutcome &outcome, const std::map &reference, + const std::optional &run_width, double radial_scale, + const CrystalPrior &prior) const; + // The run's spread of eta and ln width over the crystals, net of their own uncertainty. + CrystalPrior RunPrior(const std::vector &fits) const; }; From 670f77f4d074fe85dc1af998680b6d1aebb99ebb Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 13:17:23 +0100 Subject: [PATCH 06/34] Stills partiality: the run's width table is balanced on the partials before the post-refinement The rocking width is measured from moments of the excitation error through the prediction window, and its tails are where that is least sure. On rotation sweeps processed as stills, reflections predicted at under a third of their peak were 30% fainter than the model at low resolution and up to 50% brighter at high resolution, against reflections near their peak - a bias in I/p that no variance term removes. Before the post-refinement each shell of the width table is now rescaled by the factor at which the reflections predicted below half their peak and those above it put the same scale on the merged intensities (ratio of sums, searched by bisection). On synthetic data with a known Gaussian width the factors come out 0.98-1.00. On the two rotation sweeps the low-partiality/high-partiality ratio per shell goes from 0.68-0.93 and 0.65-1.49 to 0.98-1.07 and 0.92-1.14 (one outer shell of the first set over-corrected to 0.75). Quick panel against the previous commit (CCref last shell / overall): XFEL serial 18k frames 0.653/0.665 -> 0.657/0.685, 1.5k frames 0.252/0.543 -> 0.291/0.559; rotation sweeps as stills 0.689/0.464 -> 0.673/0.483, 0.644/0.274 -> 0.645/0.280, Zn site sweep peaks 20.6 -> 21.0; monochromatic jet 0.458/0.396 -> 0.464/0.355 (its lowest shell 0.33 -> 0.30). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- .../scale_merge/StillsPartialityRefine.cpp | 90 ++++++++++++++++++- .../scale_merge/StillsPartialityRefine.h | 3 + 2 files changed, 91 insertions(+), 2 deletions(-) diff --git a/image_analysis/scale_merge/StillsPartialityRefine.cpp b/image_analysis/scale_merge/StillsPartialityRefine.cpp index 89d5413db..cb69d7497 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.cpp +++ b/image_analysis/scale_merge/StillsPartialityRefine.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -25,6 +26,9 @@ namespace { // Model error of the per-crystal scale fit, as a fraction of the predicted intensity - ScaleOnTheFly's. constexpr double SCALE_MODEL_ERROR = 0.3; + // The width scales BalanceRunWidth searches: from 1/BALANCE_RANGE to BALANCE_RANGE times the table. + constexpr double BALANCE_RANGE = 3.0; + // The smallest priors on a crystal's radial scale and log width scale (RunPrior). constexpr double MIN_PRIOR_ETA = 1e-5; constexpr double MIN_PRIOR_LN_WIDTH = 0.01; @@ -743,6 +747,85 @@ std::optional StillsPartialityRefine::MeasureR return width; } +StillsPartialityRefine::RunWidth StillsPartialityRefine::BalanceRunWidth( + const std::vector &outcomes, const std::map &reference, + const std::optional &reference_cell, const RunWidth &width) const { + // The width table is measured from moments of the excitation error through the prediction window, and + // the tails of the curve are where that is least sure: on rotation sweeps processed as stills, the + // reflections recorded at under a third of their peak came out 30% fainter than the model at low + // resolution and up to 50% brighter at high resolution, against those recorded near their peak. So + // each shell of the table is scaled by the factor s at which its reflections predicted below half + // their peak and those above it put the same scale on the merged intensities: + // sum I / sum p_s I_ref is the same for both, p_s the Gaussian of width s sigma. + if (width.table_dstar.empty()) + return width; + const auto &indexing = experiment_.GetIndexingSettings(); + struct Point { double u, x, ref; }; // eps / sigma, I / G (corrected), the reference intensity + std::vector> shells(width.table_dstar.size()); + const Width w{.gamma0 = width.gamma0, .gamma_e = width.gamma_e, .bw = width.bandwidth_sigma, + .table_dstar = &width.table_dstar, .table_s = &width.table_s}; + for (const auto &o: outcomes) { + if (!o.image_scale_g || !(*o.image_scale_g > 0.0)) + continue; + if (reference_cell && !o.latt.GetUnitCell().is_close(*reference_cell, indexing.GetUnitCellDistTolerance(), + indexing.GetUnitCellAngleTolerance_deg())) + continue; + const Coord S0 = o.geom.GetScatteringVector(); + const double inv_lambda = 1.0 / o.geom.GetWavelength_A(); + for (const auto &r: o.reflections) { + if (r.on_ice_ring || !AcceptReflection(r, d_min_limit_, d_max_limit_) || !std::isfinite(r.I)) + continue; + const auto it = reference.find(hkl_key_generator_(r)); + if (it == reference.end() || !(it->second > 0.0)) + continue; + const Coord q = (o.latt.Astar() * static_cast(r.h) + o.latt.Bstar() * static_cast(r.k) + + o.latt.Cstar() * static_cast(r.l)) * static_cast(width.radial_scale); + const double dstar = VecLen(q.x, q.y, q.z); + const double de = VecLen(q.x + S0.x, q.y + S0.y, q.z + S0.z) - inv_lambda; + const double x = r.I * r.prescaling_corr * r.qe_corr * r.flight_corr / *o.image_scale_g; + if (!std::isfinite(x)) + continue; + // the table shell nearest in d* + size_t k = std::upper_bound(width.table_dstar.begin(), width.table_dstar.end(), dstar) + - width.table_dstar.begin(); + if (k == width.table_dstar.size() || (k > 0 && dstar - width.table_dstar[k - 1] < width.table_dstar[k] - dstar)) + k = k > 0 ? k - 1 : 0; + shells[k].push_back({de / std::sqrt(w.Sigma2(dstar, q.z)), x, it->second}); + } + } + // log(sum I / sum p_s I_ref) over the reflections below half the table's peak, less the same over those + // above it; falls as s grows. + auto gap = [](const std::vector &pts, double s) { + const double half = std::sqrt(2.0 * std::log(2.0)); // |u| at half the peak + double x_lo = 0.0, m_lo = 0.0, x_hi = 0.0, m_hi = 0.0; + for (const auto &p: pts) { + const double ps = std::exp(-0.5 * p.u * p.u / (s * s)); + (std::fabs(p.u) > half ? x_lo : x_hi) += p.x; + (std::fabs(p.u) > half ? m_lo : m_hi) += ps * p.ref; + } + if (!(x_lo > 0.0 && m_lo > 0.0 && x_hi > 0.0 && m_hi > 0.0)) + return std::numeric_limits::quiet_NaN(); + return std::log(x_lo / m_lo) - std::log(x_hi / m_hi); + }; + RunWidth balanced = width; + std::string table; + for (size_t k = 0; k < shells.size(); ++k) { + double lo = 1.0 / BALANCE_RANGE, hi = BALANCE_RANGE; + const double g_lo = gap(shells[k], lo), g_hi = gap(shells[k], hi); + if (shells[k].size() < 400 || !(g_lo > 0.0) || !(g_hi < 0.0)) + continue; + for (int i = 0; i < 40; ++i) { + const double mid = std::sqrt(lo * hi); + (gap(shells[k], mid) > 0.0 ? lo : hi) = mid; + } + balanced.table_s[k] *= std::sqrt(lo * hi); + table += fmt::format(" {:.2f} A x{:.2f}", 1.0 / width.table_dstar[k], std::sqrt(lo * hi)); + } + Logger("StillsPartialityRefine").Info("Stills rocking width balanced on the partials, against resolution:{}", + table.empty() ? " unchanged" : table); + return balanced; +} + StillsPartialityRefine::CrystalPrior StillsPartialityRefine::RunPrior(const std::vector &fits) const { // The spread of a parameter over the crystals: the variance of the estimates less the median variance // they carry from their own data, floored at a small value (a crystal is never held exactly). @@ -805,8 +888,6 @@ double StillsPartialityRefine::Run(std::vector &outcomes, si run_width = MeasureRunWidth(outcomes, reference_cell, radial_scale); } const double radial_scale = run_width ? run_width->radial_scale : 1.0; - if (used_width) - *used_width = run_width; // The priors on a crystal's radial scale and width are the run's own spread of them: the first cycle // refines every crystal against loose priors, and the spread of those estimates less their own @@ -821,6 +902,8 @@ double StillsPartialityRefine::Run(std::vector &outcomes, si std::map reference; for (const auto &m: merged) reference[hkl_key_generator_(m)] = m.I; + if (outer == 0 && run_width) + run_width = BalanceRunWidth(outcomes, reference, reference_cell, *run_width); std::atomic next{0}; auto worker = [&]() { @@ -848,6 +931,9 @@ double StillsPartialityRefine::Run(std::vector &outcomes, si prior = RunPrior(fits); } + if (used_width) + *used_width = run_width; + double tilt_sum = 0.0; size_t tilt_n = 0; for (const auto &f: fits) diff --git a/image_analysis/scale_merge/StillsPartialityRefine.h b/image_analysis/scale_merge/StillsPartialityRefine.h index 6f644fb8c..48c59c76a 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.h +++ b/image_analysis/scale_merge/StillsPartialityRefine.h @@ -109,6 +109,9 @@ private: CrystalFit RefineOne(IntegrationOutcome &outcome, const std::map &reference, const std::optional &run_width, double radial_scale, const CrystalPrior &prior) const; + // The run's width table, each shell rescaled so that its weak and strong partials agree on the scale. + RunWidth BalanceRunWidth(const std::vector &outcomes, const std::map &reference, + const std::optional &reference_cell, const RunWidth &width) const; // The run's spread of eta and ln width over the crystals, net of their own uncertainty. CrystalPrior RunPrior(const std::vector &fits) const; }; From 14190d94110c403f0d81518f670e203b45830fee Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 14:02:56 +0100 Subject: [PATCH 07/34] Stills merge: the rotation error-model fit with a partiality term, iterated with the means, merging down to p = 0.3 The stills error model was the fit the rotation merge replaced (medians of sigma^2, ^2 and dev^2 taken apart over bins of ^2, observed sigmas, one pass) and was applied to a variance rebuilt at the mean that it had not been fitted on. It is now fitted as the rotation model is - equal-count bins of counting I/sigma, the median normalised deviation of each, every bin relative - on the counting variance rebuilt at the reflection mean, with a third term for the partiality model's own error, c^2 ((1 - p) / p)^2 ^2, each bin split by that term's share so c is told apart from b. The means and the model are refitted in turn four times. With the term in the variance, the merge takes observations down to p = 0.3 (the scale and partiality references keep min_partiality). A term growing like 1/p in the tail fitted the scatter better than the one a Gaussian rocking curve propagates (growing like sqrt(-ln p)). Going down to p = 0.1 cost the outermost shell: on a rotation-as-serial set the CC of the last shell to the deposited model was 0.595 / 0.628 / 0.692 / 0.685 for floors 0.1 / 0.2 / 0.3 / 0.5. A frame with fewer than 20 reflections to fit its scale on is left out of the merge instead of merged at the scale 1 its prediction carries. Measured against the stills merge before the change (before the per-crystal partiality refinement): rotation-as-serial R1 CC_anom 0.399 -> 0.487, S-site peaks 4.85/3.34 -> 5.11/3.61, last-shell CC to the model 0.619 -> 0.621; R2 CC_anom 0.055 -> 0.082, peaks 6.72/2.35 -> 7.21/2.55, last-shell CC1/2 0.728 -> 0.742, R-free 0.238 -> 0.236. A Friedel-hand scramble of whole frames keeps CC_anom at ~0. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/scale_merge/ErrorModel.cpp | 90 ++++++ image_analysis/scale_merge/ErrorModel.h | 23 ++ image_analysis/scale_merge/Merge.cpp | 310 +++++++------------ image_analysis/scale_merge/Merge.h | 33 +- image_analysis/scale_merge/ScaleOnTheFly.cpp | 5 + rugnux/RugnuxScaleMerge.cpp | 4 +- rugnux/rugnux_cli.cpp | 4 +- tests/MergeScaleTest.cpp | 73 +++++ 8 files changed, 340 insertions(+), 202 deletions(-) diff --git a/image_analysis/scale_merge/ErrorModel.cpp b/image_analysis/scale_merge/ErrorModel.cpp index a129cc3cf..dc2868ad9 100644 --- a/image_analysis/scale_merge/ErrorModel.cpp +++ b/image_analysis/scale_merge/ErrorModel.cpp @@ -7,6 +7,9 @@ #include #include #include +#include + +#include #include "../../common/ParallelFor.h" #include "../../common/ThreadAffinity.h" @@ -152,3 +155,90 @@ bool ErrorModelUpdate(const std::vector &bs2, const std::vector fit.b2 = b2; return !settled; } + +StillsErrorModelFit FitStillsErrorModel(std::vector &pool) { + constexpr int n_bins = ERROR_MODEL_BINS; + constexpr int n_sub = 4; + StillsErrorModelFit fit; + if (pool.size() < static_cast(8 * n_bins * n_sub)) + return fit; + + // Bins of counting (I/sigma)^2, each split by the partiality term's share of the intensity term. Total + // orders, so which sample lands in which bin is a property of the samples alone. + auto snr = [](const StillsErrorModelSample &x) { return x.I2 / x.s2; }; + auto share = [](const StillsErrorModelSample &x) { return x.I2 > 0.0 ? x.P2 / x.I2 : 0.0; }; + std::sort(pool.begin(), pool.end(), [&](const auto &x, const auto &y) { + return std::make_tuple(snr(x), x.s2, x.I2, x.P2, x.dev2) < std::make_tuple(snr(y), y.s2, y.I2, y.P2, y.dev2); + }); + const size_t per = pool.size() / n_bins; + std::vector> groups; + for (int bin = 0; bin < n_bins; ++bin) { + const size_t lo = bin * per, hi = bin == n_bins - 1 ? pool.size() : (bin + 1) * per; + std::sort(pool.begin() + lo, pool.begin() + hi, [&](const auto &x, const auto &y) { + return std::make_tuple(share(x), x.s2, x.I2, x.dev2) < std::make_tuple(share(y), y.s2, y.I2, y.dev2); + }); + const size_t sub = (hi - lo) / n_sub; + for (int q = 0; q < n_sub; ++q) + groups.emplace_back(lo + q * sub, q == n_sub - 1 ? hi : lo + (q + 1) * sub); + } + // b only where the strongest bin reaches past the counting term, as in FitErrorModel. + std::vector top; + for (size_t i = (n_bins - 1) * per; i < pool.size(); ++i) + top.push_back(snr(pool[i])); + std::nth_element(top.begin(), top.begin() + top.size() / 2, top.end()); + const bool b_measured = top[top.size() / 2] > ERROR_MODEL_B_LEVER_MIN; + + const int ng = static_cast(groups.size()); + std::vector z2(pool.size()); + Eigen::MatrixXd X(ng, 3); + Eigen::VectorXd y(ng); + double a = 1.0, b2 = 0.0, c2 = 0.0; + for (int iter = 0; iter < 30; ++iter) { + // Per group: the three terms divided by the current variance, and the median normalised deviation + // expressed as the sum it stands for. + for (int g = 0; g < ng; ++g) { + const auto [lo, hi] = groups[g]; + double s = 0.0, I = 0.0, P = 0.0; + for (size_t i = lo; i < hi; ++i) { + const auto &x = pool[i]; + const double v = a * x.s2 + b2 * x.I2 + c2 * x.P2; + s += x.s2 / v; I += x.I2 / v; P += x.P2 / v; + z2[i] = x.dev2 / v; + } + const auto mid = z2.begin() + lo + (hi - lo) / 2; + std::nth_element(z2.begin() + lo, mid, z2.begin() + hi); + X.row(g) << s, I, P; + y(g) = *mid / ERROR_MODEL_CHI2_1_MEDIAN * static_cast(hi - lo); + } + // Every group weighs the same in relative terms; a negative term is dropped and the rest refitted. + const Eigen::VectorXd w = y.array().square().inverse(); + std::vector terms = b_measured ? std::vector{0, 1, 2} : std::vector{0, 2}; + Eigen::VectorXd sol = Eigen::VectorXd::Zero(3); + while (true) { + Eigen::MatrixXd Xs(ng, terms.size()); + for (size_t j = 0; j < terms.size(); ++j) + Xs.col(j) = X.col(terms[j]); + const Eigen::MatrixXd A = Xs.transpose() * w.asDiagonal() * Xs; + const Eigen::VectorXd s = A.ldlt().solve(Xs.transpose() * w.asDiagonal() * y); + sol.setZero(); + for (size_t j = 0; j < terms.size(); ++j) + sol(terms[j]) = s(j); + const auto neg = std::find_if(terms.begin() + 1, terms.end(), [&](int t) { return sol(t) < 0.0; }); + if (neg == terms.end()) + break; + terms.erase(neg); + } + if (!std::isfinite(sol(0)) || !std::isfinite(sol(1)) || !std::isfinite(sol(2))) + break; + const double a_new = std::clamp(sol(0), 0.25, 100.0), b2_new = sol(1), c2_new = sol(2); + const bool settled = std::fabs(a_new - a) <= 1e-4 * a && std::fabs(b2_new - b2) <= 1e-4 * b2 + && std::fabs(c2_new - c2) <= 1e-4 * c2; + a = a_new; b2 = b2_new; c2 = c2_new; + fit.active = true; + fit.b_resolved = b_measured && b2 > 0.0 && a == sol(0); + if (settled) + break; + } + fit.a = a; fit.b2 = b2; fit.c2 = c2; + return fit; +} diff --git a/image_analysis/scale_merge/ErrorModel.h b/image_analysis/scale_merge/ErrorModel.h index d6ee315ec..11898071e 100644 --- a/image_analysis/scale_merge/ErrorModel.h +++ b/image_analysis/scale_merge/ErrorModel.h @@ -60,3 +60,26 @@ ErrorModelFit FitErrorModel(const std::vector &pool, std::vect // the new (a, b^2) into a, b2 and fit. False when the fit stops, settled or no longer determined. bool ErrorModelUpdate(const std::vector &bs2, const std::vector &bI2, const std::vector &bd2, double &a, double &b2, ErrorModelFit &fit); + +// The error model of the stills merge: the rotation model plus a partiality term, +// var = a * s2 + b^2 * ^2 + c^2 * ((1 - p) / p)^2 * ^2, +// p the partiality the observation is divided by. A still records one sample of its rocking curve, and the +// partiality model's error, carried through the division by p, grows like 1/p in the tail; without the term +// the observations below p ~ 0.5 cannot be merged at all, with it they add to the merge. Measured on four +// serial and rotation-as-serial sets, a term growing like 1/p fits the scatter better than the one a +// Gaussian rocking curve propagates (|eps| sigma_eps / sigma^2, which grows only like sqrt(-ln p)) and gives +// the stronger anomalous map; the curve's real tail is heavier than its Gaussian core. +// Fitted as FitErrorModel fits the rotation model - equal-count bins of counting I/sigma, the median +// normalised deviation of each, every bin relative, iterated - with each bin split in four by +// ((1 - p) / p)^2, so that c is told apart from b by observations of one strength and different partiality. +struct StillsErrorModelSample { + double s2, I2, P2, dev2; // P2 = ((1 - p) / p)^2 * I2 +}; + +struct StillsErrorModelFit { + bool active = false; + double a = 1.0, b2 = 0.0, c2 = 0.0; + bool b_resolved = false; // as ErrorModelFit::b_resolved +}; + +StillsErrorModelFit FitStillsErrorModel(std::vector &pool); diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index 521bd212e..69b06c93e 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -20,6 +20,7 @@ #include "RfreeFlags.h" #include "FrenchWilson.h" #include "OutlierBand.h" +#include "ErrorModel.h" namespace { // Which half-set an observation prefers: a splitmix64 bit-mix of the image's stable index, so the @@ -69,11 +70,11 @@ MergeOnTheFly::MergeOnTheFly(const DiffractionExperiment &x) ? std::optional(scaling_settings.GetMinCCForImage()) : std::nullopt), min_partiality(scaling_settings.GetMinPartiality()), + merge_min_partiality(std::min(min_partiality, MERGE_MIN_PARTIALITY)), capture_coeff(scaling_settings.GetCaptureUncertaintyCoeff()), generator(scaling_settings.GetMergeFriedel(), space_group), reject_outliers(scaling_settings.GetOutlierRejectNsigma() > 0.0), - reject_nsigma(scaling_settings.GetOutlierRejectNsigma()), - stills(!x.IsRotationIndexing()) { + reject_nsigma(scaling_settings.GetOutlierRejectNsigma()) { } MergeOnTheFly &MergeOnTheFly::ReferenceCell(const std::optional &cell) { @@ -93,7 +94,7 @@ size_t MergeOnTheFly::Contributions(const IntegrationOutcome &outcome, std::vect continue; if (exclude_ice_rings && r.on_ice_ring) continue; - if (r.partiality < min_partiality) + if (r.partiality < MergeFloor()) continue; const float I_corr = r.I * r.image_scale_corr; @@ -152,7 +153,7 @@ std::vector> MergeOnTheFly::Verdicts(const std:: v.flags |= OBS_RESOLUTION; if (exclude_ice_rings && r.on_ice_ring) v.flags |= OBS_ICE_EXCLUDED; - if (r.partiality < min_partiality) + if (r.partiality < MergeFloor()) v.flags |= OBS_LOW_PARTIALITY; const float I_corr = r.I * r.image_scale_corr; const float sigma_corr = r.sigma * r.image_scale_corr; @@ -235,57 +236,54 @@ void MergeOnTheFly::AddImages(const std::vector &outcomes) { }); } +double MergeOnTheFly::CountingVariance(float sigma_corr, float image_scale_corr, float var_bkg, float partiality, + double I_mean) const { + // A weak observation's sigma^2 = a background/read part plus a Poisson signal part proportional to its + // OWN intensity; weighting the merge by 1/sigma^2 with that per-observation sigma biases the + // inverse-variance mean low at <1 photon (an up-fluctuated observation gets a larger sigma and is + // over-downweighted, so the weighted mean drifts below ). Rebuild the signal part at the reflection's + // EXPECTED intensity instead - the background part plus corr* - so the weight no longer + // correlates with the observation's own fluctuation. This mirrors the rotation combine in + // RotationScaleMerge::process_rawrun. A negative mean has no Poisson signal, hence max(0, ). + double v = static_cast(sigma_corr) * sigma_corr; + if (scaling_settings.GetExpectedVarianceMerge()) { + const double base = static_cast(image_scale_corr) * image_scale_corr * var_bkg + + static_cast(image_scale_corr) * std::max(0.0, I_mean); + if (base > 0.0) + v = base; + } + const double capture = capture_coeff * (1.0 - std::min(1.0f, partiality)) * std::max(0.0, I_mean); + return v + capture * capture; +} + +double MergeOnTheFly::PartialityTerm(float partiality) { + const double q = (1.0 - std::min(1.0f, partiality)) / partiality; + return q * q; +} + float MergeOnTheFly::CorrectedSigma(float I_corr, float sigma_corr, float image_scale_corr, float var_bkg, float partiality, uint64_t hkl_key) const { if (!error_model_active) return sigma_corr; - // Intensity for the (b*I)^2 term: the reflection's mean (constant over its - // observations), falling back to this observation only if the mean is unknown. + // The systematic terms use the reflection's mean (constant over its observations), falling back to + // this observation only if the mean is unknown: using the per-observation I_i instead would + // over-weight down-fluctuated points. const auto it = error_model_mean_I.find(hkl_key); - const double I_for_b = (it != error_model_mean_I.end()) ? it->second : I_corr; - - // Base variance for the a*sigma^2 term. A weak observation's sigma^2 = a background/read part plus a - // Poisson signal part proportional to its OWN intensity; weighting the merge by 1/sigma^2 with that - // per-observation sigma biases the inverse-variance mean low at <1 photon (an up-fluctuated observation - // gets a larger sigma and is over-downweighted, so the weighted mean drifts below ). Rebuild the - // signal part at the reflection's EXPECTED intensity instead - decompose out the background part - // and re-add corr* - so the weight no longer correlates with the observation's own fluctuation. This - // mirrors the rotation combine in RotationScaleMerge::process_rawrun and is bit-identical when the - // observation sits at its reflection mean. - // The signal part removed here is the one the observation's own sigma actually carries, so it is - // taken at max(0, I): a negative intensity has no Poisson signal to subtract, and subtracting it - // anyway ADDS to the background part and inflates the weight of exactly the down-fluctuated - // observations this correction exists to stop being mistreated. - double a_var = static_cast(sigma_corr) * sigma_corr; - if (scaling_settings.GetExpectedVarianceMerge()) { - const double bkg_var = static_cast(image_scale_corr) * image_scale_corr * var_bkg; - const double base = bkg_var + static_cast(image_scale_corr) * std::max(0.0, I_for_b); - if (base > 0.0) - a_var = base; - } - const double capture = capture_coeff * (1.0 - std::min(1.0f, partiality)) * std::max(0.0, I_for_b); - a_var += capture * capture; - - const double v = error_model_a * a_var - + (error_model_b * I_for_b) * (error_model_b * I_for_b); + const double I_mean = (it != error_model_mean_I.end()) ? it->second : I_corr; + const double v = error_model_a * CountingVariance(sigma_corr, image_scale_corr, var_bkg, partiality, I_mean) + + (error_model_b * error_model_b + error_model_c2 * PartialityTerm(partiality)) * I_mean * I_mean; return (v > 0.0) ? static_cast(std::sqrt(v)) : sigma_corr; } void MergeOnTheFly::RefineErrorModel(const std::vector &outcomes) { - // Reset to identity up front: every early return below then leaves the model - // inactive (CorrectedSigma returns sigma unchanged) rather than keeping a stale - // a/b from a previous call alongside a freshly-cleared mean map. - // Median of the chi-square(1) distribution: a single observation's squared deviation from its - // reflection mean, divided by its variance, is chi-square(1)-distributed, so its median is this - // fraction of its mean. Used both to de-bias the median-based variance fit and to normalize the - // reported median reduced chi^2 so that honestly calibrated sigmas give 1.0 (not 0.4549). - constexpr double CHI2_1_MEDIAN = 0.454936; - + // Reset to identity up front: every early return below then leaves the model inactive (CorrectedSigma + // returns sigma unchanged) rather than keeping a stale model from a previous call. error_model_active = false; error_model_a = 1.0; error_model_b = 0.0; + error_model_c2 = 0.0; error_model_chi2 = 0.0; error_model_b_resolved = true; error_model_mean_I.clear(); @@ -294,11 +292,12 @@ void MergeOnTheFly::RefineErrorModel(const std::vector &outc reject_down = 0.0; reject_count = 0; - // --- 1. Collect accepted, scaled observations grouped by symmetry-equivalent hkl, - // applying exactly the filters AddImage uses. --- - struct Obs { float I, sigma, p; }; + // --- 1. Collect accepted, scaled observations grouped by symmetry-equivalent hkl, applying the filters + // AddImage uses, down to the floor of the merge with an error model. + struct Obs { float I, sigma, corr, var_bkg, p; }; struct KeyedObs { uint64_t key; Obs obs; }; std::unordered_map> groups; + std::vector keys; // in order of first appearance, so every sum below is in a fixed order ReduceImagesInOrder( outcomes.size(), nthreads, @@ -314,44 +313,36 @@ void MergeOnTheFly::RefineErrorModel(const std::vector &outc continue; if (exclude_ice_rings && r.on_ice_ring) continue; - if (r.partiality < min_partiality) + if (r.partiality < merge_min_partiality) continue; const float I_corr = r.I * r.image_scale_corr; const float sigma_corr = r.sigma * r.image_scale_corr; if (!std::isfinite(I_corr) || !std::isfinite(sigma_corr) || sigma_corr <= 0.0f) continue; - out.push_back({generator(r).pack(), {I_corr, sigma_corr, r.partiality}}); + out.push_back({generator(r).pack(), {I_corr, sigma_corr, r.image_scale_corr, r.var_bkg, r.partiality}}); } }, [&](size_t, const std::vector &out) { - for (const auto &o: out) - groups[o.key].push_back(o.obs); + for (const auto &o: out) { + auto &g = groups[o.key]; + if (g.empty()) + keys.push_back(o.key); + g.push_back(o.obs); + } }); - // --- 2. One global pool of (sigma^2, ^2, bias-corrected squared deviation). For an - // observation in a group of n, the residual from the inverse-variance mean has - // E[(I_i - )^2] = sigma_i^2 (1 - h_i), h_i = w_i / sum_w (its leverage). The - // (b*I)^2 term uses the reflection mean, so the mean (not I_i) is the abscissa. --- - struct Sample { double s2, I2, dev2; }; - std::vector samples; - std::vector> log_spread; // |ln(I / median)|, weight - - for (const auto &[key, obs]: groups) { - if (obs.size() < 2) - continue; + // --- 2. The starting mean of each reflection (counting-sigma weights) and, where there are three + // observations for it to stand on, the median - the robust centre of the outlier test. + std::unordered_map mean; + for (const auto key: keys) { + const auto &obs = groups[key]; double sum_w = 0.0, sum_wI = 0.0; for (const auto &o: obs) { const double w = 1.0 / (static_cast(o.sigma) * o.sigma); sum_w += w; sum_wI += w * o.I; } - if (!(sum_w > 0.0)) - continue; - const double mean = sum_wI / sum_w; - error_model_mean_I[key] = static_cast(mean); - // Robust centre for outlier rejection: the median intensity (resists the very - // outliers the inverse-variance mean is being protected from). Only when active, and only - // where there are three observations for a median to stand on. + mean[key] = sum_wI / sum_w; if (reject_outliers && obs.size() >= 3) { std::vector iv; iv.reserve(obs.size()); @@ -360,144 +351,81 @@ void MergeOnTheFly::RefineErrorModel(const std::vector &outc std::nth_element(iv.begin(), iv.begin() + iv.size() / 2, iv.end()); reject_median_I[key] = iv[iv.size() / 2]; } - const double I2 = mean * mean; - for (const auto &o: obs) { - const double w = 1.0 / (static_cast(o.sigma) * o.sigma); - const double factor = 1.0 - w / sum_w; - if (factor < 0.05) - continue; - const double resid = static_cast(o.I) - mean; - const double capture = capture_coeff * (1.0 - std::min(1.0f, o.p)) * std::max(0.0, mean); - const double s2 = static_cast(o.sigma) * o.sigma + capture * capture; - samples.push_back({s2, I2, resid * resid / factor}); - const auto mit = reject_median_I.find(key); - if (mit != reject_median_I.end() && o.I > 0.0f && mit->second > 0.0f && mean > 0.0) - log_spread.emplace_back(std::fabs(std::log(o.I / mit->second)), I2 / s2); - } } - // The spread of ln(I / median) the band's multiplicative part is drawn from: its weighted median - // absolute value, weighted by (/sigma)^2 so the observations whose ratio counting noise does not - // blur carry it - as the rotation merge measures it. - if (!log_spread.empty()) { - std::sort(log_spread.begin(), log_spread.end()); - double total = 0.0; - for (const auto &[v, w]: log_spread) total += w; - double run = 0.0; - size_t j = 0; - for (; j + 1 < log_spread.size(); ++j) { - run += log_spread[j].second; - if (run >= 0.5 * total) break; - } - const double s = 1.4826 * log_spread[j].first; - reject_up = std::exp(reject_nsigma * s) - 1.0; - reject_down = 1.0 - std::exp(-reject_nsigma * s); - } - - // --- 3. Fit global dev2 = a*sigma^2 + b^2*^2. Bin by intensity (the per-observation - // dev2 is chi-square-1 noisy) and take medians; weight the bins by 1/dev2^2 so it - // is a *relative* fit - otherwise the strong bins (which fix b) swamp the weak - // bins (which fix a) and the weak sigmas stay over-confident. --- - constexpr int n_bins = 16; - if (samples.size() < static_cast(8 * n_bins)) - return; // too little multiplicity to fit -> leave identity - - std::sort(samples.begin(), samples.end(), - [](const Sample &p, const Sample &q) { return p.I2 < q.I2; }); - - auto median = [](std::vector &v) { - std::nth_element(v.begin(), v.begin() + v.size() / 2, v.end()); - return v[v.size() / 2]; + // --- 3. Fit the model (ErrorModel.h, FitStillsErrorModel) on the scatter about the reflection means, + // the counting variance rebuilt at the mean, and refit with the means the fitted model weights: + // the means and the model each depend on the other, and a single pass leaves the weights of + // the strong reflections at the counting-sigma means the fit started from. Measured on serial + // and rotation-as-serial sets, a few rounds raise the anomalous signal of the merge; they have + // settled after four. + constexpr int ROUNDS = 4; + std::vector samples; + StillsErrorModelFit fit; + double a = 1.0, b2 = 0.0, c2 = 0.0; + auto variance = [&](const Obs &o, double m) { + return a * CountingVariance(o.sigma, o.corr, o.var_bkg, o.p, m) + (b2 + c2 * PartialityTerm(o.p)) * m * m; }; - - // Per-intensity-bin medians of (sigma^2, ^2, dev2). - std::vector bs2, bI2, bd2; - bs2.reserve(n_bins); bI2.reserve(n_bins); bd2.reserve(n_bins); - const size_t per = samples.size() / n_bins; - for (int bin = 0; bin < n_bins; ++bin) { - const size_t lo = bin * per; - const size_t hi = (bin == n_bins - 1) ? samples.size() : lo + per; - std::vector vs2, vI2, vd2; - vs2.reserve(hi - lo); vI2.reserve(hi - lo); vd2.reserve(hi - lo); - for (size_t i = lo; i < hi; ++i) { - vs2.push_back(samples[i].s2); - vI2.push_back(samples[i].I2); - vd2.push_back(samples[i].dev2); + for (int round = 0; round < ROUNDS; ++round) { + samples.clear(); + for (const auto key: keys) { + const auto &obs = groups[key]; + if (obs.size() < 2) + continue; + const double m = mean[key]; + // The residual from the weighted mean has E[(I_i - )^2] = var_i (1 - h_i), h_i = w_i / sum_w its + // leverage, with the weights of the model that made the mean. + double sum_w = 0.0; + for (const auto &o: obs) + sum_w += 1.0 / variance(o, m); + for (const auto &o: obs) { + const double factor = 1.0 - 1.0 / (variance(o, m) * sum_w); + if (factor < 0.05) + continue; + const double resid = static_cast(o.I) - m; + samples.push_back({CountingVariance(o.sigma, o.corr, o.var_bkg, o.p, m), m * m, + PartialityTerm(o.p) * m * m, resid * resid / factor}); + } + } + fit = FitStillsErrorModel(samples); + if (!fit.active) + return; // too little multiplicity to fit -> leave identity + a = fit.a; b2 = fit.b2; c2 = fit.c2; + for (const auto key: keys) { + double sum_w = 0.0, sum_wI = 0.0; + for (const auto &o: groups[key]) { + const double w = 1.0 / variance(o, mean[key]); + sum_w += w; + sum_wI += w * o.I; + } + mean[key] = sum_wI / sum_w; } - bs2.push_back(median(vs2)); - bI2.push_back(median(vI2)); - // The per-observation dev2 is sigma^2 * chi-square(1)-distributed, whose MEDIAN is 0.4549 of - // its mean. Fitting the model to the robust median would therefore calibrate the variances to - // 0.4549x their true value (reduced chi^2 ~ 1/0.4549 = 2.2). Divide the median by that constant - // to recover an unbiased estimate of the mean (E[dev2] = sigma^2), keeping the robustness of - // the median while targeting reduced chi^2 = 1. - bd2.push_back(median(vd2) / CHI2_1_MEDIAN); } - - // Relative-weighted (1/dev2^2) least squares for (a, b^2). Floor the weight's dev2 at a - // small fraction of the typical bin dev2: an absolute floor (1e-30) does not stop a - // near-zero-scatter bin from acquiring a runaway weight and hijacking the fit, so the - // floor must scale with the data. The regression target keeps the unfloored dev2. - std::vector bd2_sorted = bd2; - const double dev2_floor = std::max(1e-30, 1e-3 * median(bd2_sorted)); - double Ass = 0, AsI = 0, AII = 0, Bs = 0, BI = 0; - for (int bin = 0; bin < n_bins; ++bin) { - const double s2 = bs2[bin], I2 = bI2[bin], d2 = bd2[bin]; - const double d2w = std::max(d2, dev2_floor); - const double wgt = 1.0 / (d2w * d2w); - Ass += wgt * s2 * s2; - AsI += wgt * s2 * I2; - AII += wgt * I2 * I2; - Bs += wgt * s2 * d2; - BI += wgt * I2 * d2; - } - - // Reject a near-collinear (ill-conditioned) system *relatively*: det lies in - // [0, Ass*AII] by Cauchy-Schwarz, so compare against that scale rather than 1e-30. - const double det = Ass * AII - AsI * AsI; - if (!(det > 1e-10 * Ass * AII)) - return; - const double a_fit = (Bs * AII - BI * AsI) / det; - const double b2_fit = (Ass * BI - AsI * Bs) / det; - const double a = std::clamp(a_fit, 0.25, 100.0); - const double b2 = std::max(b2_fit, 0.0); - // Whether 1/b can be reported as ISa, decided as the rotation fit decides it (ErrorModelUpdate): - // b^2 two of its standard errors clear of zero, the error from the bins' scatter about the line, - // and a inside its bounds - beside a clamped a the b^2 above is not the solution of any fit, and - // it is a near-zero-scatter bin that puts a on its bound (see the weight floor above). - double chi = 0.0; - for (int bin = 0; bin < n_bins; ++bin) { - if (!(bd2[bin] > 0.0)) continue; - const double r = (bd2[bin] - a_fit * bs2[bin] - b2_fit * bI2[bin]) / bd2[bin]; - chi += r * r; - } - error_model_b_resolved = a == a_fit && b2_fit >= 2.0 * std::sqrt(chi / (n_bins - 2) * Ass / det); + for (const auto key: keys) + error_model_mean_I[key] = static_cast(mean[key]); error_model_a = a; error_model_b = std::sqrt(b2); + error_model_c2 = c2; + error_model_b_resolved = fit.b_resolved; error_model_active = true; // A still's observations of one reflection spread by their partiality and per-crystal scale, not by a - // multiplicative factor about the median: many read near zero, so the spread of ln(I / median) is - // wide and the band drawn from it lets through even a powder grain or a neighbour's spot at hundreds - // of sigma. The error model's b is that spread measured as it is, so stills test the plain band of - // n sigma about the median. - if (stills) { - reject_up = reject_nsigma * error_model_b; - reject_down = reject_nsigma * error_model_b; - } + // multiplicative factor about the median: many read near zero, so a band drawn from the spread of + // ln(I / median) would let through even a powder grain or a neighbour's spot at hundreds of sigma. + // The band is the plain one of n sigma about the median. + reject_up = reject_nsigma * error_model_b; + reject_down = reject_nsigma * error_model_b; - // Achieved goodness of fit: the median of the per-observation dev2/(a*sigma^2 + (b*)^2). That - // ratio is chi-square(1)-distributed (median 0.4549) when the sigmas are correct, so normalize by - // CHI2_1_MEDIAN to report a median reduced chi^2 that targets 1.0. The median (not mean) keeps it - // robust to the heavy outlier tail of serial data. + // Achieved goodness of fit: the median of dev2 / var over the samples, which is chi^2(1)-distributed + // (median 0.4549) when the sigmas are right; normalised so that honest sigmas report 1.0. std::vector chi2; chi2.reserve(samples.size()); - for (const auto &s: samples) { - const double v = a * s.s2 + b2 * s.I2; - if (v > 0.0) - chi2.push_back(s.dev2 / v); + for (const auto &x: samples) + chi2.push_back(x.dev2 / (a * x.s2 + b2 * x.I2 + c2 * x.P2)); + if (!chi2.empty()) { + std::nth_element(chi2.begin(), chi2.begin() + chi2.size() / 2, chi2.end()); + error_model_chi2 = chi2[chi2.size() / 2] / ERROR_MODEL_CHI2_1_MEDIAN; } - error_model_chi2 = chi2.empty() ? 0.0 : median(chi2) / CHI2_1_MEDIAN; } bool MergeOnTheFly::Mask(const IntegrationOutcome &outcome) const { @@ -842,7 +770,7 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m continue; if (!AcceptReflection(r, d_min_limit_A, d_max_limit_A)) continue; - if (r.partiality < min_partiality) + if (r.partiality < MergeFloor()) continue; const float I_corr = r.I * r.image_scale_corr; diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index 3e1f22ddd..c30887d18 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -309,7 +309,19 @@ class MergeOnTheFly { // Apply image_cc_limit in Mask(). One flag for the whole engine, not a per-call argument, so the // merge, the error model and MergeStats can never disagree about which images are in. bool filter_by_image_cc = false; + // The partiality below which an observation is not merged. A merge with a fitted error model carries + // the partiality model's own error in each variance (ErrorModel.h, the stills model's c term), which + // weights a low-partiality observation by what it is worth, so it takes them down to + // MERGE_MIN_PARTIALITY; a merge without one - the references the scale and the partiality are + // refined against - stops at min_partiality. Measured on serial and rotation-as-serial sets: merging + // down to 0.3 with the term raises the anomalous map at the sulfur sites, and without the term those + // observations make it worse; going further, to 0.1, raises it a little more but costs the outermost + // shell (CC to the deposited model 0.69 -> 0.60 on one set) - the partiality model's error in the + // tail is not only a variance there. + static constexpr double MERGE_MIN_PARTIALITY = 0.3; double min_partiality = 0.02; + double merge_min_partiality = 0.02; + [[nodiscard]] double MergeFloor() const { return error_model_active ? merge_min_partiality : min_partiality; } // The share of a partial the model could not see, (1 - p), carried in its variance as // (capture_coeff * (1 - p) * )^2 - the rotation combine's capture term, per observation. double capture_coeff = 0.0; @@ -324,20 +336,27 @@ class MergeOnTheFly { std::unordered_map accumulator; // exported in key order - // Global error model (XDS form): sigma_corr^2 = a*sigma^2 + (b*)^2. a rescales the - // (under-estimated) counting variance; the (b*)^2 term adds the intensity- - // proportional systematic error that counting statistics miss, so strong reflections - // are no longer over-weighted. ISa = 1/b is the asymptotic I/sigma. Refined from the - // scatter of symmetry equivalents (RefineErrorModel); identity until then. + // Global error model (XDS form plus a partiality term, ErrorModel.h): + // sigma_corr^2 = a*sigma^2 + (b*)^2 + c^2 ((1 - p) / p)^2 ^2. a rescales the (under-estimated) + // counting variance; the (b*)^2 term adds the intensity-proportional systematic error that counting + // statistics miss, so strong reflections are no longer over-weighted; the c term is the partiality + // model's error. ISa = 1/b is the asymptotic I/sigma. Refined from the scatter of symmetry equivalents + // (RefineErrorModel); identity until then. bool error_model_active = false; double error_model_a = 1.0; double error_model_b = 0.0; + double error_model_c2 = 0.0; // the partiality term, c^2 ((1 - p) / p)^2 ^2 (ErrorModel.h) double error_model_chi2 = 0.0; // achieved median reduced chi^2 (~1.0 = honestly calibrated sigmas) bool error_model_b_resolved = true; // as ErrorModelFit::b_resolved; 1/b is reported as ISa only then // The (b*I)^2 term uses the reflection's *mean* intensity (constant over its // observations), so it inflates sigma without biasing the inverse-variance weights - // using the per-observation I_i instead would over-weight down-fluctuated points. std::unordered_map error_model_mean_I; + // An observation's counting variance on the merge's scale, its signal part rebuilt at the reflection + // mean I_mean (plus the capture term), and the partiality term's factor ((1 - p) / p)^2. + [[nodiscard]] double CountingVariance(float sigma_corr, float image_scale_corr, float var_bkg, float partiality, + double I_mean) const; + static double PartialityTerm(float partiality); [[nodiscard]] float CorrectedSigma(float I_corr, float sigma_corr, float image_scale_corr, float var_bkg, float partiality, uint64_t hkl_key) const; @@ -351,10 +370,9 @@ class MergeOnTheFly { double reject_nsigma = 6.0; std::unordered_map reject_median_I; // The asymmetric band of OutlierBand.h: exp(+-n s) - 1 about the reflection mean, s the spread of - // ln(I / median) measured on this merge's observations; on stills n b, the linear band. + // ln(I / median); the stills merge sets both to n b, the linear band. double reject_up = 0.0, reject_down = 0.0; size_t reject_count = 0; - bool stills = false; bool Mask(const IntegrationOutcome &outcome) const; @@ -383,6 +401,7 @@ public: [[nodiscard]] bool ErrorModelActive() const { return error_model_active; } [[nodiscard]] double ErrorModelA() const { return error_model_a; } [[nodiscard]] double ErrorModelB() const { return error_model_b; } + [[nodiscard]] double ErrorModelC() const { return std::sqrt(error_model_c2); } [[nodiscard]] double ErrorModelChi2() const { return error_model_chi2; } [[nodiscard]] bool ErrorModelBResolved() const { return error_model_b_resolved; } diff --git a/image_analysis/scale_merge/ScaleOnTheFly.cpp b/image_analysis/scale_merge/ScaleOnTheFly.cpp index 5cce4b17d..9fcf83d8d 100644 --- a/image_analysis/scale_merge/ScaleOnTheFly.cpp +++ b/image_analysis/scale_merge/ScaleOnTheFly.cpp @@ -137,8 +137,13 @@ void ScaleOnTheFly::Scale(IntegrationOutcome &integration_outcome) const { obs.push_back({coeff, static_cast(r.I), sigma2, var_bkg}); } + // Too few reflections to put the image on the merge's scale: it is left out of the merge (a + // non-finite correction is what the merge skips on) rather than merged at the scale of 1 its + // prediction carries, which is no scale at all. if (obs.size() < MIN_REFLECTIONS) { clear_scale(); + for (auto &r: integration_outcome.reflections) + r.image_scale_corr = NAN; return; } diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 77bb852d4..3ff2091d3 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -1764,8 +1764,8 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { // XDS convention, so the pair reads directly against a CORRECT.LP; the stills ISa is // already the whole-range 1/b, there being no asymptote tier on this path. const auto em = ToXdsErrorModel(merge_engine.ErrorModelA(), merge_engine.ErrorModelB()); - logger.Info("Error model (XDS convention): a={:.3f} b={:.3e} ISa={:.1f} chi2={:.2f}", - em.a, em.b, em.isa, merge_engine.ErrorModelChi2()); + logger.Info("Error model (XDS convention): a={:.3f} b={:.3e} ISa={:.1f} partiality c={:.3f} chi2={:.2f}", + em.a, em.b, em.isa, merge_engine.ErrorModelC(), merge_engine.ErrorModelChi2()); } merge_engine.AddImages(merge_input); if (config_.export_observations && !for_search && write_output) diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 6d98e9a00..cc30c1bb5 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -2589,8 +2589,8 @@ int RugnuxInputRun::PrepareRugnuxInput() { // XDS convention, so the pair reads directly against a CORRECT.LP; the stills ISa is // already the whole-range 1/b, there being no asymptote tier on this path. const auto em = ToXdsErrorModel(merge_engine.ErrorModelA(), merge_engine.ErrorModelB()); - logger.Info("Error model (XDS convention): a={:.3f} b={:.3e} ISa={:.1f} chi2={:.2f}", - em.a, em.b, em.isa, merge_engine.ErrorModelChi2()); + logger.Info("Error model (XDS convention): a={:.3f} b={:.3e} ISa={:.1f} partiality c={:.3f} chi2={:.2f}", + em.a, em.b, em.isa, merge_engine.ErrorModelC(), merge_engine.ErrorModelChi2()); } merge_engine.AddImages(reflections); merged_reflections = merge_engine.ExportReflections(); diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index 1e9050001..938539f10 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -527,6 +527,50 @@ TEST_CASE("ErrorModel_SameOnAnyNumberOfThreads") { CHECK(one.b2 == many.b2); } +// The stills model: samples drawn with a known a, b and partiality term c - c^2 ((1 - p) / p)^2 ^2, p spread +// over [0.1, 1] - give those three back. +TEST_CASE("StillsErrorModel_RecoversThePartialityTerm") { + std::mt19937 rng(7); + std::uniform_real_distribution u(0.0, 1.0); + std::normal_distribution z(0.0, 1.0); + const double a = 1.2, b = 0.08, c = 0.15; + std::vector pool; + for (int i = 0; i < 200000; ++i) { + const double s2 = std::pow(10.0, 2.0 * u(rng)); + const double snr = 300.0 * std::pow(10.0, -4.0 * u(rng)); + const double I2 = snr * snr * s2; + const double p = 0.1 + 0.9 * u(rng); + const double P2 = std::pow((1.0 - p) / p, 2.0) * I2; + const double e = z(rng); + pool.push_back({s2, I2, P2, (a * s2 + b * b * I2 + c * c * P2) * e * e}); + } + const auto fit = FitStillsErrorModel(pool); + REQUIRE(fit.active); + CHECK(fit.a == Catch::Approx(a).epsilon(0.05)); + CHECK(std::sqrt(fit.b2) == Catch::Approx(b).epsilon(0.05)); + CHECK(std::sqrt(fit.c2) == Catch::Approx(c).epsilon(0.05)); +} + +// Without a partiality error in the data the term stays at (near) zero. +TEST_CASE("StillsErrorModel_NoPartialityErrorNoTerm") { + std::mt19937 rng(8); + std::uniform_real_distribution u(0.0, 1.0); + std::normal_distribution z(0.0, 1.0); + std::vector pool; + for (int i = 0; i < 200000; ++i) { + const double s2 = std::pow(10.0, 2.0 * u(rng)); + const double snr = 300.0 * std::pow(10.0, -4.0 * u(rng)); + const double I2 = snr * snr * s2; + const double p = 0.1 + 0.9 * u(rng); + const double e = z(rng); + pool.push_back({s2, I2, std::pow((1.0 - p) / p, 2.0) * I2, (1.1 * s2 + 0.0025 * I2) * e * e}); + } + const auto fit = FitStillsErrorModel(pool); + REQUIRE(fit.active); + CHECK(std::sqrt(fit.c2) < 0.01); + CHECK(std::sqrt(fit.b2) == Catch::Approx(0.05).epsilon(0.05)); +} + // Data that never reach where b could be seen: a is fitted alone and b held at 0. TEST_CASE("ErrorModel_BNotMeasuredOnWeakData") { const auto pool = SyntheticErrorModelSamples(0.9, 0.05, 1.5, 0.0); @@ -1266,3 +1310,32 @@ TEST_CASE("ScalingSettings_MinPartialityRange") { CHECK_NOTHROW(s.MinPartiality(0.5)); CHECK(s.GetMinPartiality() == 0.5); } + +// An image with too few reflections to fit its scale on is left out of the merge, not merged at the scale +// of 1 it was predicted with. +TEST_CASE("ScaleOnTheFly_TooFewReflectionsLeavesTheImageOut") { + DiffractionExperiment x; + x.SetSpaceGroup(gemmi::get_spacegroup_by_name("P 1")); + std::vector outcomes(3); + for (size_t i = 0; i < outcomes.size(); ++i) { + const int n = i == 2 ? 10 : 50; + for (int h = 1; h <= n; ++h) { + Reflection r{}; + r.h = h; r.k = 1; r.l = 1; + r.d = 3.0f; + r.I = 100.0f * static_cast(h) * (i == 1 ? 2.0f : 1.0f); + r.sigma = 1.0f; + r.var_bkg = 0.5f; + r.partiality = 1.0f; + r.prescaling_corr = 1.0f; + r.image_scale_corr = 1.0f; + outcomes[i].reflections.push_back(r); + } + } + ScaleOnTheFly(x, MergeAll(x, outcomes, 1, std::nullopt)).Scale(outcomes, 1); + CHECK(outcomes[0].image_scale_g.has_value()); + CHECK(std::isfinite(outcomes[1].reflections[0].image_scale_corr)); + CHECK(!outcomes[2].image_scale_g.has_value()); + for (const auto &r: outcomes[2].reflections) + CHECK(std::isnan(r.image_scale_corr)); +} From 06fe88a86251759cc4d2d8278bc48b52c5508013 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 09:10:05 +0100 Subject: [PATCH 08/34] Stills given cell: groups of cells less than 1 % apart in volume are one crystal form The given-cell population filter left out every frame beyond a valley of the refined-volume histogram. One crystal form whose cell drifts - a crystal swelling with dose, a cell refined a little differently in different orientations - also falls into groups, and the filter then threw away part of one structure's data: a rotation sweep processed as stills, its cell 0.25 % larger in volume in its second half, lost 841 of 1800 frames, and its two half-datasets kept opposite groups. Groups whose mean volumes differ by less than 1 % are now merged as one form; the two forms of the stills battery's two-form sample are 9 % apart and are still separated. (A test on the groups' merged intensities was tried first: on that sweep the two groups correlate only 0.66-0.88 in their noise-corrected true intensities, which draws no clearer line than the cells.) That sweep's every-3rd-frame subset: S-site anomalous peaks 1.14/-0.96 -> 1.50/0.36. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- rugnux/RugnuxScaleMerge.cpp | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 3ff2091d3..ff164fb62 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -92,6 +92,9 @@ using namespace rugnux_internal; namespace { + // Two groups of a given cell's frames whose mean volumes differ by less than this (relative) are one + // crystal form (see the given-cell population below). + constexpr double MIN_FORM_SEPARATION = 0.01; // The space-group search's resolution cut on a P1 merge: where the best non-increasing fit of its // thin-shell drops under 1 (0: no cut). See the search below. double SearchCut(const std::vector &merged) { @@ -700,11 +703,35 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { for (const auto &o : outcomes) if (!o.reflections.empty()) rel.push_back(o.latt.CalcVolume() / given - 1.0); - const auto [low, high] = GivenCellPopulation(rel); + auto [low, high] = GivenCellPopulation(rel); std::string all; for (double r : rel) all += fmt::format(" {:.4f}", r); logger.Debug("Given cell: refined volumes relative to it:{}", all); + // A histogram valley alone does not make two crystal forms: one form whose cell drifts - a crystal + // swelling with dose, a cell refined a little differently in different orientations - falls into + // groups too (a drift through its range piles up at its two ends), and leaving one group out throws + // away part of the data of one structure. Measured: a rotation sweep processed as stills, its cell + // 0.25 % larger in volume in the second half than in the first, lost 841 of 1800 frames to the + // valley. Two forms of one protein that index on one cell differ by far more (9 % between the two + // forms of the battery's two-form sample), so groups closer than MIN_FORM_SEPARATION are one form. + // Their intensities would be the better judge, but a stills merge of groups taken from different + // orientations of one rotating crystal correlates only 0.8 with itself (0.66-0.88 measured), which + // is no clearer a line than the cells draw. + if (std::isfinite(low) || std::isfinite(high)) { + double sum_in = 0.0, sum_out = 0.0; + size_t n_in = 0, n_out = 0; + for (double r : rel) + (r >= low && r <= high) ? (sum_in += r, ++n_in) : (sum_out += r, ++n_out); + const double separation = std::fabs(sum_out / n_out - sum_in / n_in); + if (separation < MIN_FORM_SEPARATION) { + logger.Info("Given cell: the frames outside its volume population ({:+.2f}% to {:+.2f}%) are " + "{:.2f}% from it in volume, too close to be another crystal form - every frame is merged", + 100 * low, 100 * high, 100 * separation); + low = -std::numeric_limits::infinity(); + high = std::numeric_limits::infinity(); + } + } size_t below = 0, above = 0; for (auto &o : outcomes) { if (o.reflections.empty()) From c00c757c04c3dd75143906facf20ac0d3d809a19 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 09:39:00 +0100 Subject: [PATCH 09/34] Stills merge statistics: CC_anom and SigAno where the Friedel hands are kept apart The stills merge reported no anomalous statistics. With -A it now gives, per shell and overall, CC_anom between the frame-whole half-sets the stills CC1/2 and R_split use (so the two anomalous differences come from different crystals) and SigAno against the merged sigmas, over the acentric reflections measured in both hands. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/scale_merge/Merge.cpp | 40 ++++++++++++++++++++++++++++ image_analysis/scale_merge/Merge.h | 5 ++-- tests/MergeScaleTest.cpp | 34 +++++++++++++++++++++++ 3 files changed, 77 insertions(+), 2 deletions(-) diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index 69b06c93e..b00142b75 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -6,6 +6,8 @@ #include #include #include +#include +#include #include #include @@ -524,6 +526,9 @@ struct ShellAccum { CorrelationCoefficient cc_ref; CorrelationCoefficient cc_split; + CorrelationCoefficient cc_anom; + double anom_abs_diff = 0.0, anom_sigma = 0.0; + int anom_pairs = 0; double split_abs_diff = 0.0; double split_sum = 0.0; int split_pairs = 0; @@ -745,6 +750,37 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m } } + // Anomalous statistics where the merge keeps the Friedel hands apart: an I(-) is stored at the negated + // index of its I(+). CC_anom between the frame-whole half-sets (I_split), so the two differences come + // from different crystals; SigAno against the merged sigmas. Acentric reflections only. + if (!scaling_settings.GetMergeFriedel()) { + std::map, const MergedReflection *> by_index; + for (const auto &m: merged) + by_index[{m.h, m.k, m.l}] = &m; + const auto &ops = space_group.operations(); + for (const auto &[hkl, plus]: by_index) { + const auto it = by_index.find({-plus->h, -plus->k, -plus->l}); + if (it == by_index.end() || hkl <= it->first) // each pair once + continue; + const MergedReflection &minus = *it->second; + if (ops.is_reflection_centric({plus->h, plus->k, plus->l})) + continue; + const auto shell = shells.GetShell(plus->d); + if (!shell.has_value() || *shell < 0 || *shell >= n_shells) + continue; + for (auto *a: {&acc[*shell], &split_overall}) { + if (std::isfinite(plus->I) && std::isfinite(minus.I) && plus->sigma > 0.0f && minus.sigma > 0.0f) { + a->anom_abs_diff += std::fabs(plus->I - minus.I); + a->anom_sigma += std::hypot(plus->sigma, minus.sigma); + a->anom_pairs++; + } + if (std::isfinite(plus->I_split[0]) && std::isfinite(plus->I_split[1]) + && std::isfinite(minus.I_split[0]) && std::isfinite(minus.I_split[1])) + a->cc_anom.Add(plus->I_split[0] - minus.I_split[0], plus->I_split[1] - minus.I_split[1]); + } + } + } + // Per-reflection mean , and a per-reflection accumulator for R_meas - it needs |I_i - |, // so the observations are visited again now that the means are known. std::unordered_map merged_I; @@ -826,6 +862,10 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m // R_split as White et al. (2012) J. Appl. Cryst. 45, 335-341 define it (CrystFEL's compare_hkl). auto set_split = [](const ShellAccum &a, MergeStatisticsShell &ss) { + if (a.anom_pairs > 0) { + ss.abs_diff_over_sigma_anomalous = a.anom_abs_diff / a.anom_sigma; + ss.cc_anom = a.cc_anom.GetCC(); + } ss.n_split_pairs = a.split_pairs; if (a.split_pairs == 0) return; diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index c30887d18..ea701e683 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -48,14 +48,15 @@ struct MergeStatisticsShell { // Anomalous signal-to-noise (XDS "SigAno" / mmCIF pdbx_absDiff_over_sigma_anomalous): // <|I(+)-I(-)|> / over acentric reflections measured in both hands. NaN when - // there is no anomalous split (e.g. Friedel-merged with no mates, or the stills path). + // there is no anomalous split (e.g. Friedel-merged with no mates). double abs_diff_over_sigma_anomalous = NAN; // CCanom (AIMLESS "CCanom", phenix "cc_anom"): the correlation of the anomalous difference // measured twice - I(+)-I(-) from one half of the observations against the same difference from // the other half, over acentric reflections whose BOTH mates split into two non-empty halves. // Unlike SigAno it is not a ratio against the error model, so a mis-estimated sigma cannot move - // it. NaN when no reflection has an anomalous split (the stills path), which is not the same + // it. On stills the halves are the frame-whole ones of R_split, so the two differences come from + // different crystals. NaN when no reflection has an anomalous split, which is not the same // statement as zero - zero means measured and absent. double cc_anom = NAN; diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index 938539f10..74b9e4825 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -1339,3 +1339,37 @@ TEST_CASE("ScaleOnTheFly_TooFewReflectionsLeavesTheImageOut") { for (const auto &r: outcomes[2].reflections) CHECK(std::isnan(r.image_scale_corr)); } + +// Stills with the Friedel hands kept apart report CC_anom (between frame-whole half-sets) and SigAno: a +// consistent anomalous difference far above the noise correlates close to 1. +TEST_CASE("MergeOnTheFly_StillsAnomalousStatistics") { + DiffractionExperiment x; + x.SetSpaceGroup(gemmi::get_spacegroup_by_name("P 1")); + ScalingSettings s = x.GetScalingSettings(); + s.MergeFriedel(false); + x.ImportScalingSettings(s); + std::mt19937 rng(11); + std::normal_distribution gauss(0.0f, 1.0f); + std::vector dano(61); + for (auto &v: dano) v = 0.2f * gauss(rng); + std::vector outcomes(40); + for (size_t i = 0; i < outcomes.size(); ++i) + for (int h = 1; h <= 60; ++h) + for (int sign: {1, -1}) { + Reflection r{}; + r.h = sign * h; r.k = sign * 2; r.l = sign * 3; + r.d = 30.0f / std::sqrt(static_cast(h * h + 13)); + r.I = 1000.0f * (1.0f + 0.5f * sign * dano[h]) + 5.0f * gauss(rng); + r.sigma = 5.0f; + r.partiality = 1.0f; + r.prescaling_corr = 1.0f; + r.image_scale_corr = 1.0f; + outcomes[i].reflections.push_back(r); + } + MergeOnTheFly merge(x); + merge.AddImages(outcomes); + const auto merged = merge.ExportReflections(); + const auto stats = merge.MergeStats(merged, outcomes); + CHECK(stats.overall.cc_anom > 0.9); + CHECK(stats.overall.abs_diff_over_sigma_anomalous > 5.0); +} From dce6bd25b68d72504abd518965ee865029f7131a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 14:47:46 +0100 Subject: [PATCH 10/34] Stills merge: the floor of the merge with the partiality term is p = 0.2 With the run's width table balanced on the partials, the low-partiality observations no longer carry the high-resolution bias that set the floor at 0.3. On the stills panel (one run per set, the deposited model's anomalous sites and CC to its intensities) 0.2 keeps the anomalous gain of the lower floors on real serial data and rotation frames processed as stills, with the same outermost shell as 0.3 and 0.1; merging every observation (no floor) breaks one rotation-as-serial set. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/scale_merge/Merge.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index ea701e683..8f4206ed3 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -315,11 +315,10 @@ class MergeOnTheFly { // weights a low-partiality observation by what it is worth, so it takes them down to // MERGE_MIN_PARTIALITY; a merge without one - the references the scale and the partiality are // refined against - stops at min_partiality. Measured on serial and rotation-as-serial sets: merging - // down to 0.3 with the term raises the anomalous map at the sulfur sites, and without the term those - // observations make it worse; going further, to 0.1, raises it a little more but costs the outermost - // shell (CC to the deposited model 0.69 -> 0.60 on one set) - the partiality model's error in the - // tail is not only a variance there. - static constexpr double MERGE_MIN_PARTIALITY = 0.3; + // down to 0.2 with the term raises the anomalous map at the sulfur sites, and without the term those + // observations make it worse. With the run's width table balanced on the partials, 0.2 and 0.1 give + // the same outermost shell; merging every observation (no floor) breaks a rotation-as-serial set. + static constexpr double MERGE_MIN_PARTIALITY = 0.2; double min_partiality = 0.02; double merge_min_partiality = 0.02; [[nodiscard]] double MergeFloor() const { return error_model_active ? merge_min_partiality : min_partiality; } From b349f95e9dc3603631501cf46719e9e0bb80ac7c Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 14:50:19 +0100 Subject: [PATCH 11/34] Still battery: the float32 SwissFEL sets are dropped, their int16 copies stay The float32 Alvra sets merge to the same numbers as their int16 copies and only test reading the uncompressed float32 format, at many times the I/O; the battery keeps the int16 copies, and the data page keeps the originals as the source of the conversion. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- docs/EXTERNAL_TEST_DATA_STILLS.md | 14 +++++++------- tools/battery/still_manifest.json | 2 -- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/EXTERNAL_TEST_DATA_STILLS.md b/docs/EXTERNAL_TEST_DATA_STILLS.md index b68d40972..859a752a9 100644 --- a/docs/EXTERNAL_TEST_DATA_STILLS.md +++ b/docs/EXTERNAL_TEST_DATA_STILLS.md @@ -30,8 +30,8 @@ the paper of the entry. | `lcls242_s`, `lcls242_t` | [242](https://cxidb.org/id-242.html), [10.11577/3374214](https://doi.org/10.11577/3374214) | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | [9O4S](https://www.rcsb.org/structure/9O4S) (small cell), [9O4T](https://www.rcsb.org/structure/9O4T) (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) | | `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) | | `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) | -| `alvra103`, `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) | -| `alvra213`, `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) | +| `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) | +| `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) | Every set of the manifest is in the table and every set in the table is in the manifest, with the same CXIDB entry, PDB entry, space group and dmin; `tools/battery/doc_check.py` checks @@ -63,8 +63,8 @@ broken download resumes with `curl -C -`. | `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | **subset**: runs 256-267 (44 of about 220 stream files; ~610 GB of ~3 TB) | ~700 GB | | `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | **subset**: 3 files; the deposited stream has 14,445 frames | 4.7 GB | | `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | **subset**: 9 runs; the deposited stream has 37,006 frames | 5.7 GB | -| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry; the set reads `swissfel_103/ref/jf-16M_mask.geom`, a copy with `mask_file` pointed at that mask (see below) | ~11,000 | **subset**: 15 runs of one day; the deposition indexed 199,136 of 2,251,787 images | ~750 GB | -| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs, through `swissfel_213/ref/mbco_march_mask.geom`, a copy with `mask_file` pointed at it (see below) | ~38,000 | all 24 dark runs of the March 2019 series | ~430 GB | +| `alvra103_int16` (source) | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry; the set reads `swissfel_103/ref/jf-16M_mask.geom`, a copy with `mask_file` pointed at that mask (see below) | ~11,000 | **subset**: 15 runs of one day; the deposition indexed 199,136 of 2,251,787 images | ~750 GB | +| `alvra213_int16` (source) | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs, through `swissfel_213/ref/mbco_march_mask.geom`, a copy with `mask_file` pointed at it (see below) | ~38,000 | all 24 dark runs of the March 2019 series | ~430 GB | **The masks of the SwissFEL geometry files.** rugnux takes the pixel masks and bad regions a CrystFEL geometry file names. The SwissFEL files name their mask with an absolute path on the facility's disk, @@ -78,9 +78,9 @@ that line differs. **The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV, 68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The `_int16` sets are copies with the same layout in which `/data/data` holds whole photons (keV divided -by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; the battery runs -them with the same arguments as the float32 originals (rugnux measures the gain from the frames), and -they merge to the same numbers. Make them with +by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller. The battery runs only the +copies: they merge to the same numbers as the float32 originals (rugnux measures the gain from the +frames), and the originals are kept only as the source of the conversion and as a test of the format. Make them with `tools/battery/still/float32_to_int16.py ` (h5py and hdf5plugin), into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV, keeping the tarball's directory tree), named `_int16.h5`. One MbCO dark run (`run_000238`) holds no frames and has no copy. diff --git a/tools/battery/still_manifest.json b/tools/battery/still_manifest.json index 531806319..92c7aa214 100644 --- a/tools/battery/still_manifest.json +++ b/tools/battery/still_manifest.json @@ -10,8 +10,6 @@ {"id": "lcls242_t", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4T", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2348, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--wavelength", "1.294", "-C", "96.031,94.536,50.529,90,91.18,90", "-S", "P21"], "tiers": {}}, {"id": "euxfel83", "cxidb": 83, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6GTH", "sg": "P 32 2 1", "d_cut": 1.69, "r_free_deposited": 0.2096, "images_deposited": 14445, "inputs": ["euxfel_83/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_83/ref/agipd_mar18_v10.geom", "--wavelength", "1.3332", "-C", "41.84,41.84,233.28,90,90,120", "-S", "P3221"], "tiers": {"smoke": {"images": 2000}}}, {"id": "euxfel152", "cxidb": 152, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6U57", "sg": "I 2 3", "d_cut": 2.8, "r_free_deposited": 0.2489, "images_deposited": 37006, "inputs": ["euxfel_152/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_152/ref/kdo_embedded.geom", "--wavelength", "1.3332", "-C", "118,118,118,90,90,90", "-S", "I23"], "tiers": {"smoke": {"images": 2000}}}, - {"id": "alvra103", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/raw/run*.JF06T32V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M_mask.geom", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221"], "tiers": {}}, {"id": "alvra103_int16", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/conv/run*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M_mask.geom", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221"], "tiers": {}}, - {"id": "alvra213", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march_mask.geom", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21"], "tiers": {}}, {"id": "alvra213_int16", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/conv/march_dark_MbCO_II-March-06/data/dark/run_*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march_mask.geom", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21"], "tiers": {}} ]} From 3b84fd57308cc36e13780bb341288293845adf90 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 19:29:20 +0100 Subject: [PATCH 12/34] Stills observation export: the panel of a prediction is looked up on the detector plane The PANEL column of the observation table matched the predicted position, which is on the detector plane, against the panels' boxes in the stored image. Where the modules are not placed as imaged (every CrystFEL-geometry input: stacked JUNGFRAU and AGIPD panels) nearly every observation came out as -2 (off every panel). It is now the panel whose placed footprint holds the point. Only the diagnostic export was affected; spot finding, integration and module placement already used the right frame. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/WriteReflections.cpp | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/image_analysis/WriteReflections.cpp b/image_analysis/WriteReflections.cpp index 92055a8dd..8bb1b8789 100644 --- a/image_analysis/WriteReflections.cpp +++ b/image_analysis/WriteReflections.cpp @@ -8,7 +8,6 @@ #include "scale_merge/TwinningAnalysis.h" #include "bragg_integration/SystematicAbsence.h" #include "../common/ParallelFor.h" -#include "../common/PanelBoxes.h" #include "../writer/WriteRugnuxFiles.h" #include @@ -1400,12 +1399,11 @@ MtzTable ObservationMtzTable(const std::vector &outcomes, for (const auto &l : labels) mtz.columns.push_back({l, (l == "H" || l == "K" || l == "L") ? 'H' : (l == "FLAGS" || l == "USED" ? 'I' : 'R'), 0, {}}); - const auto panels = PanelBoxes(experiment.GetModules(), experiment.GetXPixelsNum(), experiment.GetYPixelsNum()); + // Predictions are on the detector plane, so the panel is the one whose placed footprint holds the + // point (-1 for a detector without panels, -2 in a gap or off every panel). const auto panel_of = [&](float x, float y) { - for (const auto &b : panels) - if (x >= b.x0 && x < b.x1 && y >= b.y0 && y < b.y1) - return static_cast(b.panel); - return -2.0f; + const auto p = experiment.GetModules().PlaneToPanel(x, y); + return p ? static_cast(p->panel) : -2.0f; }; // The run's rocking width at d*, as StillsPartialityRefine evaluates it: its table, linear in d* and held // beyond the end shells, else the three-term formula. From 583ad24a23798cc8963aa25b0c18607361e1520f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 20:11:34 +0100 Subject: [PATCH 13/34] Stills prediction: a reflection off the Ewald sphere is placed where its radially smeared node diffracts On SASE stills the rocking width is partly radial (the shot's photon-energy spread, or a cell spread): a node outside the sphere is recorded by the higher energies, at a lower Bragg angle, and one inside at a higher angle. The prediction put every reflection at the direction of q + S0, so the spots of a 4.6 keV serial run sat up to +-3 px off their predictions at 45 deg 2theta, growing with the excitation error and with resolution (the same for every module and every azimuth). The run's strong spots now measure the radial share of the rocking width - one relative spread s, fitted to observed minus predicted centroids on the window-sample frames - and each prediction is placed as by the wavenumber change dk/k = s^2 |q_z| eps / sigma^2 (the Gaussian expectation, never past the change that puts the node on the sphere). Measured s: 2.1e-3 on the 4.6 keV JUNGFRAU serial set (radial centroid scatter 2.4 -> 1.0 px at high resolution), ~1e-3 on the 12 keV one, 8e-4 on an AGIPD set, 0 on a 1 A JUNGFRAU 4M set. 4.6 keV set (mid subset): CCref last shell 0.673 -> 0.748, S-site peaks 7.79/1.32 -> 7.96/1.89. 12 keV set and AGIPD set unchanged within noise. Rotation data are not touched (the stills window is never set with a goniometer). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/IndexAndRefine.cpp | 1 + image_analysis/IndexAndRefine.h | 1 + .../bragg_prediction/BraggPrediction.cpp | 8 +++ .../bragg_prediction/BraggPrediction.h | 6 ++ .../bragg_prediction/BraggPredictionGPU.cu | 9 +++ .../bragg_prediction/BraggPredictionGPU.h | 1 + .../scale_merge/StillsPartialityRefine.cpp | 61 +++++++++++++++++++ .../scale_merge/StillsPartialityRefine.h | 10 +++ rugnux/RugnuxImagePass.cpp | 6 +- tests/CalcBraggPredictionTest.cpp | 43 +++++++++++++ 10 files changed, 145 insertions(+), 1 deletion(-) diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index eb04815f4..40d436577 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -642,6 +642,7 @@ void IndexAndRefine::QuickPredictAndIntegrate(DataMessage &msg, settings_prediction.window_ang = stills_window_->ang; settings_prediction.window_bw = stills_window_->bw; settings_prediction.window_q_scale = stills_window_->q_scale; + settings_prediction.window_radial = stills_window_->radial; } // Online is bounded by what the image-buffer slot can carry, offline by the prediction limit; both diff --git a/image_analysis/IndexAndRefine.h b/image_analysis/IndexAndRefine.h index 5eaae7f43..05c9a039a 100644 --- a/image_analysis/IndexAndRefine.h +++ b/image_analysis/IndexAndRefine.h @@ -97,6 +97,7 @@ struct StillsWindow { float ang = 0.0f; float bw = 0.0f; float q_scale = 1.0f; // centred on the excitation error of the reciprocal lattice scaled by this + float radial = 0.0f; // where an off-sphere reflection is placed (BraggPredictionSettings::window_radial) }; class IndexAndRefine { diff --git a/image_analysis/bragg_prediction/BraggPrediction.cpp b/image_analysis/bragg_prediction/BraggPrediction.cpp index 20dd30fb0..14b1cd5be 100644 --- a/image_analysis/bragg_prediction/BraggPrediction.cpp +++ b/image_analysis/bragg_prediction/BraggPrediction.cpp @@ -167,6 +167,14 @@ int BraggPrediction::Calc(const DiffractionExperiment &experiment, const Crystal } if (window_dist <= radial_cutoff) { + if (settings.window_radial > 0.0f) { + const float az = std::fabs(recip_z); + const float f = dist_ewald_sphere * std::fmin(settings.window_radial * az / (radial_cutoff * radial_cutoff), + 1.0f / az); + S_x += f * S0.x; + S_y += f * S0.y; + S_z += f * S0.z; + } const float s0_p0 = S0.x * recip_x + S0.y * recip_y + S0.z * recip_z; const float val = s0_sq * recip_sq - s0_p0 * s0_p0; diff --git a/image_analysis/bragg_prediction/BraggPrediction.h b/image_analysis/bragg_prediction/BraggPrediction.h index f93ab8ef6..f0e82033e 100644 --- a/image_analysis/bragg_prediction/BraggPrediction.h +++ b/image_analysis/bragg_prediction/BraggPrediction.h @@ -45,6 +45,12 @@ struct BraggPredictionSettings { // The window is centred where the rocking curve is: on the excitation error of the reciprocal lattice // scaled by this (StillsPartialityRefine::MeasureRadialScale). dist_ewald stays the lattice's own. float window_q_scale = 1.0f; + // Stills, with the window: a reflection off the Ewald sphere is placed where the radial part of its + // smeared node diffracts (StillsPartialityRefine::MeasureRadialSpread) - its S moved along S0 by + // dk / k = window_radial |q_z| eps / W^2, W the window at the reflection, but never past the change that + // puts the node on the sphere (eps / |q_z|). window_radial = s^2 times the window's width in sigmas + // squared, s the run's relative radial spread. 0 leaves the positions as they were. + float window_radial = 0.0f; }; class BraggPrediction { diff --git a/image_analysis/bragg_prediction/BraggPredictionGPU.cu b/image_analysis/bragg_prediction/BraggPredictionGPU.cu index b6b1954c8..274ce58b3 100644 --- a/image_analysis/bragg_prediction/BraggPredictionGPU.cu +++ b/image_analysis/bragg_prediction/BraggPredictionGPU.cu @@ -123,6 +123,14 @@ namespace { window_dist = fabsf(sqrtf(Scx * Scx + Scy * Scy + Scz * Scz) - C.one_over_wavelength); } if (window_dist > radial_cutoff) return false; + if (C.window_radial > 0.0f) { // see the CPU path + const float az = fabsf(recip_z); + const float f = dist_ewald * fminf(C.window_radial * az / (radial_cutoff * radial_cutoff), 1.0f / az); + Sx += f * C.S0.x; + Sy += f * C.S0.y; + Sz += f * C.S0.z; + S_len = sqrtf(Sx * Sx + Sy * Sy + Sz * Sz); + } float Srx = C.rot[0] * Sx + C.rot[1] * Sy + C.rot[2] * Sz; float Sry = C.rot[3] * Sx + C.rot[4] * Sy + C.rot[5] * Sz; float Srz = C.rot[6] * Sx + C.rot[7] * Sy + C.rot[8] * Sz; @@ -253,6 +261,7 @@ int BraggPredictionGPU::Calc(const DiffractionExperiment &experiment, hK.window_ang = settings.window_ang; hK.window_bw = settings.window_bw; hK.window_q_scale = settings.window_q_scale; + hK.window_radial = settings.window_radial; cudaMemcpyAsync(dK, &hK, sizeof(KernelConsts), cudaMemcpyHostToDevice, stream); cudaMemsetAsync(d_count, 0, sizeof(int), stream); diff --git a/image_analysis/bragg_prediction/BraggPredictionGPU.h b/image_analysis/bragg_prediction/BraggPredictionGPU.h index 427b8d043..86d570d7a 100644 --- a/image_analysis/bragg_prediction/BraggPredictionGPU.h +++ b/image_analysis/bragg_prediction/BraggPredictionGPU.h @@ -18,6 +18,7 @@ struct KernelConsts { float one_over_dmax_sq; float ewald_cutoff; float window_c0, window_ang, window_bw, window_q_scale; // the stills window curve; all zero: ewald_cutoff (see BraggPredictionSettings) + float window_radial; // where an off-sphere reflection is placed (see BraggPredictionSettings) float bandwidth_sigma; // relative Δλ/λ (sigma); 0 = monochromatic // Angle-dependent sensor efficiency, reduced to the two per-dataset numbers the kernel needs: // the optical thickness at normal incidence and the absorbed fraction there. qe_a0 = 0 leaves diff --git a/image_analysis/scale_merge/StillsPartialityRefine.cpp b/image_analysis/scale_merge/StillsPartialityRefine.cpp index cb69d7497..c15a7fdf5 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.cpp +++ b/image_analysis/scale_merge/StillsPartialityRefine.cpp @@ -519,6 +519,67 @@ double StillsPartialityRefine::MeasureRadialScale(const std::vector &outcomes, + const RunWidth &width) const { + // Strong spots only: their centroids are good to a fraction of a pixel. + constexpr double MIN_I_OVER_SIGMA = 10.0; + constexpr size_t MIN_SPOTS = 200; + constexpr float STEP = 1e-3f; // of the wavenumber, for the direction a spot moves in + // Per spot: m = how far (px, on the plane) the model moves it per unit s^2, and d = observed - predicted. + struct Spot { double mx, my, dx, dy; }; + std::vector spots; + for (const auto &o: outcomes) { + if (!o.image_scale_g || !(*o.image_scale_g > 0.0)) + continue; + const Coord S0 = o.geom.GetScatteringVector(); + for (const auto &r: o.reflections) { + if (r.on_ice_ring || !AcceptReflection(r, d_min_limit_, d_max_limit_) || !(r.d > 0.0f) + || !(r.I > MIN_I_OVER_SIGMA * r.sigma) || !std::isfinite(r.observed_x) || !std::isfinite(r.observed_y) + || !std::isfinite(r.dist_ewald)) + continue; + const Coord q = o.latt.Astar() * static_cast(r.h) + o.latt.Bstar() * static_cast(r.k) + + o.latt.Cstar() * static_cast(r.l); + const double dstar = 1.0 / r.d, qz = std::fabs(q.z); + const double a = width.gamma_e * dstar, b = width.bandwidth_sigma * qz; + const double sigma2 = width.gamma0 * width.gamma0 + a * a + b * b; + if (!(sigma2 > 0.0)) + continue; + const auto p0 = o.geom.RecipToDetector(q); + const auto p1 = o.geom.RecipToDetector(q + S0 * STEP); + const double f = qz * r.dist_ewald / sigma2 / STEP; + Spot s{f * (p1.first - p0.first), f * (p1.second - p0.second), + r.observed_x - r.predicted_x, r.observed_y - r.predicted_y}; + if (std::isfinite(s.mx) && std::isfinite(s.my)) + spots.push_back(s); + } + } + if (spots.size() < MIN_SPOTS) + return 0.0; + // Least squares, reweighted against the spots the model does not describe (overlaps, split spots): + // Cauchy weights at three times the typical residual. + double s2 = 0.0; + std::vector r2(spots.size()); + for (int iter = 0; iter < 5; ++iter) { + for (size_t i = 0; i < spots.size(); ++i) { + const double ex = spots[i].dx - s2 * spots[i].mx, ey = spots[i].dy - s2 * spots[i].my; + r2[i] = ex * ex + ey * ey; + } + std::vector sorted = r2; + std::nth_element(sorted.begin(), sorted.begin() + sorted.size() / 2, sorted.end()); + const double scale2 = 9.0 * sorted[sorted.size() / 2] / 1.386; // median r^2 = 1.386 sigma^2 in 2D + double num = 0.0, den = 0.0; + for (size_t i = 0; i < spots.size(); ++i) { + const double w = 1.0 / (1.0 + r2[i] / scale2); + num += w * (spots[i].mx * spots[i].dx + spots[i].my * spots[i].dy); + den += w * (spots[i].mx * spots[i].mx + spots[i].my * spots[i].my); + } + if (!(den > 0.0)) + return 0.0; + s2 = num / den; + } + return std::max(0.0, s2); +} + std::optional StillsPartialityRefine::MeasureRunWidth( const std::vector &outcomes, const std::optional &reference_cell, double radial_scale, const RunWidth *window_width) const { diff --git a/image_analysis/scale_merge/StillsPartialityRefine.h b/image_analysis/scale_merge/StillsPartialityRefine.h index 48c59c76a..4bc6cb85f 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.h +++ b/image_analysis/scale_merge/StillsPartialityRefine.h @@ -85,6 +85,16 @@ public: double MeasureRadialScale(const std::vector &outcomes, const std::optional &reference_cell, double radial_scale) const; + // How far the run's spots move radially with their excitation error, as the relative spread s of the + // radial scale of the reciprocal lattice (a photon-energy spread, or a cell spread) that would move them + // so. A node off the Ewald sphere is recorded where the part of its smeared node that IS on the sphere + // diffracts: along the radial share of its rocking width, that is where the photon energy that puts the + // node on the sphere sends it. With the rocking width sigma (width's three-term formula) made of a radial + // part s |q_z| and the rest, the expected move is that of a change of the beam's wavenumber by + // dk / k = s^2 |q_z| eps / sigma^2 - fitted by least squares to where the strong spots were observed + // against where they were predicted. Returns s^2; 0 when the run has too few strong spots. + double MeasureRadialSpread(const std::vector &outcomes, const RunWidth &width) const; + private: const DiffractionExperiment experiment_; const Settings settings_{}; diff --git a/rugnux/RugnuxImagePass.cpp b/rugnux/RugnuxImagePass.cpp index f9509795e..7e1886f85 100644 --- a/rugnux/RugnuxImagePass.cpp +++ b/rugnux/RugnuxImagePass.cpp @@ -449,6 +449,7 @@ void Rugnux::ImagePass(PipelineLocals &p) { for (int i = 0; i < 3; ++i) radial_scale = refine.MeasureRadialScale(outcomes, stills_user_cell_, radial_scale); stills_run_width_ = refine.MeasureRunWidth(outcomes, stills_user_cell_, radial_scale); + const double radial_spread2 = stills_run_width_ ? refine.MeasureRadialSpread(outcomes, *stills_run_width_) : 0.0; for (const int o: window_frames) outcomes[o] = IntegrationOutcome{}; if (stills_run_width_) { @@ -456,9 +457,12 @@ void Rugnux::ImagePass(PipelineLocals &p) { indexer->SetStillsWindow(StillsWindow{.c0 = k * static_cast(stills_run_width_->gamma0), .ang = k * static_cast(stills_run_width_->gamma_e), .bw = k * static_cast(stills_run_width_->bandwidth_sigma), - .q_scale = static_cast(stills_run_width_->radial_scale)}); + .q_scale = static_cast(stills_run_width_->radial_scale), + .radial = k * k * static_cast(radial_spread2)}); Note(fmt::format("prediction window {:.1f} x the rocking width measured on {} frames, centred at " "radial scale {:+.2e}", config_.stills_window_sigmas, window_frames.size(), radial_scale - 1.0)); + Note(fmt::format("spots placed with their excitation error as by a radial spread of {:.1e}", + std::sqrt(radial_spread2))); } } for (int i = 0; i < image_workers; ++i) diff --git a/tests/CalcBraggPredictionTest.cpp b/tests/CalcBraggPredictionTest.cpp index 9c071c358..831a7cb9c 100644 --- a/tests/CalcBraggPredictionTest.cpp +++ b/tests/CalcBraggPredictionTest.cpp @@ -451,6 +451,49 @@ TEST_CASE("BraggPrediction_WindowFollowsResolution", "[portable]") { CHECK(got_scaled != got); } +TEST_CASE("BraggPrediction_WindowRadialPlacement", "[portable]") { + // With window_radial a reflection off the Ewald sphere is placed as by the wavenumber change + // dk / k = window_radial |q_z| eps / W^2, capped where it puts the node on the sphere: an outside node + // (eps > 0) moves toward the beam, an inside one away, by the amount the formula says. + DiffractionExperiment experiment(DetJF4M()); + experiment.DetectorDistance_mm(100.0).BeamX_pxl(1500.0).BeamY_pxl(1000.0).IncidentEnergy_keV(12.0); + const CrystalLattice lattice(Coord{30, 10, 0}, Coord{-15, 45, 0}, Coord{0, 0, 150}); + BraggPredictionSettings plain{.high_res_A = 1.8, .max_h = 40, .max_k = 40, .max_l = 90}; + plain.window_c0 = 1e-4f; + plain.window_ang = 0.06f * static_cast(PI) / 180.0f; + BraggPredictionSettings moved = plain; + moved.window_radial = 4.0f * 2e-3f * 2e-3f; + + BraggPrediction pred_plain, pred_moved; + const int n = pred_plain.Calc(experiment, lattice, plain); + REQUIRE(n > 100); + REQUIRE(pred_moved.Calc(experiment, lattice, moved) == n); + + const auto geom = experiment.GetDiffractionGeometry(); + const Coord S0 = geom.GetScatteringVector(); + int inward = 0, outward = 0; + for (int i = 0; i < n; ++i) { + const auto &a = pred_plain.GetReflections()[i], &b = pred_moved.GetReflections()[i]; + REQUIRE(a.h == b.h); + REQUIRE(a.k == b.k); + REQUIRE(a.l == b.l); + CHECK(b.dist_ewald == a.dist_ewald); + const auto q = a.h * lattice.Astar() + a.k * lattice.Bstar() + a.l * lattice.Cstar(); + const float W2 = plain.window_c0 * plain.window_c0 + plain.window_ang * plain.window_ang * (q * q); + const float az = std::fabs(q.z); + const float f = a.dist_ewald * std::fmin(moved.window_radial * az / W2, 1.0f / az); + const auto [ex, ey] = geom.RecipToDetector(q + S0 * f); + CHECK(b.predicted_x == Catch::Approx(ex).margin(1e-3)); + CHECK(b.predicted_y == Catch::Approx(ey).margin(1e-3)); + const float ra = std::hypot(a.predicted_x - 1500.0f, a.predicted_y - 1000.0f); + const float rb = std::hypot(b.predicted_x - 1500.0f, b.predicted_y - 1000.0f); + if (a.dist_ewald > 0 && rb < ra) ++inward; + if (a.dist_ewald < 0 && rb > ra) ++outward; + } + CHECK(inward > 10); + CHECK(outward > 10); +} + TEST_CASE("BraggPrediction_GrowsPastStartingCapacity", "[portable]") { // A frame predicting more than the buffer starts with must come back whole - the same reflections a // buffer large enough from the outset returns - and not as whichever the h/k/l walk reached first. From e35e8467aebcbf2c75d4fe27c44b26575d6d531e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 18:57:58 +0100 Subject: [PATCH 14/34] Stills merge: the error-model fit and its rounds with the means run in a fraction of the time The stills error model (fit + means, iterated) had made the "error model and merge" phase 8-18x slower than before it. Now: - the fit puts its bin boundaries in place with nth_element on compact (key, position) pairs, the two halves of each split side by side, instead of sorting the whole pool by a five-field tuple; the per-group passes of its iteration run on the workers; - each round's fit starts from the previous round's model, so it settles in a few iterations; - the rounds stop once a round leaves the model where the previous one put it (to 1e-3); - those rounds only carry the means to where the model settles, so on a large run they fit every stride-th observation (about four million); one last round fits them all and makes the final means; - the observations are grouped by one parallel sort on (reflection, image order) into one flat array instead of a hash map of vectors, and the per-reflection passes (means, samples) run on the workers. Every sum runs in a fixed order over fixed pieces, so the result is the same on any number of threads (the fit is checked at 1 and 16 threads; the merge test checks 1 against 8). On the sets tried the merged data are unchanged to within the last round's convergence. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/scale_merge/ErrorModel.cpp | 86 +++++++---- image_analysis/scale_merge/ErrorModel.h | 5 +- image_analysis/scale_merge/Merge.cpp | 176 +++++++++++++++------- tests/MergeScaleTest.cpp | 4 + 4 files changed, 185 insertions(+), 86 deletions(-) diff --git a/image_analysis/scale_merge/ErrorModel.cpp b/image_analysis/scale_merge/ErrorModel.cpp index dc2868ad9..d6b8ed5e1 100644 --- a/image_analysis/scale_merge/ErrorModel.cpp +++ b/image_analysis/scale_merge/ErrorModel.cpp @@ -7,7 +7,6 @@ #include #include #include -#include #include @@ -156,60 +155,93 @@ bool ErrorModelUpdate(const std::vector &bs2, const std::vector return !settled; } -StillsErrorModelFit FitStillsErrorModel(std::vector &pool) { +StillsErrorModelFit FitStillsErrorModel(const std::vector &pool, size_t nthreads, + const StillsErrorModelFit &start) { constexpr int n_bins = ERROR_MODEL_BINS; constexpr int n_sub = 4; StillsErrorModelFit fit; if (pool.size() < static_cast(8 * n_bins * n_sub)) return fit; - // Bins of counting (I/sigma)^2, each split by the partiality term's share of the intensity term. Total - // orders, so which sample lands in which bin is a property of the samples alone. - auto snr = [](const StillsErrorModelSample &x) { return x.I2 / x.s2; }; - auto share = [](const StillsErrorModelSample &x) { return x.I2 > 0.0 ? x.P2 / x.I2 : 0.0; }; - std::sort(pool.begin(), pool.end(), [&](const auto &x, const auto &y) { - return std::make_tuple(snr(x), x.s2, x.I2, x.P2, x.dev2) < std::make_tuple(snr(y), y.s2, y.I2, y.P2, y.dev2); + // Bins of counting (I/sigma)^2, each split by the partiality term's share of the intensity term. The + // boundaries are put in place with nth_element on (key, position in the pool) - a total order, so which + // sample lands in which bin depends on the samples and the order the caller gave them in alone. The keys + // are computed once and carried in the comparison, never recomputed inside it. + struct Key { double v; size_t i; }; + const auto by_key = [](const Key &p, const Key &q) { return p.v < q.v || (p.v == q.v && p.i < q.i); }; + std::vector key(pool.size()); + ParallelChunks(static_cast(pool.size()), ThreadsForWork(pool.size(), nthreads), [&](int lo, int hi) { + for (int i = lo; i < hi; ++i) + key[i] = {pool[i].I2 / pool[i].s2, static_cast(i)}; }); const size_t per = pool.size() / n_bins; - std::vector> groups; - for (int bin = 0; bin < n_bins; ++bin) { - const size_t lo = bin * per, hi = bin == n_bins - 1 ? pool.size() : (bin + 1) * per; - std::sort(pool.begin() + lo, pool.begin() + hi, [&](const auto &x, const auto &y) { - return std::make_tuple(share(x), x.s2, x.I2, x.dev2) < std::make_tuple(share(y), y.s2, y.I2, y.dev2); - }); - const size_t sub = (hi - lo) / n_sub; - for (int q = 0; q < n_sub; ++q) - groups.emplace_back(lo + q * sub, q == n_sub - 1 ? hi : lo + (q + 1) * sub); - } + // The split runs down the middle, and the two halves of a split are disjoint ranges, so they run side + // by side while the range is still worth a thread (as in FitErrorModel). + constexpr size_t SPLIT_MIN_PARALLEL = 1u << 17; + const std::function split = [&](size_t lo, size_t hi, int b0, int b1, int depth) { + if (b0 >= b1) return; + const int mid = (b0 + b1) / 2; + const size_t k = static_cast(mid) * per; + std::nth_element(key.begin() + lo, key.begin() + k, key.begin() + hi, by_key); + if (depth > 0 && hi - lo > SPLIT_MIN_PARALLEL) { + auto left = std::async(std::launch::async, [&, lo, k, b0, mid, depth] { + RestoreThreadAffinity(); + split(lo, k, b0, mid, depth - 1); + }); + split(k, hi, mid + 1, b1, depth - 1); + left.get(); + } else { + split(lo, k, b0, mid, 0); + split(k, hi, mid + 1, b1, 0); + } + }; + split(0, key.size(), 1, n_bins, nthreads > 1 ? 3 : 0); // b only where the strongest bin reaches past the counting term, as in FitErrorModel. std::vector top; - for (size_t i = (n_bins - 1) * per; i < pool.size(); ++i) - top.push_back(snr(pool[i])); + for (size_t j = (n_bins - 1) * per; j < key.size(); ++j) + top.push_back(key[j].v); std::nth_element(top.begin(), top.begin() + top.size() / 2, top.end()); const bool b_measured = top[top.size() / 2] > ERROR_MODEL_B_LEVER_MIN; + // Each bin in four by the share, and the samples copied out in that order. + std::vector> groups(n_bins * n_sub); + std::vector x(pool.size()); + ParallelFor(n_bins, nthreads, [&](int bin) { + const size_t lo = bin * per, hi = bin == n_bins - 1 ? key.size() : (bin + 1) * per; + for (size_t j = lo; j < hi; ++j) { + const auto &o = pool[key[j].i]; + key[j].v = o.I2 > 0.0 ? o.P2 / o.I2 : 0.0; + } + const size_t sub = (hi - lo) / n_sub; + for (int q = 1; q < n_sub; ++q) + std::nth_element(key.begin() + lo + (q - 1) * sub, key.begin() + lo + q * sub, key.begin() + hi, by_key); + for (int q = 0; q < n_sub; ++q) + groups[bin * n_sub + q] = {lo + q * sub, q == n_sub - 1 ? hi : lo + (q + 1) * sub}; + for (size_t j = lo; j < hi; ++j) + x[j] = pool[key[j].i]; + }); + key = {}; const int ng = static_cast(groups.size()); std::vector z2(pool.size()); Eigen::MatrixXd X(ng, 3); Eigen::VectorXd y(ng); - double a = 1.0, b2 = 0.0, c2 = 0.0; + double a = start.a, b2 = start.b2, c2 = start.c2; for (int iter = 0; iter < 30; ++iter) { // Per group: the three terms divided by the current variance, and the median normalised deviation // expressed as the sum it stands for. - for (int g = 0; g < ng; ++g) { + ParallelFor(ng, ThreadsForWork(pool.size(), nthreads), [&](int g) { const auto [lo, hi] = groups[g]; double s = 0.0, I = 0.0, P = 0.0; for (size_t i = lo; i < hi; ++i) { - const auto &x = pool[i]; - const double v = a * x.s2 + b2 * x.I2 + c2 * x.P2; - s += x.s2 / v; I += x.I2 / v; P += x.P2 / v; - z2[i] = x.dev2 / v; + const double v = a * x[i].s2 + b2 * x[i].I2 + c2 * x[i].P2; + s += x[i].s2 / v; I += x[i].I2 / v; P += x[i].P2 / v; + z2[i] = x[i].dev2 / v; } const auto mid = z2.begin() + lo + (hi - lo) / 2; std::nth_element(z2.begin() + lo, mid, z2.begin() + hi); X.row(g) << s, I, P; y(g) = *mid / ERROR_MODEL_CHI2_1_MEDIAN * static_cast(hi - lo); - } + }); // Every group weighs the same in relative terms; a negative term is dropped and the rest refitted. const Eigen::VectorXd w = y.array().square().inverse(); std::vector terms = b_measured ? std::vector{0, 1, 2} : std::vector{0, 2}; diff --git a/image_analysis/scale_merge/ErrorModel.h b/image_analysis/scale_merge/ErrorModel.h index 11898071e..d76981479 100644 --- a/image_analysis/scale_merge/ErrorModel.h +++ b/image_analysis/scale_merge/ErrorModel.h @@ -82,4 +82,7 @@ struct StillsErrorModelFit { bool b_resolved = false; // as ErrorModelFit::b_resolved }; -StillsErrorModelFit FitStillsErrorModel(std::vector &pool); +// start: where the iteration begins - identity by default, or the fit of a previous round on a pool that +// has moved little since. +StillsErrorModelFit FitStillsErrorModel(const std::vector &pool, size_t nthreads = 1, + const StillsErrorModelFit &start = {}); diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index b00142b75..d6924587a 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -298,9 +299,7 @@ void MergeOnTheFly::RefineErrorModel(const std::vector &outc // AddImage uses, down to the floor of the merge with an error model. struct Obs { float I, sigma, corr, var_bkg, p; }; struct KeyedObs { uint64_t key; Obs obs; }; - std::unordered_map> groups; - std::vector keys; // in order of first appearance, so every sum below is in a fixed order - + std::vector all; // in image order ReduceImagesInOrder( outcomes.size(), nthreads, [&](size_t i, std::vector &out) { @@ -325,86 +324,147 @@ void MergeOnTheFly::RefineErrorModel(const std::vector &outc } }, [&](size_t, const std::vector &out) { - for (const auto &o: out) { - auto &g = groups[o.key]; - if (g.empty()) - keys.push_back(o.key); - g.push_back(o.obs); - } + all.insert(all.end(), out.begin(), out.end()); }); + // The observations of each reflection side by side, in key order and within a reflection in image + // order - a sort on (key, place), a total order, so the same on any number of workers. Every sum + // below runs over them in that fixed order. + struct KeyPlace { uint64_t key; size_t place; }; + std::vector order(all.size()); + const size_t obs_threads = ThreadsForWork(all.size(), nthreads); + ParallelChunks(static_cast(all.size()), obs_threads, [&](int lo, int hi) { + for (int i = lo; i < hi; ++i) + order[i] = {all[i].key, static_cast(i)}; + }); + ParallelSort(order.begin(), order.end(), nthreads, [](const KeyPlace &x, const KeyPlace &y) { + return x.key < y.key || (x.key == y.key && x.place < y.place); + }); + std::vector flat(all.size()); + ParallelChunks(static_cast(all.size()), obs_threads, [&](int lo, int hi) { + for (int i = lo; i < hi; ++i) + flat[i] = all[order[i].place].obs; + }); + all = {}; + std::vector keys; + std::vector start; // reflection k is flat[start[k] .. start[k + 1]) + for (size_t i = 0; i < order.size(); ++i) + if (i == 0 || order[i].key != order[i - 1].key) { + keys.push_back(order[i].key); + start.push_back(i); + } + start.push_back(order.size()); + order = {}; + const auto group = [&](int k) { return std::span(flat.data() + start[k], start[k + 1] - start[k]); }; + const int n_keys = static_cast(keys.size()); + const size_t key_threads = ThreadsForWork(keys.size(), nthreads, 1024); + // --- 2. The starting mean of each reflection (counting-sigma weights) and, where there are three // observations for it to stand on, the median - the robust centre of the outlier test. - std::unordered_map mean; - for (const auto key: keys) { - const auto &obs = groups[key]; - double sum_w = 0.0, sum_wI = 0.0; - for (const auto &o: obs) { - const double w = 1.0 / (static_cast(o.sigma) * o.sigma); - sum_w += w; - sum_wI += w * o.I; + std::vector mean(n_keys); + std::vector median(n_keys, NAN); + ParallelChunks(n_keys, key_threads, [&](int lo, int hi) { + for (int k = lo; k < hi; ++k) { + const auto obs = group(k); + double sum_w = 0.0, sum_wI = 0.0; + for (const auto &o: obs) { + const double w = 1.0 / (static_cast(o.sigma) * o.sigma); + sum_w += w; + sum_wI += w * o.I; + } + mean[k] = sum_wI / sum_w; + if (reject_outliers && obs.size() >= 3) { + std::vector iv; + iv.reserve(obs.size()); + for (const auto &o: obs) + iv.push_back(o.I); + std::nth_element(iv.begin(), iv.begin() + iv.size() / 2, iv.end()); + median[k] = iv[iv.size() / 2]; + } } - mean[key] = sum_wI / sum_w; - if (reject_outliers && obs.size() >= 3) { - std::vector iv; - iv.reserve(obs.size()); - for (const auto &o: obs) - iv.push_back(o.I); - std::nth_element(iv.begin(), iv.begin() + iv.size() / 2, iv.end()); - reject_median_I[key] = iv[iv.size() / 2]; - } - } + }); + for (int k = 0; k < n_keys; ++k) + if (!std::isnan(median[k])) + reject_median_I[keys[k]] = median[k]; // --- 3. Fit the model (ErrorModel.h, FitStillsErrorModel) on the scatter about the reflection means, // the counting variance rebuilt at the mean, and refit with the means the fitted model weights: // the means and the model each depend on the other, and a single pass leaves the weights of // the strong reflections at the counting-sigma means the fit started from. Measured on serial // and rotation-as-serial sets, a few rounds raise the anomalous signal of the merge; they have - // settled after four. + // settled after four, most often already after two or three: a round that leaves the model where + // the previous one put it (to 1e-3) would only reproduce the same means, and ends the iteration. + // Those rounds only carry the means to where the model settles, so they fit every stride-th + // observation (some four million of them); one last round fits them all and makes the final means. + // Each observation's sample goes to its own slot, so the pool is in key order whatever the number + // of workers; the slots left empty (single observations, the leverage cut) are dropped after. constexpr int ROUNDS = 4; + const size_t stride = std::max(1, flat.size() >> 22); + bool last_round = false; std::vector samples; StillsErrorModelFit fit; double a = 1.0, b2 = 0.0, c2 = 0.0; auto variance = [&](const Obs &o, double m) { return a * CountingVariance(o.sigma, o.corr, o.var_bkg, o.p, m) + (b2 + c2 * PartialityTerm(o.p)) * m * m; }; - for (int round = 0; round < ROUNDS; ++round) { - samples.clear(); - for (const auto key: keys) { - const auto &obs = groups[key]; - if (obs.size() < 2) - continue; - const double m = mean[key]; - // The residual from the weighted mean has E[(I_i - )^2] = var_i (1 - h_i), h_i = w_i / sum_w its - // leverage, with the weights of the model that made the mean. - double sum_w = 0.0; - for (const auto &o: obs) - sum_w += 1.0 / variance(o, m); - for (const auto &o: obs) { - const double factor = 1.0 - 1.0 / (variance(o, m) * sum_w); - if (factor < 0.05) + for (int round = 0; ; ++round) { + const size_t step = last_round ? 1 : stride; + samples.assign((flat.size() + step - 1) / step, StillsErrorModelSample{NAN, NAN, NAN, NAN}); + ParallelChunks(n_keys, key_threads, [&](int lo, int hi) { + std::vector var; + for (int k = lo; k < hi; ++k) { + const auto obs = group(k); + if (obs.size() < 2) continue; - const double resid = static_cast(o.I) - m; - samples.push_back({CountingVariance(o.sigma, o.corr, o.var_bkg, o.p, m), m * m, - PartialityTerm(o.p) * m * m, resid * resid / factor}); + const double m = mean[k]; + // The residual from the weighted mean has E[(I_i - )^2] = var_i (1 - h_i), h_i = w_i / sum_w + // its leverage, with the weights of the model that made the mean. + var.resize(obs.size()); + double sum_w = 0.0; + for (size_t j = 0; j < obs.size(); ++j) { + var[j] = variance(obs[j], m); + sum_w += 1.0 / var[j]; + } + for (size_t j = 0; j < obs.size(); ++j) { + if ((start[k] + j) % step != 0) + continue; + const auto &o = obs[j]; + const double factor = 1.0 - 1.0 / (var[j] * sum_w); + if (factor < 0.05) + continue; + const double resid = static_cast(o.I) - m; + samples[(start[k] + j) / step] = {CountingVariance(o.sigma, o.corr, o.var_bkg, o.p, m), m * m, + PartialityTerm(o.p) * m * m, resid * resid / factor}; + } } - } - fit = FitStillsErrorModel(samples); + }); + samples.erase(std::remove_if(samples.begin(), samples.end(), + [](const StillsErrorModelSample &x) { return std::isnan(x.dev2); }), + samples.end()); + const StillsErrorModelFit last = fit; + fit = FitStillsErrorModel(samples, nthreads, last); // each round starts from the last one's model if (!fit.active) return; // too little multiplicity to fit -> leave identity a = fit.a; b2 = fit.b2; c2 = fit.c2; - for (const auto key: keys) { - double sum_w = 0.0, sum_wI = 0.0; - for (const auto &o: groups[key]) { - const double w = 1.0 / variance(o, mean[key]); - sum_w += w; - sum_wI += w * o.I; + const auto close = [](double x, double y) { return std::fabs(x - y) <= 1e-3 * std::fabs(y); }; + const bool settled = round > 0 && close(a, last.a) && close(b2, last.b2) && close(c2, last.c2); + ParallelChunks(n_keys, key_threads, [&](int lo, int hi) { + for (int k = lo; k < hi; ++k) { + double sum_w = 0.0, sum_wI = 0.0; + for (const auto &o: group(k)) { + const double w = 1.0 / variance(o, mean[k]); + sum_w += w; + sum_wI += w * o.I; + } + mean[k] = sum_wI / sum_w; } - mean[key] = sum_wI / sum_w; - } + }); + if (last_round || ((settled || round + 1 == ROUNDS) && step == 1)) + break; + last_round = settled || round + 1 == ROUNDS; } - for (const auto key: keys) - error_model_mean_I[key] = static_cast(mean[key]); + for (int k = 0; k < n_keys; ++k) + error_model_mean_I[keys[k]] = static_cast(mean[k]); error_model_a = a; error_model_b = std::sqrt(b2); diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index 74b9e4825..e9ae92b10 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -546,6 +546,10 @@ TEST_CASE("StillsErrorModel_RecoversThePartialityTerm") { } const auto fit = FitStillsErrorModel(pool); REQUIRE(fit.active); + const auto many = FitStillsErrorModel(pool, 16); // the same numbers on any number of threads + CHECK(many.a == fit.a); + CHECK(many.b2 == fit.b2); + CHECK(many.c2 == fit.c2); CHECK(fit.a == Catch::Approx(a).epsilon(0.05)); CHECK(std::sqrt(fit.b2) == Catch::Approx(b).epsilon(0.05)); CHECK(std::sqrt(fit.c2) == Catch::Approx(c).epsilon(0.05)); From e11211b43556b7b85e4be1cfc47d8713099a04ae Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 20:26:35 +0100 Subject: [PATCH 15/34] Stills merge: CC1/2 and R_split weigh a reflection by how far its observations were scaled up; the error-model means leave out the outliers On the CCD sets the overall CC1/2 of the new stills error model fell (0.99 -> 0.66 on one) while every shell but one improved and R-free held. Two mechanisms, both in a few reflections: - A few in a hundred reflections are measured only a handful of times, at the beam stop's edge and in the detector's corners (where the written range now reaches), and only on weak frames or at low partiality, so every observation is scaled up 100-500-fold. Their half-set means are single observations whose noise, in absolute terms, exceeds the whole signal variance of the strongest shell: reflections with five or fewer observations carried 97-98% of the half-set disagreement. CC1/2 and R_split (and CC1/2 of the frame-whole halves) now weigh each reflection as the rotation merge does (cc_weight): the counting precision its halves would have had with every observation at one scale, over the precision they have - 1/n against sum(1/corr^2). Nothing intensity-dependent. The automatic resolution cutoff keeps the plain CC1/2: weighted, it moved a photon-counting set's cut outwards and its R-free up. - The innermost reflections next to the beam stop carry, on weak frames, a flare the frame's scale does not: raw counts that stay put while G falls, read at 10-30x the median. The error-model means took them in; every systematic variance term scales with the mean, so the outlier band about the median widened by as much as the mean was off and kept them, at the weight of the rest. The means of the error-model rounds now leave out what the merge's outlier test will leave out. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- image_analysis/scale_merge/Merge.cpp | 49 ++++++++++++++++++++++++---- image_analysis/scale_merge/Merge.h | 4 +++ tests/MergeScaleTest.cpp | 43 ++++++++++++++++++++++++ 3 files changed, 90 insertions(+), 6 deletions(-) diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index d6924587a..97c2232f2 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -39,6 +39,21 @@ namespace { return static_cast(z & 1ULL); } + // The weight of a reflection in CC1/2 and R_split, as the rotation merge gives it (MergedReflection:: + // cc_weight): the counting precision its two half-sets would have had with every observation at one + // common scale, over the precision they have. An observation scaled up by image_scale_corr = 1/(G p) + // carries that factor squared in counting variance, so a half of n observations has a variance of 1/n + // at a common scale and 1/q, q = sum(1/corr^2), as measured. Nothing intensity-dependent, and no credit + // for multiplicity: only how far the observations were scaled up. Without it the reflections measured a + // handful of times on weak frames or at low partiality - at the beam stop's edge and in the detector's + // corners, a few in a hundred - hold half-set means whose scaled-up noise exceeds the whole signal + // variance of the strongest shell, and the plain Pearson CC1/2 and R_split are theirs. + double HalfSetWeight(const size_t n[2], const double q[2]) { + if (n[0] == 0 || n[1] == 0) + return 1.0; + return (1.0 / n[0] + 1.0 / n[1]) / (1.0 / q[0] + 1.0 / q[1]); + } + // Calls reduce(i, out) for the images in parallel, a block of them at a time, and then consume(i, out) // for each image of the block in image order - the order that every floating-point sum, every // half-set choice and every hash-map insertion of the merge depends on. The result is therefore the @@ -134,6 +149,7 @@ size_t MergeOnTheFly::Contributions(const IntegrationOutcome &outcome, std::vect .w = w, .wI = w * I_corr, .d = r.d, + .inv_corr2 = 1.0f / (r.image_scale_corr * r.image_scale_corr), .on_ice_ring = r.on_ice_ring }); } @@ -202,8 +218,11 @@ void MergeOnTheFly::Accumulate(const std::vector &contributions, i it->second.sum_wI_half[half] += c.wI; it->second.sum_w_half[half] += c.w; it->second.n_half[half]++; + it->second.q_half[half] += c.inv_corr2; it->second.sum_wI_split[prefer] += c.wI; it->second.sum_w_split[prefer] += c.w; + it->second.n_split[prefer]++; + it->second.q_split[prefer] += c.inv_corr2; if (c.on_ice_ring) it->second.on_ice_ring = true; @@ -448,11 +467,22 @@ void MergeOnTheFly::RefineErrorModel(const std::vector &outc a = fit.a; b2 = fit.b2; c2 = fit.c2; const auto close = [](double x, double y) { return std::fabs(x - y) <= 1e-3 * std::fabs(y); }; const bool settled = round > 0 && close(a, last.a) && close(b2, last.b2) && close(c2, last.c2); + // The new means leave out what the merge's outlier test will leave out (the band of OutlierBand.h + // about the median, as Contributions draws it): the systematic terms of every variance scale with the + // mean, so a mean that took in the outliers - observations of the innermost reflections that carry a + // beam-stop flare the frame's scale does not, read at 10-30x the median on weak frames - widens the + // band by as much as it is off, keeps the outliers it should drop, and weights them like the rest. + const double band = reject_nsigma * std::sqrt(b2); ParallelChunks(n_keys, key_threads, [&](int lo, int hi) { for (int k = lo; k < hi; ++k) { double sum_w = 0.0, sum_wI = 0.0; for (const auto &o: group(k)) { - const double w = 1.0 / variance(o, mean[k]); + const double v = variance(o, mean[k]); + if (reject_outliers && !std::isnan(median[k]) + && OutsideOutlierBand(o.I, median[k], reject_nsigma, v, mean[k], std::sqrt(b2) * mean[k], + band, band)) + continue; + const double w = 1.0 / v; sum_w += w; sum_wI += w * o.I; } @@ -792,16 +822,23 @@ MergeStatistics MergeOnTheFly::MergeStats(const std::vector &m } } + // The statistics weigh each reflection by HalfSetWeight; the resolution cutoff, which reads + // MergedReflection::cc_weight, does not - on a photon-counting set the weighted fall-off + // moves the cut outwards, past where the plain one finds the data useful to refinement. + const auto it = accumulator.find(generator(m).pack()); + const bool counted = it != accumulator.end(); + const double w_half = counted ? HalfSetWeight(it->second.n_half, it->second.q_half) : 1.0; + const double w_split = counted ? HalfSetWeight(it->second.n_split, it->second.q_split) : 1.0; if (std::isfinite(m.I_half[0]) && std::isfinite(m.I_half[1])) { - acc[s].cc_half.Add(m.I_half[0], m.I_half[1]); - cc_half_overall.Add(m.I_half[0], m.I_half[1]); + acc[s].cc_half.Add(m.I_half[0], m.I_half[1], w_half); + cc_half_overall.Add(m.I_half[0], m.I_half[1], w_half); } if (std::isfinite(m.I_split[0]) && std::isfinite(m.I_split[1])) { for (auto *a: {&acc[s], &split_overall}) { - a->cc_split.Add(m.I_split[0], m.I_split[1]); - a->split_abs_diff += std::fabs(m.I_split[0] - m.I_split[1]); - a->split_sum += 0.5 * (m.I_split[0] + m.I_split[1]); + a->cc_split.Add(m.I_split[0], m.I_split[1], w_split); + a->split_abs_diff += w_split * std::fabs(m.I_split[0] - m.I_split[1]); + a->split_sum += w_split * 0.5 * (m.I_split[0] + m.I_split[1]); a->split_pairs++; } } diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index 8f4206ed3..83a625ed2 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -253,9 +253,12 @@ struct MergeAccum { double sum_wI_half[2] = {0.0, 0.0}; double sum_w_half[2] = {0.0, 0.0}; size_t n_half[2] = {0, 0}; + double q_half[2] = {0.0, 0.0}; // sum of 1 / image_scale_corr^2 over the half (HalfSetWeight) // The same sums with each frame's observations kept whole in the frame's half (R_split). double sum_wI_split[2] = {0.0, 0.0}; double sum_w_split[2] = {0.0, 0.0}; + size_t n_split[2] = {0, 0}; + double q_split[2] = {0.0, 0.0}; }; // XDS's error-model convention (Diederichs, Acta Cryst. D66 (2010) 733) is sigma^2 = a*(sigma0^2 + @@ -381,6 +384,7 @@ class MergeOnTheFly { uint64_t key; int32_t h, k, l; // the reflection's own index, as the accumulator records it float w, wI, d; + float inv_corr2; // 1 / image_scale_corr^2: how far the observation was scaled up (HalfSetWeight) bool on_ice_ring; }; // The observations of one image that reach the merge; returns how many the outlier test dropped. diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index e9ae92b10..ff3778481 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -1269,6 +1269,49 @@ TEST_CASE("MergeOnTheFly_StillsRejectsAGrainAmongPartials") { CHECK(merge.RejectedCount() == 1); } +// A reflection measured twice, on observations scaled up 300-fold (weak frames, low partiality), holds two +// half-set means whose noise dwarfs the spread of every well-measured reflection; CC1/2 and R_split weigh +// it by how far its observations were scaled up, so it does not decide them. +TEST_CASE("MergeOnTheFly_StillsCCHalfNotTakenByScaledUpPairs") { + DiffractionExperiment x; + x.SetSpaceGroup(gemmi::get_spacegroup_by_name("P 1")); + REQUIRE(!x.IsRotationIndexing()); + + std::mt19937 rng(11); + std::normal_distribution gauss(0.0f, 1.0f); + std::vector outcomes(40); + for (size_t i = 0; i < outcomes.size(); ++i) { + for (int h = 1; h <= 60; ++h) { + Reflection r{}; + r.h = h; r.k = 1; r.l = 0; + r.d = 2.0f + 0.01f * static_cast(h); + r.I = 100.0f * static_cast(h % 13 + 1) + 10.0f * gauss(rng); + r.sigma = 10.0f; + r.prescaling_corr = 1.0f; + r.partiality = 1.0f; + r.image_scale_corr = 1.0f; + outcomes[i].reflections.push_back(r); + } + if (i < 2) { + Reflection r{}; + r.h = 1; r.k = 2; r.l = 0; + r.d = 2.3f; + r.I = i == 0 ? 50.0f : -50.0f; + r.sigma = 30.0f; + r.prescaling_corr = 1.0f; + r.partiality = 1.0f; + r.image_scale_corr = 300.0f; + outcomes[i].reflections.push_back(r); + } + } + MergeOnTheFly merge(x); + merge.AddImages(outcomes); + const auto merged = merge.ExportIntensities(); + const auto stats = merge.MergeStats(merged, outcomes); + CHECK(stats.overall.cc_half > 0.99); + CHECK(stats.overall.r_split < 0.05); +} + // The capture term widens the sigma of a partial by (c (1 - p) ): a partial merges at a lower weight // than a full of the same intensity, and not at all differently with c = 0. TEST_CASE("MergeOnTheFly_CaptureTermDownweightsPartials") { From 90822803112f3795e78ce6b05161a238845a5c74 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 21:18:33 +0100 Subject: [PATCH 16/34] Report docs: the stills CC1/2 and R_split are weighted, FITTED_RESOLUTION reads the plain fall-off Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- docs/RUGNUX_REPORT.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 9994522b6..f0601edf1 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -71,9 +71,11 @@ absent where CC1/2 is not positive. All three are also columns of the shell tabl a "Serial statistics" table beside the shell table. Every frame's observations go whole into one of two half-sets - as CrystFEL's partialator deals out crystals - and R_split = (1/sqrt 2) sum|I_1 - I_2| / (1/2 sum (I_1 + I_2)) (White et al., 2012) and CC1/2 are taken over the reflections both halves -measured, so both compare directly with CrystFEL's `compare_hkl`. `CC_HALF=` deals each reflection's -observations out evenly instead, as XDS and cctbx do, and is the one the processing decides on; the two -agree overall and can differ in a single shell. On stills `R_MEAS=` measures the scatter of partial +measured. `CC_HALF=` deals each reflection's observations out evenly instead, as XDS and cctbx do, and is +the one the processing decides on; the two agree overall and can differ in a single shell. On stills all +three, overall and per shell, weigh each reflection by how far its observations were scaled up (as the +rotation CC1/2 does), so a handful of reflections seen only on weak frames cannot decide them - unlike +CrystFEL's plain sums; `FITTED_RESOLUTION` is still read off the unweighted CC1/2 fall-off. On stills `R_MEAS=` measures the scatter of partial observations, not the error of the merged intensity, and is not the number to quote. **`OBSERVATIONS_REJECTED=`** counts the observations the merge's outlier tests removed, and From 10bdf1d115a7897abc9aaf51f596c5e1fa058a0d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 21:48:01 +0100 Subject: [PATCH 17/34] JUNGFRAU modules for Jungfraujoch data: written into new files, rebuilt from the mask of old ones Stills module placement needs the detector's modules, which came only from NXdetector_module groups, CrystFEL panels or --module-geometry; a Jungfraujoch master carries none (or one whole-image module), so placement never ran on our own JUNGFRAU 4M/8M/9M data. - A JUNGFRAU experiment that assembles its image now puts the modules it places into the start message, so the master file holds module_0..N (placed as imaged) and the reader gives them back. The broker's own processing is unchanged. - A JUNGFRAU file without module groups (decided from the detector description) gets its modules from the pixel mask: every region clear of the module-gap bit must be one whole 1030 x 514 module either way round, else none are taken. DECTRIS files, whose masks use the same bit, are not touched. - Rotation runs keep processing the image as one panel where the modules sit as imaged (no module step there); merged data bit-identical on a JUNGFRAU 4M rotation set. On the 8M and 9M serial sets 16/18 modules are found; the held-out test favours placement on all three (capped rms 1.651 -> 1.615, 1.367 -> 1.350, 1.498 -> 1.365 px) but HDF5 input keeps the existing measure-and-report rule, so only the per-panel spot finding / integration changes: CCref last shell 0.273 -> 0.272, 0.421 -> 0.426, 0.443 -> 0.444 (noise level). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- common/DetectorModules.cpp | 37 +++++++++++++++++++++++ common/DetectorModules.h | 6 ++++ common/DiffractionExperiment.cpp | 4 +++ reader/HDF5MetadataSource.cpp | 10 +++++++ rugnux/Rugnux.cpp | 6 ++++ tests/DetectorModulesTest.cpp | 28 ++++++++++++++++++ tests/JFJochReaderTest.cpp | 51 ++++++++++++++++++++++++++++++++ 7 files changed, 142 insertions(+) diff --git a/common/DetectorModules.cpp b/common/DetectorModules.cpp index 7b313ed65..449b965f9 100644 --- a/common/DetectorModules.cpp +++ b/common/DetectorModules.cpp @@ -4,6 +4,7 @@ #include "DetectorModules.h" #include "DetectorGeometry.h" #include "Definitions.h" +#include "PixelMask.h" #include #include @@ -193,3 +194,39 @@ DetectorModules ModulesAsImaged(const DetectorGeometry &geom) { } return DetectorModules(out); } + +DetectorModules ModulesFromGapMask(const std::vector &mask, int64_t width, int64_t height) { + if (width <= 0 || height <= 0 || mask.size() != static_cast(width * height)) + return {}; + const auto clear = [&](int64_t x, int64_t y) { + return x >= 0 && y >= 0 && x < width && y < height + && (mask[y * width + x] & (1u << PixelMask::ModuleGapPixelBit)) == 0; + }; + std::vector taken(mask.size(), 0); + std::vector out; + for (int64_t y = 0; y < height; y++) + for (int64_t x = 0; x < width; x++) { + if (!clear(x, y) || taken[y * width + x]) + continue; + // The first clear pixel of a region not yet taken is a module's corner: the module runs to the + // gap along the row and down the column, and has to be whole and bounded by gap all round. + int64_t w = 0, h = 0; + while (clear(x + w, y)) w++; + while (clear(x, y + h)) h++; + if (!((w == CONVERTED_MODULE_COLS && h == CONVERTED_MODULE_LINES) + || (w == CONVERTED_MODULE_LINES && h == CONVERTED_MODULE_COLS))) + return {}; + for (int64_t yy = y - 1; yy <= y + h; yy++) + for (int64_t xx = x - 1; xx <= x + w; xx++) { + const bool inside = xx >= x && xx < x + w && yy >= y && yy < y + h; + if (clear(xx, yy) != inside) + return {}; + if (inside) + taken[yy * width + xx] = 1; + } + out.push_back(DetectorModule::AsImaged(x, y, w, h)); + } + if (out.size() < 2) + return {}; + return DetectorModules(out); +} diff --git a/common/DetectorModules.h b/common/DetectorModules.h index d38649d4f..4f979b952 100644 --- a/common/DetectorModules.h +++ b/common/DetectorModules.h @@ -121,3 +121,9 @@ public: // CONVERTED_MODULE_COLS x CONVERTED_MODULE_LINES turned the way the geometry turns it, placed as imaged. // Empty for a detector of one module. DetectorModules ModulesAsImaged(const DetectorGeometry &geom); + +// The modules of a JUNGFRAU image where only its pixel mask says where they are: every region of pixels +// clear of the module-gap bit (PixelMask::ModuleGapPixelBit) has to be one whole module, +// CONVERTED_MODULE_COLS x CONVERTED_MODULE_LINES either way round, and is taken as placed as imaged. +// Empty where any region is not, or where there are fewer than two. The mask is row-major, width x height. +DetectorModules ModulesFromGapMask(const std::vector &mask, int64_t width, int64_t height); diff --git a/common/DiffractionExperiment.cpp b/common/DiffractionExperiment.cpp index 488a8f1fb..d8bb68ff4 100644 --- a/common/DiffractionExperiment.cpp +++ b/common/DiffractionExperiment.cpp @@ -653,6 +653,10 @@ SpotFindingSettings DiffractionExperiment::DefaultDataProcessingSettings() { void DiffractionExperiment::FillMessage(StartMessage &message) const { message.images_per_file = GetImagesPerFile(); message.detector_modules = GetModules().Get(); + // A JUNGFRAU image assembled here is made of modules this experiment places itself: they go into + // the file one by one, so that what reads it back knows them. + if (message.detector_modules.empty() && GetDetectorType() == DetectorType::JUNGFRAU && IsGeometryTransformed()) + message.detector_modules = ModulesAsImaged(detector.GetGeometry()).Get(); message.beam_center_x = GetBeamX_pxl(); message.beam_center_y = GetBeamY_pxl(); const auto direct_beam = GetDiffractionGeometry().GetDirectBeam_pxl(); diff --git a/reader/HDF5MetadataSource.cpp b/reader/HDF5MetadataSource.cpp index d53bee837..0d6fa3b6d 100644 --- a/reader/HDF5MetadataSource.cpp +++ b/reader/HDF5MetadataSource.cpp @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute // SPDX-License-Identifier: GPL-3.0-only +#include +#include #include #include #include @@ -1148,6 +1150,14 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen } if (mask_tmp.empty()) mask_tmp = std::vector(image_size_x * image_size_y); + // A JUNGFRAU file that does not state its modules one by one still marks the gaps between + // them in its mask: each region clear of the gap bit is one module, placed as imaged. Only for + // JUNGFRAU - a DECTRIS mask uses the same bit for its gaps, and its modules are not read here. + std::string detector_upper = detector_name; + std::transform(detector_upper.begin(), detector_upper.end(), detector_upper.begin(), + [](unsigned char c) { return static_cast(std::toupper(c)); }); + if (dataset->experiment.GetModules().empty() && detector_upper.find("JUNGFRAU") != std::string::npos) + dataset->experiment.Modules(ModulesFromGapMask(mask_tmp, image_size_x, image_size_y)); dataset->pixel_mask = std::make_shared(mask_tmp); } diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 89efe8b2d..bcd85a963 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -387,6 +387,12 @@ Rugnux::Rugnux(JFJochReader &reader, DiffractionExperiment experiment, // and DIALS decide on. The anomalous signal is not lost: the merge always keeps the Bijvoet split, // and I(+)/I(-), SigAno and CCanom come from it in either mode. Stills keep -A as it was - their // merge has no separate split. + // Modules where the image already has them (a JUNGFRAU file's own, or rebuilt from its mask) tell + // the stills module placement what may move; a rotation run has no such step and processes the + // image as one panel, as it always has. + if (experiment_.IsRotationIndexing() && !experiment_.GetModules().Moved()) + experiment_.Modules({}); + if (experiment_.IsRotationIndexing() && !experiment_.GetScalingSettings().GetMergeFriedel()) { ScalingSettings s = experiment_.GetScalingSettings(); experiment_.ImportScalingSettings(s.MergeFriedel(true)); diff --git a/tests/DetectorModulesTest.cpp b/tests/DetectorModulesTest.cpp index 9a3478a29..bcb823c72 100644 --- a/tests/DetectorModulesTest.cpp +++ b/tests/DetectorModulesTest.cpp @@ -9,6 +9,7 @@ #include "../common/DetectorModules.h" #include "../common/DiffractionExperiment.h" #include "../common/JFJochMath.h" +#include "../common/PixelMask.h" #include "../reader/NXmxModules.h" #include "../writer/NXmxModules.h" @@ -110,3 +111,30 @@ TEST_CASE("DetectorModules_NXmxRoundTrip", "[DetectorModules]") { CHECK(b.sy == Catch::Approx(a.sy).margin(1e-6)); } } + +// A JUNGFRAU file that marks only the gaps between its modules in the mask: the modules come back from +// the mask, one per gap-bounded region; a region that is not one whole module gives none at all. +TEST_CASE("DetectorModules_FromGapMask", "[DetectorModules]") { + const DetectorGeometryModular geom(4, 2, 8, 36, false); + const DetectorModules expected = ModulesAsImaged(geom); + const int64_t width = geom.GetWidth(true), height = geom.GetHeight(true); + std::vector mask(width * height, 1u << PixelMask::ModuleGapPixelBit); + expected.ForEachPixel(width, height, [&](int64_t, size_t i, double, double) { mask[i] = 0; }); + mask[5] |= 1u << PixelMask::ErrorPixelBit; // other bits do not matter + + const DetectorModules modules = ModulesFromGapMask(mask, width, height); + REQUIRE(modules.size() == expected.size()); + for (size_t i = 0; i < modules.size(); i++) { + CHECK(modules[i].x0 == expected[i].x0); + CHECK(modules[i].y0 == expected[i].y0); + CHECK(modules[i].width == expected[i].width); + CHECK(modules[i].height == expected[i].height); + } + + // A gap pixel inside a module: no longer whole modules. + auto holed = mask; + holed[100 * width + 100] |= 1u << PixelMask::ModuleGapPixelBit; + CHECK(ModulesFromGapMask(holed, width, height).empty()); + // No gaps at all: one region the size of the image, not a module. + CHECK(ModulesFromGapMask(std::vector(width * height, 0), width, height).empty()); +} diff --git a/tests/JFJochReaderTest.cpp b/tests/JFJochReaderTest.cpp index fa3117bc4..02364ce78 100644 --- a/tests/JFJochReaderTest.cpp +++ b/tests/JFJochReaderTest.cpp @@ -5168,3 +5168,54 @@ TEST_CASE("JFJochReader_PanelId", "[HDF5][PanelId]") { REQUIRE(H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0); } + +// A JUNGFRAU image assembled by the experiment is written with its modules one by one (module_0, ...), +// and the reader gives them back where the assembly put them; a DECTRIS detector is written without. +TEST_CASE("JFJochReader_JungfrauModules", "[HDF5][Full]") { + RegisterHDF5Filter(); + for (const bool jungfrau: {true, false}) { + DiffractionExperiment x(jungfrau ? DetJF(4) : DetDECTRIS(2068, 1064, "Detector", {})); + x.FilePrefix("test_modules").ImagesPerTrigger(1).OverwriteExistingFiles(true); + x.BeamX_pxl(100).BeamY_pxl(200).DetectorDistance_mm(150) + .IncidentEnergy_keV(WVL_1A_IN_KEV).PixelSigned(false).BitDepthImage(16); + if (jungfrau) + x.FrameTime(std::chrono::microseconds(500), std::chrono::microseconds(10)); + const DetectorModules expected = jungfrau ? ModulesAsImaged(x.GetDetectorSetup().GetGeometry()) + : DetectorModules(); + CHECK(expected.size() == (jungfrau ? 4 : 0)); + ScanResultGenerator generator(x); + std::vector image(x.GetPixelsNum(), 0); + { + StartMessage start_message; + x.FillMessage(start_message); + FileWriter file_set(start_message); + DataMessage message{}; + message.image = CompressedImage(image, x.GetXPixelsNum(), x.GetYPixelsNum()); + message.number = 0; + REQUIRE_NOTHROW(file_set.WriteHDF5(message)); + REQUIRE_NOTHROW(generator.Add(message)); + EndMessage end_message; + end_message.max_image_number = 1; + generator.FillEndMessage(end_message); + file_set.WriteHDF5(end_message); + file_set.Finalize(); + } + { + JFJochHDF5Reader reader; + reader.ReadFile("test_modules_master.h5"); + const DetectorModules &back = reader.GetDataset()->experiment.GetModules(); + REQUIRE(back.size() == expected.size()); + for (size_t i = 0; i < back.size(); i++) { + CHECK(back[i].x0 == expected[i].x0); + CHECK(back[i].y0 == expected[i].y0); + CHECK(back[i].width == expected[i].width); + CHECK(back[i].height == expected[i].height); + CHECK(back[i].ox == Catch::Approx(expected[i].ox).margin(1e-3)); + CHECK(back[i].oy == Catch::Approx(expected[i].oy).margin(1e-3)); + } + CHECK_FALSE(back.Moved()); + } + remove("test_modules_master.h5"); + } + REQUIRE(H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0); +} From da641c603ac4ce44eb1183dda6317fce08bd1a32 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 31 Oct 2026 22:04:39 +0100 Subject: [PATCH 18/34] Stills: --model no longer changes the processing - its intensities are not the per-image reference Since rc164 a serial-stills run given --model with -C/-S computed structure factors from the model and handed them to the image pass as the reference intensities (ReferenceIntensities), the same path a reference MTZ takes. That reference also feeds the per-image scale and CC, so the frames dropped as collapsed, the radial scale, the indexing-ambiguity choice and from there the whole merge differed with and without the model: on a PAL CCD set CC1/2 0.976 without it, 0.893 with it, at the same code. A model is placed and validated after the merge; only a reference MTZ (-z) is a processing input now. The data-only ambiguity resolution (space group given) runs as without a model, and validation still relabels the finished merge into the model's indexing where it wins. With and without --model the merged reflections of that set are now identical, bit for bit. Docs updated. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- docs/RUGNUX_ADVANCED.md | 16 ++++++++-------- docs/RUGNUX_CLI.md | 2 +- docs/RUGNUX_TUTORIAL.md | 6 +++--- rugnux/Rugnux.cpp | 25 +++++-------------------- rugnux/RugnuxScaleMerge.cpp | 5 ++--- 5 files changed, 19 insertions(+), 35 deletions(-) diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index d387a1436..b32c70d43 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -46,10 +46,10 @@ against a foreign dataset injects that dataset's systematics — so `-z` never p errors into the intensities. `--reference-column` picks the column to read where the automatic choice (`F-model`, else `IMEAN`/`I`, else `FP`/`FOBS`/`F`) is not the right one. -For the indexing ambiguity an atomic model does as well: `--model` computes the intensities it -needs from the structure (a model that fits also sets the hand label and the setting, see -[Validating against a model](#validating-against-a-model-rugnux---model)). Where a reference dataset exists, prefer it; -where only a model does, it resolves the ambiguity just the same. +An atomic model does not take part in the processing: `--model` is placed and validated after the merge, +and a run with one merges exactly what a run without one does. A model that fits then sets the frame the +merge is written in - the hand label, the setting and the indexing (see +[Validating against a model](#validating-against-a-model-rugnux---model)) - by relabelling it, never by re-merging. ### The indexing ambiguity @@ -102,15 +102,15 @@ How to resolve it: | **Rotation**, a reference dataset exists | `-z reference.mtz`. Once the space group is settled, each candidate reindexing of the merged intensities is correlated with the reference and the best-correlating one is re-merged. Only the *hkl* labels change; the cell does not | | **Serial stills**, a reference dataset exists | `-z reference.mtz`. Resolved **per image**, at integration time, by correlating each crystal's intensities with the reference — so the merge never mixes hands in the first place. Where the crystals' own cells tell the indexings apart (pseudo-merohedry), a crystal its cell decides keeps that indexing and the reference decides the rest. The data-only resolution below runs as a check: `AMBIGUITY_DATA_DISAGREES=` | | **Rotation**, only a model | `--model model.pdb`. The merged data are fitted to the model in each candidate indexing and the lowest R-free wins — provided the model first beats its random-orientation null (`MODEL_FIT= ACCEPTED`) and its lead over the runner-up beats the lead a random placement of the same model takes; the written reflections are then reindexed into it, so the file, the R-factors and the maps agree. Where either bar is missed the data keep the indexing they were merged in, and the report says by how much (`MODEL_INDEXING_MARGIN_SIGMA`). It is the **data** that are relabelled, never the model: every dataset of one crystal form processed against the same model comes out in that model's indexing, so the reflection files and maps of a screening campaign compare directly. The relabelling is a proper rotation of the lattice, so the merging statistics are unchanged and I(+)/I(-) keep their hand | -| **Serial stills**, only a model | `--model model.pdb`, with the cell and space group given (`-C` / `-S`). Structure factors are computed from the model up front and used as the reference for the per-image test, exactly as a reference MTZ would be, the crystals their own cell decides again excepted. The data-only resolution below is then run only as a check: `AMBIGUITY_DATA_DISAGREES=` counts the crystals on which it would have chosen otherwise | -| **Serial stills**, neither, space group given (`-S`) | Nothing to do: the crystals are made consistent with each other from the data alone (`INDEXING_AMBIGUITY_RESOLVED= data`), in an indexing that is arbitrary against other data sets. A model or reference still settles the weakest crystals more reliably than the other crystals can, and puts the merge in its own indexing | +| **Serial stills**, only a model, space group given (`-S`) | The crystals are made consistent with each other from the data alone, as in the row below - the model plays no part in the processing. Validation then fits the merge to the model in each candidate indexing, as for rotation data, and relabels the written reflections into the model's indexing where it wins | +| **Serial stills**, neither, space group given (`-S`) | Nothing to do: the crystals are made consistent with each other from the data alone (`INDEXING_AMBIGUITY_RESOLVED= data`), in an indexing that is arbitrary against other data sets. A reference still settles the weakest crystals more reliably than the other crystals can, and puts the merge in its own indexing | | Neither | The run warns and merges in whichever indexing it found: for rotation data a usable dataset in an arbitrary frame; for stills whose space group the run determined, a merge of mixed indexings, which looks like a perfect twin and can pass for higher symmetry than the crystal has. Run again with `-S` | Two things to know about where the choice lands: - **`--mode scale` cannot repair a stills run after the fact.** The per-image test happens at - integration time, so a `_process.h5` whose images were integrated without a reference — an MTZ or a - model — has already lost the distinction, and no re-merge brings it back; the data-only resolution + integration time, so a `_process.h5` whose images were integrated without a reference MTZ has + already lost the distinction, and no re-merge brings it back; the data-only resolution runs in the full pipeline, not in `--mode scale`. On rotation data, where the ambiguity is one choice for the whole dataset, `--mode scale --model model.pdb` does resolve it. - **The choice reaches the written reflections**, not only the R-factors and the maps: the merged diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index 45674058f..8f91c26ea 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -304,7 +304,7 @@ serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rota | `-z, --reference ` | Reference data of the same crystal form — an MTZ or a PDB structure-factor mmCIF (`-sf.cif`), gzipped or not, recognised by content; `--reference-mtz` is the same option: fixes the space group, its setting and the cell, resolves the [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity), hands over the R-free set and reports CCref. Not a scale anchor | | `--free-flags ` | Take only the R-free test set of this MTZ or structure-factor mmCIF - e.g. the deposited reflections of the `--model` - for the validation, the refinement and the merged files; nothing else of the file is used, so unlike `-z` it fixes no space group, cell or frame. Taken on the model's axes where the merge matches the file's intensities (`R_FREE_FLAGS=`, see [Free set and model bias](RUGNUX_MODEL_AND_REFERENCE.md#free-set-and-model-bias)) | | `--reference-column