jungfrauandClaude Opus 5
2308bbad8c
A crystal integrated to the detector corner but merged well short of it carries observations through the whole merge that the merge then discards. On the heaviest dataset in the rotation test set that is 63.3 M partials of which 6.4 M are ever used: the other nine tenths are sorted, uploaded, scaled, combined, error-modelled and post-refined before anything looks at their resolution. Ingest copied every one of them unconditionally, and the d_min limit was first applied far downstream, in the ASU grouping. They are now dropped at ingest, immediately after the one big sort: - WHOLE raw-hkl runs are dropped, on the same rawrun_d the ASU grouping already tests. A per-observation test is not equivalent - a run is in or out today by one member's d - and using a different rule here would put the two out of step. - The drop happens AFTER the flux meter, which takes each frame's mean background over every reflection on it, and after the sort, so neither changes. - The compaction runs in index order, so a frame's observations stay contiguous and keep their order, and every per-frame sum keeps its sequence of roundings. Post-refinement reads the integration outcomes rather than the merge arrays, so it still sees every reflection. That is the point: --integration-high-resolution buys the same time by never integrating the reflections, and pays for it in the per-frame geometry, which wants them. Two more passes over the observation array go with it. The incident-flux divide was 15.8 % of all user cycles to read one int and divide one float across 5 GB; the per-frame mean it needs is now accumulated by the ingest fill loop - one frame, one thread, same order, so bit-exact - and the divide rides on the finiteness pass that already touches that field. And the geometry post-refinement is fitted on a bounded sample of partials, selected by a hash of the raw hkl so whole rocking events are kept or dropped together and the sweep and the detector are thinned uniformly. The sample size is 8 M and the reason it is not smaller is measured. Over a 126x thinning the fitted rotation scale is flat to 2e-5 and the beam centre moves 0.03 px, but the CELL scale breaks between 8 M and 4 M: the axis step keeps the 20 000 strongest events, so once the pool approaches that size it starts fitting weaker ones and the second pass's cell shifts by ~0.1 %. Measured on the heaviest crystal, three A/B pairs with the order alternated: 68.4 s -> 37.3 s wall, 530 s -> 221 s of CPU. Whole battery 8m07s -> 7m15s, space group 21/24 with the same three disagreements as before, no failures. Bit-identical is not available on the GPU path - the resident reductions and the fulls emit order depend on array length - so what is shown is that every difference sits inside the spread the unmodified binary has against itself between two runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
70 lines
4.3 KiB
C++
70 lines
4.3 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <vector>
|
|
|
|
#include "../../common/DiffractionGeometry.h"
|
|
#include "../../common/CrystalLattice.h"
|
|
#include "../../common/GoniometerAxis.h"
|
|
#include "../../common/UnitCell.h"
|
|
#include "../../common/Logger.h"
|
|
#include "../IntegrationOutcome.h"
|
|
#include "gemmi/symmetry.hpp"
|
|
|
|
// Post-integration geometry refinement for rotation data. Unlike the at-indexing XtalOptimizer, this runs
|
|
// AFTER integration/merge, where each reflection has an OBSERVED rocking centroid phi_obs (the intensity-
|
|
// weighted mean goniometer angle over the frames it spans) and an observed spot position. It refines one
|
|
// shared crystal orientation + cell (+ optionally the detector distance) against two residuals:
|
|
// * an Ewald excitation residual evaluated at phi_obs (distance-independent) -> pins the absolute cell
|
|
// scale that the positional residual leaves degenerate with the distance. Because phi_obs is the real
|
|
// rocking angle (not a frame centre) it is unbiased.
|
|
// * the positional detector<->reciprocal residual at each partial's observed spot -> pins the distance.
|
|
// Reflections are weighted by their merged I/sigma (strong, well-measured reflections dominate).
|
|
|
|
struct PostRefineResult {
|
|
bool ok = false;
|
|
DiffractionGeometry geom; // refined (distance / beam left as configured)
|
|
UnitCell cell{}; // refined unit cell
|
|
int events_used = 0;
|
|
int obs_used = 0;
|
|
double distance_before_mm = 0.0, distance_after_mm = 0.0;
|
|
double beam_x_before_px = 0.0, beam_x_after_px = 0.0; // refined beam centre (GEOM mode)
|
|
double beam_y_before_px = 0.0, beam_y_after_px = 0.0;
|
|
bool cell_refined = false; // GEOM step A (cell scale + axis) passed cross-validation
|
|
bool detector_refined = false; // GEOM step B (distance + beam) passed cross-validation
|
|
// GONIOMETER ROTATION SCALE: the factor by which the stage actually turned relative to the angle
|
|
// stored in the file (which is the COMMANDED value, hence a stage calibration error is invisible in
|
|
// the header). Fitted after step A as a single free parameter, with the cell scale and the axis
|
|
// direction held at their committed values. Always the fitted value; 1.0 = header and stage agree.
|
|
double rotation_scale = 1.0;
|
|
// Whether the fit passed every test needed to ACT on it: enough sweep and events, a significant and
|
|
// physically relevant size, and the same k from every fifth of the sweep. Only then is it applied.
|
|
bool rotation_scale_suspect = false;
|
|
};
|
|
|
|
struct PostRefineSettings {
|
|
gemmi::CrystalSystem crystal_system = gemmi::CrystalSystem::Triclinic;
|
|
bool refine_geometry = false; // XtalOptimizer-equivalent: cell scale + axis (from phi_obs) and detector
|
|
// distance + beam centre (from the observed spot positions X,Y), as two
|
|
// separate cross-validated steps. The only supported refinement mode.
|
|
double excitation_weight = 1.0; // weight of the phi/excitation residual vs the positional one
|
|
int min_events = 50;
|
|
// Cap on how many partials are gathered, reached by thinning whole raw hkls out of the set (see the
|
|
// .cpp). Five geometry parameters do not need tens of millions of observations, and the two solver
|
|
// steps already cap themselves an order of magnitude below this.
|
|
size_t max_partials = 8000000;
|
|
int num_threads = 1;
|
|
};
|
|
|
|
// nominal_geom / reference_latt: the current detector geometry and the phi=0 reference lattice (orientation
|
|
// + cell) from rotation indexing. outcomes: the per-image integrated reflections (observed_x/y, I, sigma,
|
|
// image_number). axis: the goniometer. Returns ok=false (geometry untouched) on failure.
|
|
PostRefineResult PostRefineRotationGeometry(const std::vector<IntegrationOutcome> &outcomes,
|
|
const GoniometerAxis &axis,
|
|
const DiffractionGeometry &nominal_geom,
|
|
const CrystalLattice &reference_latt,
|
|
const PostRefineSettings &settings,
|
|
Logger &logger);
|