mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 92 Wiki Activity

rc179: everything on rc178 after its merge into main (#88), except the CHANGELOG

Browse Source 
The rc178 branch continued after #88 was squash-merged; this is its content from b9ada7914 (the tree
#88 squashed) to 094eee31a, as one change on main. The individual commits stay on the rc178 branch:

- still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes
- AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets
- DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked
- tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout
- Dense per-pixel maps through the panel table (panel layout stage 1)
- Panel id on spots and the panel table in the start message
- Viewer and preview show a panel detector as it lies on the detector plane
- Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster
- Panel layout stage 2: prediction and Bragg integration through the panel table
- XTC reader: frames handed up as stored, streams indexed in parallel
- XFEL reader: integer frames stay integer, one gain, placement by panel
- EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery"
- still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed
- float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count
- CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them
- Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table
- Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane
- still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good
- still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file
- still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition
- stills fixed cost: geometry refinement and pre-scan do the same work in less wall time
- stills geometry sample and trial spot finding run as many workers as the image pass
- Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame
- StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's
- MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in)
- Stills given a cell: the written cell is the mean over the frames that are merged
- Stills: the radial curvature correction decided per run on the same empty apertures
- StillsPartialityRefine: measure the run width only on shells with signal
- Stills pre-scan: report the paired z of the radial-correction comparison
- module placement: the five concurrent fits each keep every thread
- still battery: score R-free against the deposited data through the same dimple protocol
- still battery: the R-free target is the lower of the deposition's dimple and published values
- Stills: background clip follows the radial curve; signal disk from the lattice spacing
- Stills keep an overloaded observation as a lower bound instead of dropping it
- Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip
- battery: R-free against the deposition without refinement only, one protocol for rotation and stills
- still battery: EuXFEL sets run on a 2,000-image subset for now
- stills geometry refinement: fit the detector distance and the cell together
- stills geometry refinement: a distance given with --detector-distance is held in the joint fit
- Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count
- Metal integration test: the CPU comparison runs on a rotation experiment
- Stills: kept overloads go back to the drop where they are not concentrated at low resolution
- stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated
- battery results: XFEL still battery 20261022-1358 on rc178 0cf62452c - 12 sets, cold, R_free unrefined
- battery results: still summary gains the resolution of our merge and of the deposition
- Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value
- Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed
- XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock
- stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes
- rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid
- stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree
- CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get
- broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4)
- frontend: Mol* viewer of the placed model and its density on the Rugnux live page
- writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does
- rugnux live maps: one unit cell centred on the model; the viewer opens on all of it
- Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer
- Mol* viewer: open with the maps drawn within 10 A of the centre of view
- Mol* viewer: aaredb's structure-viewer setup, on a black stage
- rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons
- Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2)
- ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI)
- stills tilt decision: the modules stay where the run placed them
- still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it
- battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC
- Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons
- BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it
- ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization
- BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain
- ShadowFinder: measure the modulation robustly under a wide one-sided shadow
- ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation
- ShadowFinder: back to the predicted polarization factor, in the plane the background shows
- MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again
- battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables
- XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel
- stills geometry trial: the hypotheses a search asks are indexed side by side
- stills tilt decision: the fits start beside the pre-scan and the geometry trial
- XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache
- XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step
- GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block
- GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart
- docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts
- battery: doc_check.py cross-checks the manifests against the dataset pages
- stills module placement: the placement fit gets half the threads, each held-out fit an eighth
- still battery: the EuXFEL sets run on all their frames, as the published still runs did
- battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table
- macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-23 17:56:42 +02:00
1 parent afbe844b20
commit 0a515bb086
218 files changed
+55733 -1764
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

image_analysis/geom_refinement/GeometryRefiner.cpp
+288 -90
View File
@@ -9,6 +9,7 @@
#include <vector>
#include "../../common/JFJochMath.h" // PI
#include "../../common/ParallelFor.h"
#include "LatticeReduction.h"
#include "XtalResidual.h"
#include "ceres/ceres.h"
@@ -138,24 +139,171 @@ void PlaceOnModule(const ModuleShift &m, const double move[3], double x, double
py = m.y0 + s * ux + c * uy + move[1];
}
// XtalResidual for a spot on a module that is being placed: block `move` = (dx, dy, rot) of the module,
// (ux, uy) the spot's offset from the module centre. Moving the spot by R(rot) u - u + (dx, dy) shifts its
// offset from the beam exactly as moving the beam the other way does, so the beam handed to XtalResidual
// carries the move and everything else is the residual of an unmoved spot.
struct ModuleXtalResidual {
XtalResidual residual;
double ux, uy;
// How a residual reads the beam block. Without the tilt refined it is the PONI, as everywhere else. With the
// tilt refined it is the DIRECT BEAM, the point the undeflected beam lands on, and the PONI is worked out from
// it: a tilt moves the whole pattern by D tan(tilt)/pixel, exactly as a moved PONI does, so in PONI terms
// the two trade tens of pixels for each other (and the beam's box cannot hold the trade), while the direct
// beam does not move with the tilt and the tilt is left with only what no beam can imitate - the keystone.
// In XtalResidual's frame the direct beam is where M (u, v) = (-D tan rot1, -D tan rot2 / cos rot1), with
// (u, v) its offset from the PONI in mm and M the image orientation (a mirror and quarter turns, so M^-1 = M^T).
struct BeamBlock {
bool direct = false;
double m00 = 1, m01 = 0, m10 = 0, m11 = 1; // DetectorOrientation::Matrix(), as XtalResidual takes it
double pixel = 1;
template <typename T>
bool operator()(const T *const beam, const T *const distance_mm, const T *const detector_rot,
const T *const rotation_axis, const T *const p0, const T *const p1, const T *const p2,
const T *const move, T *out) const {
template <typename T, typename R>
void Poni(const T *const beam, const T &distance_mm, const R *const detector_rot, T *poni) const {
if (!direct) {
poni[0] = beam[0];
poni[1] = beam[1];
return;
}
using std::cos;
using std::tan;
const T dx = -distance_mm * tan(detector_rot[0]);
const T dy = -distance_mm * tan(detector_rot[1]) / cos(detector_rot[0]);
poni[0] = beam[0] - (T(m00) * dx + T(m10) * dy) / T(pixel);
poni[1] = beam[1] - (T(m01) * dx + T(m11) * dy) / T(pixel);
}
};
// The residual of one spot (XtalResidual) on the blocks this refinement can move, in this order: the beam
// (read through BeamBlock) and the distance as one block, the tilt where it is refined, the frame's orientation,
// the cell lengths, and the module's move (dx, dy, rot) where the spot's module is placed. The rotation axis, the
// cell angles and a held tilt enter as plain numbers.
//
// The residual is the difference of two halves that share no block: the observed reciprocal vector (beam,
// distance, tilt, move) and the predicted one (orientation, cell). Ceres' AutoDiffCostFunction would carry the
// derivatives of every block through both halves; here each half is differentiated on its own blocks only, which
// gives the same values and the same derivatives with half as many of them per number.
//
// A placed module: moving the spot by R(rot) u - u + (dx, dy), with (ux, uy) its offset from the module
// centre, shifts its offset from the beam exactly as moving the beam the other way does, so the beam handed
// to XtalResidual carries the move and everything else is the residual of an unmoved spot.
class SpotCost : public ceres::CostFunction {
public:
SpotCost(const XtalResidual &residual, const BeamBlock &beam_block, const double *detector_rot,
const double *rotation_axis, const double *cell_ang, bool tilt, bool module, double ux = 0,
double uy = 0)
: residual(residual), beam_block(beam_block), detector_rot(detector_rot), rotation_axis(rotation_axis),
cell_ang(cell_ang), tilt(tilt), module(module), ux(ux), uy(uy) {
set_num_residuals(3);
std::vector<int32_t> &sizes = *mutable_parameter_block_sizes();
sizes.push_back(3);
if (tilt)
sizes.push_back(2);
sizes.push_back(3);
sizes.push_back(3);
if (module)
sizes.push_back(3);
}
bool Evaluate(double const *const *parameters, double *residuals, double **jacobians) const override {
if (tilt)
return module ? EvaluateWith<true, true>(parameters, residuals, jacobians)
: EvaluateWith<true, false>(parameters, residuals, jacobians);
return module ? EvaluateWith<false, true>(parameters, residuals, jacobians)
: EvaluateWith<false, false>(parameters, residuals, jacobians);
}
private:
template <bool TILT, bool MODULE>
bool EvaluateWith(double const *const *p, double *out, double **jac) const {
constexpr int N_OBS = 3 + (TILT ? 2 : 0) + (MODULE ? 3 : 0);
using JO = ceres::Jet<double, N_OBS>;
using JP = ceres::Jet<double, 6>;
constexpr int TILT_BLOCK = 1, P0_BLOCK = TILT ? 2 : 1, P1_BLOCK = P0_BLOCK + 1, MOVE_BLOCK = P1_BLOCK + 1;
// The observed half, on the beam and distance, the tilt and the move.
JO bd[3], t[2], m[3];
for (int j = 0; j < 3; ++j)
bd[j] = JO(p[0][j], j);
if constexpr (TILT)
for (int j = 0; j < 2; ++j)
t[j] = JO(p[TILT_BLOCK][j], 3 + j);
if constexpr (MODULE)
for (int j = 0; j < 3; ++j)
m[j] = JO(p[MOVE_BLOCK][j], N_OBS - 3 + j);
JO obs[3];
if constexpr (TILT)
Observed(bd, t, MODULE ? m : nullptr, obs);
else
Observed(bd, detector_rot, MODULE ? m : nullptr, obs);
// The predicted half, on the orientation and the cell lengths.
JP p0[3], p1[3];
for (int j = 0; j < 3; ++j) {
p0[j] = JP(p[P0_BLOCK][j], j);
p1[j] = JP(p[P1_BLOCK][j], 3 + j);
}
const JP p2[3] = {JP(cell_ang[0]), JP(cell_ang[1]), JP(cell_ang[2])};
JP recip_unrot[3];
residual.PredictedRecipUnrot(p1, p2, recip_unrot);
JP pred[3];
const AngleAxisRotator<JP> rot_p0(p0);
rot_p0.Rotate(recip_unrot, pred);
for (int r = 0; r < 3; ++r)
out[r] = obs[r].a - pred[r].a;
if (!jac)
return true;
// Row-major 3 x size blocks; a block held constant has no Jacobian asked for.
auto fill = [](double *block, int size, auto derivative) {
if (block)
for (int r = 0; r < 3; ++r)
for (int c = 0; c < size; ++c)
block[r * size + c] = derivative(r, c);
};
fill(jac[0], 3, [&](int r, int c) { return obs[r].v[c]; });
if constexpr (TILT)
fill(jac[TILT_BLOCK], 2, [&](int r, int c) { return obs[r].v[3 + c]; });
if constexpr (MODULE)
fill(jac[MOVE_BLOCK], 3, [&](int r, int c) { return obs[r].v[N_OBS - 3 + c]; });
fill(jac[P0_BLOCK], 3, [&](int r, int c) { return -pred[r].v[c]; });
fill(jac[P1_BLOCK], 3, [&](int r, int c) { return -pred[r].v[3 + c]; });
return true;
}
template <typename T, typename R>
void Observed(const T *const beam_distance, const R *const tilt, const T *const move, T *recip_obs) const {
using std::cos;
using std::sin;
const T c = cos(move[2]), s = sin(move[2]);
const T moved_beam[2] = {beam[0] - ((c - T(1)) * T(ux) - s * T(uy) + move[0]),
beam[1] - (s * T(ux) + (c - T(1)) * T(uy) + move[1])};
return residual(moved_beam, distance_mm, detector_rot, rotation_axis, p0, p1, p2, out);
T b[2];
beam_block.Poni(beam_distance, beam_distance[2], tilt, b);
if (move) {
const T c = cos(move[2]), s = sin(move[2]);
b[0] = b[0] - ((c - T(1)) * T(ux) - s * T(uy) + move[0]);
b[1] = b[1] - (s * T(ux) + (c - T(1)) * T(uy) + move[1]);
}
const R axis[3] = {R(rotation_axis[0]), R(rotation_axis[1]), R(rotation_axis[2])};
residual.ObservedRecip(b, beam_distance[2], tilt, axis, recip_obs);
}
const XtalResidual residual;
const BeamBlock beam_block;
const double *detector_rot; // the tilt where it is held
const double *rotation_axis;
const double *cell_ang;
const bool tilt, module;
const double ux, uy;
};
// The residual of a fit that moves only the crystal orientations (refine_beam_distance off, no module
// placed): the observed reciprocal vector and the unrotated prediction are then constants of the spot, taken
// once in plain numbers, and only the orientation's three derivatives are carried.
struct OrientationResidual {
double recip_obs[3];
double recip_unrot[3];
template <typename T>
bool operator()(const T *const p0, T *out) const {
T e_pred_recip[3];
const AngleAxisRotator<T> rot_p0(p0);
rot_p0.Rotate(recip_unrot, e_pred_recip);
out[0] = recip_obs[0] - e_pred_recip[0];
out[1] = recip_obs[1] - e_pred_recip[1];
out[2] = recip_obs[2] - e_pred_recip[2];
return true;
}
};
@@ -166,36 +314,30 @@ struct ModuleXtalResidual {
// weighted by the spots each module holds: a module the frames barely reach is placed poorly, and an
// unweighted sum would hand its error to all the others.
//
// With the tilt refined, the two tilt-like patterns are held at zero as well: a detector tilted by a small
// angle about y moves a point (x, y) (from the beam) by (x^2, x y) / D times the angle, one about x by
// (x y, y^2) / D. Each is taken as the modules' moves projected on that field, in units of the lever.
// The detector tilt is not among them. Within one module a tilt is a gradient (a scale and a shear across
// it), which no rigid move of the module makes, and with the tilt refined the placement gains no new
// direction it cannot tell apart; holding the modules' moves to no tilt-like pattern as well only cost
// accuracy on frames the fit had not seen.
struct ModuleGauge {
std::vector<std::array<double, 2>> outward; // unit vector from the beam to each module centre
std::vector<std::array<double, 2>> centre; // each module centre from the beam, over lever_px
std::vector<double> share; // each module's fraction of the spots
double lever_px; // turns are weighed at this distance from the centre
double weight;
bool tilt = false;
template <typename T>
bool operator()(T const *const *move, T *out) const {
T sx(0), sy(0), sr(0), so(0), t1(0), t2(0);
T sx(0), sy(0), sr(0), so(0);
for (size_t i = 0; i < outward.size(); ++i) {
const T w(share[i]);
sx += w * move[i][0];
sy += w * move[i][1];
sr += w * move[i][2] * T(lever_px);
so += w * (move[i][0] * T(outward[i][0]) + move[i][1] * T(outward[i][1]));
const double x = centre[i][0], y = centre[i][1];
t1 += w * (move[i][0] * T(x * x) + move[i][1] * T(x * y));
t2 += w * (move[i][0] * T(x * y) + move[i][1] * T(y * y));
}
out[0] = T(weight) * sx;
out[1] = T(weight) * sy;
out[2] = T(weight) * sr;
out[3] = T(weight) * so;
out[4] = tilt ? T(weight) * t1 : T(0);
out[5] = tilt ? T(weight) * t2 : T(0);
return true;
}
};
@@ -210,9 +352,14 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
try {
const gemmi::CrystalSystem system = MapSystem(settings.crystal_system);
// Shared geometry parameter blocks (refined). Detector tilt and the rotation axis are constant.
double beam[2] = {nominal_geom.GetBeamX_pxl(), nominal_geom.GetBeamY_pxl()};
double distance_mm = nominal_geom.GetDetectorDistance_mm();
// Shared geometry parameter blocks (refined). The rotation axis is constant, and so is the detector tilt
// unless settings.refine_tilt. The beam and the distance are one block of three: every block refined
// beside it has three parameters too (orientations, cell lengths, module moves), so Ceres eliminates the
// orientations with its fixed-size Schur code rather than the general one (1.7x faster on one thread).
double beam_distance[3] = {nominal_geom.GetBeamX_pxl(), nominal_geom.GetBeamY_pxl(),
nominal_geom.GetDetectorDistance_mm()};
double *const beam = beam_distance;
double &distance_mm = beam_distance[2];
double detector_rot[2] = {nominal_geom.GetPoniRot1_rad(), nominal_geom.GetPoniRot2_rad()};
double rot_vec[3] = {1.0, 0.0, 0.0};
const double rot3 = nominal_geom.GetPoniRot3_rad();
@@ -222,6 +369,28 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
const double lambda = nominal_geom.GetWavelength_A();
const double pixel = nominal_geom.GetPixelSize_mm();
// With the tilt refined, the beam block is the direct beam (BeamBlock), started and bounded where the
// nominal geometry puts the direct beam.
const bool refine_tilt = settings.refine_tilt && settings.refine_beam_distance;
BeamBlock beam_block;
if (refine_tilt) {
const RotMatrix m = orientation.Matrix();
beam_block = {true, m.Column(0).x, m.Column(1).x, m.Column(0).y, m.Column(1).y, pixel};
double poni_at_zero[2];
const double zero[2] = {0, 0};
beam_block.Poni(zero, distance_mm, detector_rot, poni_at_zero);
beam[0] -= poni_at_zero[0];
beam[1] -= poni_at_zero[1];
}
const double beam0[2] = {beam[0], beam[1]};
// The PONI the beam block stands for, at the current tilt and distance.
auto poni = [&](double &x, double &y) {
double p[2];
beam_block.Poni(beam, distance_mm, detector_rot, p);
x = p[0];
y = p[1];
};
// Shared cell blocks. Lengths refined (regularized); angles constant.
double cell_len[3] = {input_cell.a, input_cell.b, input_cell.c};
const double cell_len0[3] = {input_cell.a, input_cell.b, input_cell.c};
@@ -280,6 +449,11 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
if (!refined[m])
move[m] = {0, 0, 0};
// Where nothing but the crystal orientations moves, each spot's residual is reduced to them
// (OrientationResidual).
const bool orientation_only = !settings.refine_beam_distance
&& std::none_of(refined.begin(), refined.end(), [](bool r) { return r; });
auto place = [&](const GeomRefineSpot &s, double &x, double &y) {
if (s.module >= 0 && static_cast<size_t>(s.module) < modules.size())
PlaceOnModule(modules[s.module], move[s.module].data(), s.x, s.y, x, y);
@@ -296,6 +470,8 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
return 0.3 - (0.3 - 0.1) * t;
};
ceres::CauchyLoss loss(loss_scale);
// Whether the last solve that ran produced a usable solution; folded into result.ok below,
// which otherwise reports success on spot and frame counts alone.
bool solve_usable = false;
@@ -303,14 +479,19 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
for (int round = 0; round < settings.rounds; ++round) {
EffectiveCell(system, cell_len, cell_ang, eff_len, eff_alpha, eff_beta, eff_gamma);
double poni_x, poni_y;
poni(poni_x, poni_y);
DiffractionGeometry cur = nominal_geom;
cur.BeamX_pxl(beam[0]).BeamY_pxl(beam[1]).DetectorDistance_mm(distance_mm)
cur.BeamX_pxl(poni_x).BeamY_pxl(poni_y).DetectorDistance_mm(distance_mm)
.PoniRot1_rad(static_cast<float>(detector_rot[0])).PoniRot2_rad(static_cast<float>(detector_rot[1]));
const double tol = round_tol(round);
const double tol_sq = tol * tol;
ceres::Problem problem;
// One robust loss for every spot, kept here rather than one allocated (and freed) per spot.
ceres::Problem::Options problem_options;
problem_options.loss_function_ownership = ceres::DO_NOT_TAKE_OWNERSHIP;
ceres::Problem problem(problem_options);
auto ordering = std::make_shared<ceres::ParameterBlockOrdering>();
int spots_used = 0, frames_used = 0;
@@ -330,23 +511,36 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
if (dn > tol_sq)
continue;
if (s.module >= 0 && static_cast<size_t>(s.module) < modules.size() && refined[s.module]) {
const XtalResidual xr(x, y, lambda, pixel, cos_rot3, sin_rot3, 0.0, h, k, l, system, orientation);
if (orientation_only) {
auto *r = new OrientationResidual;
double b[2];
beam_block.Poni(beam, distance_mm, detector_rot, b);
xr.ObservedRecip(b, distance_mm, detector_rot, rot_vec, r->recip_obs);
xr.PredictedRecipUnrot(cell_len, cell_ang, r->recip_unrot);
problem.AddResidualBlock(new ceres::AutoDiffCostFunction<OrientationResidual, 3, 3>(r),
&loss, orient[i].data());
} else if (s.module >= 0 && static_cast<size_t>(s.module) < modules.size() && refined[s.module]) {
// The spot as found: the module's move is a block of the residual.
const auto &m = modules[s.module];
problem.AddResidualBlock(
new ceres::AutoDiffCostFunction<ModuleXtalResidual, 3, 2, 1, 2, 3, 3, 3, 3, 3>(
new ModuleXtalResidual{XtalResidual(s.x, s.y, lambda, pixel, cos_rot3, sin_rot3,
0.0, h, k, l, system, orientation),
s.x - m.x0, s.y - m.y0}),
new ceres::CauchyLoss(loss_scale),
beam, &distance_mm, detector_rot, rot_vec, orient[i].data(), cell_len, cell_ang,
move[s.module].data());
} else
problem.AddResidualBlock(
new ceres::AutoDiffCostFunction<XtalResidual, 3, 2, 1, 2, 3, 3, 3, 3>(
new XtalResidual(x, y, lambda, pixel, cos_rot3, sin_rot3, 0.0,
h, k, l, system, orientation)),
new ceres::CauchyLoss(loss_scale),
beam, &distance_mm, detector_rot, rot_vec, orient[i].data(), cell_len, cell_ang);
const XtalResidual found(s.x, s.y, lambda, pixel, cos_rot3, sin_rot3, 0.0, h, k, l, system,
orientation);
auto *cost = new SpotCost(found, beam_block, detector_rot, rot_vec, cell_ang, refine_tilt, true,
s.x - m.x0, s.y - m.y0);
if (refine_tilt)
problem.AddResidualBlock(cost, &loss, beam_distance, detector_rot, orient[i].data(),
cell_len, move[s.module].data());
else
problem.AddResidualBlock(cost, &loss, beam_distance, orient[i].data(), cell_len,
move[s.module].data());
} else {
auto *cost = new SpotCost(xr, beam_block, detector_rot, rot_vec, cell_ang, refine_tilt, false);
if (refine_tilt)
problem.AddResidualBlock(cost, &loss, beam_distance, detector_rot, orient[i].data(),
cell_len);
else
problem.AddResidualBlock(cost, &loss, beam_distance, orient[i].data(), cell_len);
}
++used_in_frame;
++spots_used;
}
@@ -374,9 +568,11 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
new ceres::AutoDiffCostFunction<CellLengthRegularizer, 3, 3>(reg), nullptr, cell_len);
}
// Shared blocks go in the second elimination group; constant ones are stripped by Ceres.
for (double *b : {beam, &distance_mm, detector_rot, rot_vec, cell_len, cell_ang})
ordering->AddElementToGroup(b, 1);
// Shared blocks go in the second elimination group (those the problem has; constant ones are
// stripped by Ceres).
for (double *b : {beam_distance, detector_rot, cell_len})
if (problem.HasParameterBlock(b))
ordering->AddElementToGroup(b, 1);
// The refined modules' moves: bounded to what a misplaced module plausibly is, and held to
// zero in sum by the gauge (ModuleGauge). A 1 px mean shift costs as much there as moving
@@ -395,22 +591,18 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
const double oy = modules[m].y0 - nominal_geom.GetBeamY_pxl();
const double r = std::max(1.0, std::hypot(ox, oy));
gauge->outward.push_back({ox / r, oy / r});
gauge->centre.push_back({ox, oy});
gauge->share.push_back(module_spots[m] / total);
lever += r;
}
gauge->lever_px = lever / static_cast<double>(moved.size());
for (auto &c : gauge->centre)
c = {c[0] / gauge->lever_px, c[1] / gauge->lever_px};
gauge->weight = 10.0 * recip_per_px * sqrtN;
gauge->tilt = settings.refine_tilt;
auto *cost = new ceres::DynamicAutoDiffCostFunction<ModuleGauge, 4>(gauge);
std::vector<double *> blocks;
for (size_t m : moved) {
cost->AddParameterBlock(3);
blocks.push_back(move[m].data());
}
cost->SetNumResiduals(6);
cost->SetNumResiduals(4);
problem.AddResidualBlock(cost, nullptr, blocks);
}
for (size_t m : moved) {
@@ -425,22 +617,27 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
// Beam + distance bounds around the nominal value; cell lengths loosely bounded (the
// regularizer does the anchoring).
problem.SetParameterLowerBound(beam, 0, nominal_geom.GetBeamX_pxl() - settings.beam_range_px);
problem.SetParameterUpperBound(beam, 0, nominal_geom.GetBeamX_pxl() + settings.beam_range_px);
problem.SetParameterLowerBound(beam, 1, nominal_geom.GetBeamY_pxl() - settings.beam_range_px);
problem.SetParameterUpperBound(beam, 1, nominal_geom.GetBeamY_pxl() + settings.beam_range_px);
problem.SetParameterLowerBound(&distance_mm, 0,
std::max(1.0, nominal_geom.GetDetectorDistance_mm() - settings.distance_range_mm));
problem.SetParameterUpperBound(&distance_mm, 0,
nominal_geom.GetDetectorDistance_mm() + settings.distance_range_mm);
for (int j = 0; j < 3; ++j) {
problem.SetParameterLowerBound(cell_len, j, 0.7 * cell_len0[j]);
problem.SetParameterUpperBound(cell_len, j, 1.3 * cell_len0[j]);
}
if (!orientation_only) {
problem.SetParameterLowerBound(beam_distance, 0, beam0[0] - settings.beam_range_px);
problem.SetParameterUpperBound(beam_distance, 0, beam0[0] + settings.beam_range_px);
problem.SetParameterLowerBound(beam_distance, 1, beam0[1] - settings.beam_range_px);
problem.SetParameterUpperBound(beam_distance, 1, beam0[1] + settings.beam_range_px);
problem.SetParameterLowerBound(beam_distance, 2,
std::max(1.0, nominal_geom.GetDetectorDistance_mm() - settings.distance_range_mm));
problem.SetParameterUpperBound(beam_distance, 2,
nominal_geom.GetDetectorDistance_mm() + settings.distance_range_mm);
for (int j = 0; j < 3; ++j) {
problem.SetParameterLowerBound(cell_len, j, 0.7 * cell_len0[j]);
problem.SetParameterUpperBound(cell_len, j, 1.3 * cell_len0[j]);
}
if (!settings.refine_distance)
problem.SetParameterBlockConstant(&distance_mm);
if (settings.refine_tilt && settings.refine_beam_distance) {
if (!settings.refine_beam_distance) {
problem.SetParameterBlockConstant(beam_distance);
problem.SetParameterBlockConstant(cell_len);
} else if (!settings.refine_distance)
problem.SetManifold(beam_distance, new ceres::SubsetManifold(3, {2}));
}
if (refine_tilt) {
const double tilt_range = 2.0 * PI / 180.0;
for (int j = 0; j < 2; ++j) {
problem.SetParameterLowerBound(detector_rot, j, (j == 0 ? nominal_geom.GetPoniRot1_rad()
@@ -448,14 +645,6 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
problem.SetParameterUpperBound(detector_rot, j, (j == 0 ? nominal_geom.GetPoniRot1_rad()
: nominal_geom.GetPoniRot2_rad()) + tilt_range);
}
} else
problem.SetParameterBlockConstant(detector_rot);
problem.SetParameterBlockConstant(rot_vec);
problem.SetParameterBlockConstant(cell_ang); // angles not refined in v1
if (!settings.refine_beam_distance) {
problem.SetParameterBlockConstant(beam);
problem.SetParameterBlockConstant(&distance_mm);
problem.SetParameterBlockConstant(cell_len);
}
ceres::Solver::Options options;
@@ -477,8 +666,10 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
// Final fit quality: median detector residual of the retained spots at the refined geometry.
EffectiveCell(system, cell_len, cell_ang, eff_len, eff_alpha, eff_beta, eff_gamma);
double poni_x, poni_y;
poni(poni_x, poni_y);
DiffractionGeometry cur = nominal_geom;
cur.BeamX_pxl(beam[0]).BeamY_pxl(beam[1]).DetectorDistance_mm(distance_mm)
cur.BeamX_pxl(poni_x).BeamY_pxl(poni_y).DetectorDistance_mm(distance_mm)
.PoniRot1_rad(static_cast<float>(detector_rot[0])).PoniRot2_rad(static_cast<float>(detector_rot[1]));
std::vector<double> residuals;
constexpr double CAP_PX = 2.0;
@@ -530,8 +721,8 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
}
result.ok = solve_usable && result.spots_used >= settings.min_spots_total && result.frames_used > 0;
result.beam_x_px = beam[0];
result.beam_y_px = beam[1];
result.beam_x_px = poni_x;
result.beam_y_px = poni_y;
result.distance_mm = distance_mm;
result.cell = BuildUnitCell(system, cell_len, cell_ang);
result.rot1_rad = detector_rot[0];
@@ -560,7 +751,7 @@ GeometryRefinerResult RefineGlobalGeometry(const DiffractionGeometry &nominal_ge
std::vector<ModuleScan> ScanModuleShifts(const DiffractionGeometry &geom,
const std::vector<GeomRefineFrame> &frames,
const std::vector<ModuleShift> &modules,
double range_px) {
double range_px, size_t nthreads) {
// A spot is on its lattice when every fractional index is within this of a whole number.
constexpr double ON_LATTICE = 0.15;
@@ -602,14 +793,21 @@ std::vector<ModuleScan> ScanModuleShifts(const DiffractionGeometry &geom,
const double step = pass == 0 ? 1.0 : 0.5;
const double half = pass == 0 ? range_px : 1.0;
const double cx = r.dx, cy = r.dy;
for (double dy = -half; dy <= half; dy += step)
for (double dx = -half; dx <= half; dx += step) {
const int n = score(on_module[m], modules[m], cx + dx, cy + dy);
if (n > r.on_lattice_at_best) {
r.on_lattice_at_best = n;
r.dx = cx + dx;
r.dy = cy + dy;
}
std::vector<double> offsets;
for (double d = -half; d <= half; d += step)
offsets.push_back(d);
// Every shift of the grid is scored on the workers, then the grid is walked in order: the first
// shift with the most spots wins, as it would on one thread.
const int n = static_cast<int>(offsets.size());
std::vector<int> on_lattice(n * n);
ParallelFor(n * n, nthreads, [&](int k) {
on_lattice[k] = score(on_module[m], modules[m], cx + offsets[k % n], cy + offsets[k / n]);
});
for (int k = 0; k < n * n; k++)
if (on_lattice[k] > r.on_lattice_at_best) {
r.on_lattice_at_best = on_lattice[k];
r.dx = cx + offsets[k % n];
r.dy = cy + offsets[k / n];
}
}
}