Post-refinement: the distance hypothesis is decided at the fit, on the excitation residual
Whether a sweep determines its detector distance at all was decided by running the canonical pass twice - once at the post-refined distance, once at the header's - and keeping whichever realised the lower held-out residual, the header's on a tie. The first-stage test that sent a run down that arm compared the free and held fits on the POOLED held-out residual, and pooling is what made it uninformative: the positional family cannot see the distance along the degenerate direction, so it dilutes the one family that can, while the unpaired standard error of a heavy-tailed mean of squares is 2-25 % of the mean against in-fit differences of 0.1-1 %. The test therefore said "cannot tell" on four fifths of the fitted sweeps and the arm ran on most of a corpus, buying by prior what it could not measure. The two hypotheses are now compared on the one residual family that can tell them apart. The excitation residual never involves the detector, so it is blind to the distance itself; what it sees is the cell scale, and a held fit at a wrong header distance is forced into a wrong cell scale by the spot positions, which the rocking angles then refuse - measured on a sweep whose header was 1.4 % long, the held fit's held-out excitation residual is seventeen times the free fit's. Where freeing the distance lowers that residual by more than its own standard error the free fit is committed as before; where it does not, the held fit is committed - header distance, refined beam, cell, orientation and axis - and the walk and the commit run held. The positional residual is deliberately not consulted: its in-fit gain along the degenerate direction is the one re-integration erases. The question is asked only while the run is still at the file's distance. A run that has walked off the header has already refuted that hypothesis by re-integrating, and asking it again at every round stalls a walk short of its fixed point, because the excitation standard error at the walk's tail is outlier-dominated (measured: a walk stopped 0.4 % early, seven passes, cell 0.65 % off against 0.26 %, ISa 10.6 against 14.0). So there is no arm, and with it go the two probe passes that measured it and the canonical pass the losing arm used to cost: the decision costs two Ceres solves. On thirteen rotation sweeps covering both verdicts, every decision the arm took by evidence or by its tie rule is reproduced at the fit, except where the excitation family sees what the pooled test could not and the free distance - the better cell against an external reference - is taken instead; merged intensities are unchanged where the verdict is. POSTREFINE_DISTANCE_HELD now means "the committed fit held the header distance", and is cleared where a later geometry walk left it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
This commit is contained in:
1 parent
2e4eabb317
commit
2fdc4d3bdf
5 files changed
+94
-158
No files matched your search
@@ -326,7 +326,7 @@ The refinement above (§7.2) runs per image against that image's spots. For rota
|
||||
|
||||
The geometry the run commits is re-fitted on all the reflections once the held-out half has approved it — a walk from where it arrived, one step wide; everything else from nominal in the wide box — and the bounds are asked again of that fit rather than only of the half that earned it. A move outside them leaves the geometry at nominal, as every other refusal does, and the refusal says so in the report (`POSTREFINE_REFUSED`, with what the fit wanted) rather than passing silently. Detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal.
|
||||
|
||||
**Whether the data determine the distance at all is asked, not assumed.** At a detector far enough away that no reflection reaches more than a few degrees of $2\theta$, a longer distance and a larger cell move every spot the same way to first order — the difference is of order $\sin^2\theta$ of the spot's own position, about 0.4 px rms per per cent of distance over a 2M detector at 820 mm against 2–3 px at the distances a crystal is usually collected at — and the rocking angles pin the cell scale only through the same $\sin\theta$. The joint fit then finds a distance/cell pair that fits its own observations a little better than the header, commits it, and the pass re-integrated there finds the next pair: a walk along the degenerate direction that the realised residual never ratifies (measured on such a sweep: a header at 820 mm walked to 846 mm with the cell 3.3 % too large, the realised held-out residual flat at every round). So the same fit is asked once more with the distance **held at the header**, every other block as free as before — the nested hypothesis "the header distance is right". Where the free distance lowers the held-out residual below the held fit's by more than that residual's own standard error, it has paid decisively and is committed exactly as before, which is every sweep whose $2\theta$ reaches tens of degrees and whose header is wrong. Where it has not, the fit cannot say — its own gain is not what decides, because a re-integration at the new distance need not realise it — so **both hypotheses are carried**: the canonical pass runs at the free geometry as before and once more at the header distance, and the run keeps the one whose *re-integrated* held-out residual is lower, the header's on a tie (the report says so: `POSTREFINE_DISTANCE_HELD`, and the pass decision names both residuals). Nothing is tuned here: the only inputs are the standard errors of the residuals themselves, the same noise the geometry walk's rounds have to beat. The wavelength is never refined on a single crystal for the same reason in its exact form: it scales the spot positions and the rocking angles identically to the cell, so no sweep can tell the two apart at any $2\theta$.
|
||||
**Whether the data determine the distance at all is asked, not assumed.** At a detector far enough away that no reflection reaches more than a few degrees of $2\theta$, a longer distance and a larger cell move every spot the same way to first order — the difference is of order $\sin^2\theta$ of the spot's own position, about 0.4 px rms per per cent of distance over a 2M detector at 820 mm against 2–3 px at the distances a crystal is usually collected at. The joint fit then finds a distance/cell pair that fits its own spot positions a little better than the header, commits it, and the pass re-integrated there finds the next pair: a walk along the degenerate direction that the realised residual never ratifies (measured on such a sweep: a header at 820 mm walked to 846 mm with the cell 3.3 % too large, the realised held-out residual flat at every round). So the same fit is asked once more with the distance **held at the header**, every other block as free as before — the nested hypothesis "the header distance is right" — and the two are compared on the one residual family that can tell them apart: the **excitation** residual. It never involves the detector, so it is blind to the distance itself; what it sees is the cell scale, and a held fit at a wrong header distance is forced into a wrong cell scale by the spot positions, which the rocking angles then refuse (measured: a header 1.4 % long leaves the held fit's excitation residual seventeen times the free fit's). Where freeing the distance lowers the held-out excitation residual below the held fit's by more than that residual's own standard error, the free fit is committed exactly as before; where it does not, the held fit is — header distance, refined beam, cell, orientation and axis — and the report says so (`POSTREFINE_DISTANCE_HELD`). The positional residual is deliberately not consulted for this: it is the family whose in-fit gain along the degenerate direction re-integration erases, and pooled with the excitation family it either drowns a decisive excitation gain in its own noise (a 54 % excitation gain read as 9 % pooled against a 9 % noise) or lends the degenerate direction a gain that is not there. Nothing is tuned here: the only input is the standard error of the residual itself, the same noise the geometry walk's rounds have to beat. The wavelength is never refined on a single crystal for the same reason in its exact form: it scales the spot positions and the rocking angles identically to the cell, so no sweep can tell the two apart at any $2\theta$.
|
||||
3. **Pass 2** re-indexes de novo and re-integrates at the committed geometry. Only the **detector distance and beam centre** carry over: the refined cell, orientation and axis are what make the distance identifiable, but pass 2 re-indexes from scratch, so they are not propagated. Where that re-index indexes too few frames the run falls back to pass 1's lattice and integrates it at the refined geometry — and the cell is then **scaled to the distance it will be used at**, since a real-space cell is measured against the distance its spots were seen at, and carrying it across a distance change otherwise scales the whole cell by the ratio of the two. The orientation is untouched.
|
||||
|
||||
The space group is determined **after** pass 2, on the geometry the run refined, and pass 1 does not search at all: a decision taken on the worse of the two passes and then carried forward is a constraint on the better one, and would have to be reconciled with what pass 2 later found. The guard that chooses which pass is written compares each pass's **first** merge — $P1$ on both sides, full resolution range, before the correction surfaces — which both passes produce anyway, so it never compares statistics computed in two different space groups. What it compares there is the **signal each pass measured**: the count of unique reflections merged at $I/\sigma \ge 2$. Self-consistency cannot do this job — against an external arbiter the signal count named the more accurate geometry on 23 of 27 arm-dataset pairs where $R_\mathrm{meas}$, $CC_{1/2}$ and ISa managed 13, a coin flip — and that merge's own $CC_{1/2}$ least of all, being pooled over the whole range, uncut and uncorrected, so the shells with no signal in them dominate it and they are exactly the shells a geometry move disturbs (it reads 0.13 on a crystal whose data merge at 0.995). The refined pass is sent back only where it merges more unique reflections than its cell can hold, where it measured decisively less signal (10 %, and only where it holds no more reflections either — a wider integration disk pulls weak reflections in and dilutes the strong fraction without measuring less), or where it lost the axial rows the systematic absences are read off. One index-time veto remains and is keyed to pass 1's **lattice** rather than its group: a centred pass-1 lattice against a primitive pass-2 one.
|
||||
|
||||
@@ -501,10 +501,12 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
// summed over below is the same weighting the solver itself applies.
|
||||
// se_out, where asked for: the standard error of the returned mean over the residual values
|
||||
// it averages.
|
||||
// exc_se_out, where asked for: the same standard error for the excitation mean alone.
|
||||
auto joint_cost = [&](Subset s, const double bm[2], const double ds[1], const double rv[3],
|
||||
const double q0[3], const double q1[3], const double q2[3],
|
||||
double &pos_out, double &exc_out, double *se_out = nullptr) {
|
||||
double cp = 0.0, ce = 0.0, c4 = 0.0;
|
||||
double &pos_out, double &exc_out, double *se_out = nullptr,
|
||||
double *exc_se_out = nullptr) {
|
||||
double cp = 0.0, ce = 0.0, c4 = 0.0, c4e = 0.0;
|
||||
size_t np = 0, ne = 0;
|
||||
for (size_t oi = 0; oi < n_obs; ++oi) {
|
||||
const Partial *pp = obs[oi];
|
||||
@@ -528,6 +530,7 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
r(rv, q0, q1, q2, &res);
|
||||
ce += res * res;
|
||||
c4 += res * res * res * res;
|
||||
c4e += res * res * res * res;
|
||||
++ne;
|
||||
}
|
||||
pos_out = np ? cp / np : 0.0;
|
||||
@@ -537,6 +540,9 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
if (se_out)
|
||||
*se_out = (np + ne) > 1 ? std::sqrt(std::max(0.0, c4 / n - mean * mean) / (n - 1.0))
|
||||
: std::numeric_limits<double>::quiet_NaN();
|
||||
if (exc_se_out)
|
||||
*exc_se_out = ne > 1 ? std::sqrt(std::max(0.0, c4e / ne - exc_out * exc_out) / (ne - 1.0))
|
||||
: std::numeric_limits<double>::quiet_NaN();
|
||||
return mean;
|
||||
};
|
||||
|
||||
@@ -663,55 +669,74 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
double distance_corr[4];
|
||||
bool convJ = solve_joint(FIT, /*wide=*/true, /*fix_distance=*/false, bm_f, ds_f, rv_f,
|
||||
q0_f, q1_f, q2_f, distance_corr);
|
||||
double cv_ref_se = NAN;
|
||||
double exc_ref_se = NAN;
|
||||
double cv_ref = convJ ? joint_cost(VAL, bm_f, ds_f, rv_f, q0_f, q1_f, q2_f,
|
||||
pos_ref, exc_ref, &cv_ref_se)
|
||||
pos_ref, exc_ref, nullptr, &exc_ref_se)
|
||||
: cv_nom;
|
||||
int steps = 1;
|
||||
|
||||
// Do these data determine the distance at all? At a detector far enough away that
|
||||
// no reflection reaches more than a few degrees of 2theta, a longer distance and a
|
||||
// larger cell move every spot the same way to first order (the difference is of
|
||||
// order sin^2 theta of the spot's own position), and the rocking angles pin the
|
||||
// cell scale only through the same sin theta. The joint fit then finds a
|
||||
// distance/cell pair that fits its own observations a little better than the
|
||||
// order sin^2 theta of the spot's own position). The joint fit then finds a
|
||||
// distance/cell pair that fits its own spot positions a little better than the
|
||||
// header, commits it, and the pass re-integrated there finds the next pair - a
|
||||
// walk along the degenerate direction that the realised residual never ratifies.
|
||||
// So the same fit is asked once more with the distance HELD at the header, every
|
||||
// other block as free as before - the nested hypothesis "the header distance is
|
||||
// right". Where the free distance lowers the held-out residual below the held fit's
|
||||
// by more than the standard error of that residual - the same noise a round of the
|
||||
// geometry walk has to beat (HeldOutResidualFell) - it has paid decisively and is
|
||||
// committed exactly as before. Where it has not, this fit cannot say: its own gain is
|
||||
// not what decides, because the pass re-integrated at the new distance need not
|
||||
// realise it (measured: a 0.6 % in-fit gain at nearly three times its paired scatter,
|
||||
// none of it realised) while a sweep at a short distance realises a gain of that size
|
||||
// several times over. So both hypotheses are carried - the free fit committed as
|
||||
// before, the held fit's beam beside it - and RunAllPasses runs the canonical pass at
|
||||
// each and keeps the one whose RE-INTEGRATED held-out residual is lower, the header's
|
||||
// on a tie.
|
||||
// right" - and the two are compared on the one residual family that can tell them
|
||||
// apart. The excitation residual never involves the detector, so it is blind to the
|
||||
// distance itself; what it sees is the cell scale, and a held fit at a wrong header
|
||||
// distance is forced into a wrong cell scale by the spot positions, which the
|
||||
// rocking angles then refuse (measured: a header 1.4 % long leaves the held fit's
|
||||
// excitation residual seventeen times the free fit's). Where freeing the distance
|
||||
// lowers the held-out excitation residual below the held fit's by more than the
|
||||
// standard error of that residual - the same noise a round of the geometry walk has
|
||||
// to beat (HeldOutResidualFell) - the free fit is committed exactly as before. Where
|
||||
// it does not, the held fit is the candidate from here on: what the walk, the gate
|
||||
// and the commit below see. The positional residual is deliberately not consulted
|
||||
// for this. Its in-fit gain along the degenerate direction is the one re-integration
|
||||
// erases, and pooled with the excitation family it either drowns a decisive
|
||||
// excitation gain in its own noise (measured: a 54 % excitation gain read as 9 %
|
||||
// pooled against a 9 % noise) or lends the degenerate direction a gain that is not
|
||||
// there.
|
||||
double bm_h[2] = {beam_x0, beam_y0}, ds_h[1] = {dist0};
|
||||
double rv_h[3], q0_h[3], q1_h[3], q2_h[3];
|
||||
for (int j = 0; j < 3; ++j) {
|
||||
rv_h[j] = ax0[j]; q0_h[j] = p0_0[j]; q1_h[j] = p1_0[j]; q2_h[j] = p2_0[j];
|
||||
}
|
||||
const bool convH = convJ && solve_joint(FIT, /*wide=*/true, /*fix_distance=*/true,
|
||||
bm_h, ds_h, rv_h, q0_h, q1_h, q2_h);
|
||||
double pos_h = 0.0, exc_h = 0.0, cv_h = cv_nom, cv_h_se = NAN;
|
||||
// A run that has already walked off the header has refuted that hypothesis by
|
||||
// re-integrating - the rounds of a walk are judged on what they realise - so the fit
|
||||
// is not asked to hold a distance that is no longer the header's.
|
||||
const bool convH = settings.distance_at_header && convJ
|
||||
&& solve_joint(FIT, /*wide=*/true, /*fix_distance=*/true, bm_h, ds_h,
|
||||
rv_h, q0_h, q1_h, q2_h);
|
||||
double pos_h = 0.0, exc_h = 0.0, cv_h = cv_nom, exc_h_se = NAN;
|
||||
if (convH)
|
||||
cv_h = joint_cost(VAL, bm_h, ds_h, rv_h, q0_h, q1_h, q2_h, pos_h, exc_h, &cv_h_se);
|
||||
const double noise = std::hypot(cv_ref_se, cv_h_se);
|
||||
const bool distance_pays = convH && std::isfinite(noise) && cv_ref < cv_h - noise;
|
||||
const bool distance_undecided = convH && !distance_pays;
|
||||
logger.Info("Post-refine GEOM distance: free {:.3f} mm gives held-out {:.3e} +- {:.1e}; held "
|
||||
"at the header {:.3f} mm gives {:.3e} +- {:.1e} (positional {:.3e}, excitation "
|
||||
"{:.3e}); free minus held {:+.3e} against the noise {:.1e} => {}; the free fit's "
|
||||
"formal sd(distance) {:.4f} %, corr(distance, cell a/b/c) {:.4f} {:.4f} {:.4f}",
|
||||
ds_f[0], cv_ref, cv_ref_se, dist0, cv_h, cv_h_se, pos_h, exc_h, cv_ref - cv_h,
|
||||
noise,
|
||||
distance_undecided ? "this fit cannot tell, both are carried to the canonical pass"
|
||||
: "the free distance pays",
|
||||
100.0 * distance_corr[0], distance_corr[1], distance_corr[2], distance_corr[3]);
|
||||
cv_h = joint_cost(VAL, bm_h, ds_h, rv_h, q0_h, q1_h, q2_h, pos_h, exc_h, nullptr,
|
||||
&exc_h_se);
|
||||
const double noise = std::hypot(exc_ref_se, exc_h_se);
|
||||
const bool hold_distance = convH && !(std::isfinite(noise) && exc_ref < exc_h - noise);
|
||||
if (settings.distance_at_header)
|
||||
logger.Info("Post-refine GEOM distance: free {:.3f} mm gives held-out excitation {:.3e} +- "
|
||||
"{:.1e} (positional {:.3e}); held at the header {:.3f} mm gives {:.3e} +- "
|
||||
"{:.1e} (positional {:.3e}); free minus held {:+.3e} against the noise {:.1e} "
|
||||
"=> {}; the free fit's formal sd(distance) {:.4f} %, corr(distance, cell "
|
||||
"a/b/c) {:.4f} {:.4f} {:.4f}",
|
||||
ds_f[0], exc_ref, exc_ref_se, pos_ref, dist0, exc_h, exc_h_se, pos_h,
|
||||
exc_ref - exc_h, noise,
|
||||
hold_distance ? "the rocking angles do not ask for it, the header distance is held"
|
||||
: convH ? "the free distance pays"
|
||||
: "the held fit did not converge, the free distance stands",
|
||||
100.0 * distance_corr[0], distance_corr[1], distance_corr[2],
|
||||
distance_corr[3]);
|
||||
if (hold_distance) {
|
||||
bm_f[0] = bm_h[0]; bm_f[1] = bm_h[1]; ds_f[0] = ds_h[0];
|
||||
for (int j = 0; j < 3; ++j) {
|
||||
rv_f[j] = rv_h[j]; q0_f[j] = q0_h[j]; q1_f[j] = q1_h[j]; q2_f[j] = q2_h[j];
|
||||
}
|
||||
cv_ref = cv_h; pos_ref = pos_h; exc_ref = exc_h;
|
||||
}
|
||||
|
||||
// A larger move is no longer refused for its size - but it is not taken as one solve
|
||||
// asserted it either. It is re-fitted as a WALK: each step seeded where the last one
|
||||
@@ -732,8 +757,8 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
for (int j = 0; j < 3; ++j) {
|
||||
rv_s[j] = rv_f[j]; q0_s[j] = q0_f[j]; q1_s[j] = q1_f[j]; q2_s[j] = q2_f[j];
|
||||
}
|
||||
if (!solve_joint(FIT, /*wide=*/false, /*fix_distance=*/false, bm_s, ds_s, rv_s,
|
||||
q0_s, q1_s, q2_s))
|
||||
if (!solve_joint(FIT, /*wide=*/false, hold_distance, bm_s, ds_s, rv_s, q0_s, q1_s,
|
||||
q2_s))
|
||||
break;
|
||||
double pos_s = 0.0, exc_s = 0.0;
|
||||
const double cv_s = joint_cost(VAL, bm_s, ds_s, rv_s, q0_s, q1_s, q2_s,
|
||||
@@ -809,7 +834,7 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
axv[j] = rv_f[j]; p0[j] = q0_f[j]; p1[j] = q1_f[j]; p2[j] = q2_f[j];
|
||||
}
|
||||
}
|
||||
solve_joint(ALL, !walked, /*fix_distance=*/false, beam, dist, axv, p0, p1, p2); // commit: re-fit on all data
|
||||
solve_joint(ALL, !walked, hold_distance, beam, dist, axv, p0, p1, p2); // commit: re-fit on all data
|
||||
ds_log[0] = dist[0]; bm_log[0] = beam[0]; bm_log[1] = beam[1];
|
||||
for (int j = 0; j < 3; ++j) { q1_log[j] = p1[j]; q2_log[j] = p2[j]; }
|
||||
refused = out_of_bounds(beam, p2);
|
||||
@@ -818,16 +843,10 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
for (int j = 0; j < 3; ++j) {
|
||||
axv[j] = ax0[j]; p0[j] = p0_0[j]; p1[j] = p1_0[j]; p2[j] = p2_0[j];
|
||||
}
|
||||
} else if (distance_undecided) {
|
||||
// The other arm, re-fitted on all the data the same way: its beam at the
|
||||
// header distance is what the canonical pass at that distance runs with.
|
||||
if (solve_joint(ALL, /*wide=*/true, /*fix_distance=*/true, bm_h, ds_h, rv_h, q0_h,
|
||||
q1_h, q2_h)
|
||||
&& out_of_bounds(bm_h, q2_h) == nullptr)
|
||||
result.held_arm_beam_px = std::array<double, 2>{bm_h[0], bm_h[1]};
|
||||
}
|
||||
}
|
||||
commit = (refused == nullptr);
|
||||
result.distance_held = commit && hold_distance;
|
||||
// Name which test refused it. Several different things reject here and the geometry that
|
||||
// comes out is the same in all of them, so a run that silently keeps its header geometry
|
||||
// says nothing about whether the fit was bad, the improvement too small, or the move too
|
||||
@@ -836,9 +855,8 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
// without the log.
|
||||
const std::string verdict = commit ? fmt::format("COMMIT ({} step{}{})", steps,
|
||||
steps == 1 ? "" : "s",
|
||||
result.held_arm_beam_px
|
||||
? ", the header distance carried as the other arm"
|
||||
: "")
|
||||
hold_distance ? ", distance held at the header"
|
||||
: "")
|
||||
: "reject (" + std::string(refused) + ")";
|
||||
// The cell the log names is the EFFECTIVE one - what the residual's B matrix builds
|
||||
// from the blocks - not the blocks themselves, whose unused components a high-symmetry
|
||||
@@ -871,7 +889,9 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations,
|
||||
// A move of more than one step is a walk, not a correction next to where it started,
|
||||
// and it is the one the old bound refused outright. It is committed here and ratified
|
||||
// where it can be: by re-indexing at it (see RunAllPasses).
|
||||
result.large_move = commit && beyond_a_step(dist, p1);
|
||||
// A held distance has nothing to walk: the walk is the distance finding the rest of an
|
||||
// error the rocking angles have just said they do not see.
|
||||
result.large_move = commit && !hold_distance && beyond_a_step(dist, p1);
|
||||
} else {
|
||||
logger.Info("Post-refine GEOM: only {} positional observations - the joint fit needs the "
|
||||
"spot positions as well as the rocking angles, so nothing is refined", n_obs);
|
||||
|
||||
@@ -49,14 +49,9 @@ struct PostRefineResult {
|
||||
// from. Such a move is not refused for its size - it is RATIFIED by re-integrating and re-indexing
|
||||
// at it, which a second lattice does not survive and a wrong header does (see RunAllPasses).
|
||||
bool large_move = false;
|
||||
// The same fit with the distance HELD at the header, where the free distance did not lower the
|
||||
// held-out residual below it by more than the noise of that residual: the beam centre that fit
|
||||
// committed, at the header distance. The free fit is what `geom`, `cell` and the *_after fields
|
||||
// carry, exactly as before; this is the other hypothesis, carried so that the canonical stage
|
||||
// can run at both and keep the one whose RE-INTEGRATED residual is lower (see RunAllPasses).
|
||||
// Absent where the free distance paid decisively, or where nothing was committed.
|
||||
std::optional<std::array<double, 2>> held_arm_beam_px;
|
||||
// Set by RunAllPasses where that comparison kept the header distance: the report names it.
|
||||
// What was committed is the fit with the distance HELD at the header - everything else as free as
|
||||
// ever - because freeing it did not lower the held-out excitation residual by more than that
|
||||
// residual's noise (see the joint fit in PostRefine.cpp). The report names it.
|
||||
bool distance_held = false;
|
||||
// The held-out residual at the geometry the fit STARTED from - what the integration this fit was
|
||||
// handed actually realises there, before the fit moves anything - and its standard error over the
|
||||
@@ -89,6 +84,11 @@ struct PostRefineSettings {
|
||||
// spot positions) and the detector distance + beam centre, in one
|
||||
// cross-validated fit. The only supported refinement mode.
|
||||
double excitation_weight = 1.0; // weight of the phi/excitation residual vs the positional one
|
||||
// The geometry handed in still has the file's distance, so "the header distance is right" is a
|
||||
// hypothesis the fit can hold and test (PostRefineResult::distance_held). Once a run has moved off
|
||||
// the header - a walk, see RunAllPasses - that hypothesis is already refuted and the distance is
|
||||
// simply free.
|
||||
bool distance_at_header = true;
|
||||
int min_events = 50;
|
||||
int num_threads = 1;
|
||||
// An independent measurement of the beam centre in pixels, where the run has one (rugnux's pre-scan
|
||||
|
||||
+13
-95
@@ -1867,6 +1867,7 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) {
|
||||
// from the same place (the beam-centre arbitration below).
|
||||
const DiffractionExperiment experiment_before_first_pass = experiment_;
|
||||
const int64_t max_spot_count_snapshot = experiment_.GetDatasetSettings().GetMaxSpotCount();
|
||||
header_distance_mm_ = experiment_.GetDetectorDistance_mm();
|
||||
prepass_detector_geometry_.reset();
|
||||
prepass_rotation_scale_.reset();
|
||||
prepass_result_.reset();
|
||||
@@ -2085,83 +2086,6 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) {
|
||||
.PoniRot1_rad(g[3]).PoniRot2_rad(g[4]);
|
||||
};
|
||||
|
||||
// Where the pre-pass could not tell whether its distance move paid (held_arm_beam_px: the
|
||||
// free distance lowered the fit's own held-out residual by less than that residual's noise),
|
||||
// the other hypothesis - the header distance, with the beam the same fit found there - is
|
||||
// measured too, and the two are judged on what they REALISE: the held-out residual a pass
|
||||
// re-integrated and re-indexed at that geometry measures there, the comparison the walk below
|
||||
// makes between its rounds. Where the free arm wins the run goes on at the post-refined
|
||||
// distance; where the header's wins, or the two cannot be told apart, the header's stands -
|
||||
// it is the hypothesis that asserts nothing, so a tie is its - and the distance walk below is
|
||||
// not entered: the data have just said they cannot place the distance. Measured on a sweep at
|
||||
// 820 mm reaching 2theta ~ 9 deg, where the fit's own gain never realises: the free arm's
|
||||
// re-integrated residual sat within 0.6 % of the header's at a noise of 8 %, and the cell it
|
||||
// carried was 1 % too large. On a sweep at 110 mm reaching 57 deg, a 0.14 % distance move the
|
||||
// pre-pass could not decide realised 3.7 % against a noise of 1.5 %, and the cell it carried
|
||||
// matched the reference to 0.04 %.
|
||||
//
|
||||
// Both arms are measured BEFORE the canonical pass, by passes that stop as soon as the
|
||||
// post-refinement has measured (postrefine_probe_only_). The residual is all the comparison
|
||||
// reads, and it is in hand before the scaling engine is built, so neither arm pays for a
|
||||
// merge, a space-group search or a set of correction surfaces - and only the arm that wins is
|
||||
// ever run as a canonical pass. Judging the arms on written passes instead cost the loser's
|
||||
// whole pass and, where the header won, a third pass that reproduced the loser's numbers
|
||||
// digit for digit.
|
||||
std::string distance_arm; // the arm decision, for the pass decision text
|
||||
bool distance_arm_kept_header = false;
|
||||
int arm_passes = 0;
|
||||
if (!cancelled_ && prepass_detector_geometry_ && pass1.post_refine
|
||||
&& pass1.post_refine->held_arm_beam_px) {
|
||||
const std::array<float, 5> free_geometry = current_geometry();
|
||||
std::array<float, 5> held_geometry = free_geometry;
|
||||
held_geometry[0] = static_cast<float>((*pass1.post_refine->held_arm_beam_px)[0]);
|
||||
held_geometry[1] = static_cast<float>((*pass1.post_refine->held_arm_beam_px)[1]);
|
||||
held_geometry[2] = header_geometry[2];
|
||||
logger.Info("Two-pass: the pre-pass could not tell whether moving the distance {:.3f} -> {:.3f} mm "
|
||||
"paid, so the header distance is re-integrated as well and the two are "
|
||||
"judged on the held-out residual each realises", header_geometry[2],
|
||||
free_geometry[2]);
|
||||
postrefine_probe_only_ = true;
|
||||
const auto free_arm = RunPipeline(observer, /*write_output=*/false, /*geometry_prepass=*/false);
|
||||
++arm_passes;
|
||||
set_geometry(held_geometry);
|
||||
const auto held_arm = RunPipeline(observer, /*write_output=*/false, /*geometry_prepass=*/false);
|
||||
++arm_passes;
|
||||
postrefine_probe_only_ = false;
|
||||
const double realised_free = free_arm.post_refine ? free_arm.post_refine->held_out_before : NAN;
|
||||
const double realised_held = held_arm.post_refine ? held_arm.post_refine->held_out_before : NAN;
|
||||
const double se_free = free_arm.post_refine ? free_arm.post_refine->held_out_before_se : NAN;
|
||||
const double se_held = held_arm.post_refine ? held_arm.post_refine->held_out_before_se : NAN;
|
||||
if (!cancelled_ && free_arm.post_refine && held_arm.post_refine
|
||||
&& HeldOutResidualFell(*held_arm.post_refine, *free_arm.post_refine)) {
|
||||
logger.Info("Two-pass: re-integrated at the post-refined distance {:.3f} mm the held-out "
|
||||
"residual is {:.3e} against {:.3e} at the header's {:.3f} mm (standard errors "
|
||||
"{:.1e} and {:.1e}) - the move is realised and stands", free_geometry[2],
|
||||
realised_free, realised_held, held_geometry[2], se_free, se_held);
|
||||
set_geometry(free_geometry);
|
||||
distance_arm = fmt::format(
|
||||
"the header distance {:.3f} mm was run as well and realised a held-out residual of "
|
||||
"{:.3e} against {:.3e} at the post-refined {:.3f} mm, so the post-refined distance "
|
||||
"stands", held_geometry[2], realised_held, realised_free, free_geometry[2]);
|
||||
} else if (!cancelled_) {
|
||||
logger.Info("Two-pass: re-integrated at the header distance {:.3f} mm the held-out residual "
|
||||
"is {:.3e}, which the post-refined {:.3f} mm does not beat ({:.3e}; standard "
|
||||
"errors {:.1e} and {:.1e}) - these data do not determine the distance, and the "
|
||||
"header's stands", held_geometry[2], realised_held, free_geometry[2],
|
||||
realised_free, se_held, se_free);
|
||||
distance_arm_kept_header = true;
|
||||
pass1.post_refine->distance_held = true;
|
||||
pass1.post_refine->beam_x_after_px = held_geometry[0];
|
||||
pass1.post_refine->beam_y_after_px = held_geometry[1];
|
||||
pass1.post_refine->distance_after_mm = held_geometry[2];
|
||||
distance_arm = fmt::format(
|
||||
"the header distance {:.3f} mm kept: re-integrated there the held-out residual is "
|
||||
"{:.3e}, which the post-refined {:.3f} mm does not beat ({:.3e}), so these data do "
|
||||
"not determine the distance", held_geometry[2], realised_held, free_geometry[2],
|
||||
realised_free);
|
||||
}
|
||||
}
|
||||
|
||||
logger.Info("Rotation two-pass geometry post-refinement: second pass (refined geometry, canonical) -> {}_*",
|
||||
base_prefix);
|
||||
auto pass2 = RunPipeline(observer, /*write_output=*/true, /*geometry_prepass=*/false);
|
||||
@@ -2210,7 +2134,7 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) {
|
||||
best_fit = *pass2.post_refine;
|
||||
walk_rounds = fmt::format("{:.3f} mm / {:.3e}", best_geometry[2], best_fit.held_out_before);
|
||||
}
|
||||
while (!cancelled_ && prepass_detector_geometry_ && !distance_arm_kept_header
|
||||
while (!cancelled_ && prepass_detector_geometry_
|
||||
&& pass2.post_refine && pass2.post_refine->detector_refined
|
||||
&& (pass2.post_refine->large_move || geometry_rounds > 0)) {
|
||||
if (geometry_rounds == MAX_GEOMETRY_ROUNDS) {
|
||||
@@ -2276,10 +2200,13 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) {
|
||||
pass1.post_refine->beam_x_after_px = best_geometry[0];
|
||||
pass1.post_refine->beam_y_after_px = best_geometry[1];
|
||||
pass1.post_refine->distance_after_mm = best_geometry[2];
|
||||
// A first pass that held the header distance and a walk that then left it: the run's
|
||||
// decision is the walk's.
|
||||
pass1.post_refine->distance_held = false;
|
||||
}
|
||||
pass2.post_refine = pass1.post_refine;
|
||||
pass2.pass_number = 2 + arm_passes + walk_passes;
|
||||
pass2.pass_count = 2 + arm_passes + walk_passes;
|
||||
pass2.pass_number = 2 + walk_passes;
|
||||
pass2.pass_count = 2 + walk_passes;
|
||||
if (geometry_rounds > 0)
|
||||
pass2.pass_decision = fmt::format(
|
||||
"post-refined geometry walked {} round{} and kept round {}, detector distance {:.3f} mm "
|
||||
@@ -2414,8 +2341,6 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) {
|
||||
pass2.pass_decision = (prepass_detector_geometry_ || prepass_rotation_scale_)
|
||||
? "post-refined geometry adopted"
|
||||
: "the post-refinement committed no geometry change, so this pass reproduces the first";
|
||||
if (!distance_arm.empty())
|
||||
pass2.pass_decision = distance_arm + "; " + pass2.pass_decision;
|
||||
pass2.geometry_not_converged = geometry_not_converged;
|
||||
return pass2;
|
||||
}
|
||||
@@ -5210,6 +5135,11 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
|
||||
prs.crystal_system = rot->search_result.system;
|
||||
prs.num_threads = static_cast<int>(config_.nthreads);
|
||||
prs.refine_geometry = true;
|
||||
// Only a pass still at the file's distance can hold it as a hypothesis; a pass
|
||||
// that has walked off it (the geometry walk in RunAllPasses) is past that question.
|
||||
prs.distance_at_header = !std::isfinite(header_distance_mm_)
|
||||
|| std::fabs(experiment_.GetDetectorDistance_mm() - header_distance_mm_)
|
||||
< 1e-3 * header_distance_mm_;
|
||||
// The centre the pre-scan measured, where the fit was precise enough to be used as
|
||||
// a centre at all (the same ceiling --estimate-beam-center adopts one on). It does
|
||||
// not move the geometry - it only tells the bound on step B's beam move where else
|
||||
@@ -5284,20 +5214,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
|
||||
// before the engine is built rather than beside it - the gathered observations and their
|
||||
// rocking events are gigabytes on a long axis, and beside the engine's own arrays they set
|
||||
// the pass's memory high-water mark.
|
||||
if (postrefine_probe_ && !geometry_prepass && is_rotation) {
|
||||
if (postrefine_probe_ && !geometry_prepass && is_rotation)
|
||||
post_refine_geometry();
|
||||
// A pass the distance arm runs to compare two geometries has now produced the only thing
|
||||
// the comparison reads - the held-out residual of the fit just measured. Stop here rather
|
||||
// than scale, search, fit surfaces and merge for a result nothing looks at.
|
||||
if (postrefine_probe_only_) {
|
||||
result.processing_time_s = std::chrono::duration<double>(
|
||||
std::chrono::steady_clock::now() - start_time).count();
|
||||
logger.Info("Distance-arm probe: {} images integrated and post-refined in {:.2f} s "
|
||||
"(the arm reads the held-out residual only, so this pass does not merge)",
|
||||
result.images_processed, result.processing_time_s);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
if (is_rotation) {
|
||||
if (experiment_.GetRefineRotationWedgeInScaling()
|
||||
|| rot_ss.GetRotationWedgeForScaling().has_value())
|
||||
|
||||
+4
-6
@@ -555,12 +555,10 @@ class Rugnux {
|
||||
// whether it has finished walking is a question only reflections integrated after the move can
|
||||
// answer; RunAllPasses turns this on for the passes that answer it.
|
||||
bool postrefine_probe_ = false;
|
||||
// Whether such a pass exists ONLY for that measurement. The distance arm reads the held-out
|
||||
// residual the probe returns and nothing else, so a pass run for it stops as soon as the probe
|
||||
// has measured - which is before the scaling engine is built. Everything below that point (the
|
||||
// merges, the space-group search, the correction surfaces, the reports) is work no comparison
|
||||
// looks at, and on a long sweep it is four fifths of the pass.
|
||||
bool postrefine_probe_only_ = false;
|
||||
// The file's detector distance, from before the first pass of the rotation two-pass: a pass whose
|
||||
// distance is still this one asks the post-refinement to test "the header distance is right"
|
||||
// (PostRefineSettings::distance_at_header); a pass that has walked off it does not.
|
||||
double header_distance_mm_ = std::numeric_limits<double>::quiet_NaN();
|
||||
|
||||
// Pre-scan: read a spread sample of frames and take three things off them - the shadow of the beam
|
||||
// stop and its holder, added to the pixel mask (config_.detect_beam_stop), the beam centre
|
||||
|
||||
Reference in new issue
Block a user