Stills rocking width: fit the window formula to the measured table, relatively

The prediction window's three-term width (constant + angular + bandwidth) was
fitted, unweighted in s^2, to its own equal-count shells, crowded at high
resolution. The finest shells decided it and the low-resolution window was an
extrapolation of them. Over 13-2 A the three terms are nearly collinear, so a
tiny input change could flip which one carries the width. Masking two hot
pixels took one run's constant term from 7.4e-5 to 0, so the window at 13 A
fell from 8.0e-5 to 2.7e-5 1/A, below the 4.5e-5 measured there. Its
lowest-shell balance factor then moved 0.97 -> 0.68 and R-free 0.1855 -> 0.192.

The formula is now fitted to the same equal-width-in-d* table the partiality
uses, each shell weighted by 1/s^4 so that every shell counts the same in
proportion. On the logged tables of six still-arm runs (rc184 and rc186
inputs) the fitted curve stays within ~10% of the table at every shell. The
13 A window no longer depends on which terms the fit keeps: 4.5e-5 vs 4.7e-5
on that run, 1.08e-4 on both inputs of another (it was 1.26e-4 vs 1.72e-4).
The terms on their own are still undetermined, which the code and docs now say.

The merge-stage balance also gives a shell it cannot balance its neighbour's
factor instead of leaving it as measured, which had left the outermost shell
at half the balanced shell inside it. The balanced table is then made
non-decreasing again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-14 22:45:36 +01:00
1 parent 595a94d4dc
commit 2f3df0f716
2 files changed
+108 -92

No files matched your search

+1 -1
View File
@@ -339,7 +339,7 @@ The partiality applied is fixed by the data type and scaling stage, not chosen f
2. **Unity** ($P_{ij}=1$): used for the scale-on-fulls refit (§10.6), where each observation is already a complete reflection.
3. **Fixed**: use the per-reflection partiality carried from prediction. Still/serial images are predicted with $P=1$, so a single-pass stills scale is effectively unity/fixed — which is exactly what `--simple-stills` keeps. By default the stills path instead **post-refines a physical partiality**: a small per-crystal orientation tilt $(\delta\psi_x,\delta\psi_y)$ about the two axes perpendicular to the beam is refined against the running merge, and every reflection's partiality is then recomputed analytically from the refined lattice through its excitation error $\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|$ and a Gaussian width $\sigma(d^*)$ — the run's rocking width, measured once on the intensity-weighted excitation errors and shared by every crystal. It is measured per group of frames of similar prediction window (four per shell), each against its own window, the shell taking the median of the groups whose curve is narrower than their window, so the frames with the widest windows do not set it everywhere; and it is measured in eight shells of equal width in $d^*$, made non-decreasing (pool-adjacent-violators, weighted by each shell's summed signal) and interpolated linearly in $d^*$ as a table. The three-term form $\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2$ — the reciprocal-lattice point's own radius, the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature — is still fitted, sets the prediction window (§8.2) and stands in for the table where fewer than three shells are measurable. The curve is centred on the run's **radial scale** of the reciprocal lattice — a wavelength or cell a little off, which the spot positions cannot see since they fix only $\lambda d^*$ — measured from where the curve's centre lies against $d^{*2}$ and applied to $\mathbf{q}$. Where the frames were predicted through the measured window (§8.2), the post-refinement uses that same curve rather than measuring it again; the curve is written to `_process.h5` (`/entry/MX/stillsRockingWidth*`, `stillsRadialScale`), so `--mode scale` merges with it too. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the *spatial* pattern of partialities — a degree of freedom the per-image scale $G$ does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with $G$. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with $G$ profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute $P$ → re-merge, repeated a few times; the refined partiality is adopted on every crystal it was refined on. The per-crystal fits of $G$ and of the tilt measure each residual against its expected variance at the model, background + counting + $(0.3\,I_\mathrm{model})^2$: a still's strong reflections scatter about the model by tens of per cent, and in counting sigmas alone every strong reflection would be an outlier and the scale would be set by the weak ones.
3. **Fixed**: use the per-reflection partiality carried from prediction. Still/serial images are predicted with $P=1$, so a single-pass stills scale is effectively unity/fixed — which is exactly what `--simple-stills` keeps. By default the stills path instead **post-refines a physical partiality**: a small per-crystal orientation tilt $(\delta\psi_x,\delta\psi_y)$ about the two axes perpendicular to the beam is refined against the running merge, and every reflection's partiality is then recomputed analytically from the refined lattice through its excitation error $\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|$ and a Gaussian width $\sigma(d^*)$ — the run's rocking width, measured once on the intensity-weighted excitation errors and shared by every crystal. It is measured per group of frames of similar prediction window (four per shell), each against its own window, the shell taking the median of the groups whose curve is narrower than their window, so the frames with the widest windows do not set it everywhere; and it is measured in eight shells of equal width in $d^*$, made non-decreasing (pool-adjacent-violators, weighted by each shell's summed signal) and interpolated linearly in $d^*$ as a table. The three-term form $\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2$ — the reciprocal-lattice point's own radius, the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature — is fitted to the same shells, each counting the same in proportion, and sets the prediction window (§8.2). Over the 13–2 Å the shells span the three terms are nearly collinear, so which of them carries the width is not determined and the fit reports one or two of them at zero on most runs; the curve itself is determined, and the curve is all the window and the partiality use. Before the post-refinement each shell of the table is rescaled so that the reflections predicted below half their peak and those above it put the same scale on the run's merge; a shell too sparse to say takes its neighbour's factor. The curve is centred on the run's **radial scale** of the reciprocal lattice — a wavelength or cell a little off, which the spot positions cannot see since they fix only $\lambda d^*$ — measured from where the curve's centre lies against $d^{*2}$ and applied to $\mathbf{q}$. Where the frames were predicted through the measured window (§8.2), the post-refinement uses that same curve rather than measuring it again; the curve is written to `_process.h5` (`/entry/MX/stillsRockingWidth*`, `stillsRadialScale`), so `--mode scale` merges with it too. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the *spatial* pattern of partialities — a degree of freedom the per-image scale $G$ does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with $G$. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with $G$ profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute $P$ → re-merge, repeated a few times; the refined partiality is adopted on every crystal it was refined on. The per-crystal fits of $G$ and of the tilt measure each residual against its expected variance at the model, background + counting + $(0.3\,I_\mathrm{model})^2$: a still's strong reflections scatter about the model by tens of per cent, and in counting sigmas alone every strong reflection would be an outlier and the scale would be set by the weak ones.
Reflections below a minimum partiality are left out of the merge (`--min-partiality`, default 0.02 on rotation and 0.5 on stills): the empirical rocking curve of a still has a floor far out on its tail, from mis-oriented crystals, and dividing such an observation by a small model $p$ makes it biased, not merely noisy. On stills this holds for the merges the scale and the partiality are refined against. The final merge, which has the fitted error model, takes every observation: the model's partiality term $c\,((1-p)/p)\,\langle I\rangle$ (§10.5) weights a low-partiality observation by what it is worth. A reflection - each Bijvoet hand under `-A` - is reported only if at least one of its observations reaches $p \ge 0.2$; one seen only far out on its rocking curve rests on the partiality model alone. The error model is fitted on the observations at $p \ge 0.2$.
@@ -110,6 +110,86 @@ namespace {
return 3.0 * u / (a * a);
}
// A wider curve at finer resolution is the physics (every term grows with d*), so the width table is made
// non-decreasing: pool adjacent violators, weighted by w.
void PoolNonDecreasing(std::vector<double> &s, const std::vector<double> &w) {
std::vector<double> block_s, block_w;
std::vector<size_t> block_n;
for (size_t i = 0; i < s.size(); ++i) {
block_s.push_back(s[i]);
block_w.push_back(w[i]);
block_n.push_back(1);
while (block_s.size() > 1 && block_s[block_s.size() - 2] > block_s.back()) {
const size_t j = block_s.size() - 2;
block_s[j] = (block_s[j] * block_w[j] + block_s.back() * block_w.back()) / (block_w[j] + block_w.back());
block_w[j] += block_w.back();
block_n[j] += block_n.back();
block_s.pop_back();
block_w.pop_back();
block_n.pop_back();
}
}
s.clear();
for (size_t j = 0; j < block_s.size(); ++j)
s.insert(s.end(), block_n[j], block_s[j]);
}
// The three-term formula s^2 = gamma0^2 + (gamma_e d*)^2 + (bw lambda d*^2 / 2)^2 fitted to the width
// table, which the prediction window uses. Least squares for s^2 = A + B d*^2 + C d*^4 over the shells,
// each weighted by 1 / s^4 so that all count the same in proportion: unweighted, the finest shells (the
// largest s^2) decided the fit, and the low-resolution end of the curve, where the window matters most,
// was their extrapolation - on one run it fell from 7.4e-5 to 2.7e-5 1/A, under the width measured there,
// when two pixels were masked. Every term is a variance, so a negative one is unphysical: of the fits on
// each subset of the three terms, the best with none negative is kept. Over 13-2 A the three terms are
// nearly collinear, so which of them carries the width is not determined - one or two come out at zero
// on most runs - but the curve is, and the curve is all the window and the partiality use.
bool FitWidthFormula(StillsRunWidth &width, double wavelength) {
const std::vector<double> &sx = width.table_dstar, &sy = width.table_s;
std::array<double, 3> best{};
double best_residual = INFINITY;
for (int mask = 1; mask < 8; ++mask) {
std::vector<int> terms;
for (int t = 0; t < 3; ++t)
if (mask & (1 << t))
terms.push_back(t);
const int m = static_cast<int>(terms.size());
Eigen::MatrixXd M = Eigen::MatrixXd::Zero(m, m);
Eigen::VectorXd v = Eigen::VectorXd::Zero(m);
for (size_t k = 0; k < sx.size(); ++k) {
const double x = sx[k] * sx[k], y = sy[k] * sy[k], w = 1.0 / (y * y);
for (int i = 0; i < m; ++i) {
const double fi = std::pow(x, terms[i]);
v(i) += w * fi * y;
for (int j = 0; j < m; ++j)
M(i, j) += w * fi * std::pow(x, terms[j]);
}
}
const Eigen::VectorXd c = M.ldlt().solve(v);
if (!c.allFinite() || (c.array() < 0.0).any())
continue;
std::array<double, 3> coef{};
for (int i = 0; i < m; ++i)
coef[terms[i]] = c(i);
double residual = 0.0;
for (size_t k = 0; k < sx.size(); ++k) {
const double x = sx[k] * sx[k], y = sy[k] * sy[k];
const double r = (y - coef[0] - coef[1] * x - coef[2] * x * x) / y;
residual += r * r;
}
if (residual < best_residual) {
best_residual = residual;
best = coef;
}
}
if (!std::isfinite(best_residual) || !(best[0] + best[1] + best[2] > 0.0))
return false;
// The d*^4 term is a bandwidth: its radial smear bw |q_z|, with |q_z| = lambda d*^2 / 2 on the sphere.
width.gamma0 = std::sqrt(best[0]);
width.gamma_e = std::sqrt(best[1]);
width.bandwidth_sigma = 2.0 * std::sqrt(best[2]) / wavelength;
return true;
}
// Robust per-crystal scale G (linear in G given the model coefficients), the objective of
// ScaleOnTheFly::SolveScaleIRLS: minimise sum Cauchy_k( (G*coeff - Iobs) / sqrt(var(G*coeff)) ) over G >= 0,
// the variances re-evaluated at each step's G.
@@ -689,30 +769,12 @@ std::optional<StillsPartialityRefine::RunWidth> StillsPartialityRefine::MeasureR
return m % 2 ? widths[m / 2] : 0.5 * (widths[m / 2 - 1] + widths[m / 2]);
};
// For the three-term formula (the prediction window): shells of equal count in d*^2.
std::vector<double> sx, sy;
for (size_t k = 0; k < N_SHELLS; ++k) {
const size_t lo = k * pts.size() / N_SHELLS, hi = (k + 1) * pts.size() / N_SHELLS;
const auto y = shell_width(lo, hi);
if (!y)
continue;
double sumx = 0.0;
for (size_t i = lo; i < hi; ++i)
sumx += pts[i].x;
const double x = sumx / static_cast<double>(hi - lo);
sx.push_back(x);
sy.push_back(*y);
logger.Debug("Stills rocking width at d = {:.2f} A: {:.2e} 1/A ({:.3f} deg)",
1.0 / std::sqrt(x), std::sqrt(*y), std::sqrt(*y / x) * kRadToDeg);
}
// For the partiality: shells of equal width in d*, kept as a table. Equal-count shells crowd where
// the reflections are - on a run predicting ~1000 reflections a frame, all eight sat at 3.8-1.8 A, so
// the low-resolution width was the formula's extrapolation, and the three-term fit flipped between
// forms with the number of frames (to a pure bandwidth at 50k frames). Measured where it is used and
// interpolated, the width needs no form. A wider curve at finer resolution is the physics (every term
// grows with d*), so the table is made non-decreasing (pool adjacent violators, weighted by the
// shells' summed signal).
// Shells of equal width in d*, kept as a table for the partiality and fitted for the formula. Equal-count
// shells crowd where the reflections are - on a run predicting ~1000 reflections a frame, all eight sat
// at 3.8-1.8 A, so the low-resolution width was the formula's extrapolation, and the three-term fit
// flipped between forms with the number of frames (to a pure bandwidth at 50k frames). A wider curve at
// finer resolution is the physics (every term grows with d*), so the table is made non-decreasing (pool
// adjacent violators, weighted by the shells' summed signal).
std::vector<double> table_dstar, table_s, table_weight;
{
const double lo_d = std::sqrt(pts.front().x), hi_d = std::sqrt(pts.back().x);
@@ -736,82 +798,23 @@ std::optional<StillsPartialityRefine::RunWidth> StillsPartialityRefine::MeasureR
}
lo = hi;
}
std::vector<double> block_s, block_w;
std::vector<size_t> block_n;
for (size_t i = 0; i < table_s.size(); ++i) {
block_s.push_back(table_s[i]);
block_w.push_back(table_weight[i]);
block_n.push_back(1);
while (block_s.size() > 1 && block_s[block_s.size() - 2] > block_s.back()) {
const size_t j = block_s.size() - 2;
block_s[j] = (block_s[j] * block_w[j] + block_s.back() * block_w.back()) / (block_w[j] + block_w.back());
block_w[j] += block_w.back();
block_n[j] += block_n.back();
block_s.pop_back();
block_w.pop_back();
block_n.pop_back();
}
}
table_s.clear();
for (size_t j = 0; j < block_s.size(); ++j)
table_s.insert(table_s.end(), block_n[j], block_s[j]);
if (table_dstar.size() < 3) {
table_dstar.clear();
table_s.clear();
}
}
if (sx.size() < 3)
if (table_dstar.size() < 3)
return std::nullopt;
PoolNonDecreasing(table_s, table_weight);
for (size_t i = 0; i < table_dstar.size(); ++i)
logger.Debug("Stills rocking width at d = {:.2f} A: {:.2e} 1/A ({:.3f} deg)", 1.0 / table_dstar[i],
table_s[i], table_s[i] * table_dstar[i] * kRadToDeg);
// Least squares for s^2 = A + B d*^2 + C d*^4 over the shells. Every term is a variance, so a negative
// one is unphysical: of the fits on each subset of the three terms, the best with none negative is kept.
std::array<double, 3> best{};
double best_residual = INFINITY;
for (int mask = 1; mask < 8; ++mask) {
std::vector<int> terms;
for (int t = 0; t < 3; ++t)
if (mask & (1 << t))
terms.push_back(t);
const int m = static_cast<int>(terms.size());
Eigen::MatrixXd M = Eigen::MatrixXd::Zero(m, m);
Eigen::VectorXd v = Eigen::VectorXd::Zero(m);
for (size_t k = 0; k < sx.size(); ++k)
for (int i = 0; i < m; ++i) {
const double fi = std::pow(sx[k], terms[i]);
v(i) += fi * sy[k];
for (int j = 0; j < m; ++j)
M(i, j) += fi * std::pow(sx[k], terms[j]);
}
const Eigen::VectorXd c = M.ldlt().solve(v);
if (!c.allFinite() || (c.array() < 0.0).any())
continue;
std::array<double, 3> coef{};
for (int i = 0; i < m; ++i)
coef[terms[i]] = c(i);
double residual = 0.0;
for (size_t k = 0; k < sx.size(); ++k) {
const double r = sy[k] - coef[0] - coef[1] * sx[k] - coef[2] * sx[k] * sx[k];
residual += r * r;
}
if (residual < best_residual) {
best_residual = residual;
best = coef;
}
}
if (!std::isfinite(best_residual) || !(best[0] + best[1] + best[2] > 0.0))
RunWidth width{.radial_scale = radial_scale, .table_dstar = table_dstar, .table_s = table_s};
if (!FitWidthFormula(width, experiment_.GetWavelength_A()))
return std::nullopt;
// The d*^4 term is a bandwidth: its radial smear bw |q_z|, with |q_z| = lambda d*^2 / 2 on the sphere.
const RunWidth width{.gamma0 = std::sqrt(best[0]), .gamma_e = std::sqrt(best[1]),
.bandwidth_sigma = 2.0 * std::sqrt(best[2]) / experiment_.GetWavelength_A(),
.radial_scale = radial_scale, .table_dstar = table_dstar, .table_s = table_s};
logger.Info("Stills rocking width measured: {:.2e} 1/A + {:.3f} deg angular + {:.4f} bandwidth (FWHM)",
width.gamma0, width.gamma_e * kRadToDeg, width.bandwidth_sigma * 2.3548);
std::string table;
for (size_t i = 0; i < table_dstar.size(); ++i)
table += fmt::format(" {:.2f} A {:.2e}", 1.0 / table_dstar[i], table_s[i]);
logger.Info("Stills rocking width for the partiality, against resolution (1/A):{}",
table.empty() ? " none, the formula is used" : table);
logger.Info("Stills rocking width for the partiality, against resolution (1/A):{}", table);
return width;
}
@@ -898,6 +901,14 @@ StillsPartialityRefine::RunWidth StillsPartialityRefine::BalanceRunWidth(
}
factor[k] = std::sqrt(lo * hi);
});
// A shell the partials cannot balance (too few, or a factor outside the range) takes its neighbour's
// factor: left as measured, the outermost shell sat at half the balanced one inside it (x2.2 on one run).
for (size_t k = 1; k < factor.size(); ++k)
if (std::isnan(factor[k]))
factor[k] = factor[k - 1];
for (size_t k = factor.size() - 1; k > 0; --k)
if (std::isnan(factor[k - 1]))
factor[k - 1] = factor[k];
RunWidth balanced = width;
std::string table;
for (size_t k = 0; k < shells.size(); ++k) {
@@ -908,6 +919,11 @@ StillsPartialityRefine::RunWidth StillsPartialityRefine::BalanceRunWidth(
}
Logger("StillsPartialityRefine").Info("Stills rocking width balanced on the partials, against resolution:{}",
table.empty() ? " unchanged" : table);
// Balanced shell by shell, the table can lose the order the physics gives it.
std::vector<double> count(shells.size());
for (size_t k = 0; k < shells.size(); ++k)
count[k] = static_cast<double>(shells[k].size());
PoolNonDecreasing(balanced.table_s, count);
return balanced;
}