diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 798b81943..90d4f25cf 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -339,7 +339,7 @@ The partiality applied is fixed by the data type and scaling stage, not chosen f 2. **Unity** ($P_{ij}=1$): used for the scale-on-fulls refit (§10.6), where each observation is already a complete reflection. -3. **Fixed**: use the per-reflection partiality carried from prediction. Still/serial images are predicted with $P=1$, so a single-pass stills scale is effectively unity/fixed — which is exactly what `--simple-stills` keeps. By default the stills path instead **post-refines a physical partiality**: a small per-crystal orientation tilt $(\delta\psi_x,\delta\psi_y)$ about the two axes perpendicular to the beam is refined against the running merge, and every reflection's partiality is then recomputed analytically from the refined lattice through its excitation error $\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|$ and a Gaussian width $\sigma(d^*)$ — the run's rocking width, measured once on the intensity-weighted excitation errors and shared by every crystal. It is measured per group of frames of similar prediction window (four per shell), each against its own window, the shell taking the median of the groups whose curve is narrower than their window, so the frames with the widest windows do not set it everywhere; and it is measured in eight shells of equal width in $d^*$, made non-decreasing (pool-adjacent-violators, weighted by each shell's summed signal) and interpolated linearly in $d^*$ as a table. The three-term form $\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2$ — the reciprocal-lattice point's own radius, the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature — is still fitted, sets the prediction window (§8.2) and stands in for the table where fewer than three shells are measurable. The curve is centred on the run's **radial scale** of the reciprocal lattice — a wavelength or cell a little off, which the spot positions cannot see since they fix only $\lambda d^*$ — measured from where the curve's centre lies against $d^{*2}$ and applied to $\mathbf{q}$. Where the frames were predicted through the measured window (§8.2), the post-refinement uses that same curve rather than measuring it again; the curve is written to `_process.h5` (`/entry/MX/stillsRockingWidth*`, `stillsRadialScale`), so `--mode scale` merges with it too. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the *spatial* pattern of partialities — a degree of freedom the per-image scale $G$ does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with $G$. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with $G$ profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute $P$ → re-merge, repeated a few times; the refined partiality is adopted on every crystal it was refined on. The per-crystal fits of $G$ and of the tilt measure each residual against its expected variance at the model, background + counting + $(0.3\,I_\mathrm{model})^2$: a still's strong reflections scatter about the model by tens of per cent, and in counting sigmas alone every strong reflection would be an outlier and the scale would be set by the weak ones. +3. **Fixed**: use the per-reflection partiality carried from prediction. Still/serial images are predicted with $P=1$, so a single-pass stills scale is effectively unity/fixed — which is exactly what `--simple-stills` keeps. By default the stills path instead **post-refines a physical partiality**: a small per-crystal orientation tilt $(\delta\psi_x,\delta\psi_y)$ about the two axes perpendicular to the beam is refined against the running merge, and every reflection's partiality is then recomputed analytically from the refined lattice through its excitation error $\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|$ and a Gaussian width $\sigma(d^*)$ — the run's rocking width, measured once on the intensity-weighted excitation errors and shared by every crystal. It is measured per group of frames of similar prediction window (four per shell), each against its own window, the shell taking the median of the groups whose curve is narrower than their window, so the frames with the widest windows do not set it everywhere; and it is measured in eight shells of equal width in $d^*$, made non-decreasing (pool-adjacent-violators, weighted by each shell's summed signal) and interpolated linearly in $d^*$ as a table. The three-term form $\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2$ — the reciprocal-lattice point's own radius, the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature — is fitted to the same shells, each counting the same in proportion, and sets the prediction window (§8.2). Over the 13–2 Å the shells span the three terms are nearly collinear, so which of them carries the width is not determined and the fit reports one or two of them at zero on most runs; the curve itself is determined, and the curve is all the window and the partiality use. Before the post-refinement each shell of the table is rescaled so that the reflections predicted below half their peak and those above it put the same scale on the run's merge; a shell too sparse to say takes its neighbour's factor. The curve is centred on the run's **radial scale** of the reciprocal lattice — a wavelength or cell a little off, which the spot positions cannot see since they fix only $\lambda d^*$ — measured from where the curve's centre lies against $d^{*2}$ and applied to $\mathbf{q}$. Where the frames were predicted through the measured window (§8.2), the post-refinement uses that same curve rather than measuring it again; the curve is written to `_process.h5` (`/entry/MX/stillsRockingWidth*`, `stillsRadialScale`), so `--mode scale` merges with it too. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the *spatial* pattern of partialities — a degree of freedom the per-image scale $G$ does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with $G$. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with $G$ profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute $P$ → re-merge, repeated a few times; the refined partiality is adopted on every crystal it was refined on. The per-crystal fits of $G$ and of the tilt measure each residual against its expected variance at the model, background + counting + $(0.3\,I_\mathrm{model})^2$: a still's strong reflections scatter about the model by tens of per cent, and in counting sigmas alone every strong reflection would be an outlier and the scale would be set by the weak ones. Reflections below a minimum partiality are left out of the merge (`--min-partiality`, default 0.02 on rotation and 0.5 on stills): the empirical rocking curve of a still has a floor far out on its tail, from mis-oriented crystals, and dividing such an observation by a small model $p$ makes it biased, not merely noisy. On stills this holds for the merges the scale and the partiality are refined against. The final merge, which has the fitted error model, takes every observation: the model's partiality term $c\,((1-p)/p)\,\langle I\rangle$ (§10.5) weights a low-partiality observation by what it is worth. A reflection - each Bijvoet hand under `-A` - is reported only if at least one of its observations reaches $p \ge 0.2$; one seen only far out on its rocking curve rests on the partiality model alone. The error model is fitted on the observations at $p \ge 0.2$. diff --git a/image_analysis/scale_merge/StillsPartialityRefine.cpp b/image_analysis/scale_merge/StillsPartialityRefine.cpp index def049fec..5d02abaf5 100644 --- a/image_analysis/scale_merge/StillsPartialityRefine.cpp +++ b/image_analysis/scale_merge/StillsPartialityRefine.cpp @@ -110,6 +110,86 @@ namespace { return 3.0 * u / (a * a); } + // A wider curve at finer resolution is the physics (every term grows with d*), so the width table is made + // non-decreasing: pool adjacent violators, weighted by w. + void PoolNonDecreasing(std::vector &s, const std::vector &w) { + std::vector block_s, block_w; + std::vector block_n; + for (size_t i = 0; i < s.size(); ++i) { + block_s.push_back(s[i]); + block_w.push_back(w[i]); + block_n.push_back(1); + while (block_s.size() > 1 && block_s[block_s.size() - 2] > block_s.back()) { + const size_t j = block_s.size() - 2; + block_s[j] = (block_s[j] * block_w[j] + block_s.back() * block_w.back()) / (block_w[j] + block_w.back()); + block_w[j] += block_w.back(); + block_n[j] += block_n.back(); + block_s.pop_back(); + block_w.pop_back(); + block_n.pop_back(); + } + } + s.clear(); + for (size_t j = 0; j < block_s.size(); ++j) + s.insert(s.end(), block_n[j], block_s[j]); + } + + // The three-term formula s^2 = gamma0^2 + (gamma_e d*)^2 + (bw lambda d*^2 / 2)^2 fitted to the width + // table, which the prediction window uses. Least squares for s^2 = A + B d*^2 + C d*^4 over the shells, + // each weighted by 1 / s^4 so that all count the same in proportion: unweighted, the finest shells (the + // largest s^2) decided the fit, and the low-resolution end of the curve, where the window matters most, + // was their extrapolation - on one run it fell from 7.4e-5 to 2.7e-5 1/A, under the width measured there, + // when two pixels were masked. Every term is a variance, so a negative one is unphysical: of the fits on + // each subset of the three terms, the best with none negative is kept. Over 13-2 A the three terms are + // nearly collinear, so which of them carries the width is not determined - one or two come out at zero + // on most runs - but the curve is, and the curve is all the window and the partiality use. + bool FitWidthFormula(StillsRunWidth &width, double wavelength) { + const std::vector &sx = width.table_dstar, &sy = width.table_s; + std::array best{}; + double best_residual = INFINITY; + for (int mask = 1; mask < 8; ++mask) { + std::vector terms; + for (int t = 0; t < 3; ++t) + if (mask & (1 << t)) + terms.push_back(t); + const int m = static_cast(terms.size()); + Eigen::MatrixXd M = Eigen::MatrixXd::Zero(m, m); + Eigen::VectorXd v = Eigen::VectorXd::Zero(m); + for (size_t k = 0; k < sx.size(); ++k) { + const double x = sx[k] * sx[k], y = sy[k] * sy[k], w = 1.0 / (y * y); + for (int i = 0; i < m; ++i) { + const double fi = std::pow(x, terms[i]); + v(i) += w * fi * y; + for (int j = 0; j < m; ++j) + M(i, j) += w * fi * std::pow(x, terms[j]); + } + } + const Eigen::VectorXd c = M.ldlt().solve(v); + if (!c.allFinite() || (c.array() < 0.0).any()) + continue; + std::array coef{}; + for (int i = 0; i < m; ++i) + coef[terms[i]] = c(i); + double residual = 0.0; + for (size_t k = 0; k < sx.size(); ++k) { + const double x = sx[k] * sx[k], y = sy[k] * sy[k]; + const double r = (y - coef[0] - coef[1] * x - coef[2] * x * x) / y; + residual += r * r; + } + if (residual < best_residual) { + best_residual = residual; + best = coef; + } + } + if (!std::isfinite(best_residual) || !(best[0] + best[1] + best[2] > 0.0)) + return false; + // The d*^4 term is a bandwidth: its radial smear bw |q_z|, with |q_z| = lambda d*^2 / 2 on the sphere. + width.gamma0 = std::sqrt(best[0]); + width.gamma_e = std::sqrt(best[1]); + width.bandwidth_sigma = 2.0 * std::sqrt(best[2]) / wavelength; + return true; + } + // Robust per-crystal scale G (linear in G given the model coefficients), the objective of // ScaleOnTheFly::SolveScaleIRLS: minimise sum Cauchy_k( (G*coeff - Iobs) / sqrt(var(G*coeff)) ) over G >= 0, // the variances re-evaluated at each step's G. @@ -689,30 +769,12 @@ std::optional StillsPartialityRefine::MeasureR return m % 2 ? widths[m / 2] : 0.5 * (widths[m / 2 - 1] + widths[m / 2]); }; - // For the three-term formula (the prediction window): shells of equal count in d*^2. - std::vector sx, sy; - for (size_t k = 0; k < N_SHELLS; ++k) { - const size_t lo = k * pts.size() / N_SHELLS, hi = (k + 1) * pts.size() / N_SHELLS; - const auto y = shell_width(lo, hi); - if (!y) - continue; - double sumx = 0.0; - for (size_t i = lo; i < hi; ++i) - sumx += pts[i].x; - const double x = sumx / static_cast(hi - lo); - sx.push_back(x); - sy.push_back(*y); - logger.Debug("Stills rocking width at d = {:.2f} A: {:.2e} 1/A ({:.3f} deg)", - 1.0 / std::sqrt(x), std::sqrt(*y), std::sqrt(*y / x) * kRadToDeg); - } - - // For the partiality: shells of equal width in d*, kept as a table. Equal-count shells crowd where - // the reflections are - on a run predicting ~1000 reflections a frame, all eight sat at 3.8-1.8 A, so - // the low-resolution width was the formula's extrapolation, and the three-term fit flipped between - // forms with the number of frames (to a pure bandwidth at 50k frames). Measured where it is used and - // interpolated, the width needs no form. A wider curve at finer resolution is the physics (every term - // grows with d*), so the table is made non-decreasing (pool adjacent violators, weighted by the - // shells' summed signal). + // Shells of equal width in d*, kept as a table for the partiality and fitted for the formula. Equal-count + // shells crowd where the reflections are - on a run predicting ~1000 reflections a frame, all eight sat + // at 3.8-1.8 A, so the low-resolution width was the formula's extrapolation, and the three-term fit + // flipped between forms with the number of frames (to a pure bandwidth at 50k frames). A wider curve at + // finer resolution is the physics (every term grows with d*), so the table is made non-decreasing (pool + // adjacent violators, weighted by the shells' summed signal). std::vector table_dstar, table_s, table_weight; { const double lo_d = std::sqrt(pts.front().x), hi_d = std::sqrt(pts.back().x); @@ -736,82 +798,23 @@ std::optional StillsPartialityRefine::MeasureR } lo = hi; } - std::vector block_s, block_w; - std::vector block_n; - for (size_t i = 0; i < table_s.size(); ++i) { - block_s.push_back(table_s[i]); - block_w.push_back(table_weight[i]); - block_n.push_back(1); - while (block_s.size() > 1 && block_s[block_s.size() - 2] > block_s.back()) { - const size_t j = block_s.size() - 2; - block_s[j] = (block_s[j] * block_w[j] + block_s.back() * block_w.back()) / (block_w[j] + block_w.back()); - block_w[j] += block_w.back(); - block_n[j] += block_n.back(); - block_s.pop_back(); - block_w.pop_back(); - block_n.pop_back(); - } - } - table_s.clear(); - for (size_t j = 0; j < block_s.size(); ++j) - table_s.insert(table_s.end(), block_n[j], block_s[j]); - if (table_dstar.size() < 3) { - table_dstar.clear(); - table_s.clear(); - } } - - if (sx.size() < 3) + if (table_dstar.size() < 3) return std::nullopt; + PoolNonDecreasing(table_s, table_weight); + for (size_t i = 0; i < table_dstar.size(); ++i) + logger.Debug("Stills rocking width at d = {:.2f} A: {:.2e} 1/A ({:.3f} deg)", 1.0 / table_dstar[i], + table_s[i], table_s[i] * table_dstar[i] * kRadToDeg); - // Least squares for s^2 = A + B d*^2 + C d*^4 over the shells. Every term is a variance, so a negative - // one is unphysical: of the fits on each subset of the three terms, the best with none negative is kept. - std::array best{}; - double best_residual = INFINITY; - for (int mask = 1; mask < 8; ++mask) { - std::vector terms; - for (int t = 0; t < 3; ++t) - if (mask & (1 << t)) - terms.push_back(t); - const int m = static_cast(terms.size()); - Eigen::MatrixXd M = Eigen::MatrixXd::Zero(m, m); - Eigen::VectorXd v = Eigen::VectorXd::Zero(m); - for (size_t k = 0; k < sx.size(); ++k) - for (int i = 0; i < m; ++i) { - const double fi = std::pow(sx[k], terms[i]); - v(i) += fi * sy[k]; - for (int j = 0; j < m; ++j) - M(i, j) += fi * std::pow(sx[k], terms[j]); - } - const Eigen::VectorXd c = M.ldlt().solve(v); - if (!c.allFinite() || (c.array() < 0.0).any()) - continue; - std::array coef{}; - for (int i = 0; i < m; ++i) - coef[terms[i]] = c(i); - double residual = 0.0; - for (size_t k = 0; k < sx.size(); ++k) { - const double r = sy[k] - coef[0] - coef[1] * sx[k] - coef[2] * sx[k] * sx[k]; - residual += r * r; - } - if (residual < best_residual) { - best_residual = residual; - best = coef; - } - } - if (!std::isfinite(best_residual) || !(best[0] + best[1] + best[2] > 0.0)) + RunWidth width{.radial_scale = radial_scale, .table_dstar = table_dstar, .table_s = table_s}; + if (!FitWidthFormula(width, experiment_.GetWavelength_A())) return std::nullopt; - // The d*^4 term is a bandwidth: its radial smear bw |q_z|, with |q_z| = lambda d*^2 / 2 on the sphere. - const RunWidth width{.gamma0 = std::sqrt(best[0]), .gamma_e = std::sqrt(best[1]), - .bandwidth_sigma = 2.0 * std::sqrt(best[2]) / experiment_.GetWavelength_A(), - .radial_scale = radial_scale, .table_dstar = table_dstar, .table_s = table_s}; logger.Info("Stills rocking width measured: {:.2e} 1/A + {:.3f} deg angular + {:.4f} bandwidth (FWHM)", width.gamma0, width.gamma_e * kRadToDeg, width.bandwidth_sigma * 2.3548); std::string table; for (size_t i = 0; i < table_dstar.size(); ++i) table += fmt::format(" {:.2f} A {:.2e}", 1.0 / table_dstar[i], table_s[i]); - logger.Info("Stills rocking width for the partiality, against resolution (1/A):{}", - table.empty() ? " none, the formula is used" : table); + logger.Info("Stills rocking width for the partiality, against resolution (1/A):{}", table); return width; } @@ -898,6 +901,14 @@ StillsPartialityRefine::RunWidth StillsPartialityRefine::BalanceRunWidth( } factor[k] = std::sqrt(lo * hi); }); + // A shell the partials cannot balance (too few, or a factor outside the range) takes its neighbour's + // factor: left as measured, the outermost shell sat at half the balanced one inside it (x2.2 on one run). + for (size_t k = 1; k < factor.size(); ++k) + if (std::isnan(factor[k])) + factor[k] = factor[k - 1]; + for (size_t k = factor.size() - 1; k > 0; --k) + if (std::isnan(factor[k - 1])) + factor[k - 1] = factor[k]; RunWidth balanced = width; std::string table; for (size_t k = 0; k < shells.size(); ++k) { @@ -908,6 +919,11 @@ StillsPartialityRefine::RunWidth StillsPartialityRefine::BalanceRunWidth( } Logger("StillsPartialityRefine").Info("Stills rocking width balanced on the partials, against resolution:{}", table.empty() ? " unchanged" : table); + // Balanced shell by shell, the table can lose the order the physics gives it. + std::vector count(shells.size()); + for (size_t k = 0; k < shells.size(); ++k) + count[k] = static_cast(shells[k].size()); + PoolNonDecreasing(balanced.table_s, count); return balanced; }