diff --git a/common/BraggIntegrationSettings.h b/common/BraggIntegrationSettings.h index 24a8f5d2..4886611a 100644 --- a/common/BraggIntegrationSettings.h +++ b/common/BraggIntegrationSettings.h @@ -57,8 +57,8 @@ class BraggIntegrationSettings { // pixel reads). Measured empty-aperture bias over 9 bands on 3 crystals: 4.33 -> 0.79 counts mean // |bias|, scatter unchanged. // - // Unset means AUTO, which is the default: apply it per image where that image's peak-excluded ice - // score says a SMOOTH powder ring is present, and not otherwise. The correction models the + // Unset means AUTO: apply it per image where that image's peak-excluded ice score says a SMOOTH + // powder ring is present, and not otherwise. NOT the default - see below. The correction models the // background as a function of radius alone, so it helps exactly where that is true and not // elsewhere. Measured against a fixed external model, band-versus-decoy-band: on a crystal with // pure smooth ice it removes 43% of the ice bands' excess amplitude, with the effect 7x stronger @@ -68,7 +68,14 @@ class BraggIntegrationSettings { // correction is gated on the smooth one. Auto only engages where a peak-excluded score exists // (adaptive spot finding); the plain profile carries the Bragg peaks and cannot support a // threshold, so without it auto stays off. - std::optional bkg_radial_correction; + // + // OFF by default. Auto targets correctly - over the rotation battery it fires on ten crystals and + // every one of them is ice-positive - but it costs 1.35x the wall clock, and on the merge + // statistics it is the familiar sign-mixed trade rather than a win: high-shell CC1/2 worse on + // three of the four crystals that move materially. The case for it rests on agreement with an + // external model, which is the better arbiter but a narrower one, so it stays opt-in until that + // is settled on its own evidence. + std::optional bkg_radial_correction = false; // Half-width of the hkl cube the predictor walks: every reflection with |h|,|k|,|l| <= this is // tested against the Ewald sphere, and nothing outside it can ever be predicted. An axis is // truncated once a/d_min exceeds this, and the GPU cost is the cube (2n+1)^3 of candidates, so diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index ea7c6a3b..425a4552 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -9,7 +9,7 @@ This is an UNSTABLE release. It includes many experimental features, as well as * Bragg prediction: How far the predictor walks the lattice is a setting (`bragg_integration_settings.max_hkl`) instead of a fixed 100, derived per crystal offline from the refined cell (`--max-hkl` overrides); the broker keeps a fixed bootstrap so a live acquisition has a predictable per-image cost. * Bragg integration: The local background ring is now made robust with a **high-side sigma clip** (`--background-clip `, default 4) instead of the symmetric trimmed mean, which is biased low on Poisson data and added ~5 counts to every partial. The trim stays reachable with `--background-trim `. Expect `` to fall and edge `R_meas` to rise - that is the removed bias, not a regression; per-shell agreement with independent processing improves. * Bragg integration: The **uncertainty of the background estimate** is now propagated into `sigma`; both engines omitted it, which understated every reflection's `sigma` by `sqrt(1 + n_signal/n_bkg)` = 1.109 with the shipped stencil. Expect `` to fall by that factor on every dataset. -* Bragg integration: New **radial background correction** (`--background-radial[=on|off|auto]`, default `auto`) for the bias a curved radial background leaves in a flat ring mean - tens of counts on a single reflection sitting on an ice ring. `auto` applies it per image where that image's ice score shows a *smooth* powder ring: measured against a fixed atomic model it removes 43 % of the ice bands' excess amplitude on smooth ice, but on ice made of discrete crystallite spots - where a radius-only background model has nothing to fit - it makes the bias worse, and the gate separates the two. +* Bragg integration: New **radial background correction** (`--background-radial[=on|off|auto]`, default off) for the bias a curved radial background leaves in a flat ring mean - tens of counts on a single reflection sitting on an ice ring. `auto` applies it per image where that image's ice score shows a *smooth* powder ring, which targets correctly (it fires on ten battery crystals, all ice-positive) but costs 1.35x the wall clock and is sign-mixed on the merge statistics, so it stays opt-in: measured against a fixed atomic model it removes 43 % of the ice bands' excess amplitude on smooth ice, but on ice made of discrete crystallite spots - where a radius-only background model has nothing to fit - it makes the bias worse, and the gate separates the two. * rugnux: **Ice-ring handling now runs only when the crystal is measured to have ice** - the eleven fixed bands cost 16-26 % of the unique reflections whether it does or not. Detection uses two channels, the spot finder's peak-excluded radial profile (`--ice-min-score`, default 1.5) and found spots on the rings against ice-free flanks (`--ice-min-spot-ratio`, default 2.0), which catch smooth and textured ice respectively; the previous score was read off the plain azimuthal profile, where strong reflections in a ring's bin score as ice. * rugnux: The merge-time **ice-ring mask has been removed**. Measured against anomalous peak height it returned -0.001 +- 0.018 sigma while deleting reflections whose mean `I/sigma` was *above* the dataset average, leaving `R_meas`, CC1/2 and ISa unchanged and the affected shell 17 completeness points short. Ice reflections are still kept out of the scale fit and the space-group search, as every established scaling program does, and are kept in the final merge. * rugnux: **Multiplicity was over-reported** by the rotation merge: the per-shell observation counter rode on the R_meas re-walk, which deliberately applies a wider filter than the merge, so observations the merge had excluded were counted against a unique count that excluded them - and a group the merge dropped entirely has a NaN resolution, which `ResolutionShells::GetShell` silently binned as the lowest-resolution shell rather than rejecting. Only counts were affected; intensities, sigmas, R_meas, CC1/2, completeness and ISa were always right. diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index a6ef1187..6069fc09 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -592,7 +592,7 @@ $ $ with $\kappa$ obtained once by azimuth-averaging the stencil and $\bar B(r)$ the image's own radial background curve. Applying it costs one short dot product per reflection and no extra pixel reads; correcting the background *scalar* means the box sum, the profile fit and the variance all pick it up. The curve is accumulated from the same annulus pixels the background pass already reads (a pixel's radius is the reflection's radius plus the pixel's projection on the beam→reflection direction, so no per-pixel square root is needed) and specifically from the **clipped** pixels, or it would carry neighbour tails and zingers — which is why the correction is inert under `--integrator boxsum`, that path having no clip pass. -The model is a function of **radius alone**, so it is applied only where that is true of the background. `--background-radial` takes `on`, `off` or `auto`, and **`auto` is the default**: each image's peak-excluded ice score (§3.3) is taken after spot detection and before integration, and the correction is applied to that image when the score reaches the same `--ice-min-score` gate. Smooth powder ice *is* a radial feature and is corrected; ice made of discrete crystallite spots — which the profile channel is blind to and the spot channel catches — leaves no smooth ring to model, and correcting it makes matters worse. Measured against a fixed atomic model, comparing ice bands with resolution-matched decoy bands carrying no ice: on a crystal with pure smooth ice the correction removes **43 % of the bands' excess amplitude**, and the improvement is **7× larger inside the bands than outside**, which is its stated mechanism; on a crystal whose ice is textured the same correction *increased* the excess amplitude by half; on a clean crystal it is inert to four decimal places. Auto engages only where a peak-excluded score exists (adaptive spot finding, §3.2) — a plain azimuthal profile carries the Bragg peaks and cannot support an absolute threshold, so without one auto leaves the correction off. +The model is a function of **radius alone**, so it is applied only where that is true of the background. `--background-radial` takes `on`, `off` or `auto`. It is **off by default**; under `auto` each image's peak-excluded ice score (§3.3) is taken after spot detection and before integration, and the correction is applied to that image when the score reaches the same `--ice-min-score` gate. Smooth powder ice *is* a radial feature and is corrected; ice made of discrete crystallite spots — which the profile channel is blind to and the spot channel catches — leaves no smooth ring to model, and correcting it makes matters worse. Measured against a fixed atomic model, comparing ice bands with resolution-matched decoy bands carrying no ice: on a crystal with pure smooth ice the correction removes **43 % of the bands' excess amplitude**, and the improvement is **7× larger inside the bands than outside**, which is its stated mechanism; on a crystal whose ice is textured the same correction *increased* the excess amplitude by half; on a clean crystal it is inert to four decimal places. Auto engages only where a peak-excluded score exists (adaptive spot finding, §3.2) — a plain azimuthal profile carries the Bragg peaks and cannot support an absolute threshold, so without one auto leaves the correction off. ### 9.3 Profile-fitted extraction (default) diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index a157b0a2..27672701 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -290,7 +290,7 @@ Integration: | `--integration-radius ` | Signal-box radius `r1`, or `r1,r2,r3` (px). One value ⇒ `r2=r1+2`, `r3=r1+4` | | `--background-clip ` | Monochromatic (rotation + still): high-side clip of the background ring at `mean + n·√mean` (default 4; 0 = off). The default background estimator — it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Broadband data always clip, at 3σ; ignored by `--integrator boxsum` | | `--background-trim ` | Use the old symmetric trimmed mean for the background ring instead of the clip, 0≤f<0.5 (`0.10` was the former default). Switches `--background-clip` off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; `0` = plain ring mean | -| `--background-radial[=on\|off\|auto]` | Correct the background ring for the **curvature** of the radial background (default `auto`). Disk and ring are concentric, so a background linear in position cancels between them and only curvature survives — which on a smooth ice ring reaches +26 counts on a single reflection. `auto` applies it per image where that image's ice score shows a *smooth* powder ring, since the model is a function of radius alone: on ice made of discrete crystallite spots there is no smooth ring and the correction makes the bias worse. Ignored by `--integrator boxsum` (no clip pass to take the curve from) | +| `--background-radial[=on\|off\|auto]` | Correct the background ring for the **curvature** of the radial background (default **off**). Disk and ring are concentric, so a background linear in position cancels between them and only curvature survives — which on a smooth ice ring reaches +26 counts on a single reflection. `auto` applies it per image where that image's ice score shows a *smooth* powder ring, since the model is a function of radius alone: on ice made of discrete crystallite spots there is no smooth ring and the correction makes the bias worse. Ignored by `--integrator boxsum` (no clip pass to take the curve from) | | `--integration-high-resolution ` | High-resolution limit for prediction and integration. Omitted (or 0) means integration extends as far as the detector reaches — which is what the predictor can place on the detector anyway, since it rejects reflections that miss it. Set a value to integrate less than the detector offers | | `--max-hkl ` | Predict reflections with \|h\|,\|k\|,\|l\| ≤ `n` (max 511). By default this is derived per crystal from the refined cell as `ceil(max(a,b,c)/d_min) + 1`, which is the exact bound: the predictor keeps only \|q\| ≤ 1/d_min and `h = a·q`, so no reflection can lie outside it and no candidate inside it is wasted on a shorter axis. Set it only to override that | | `--bandwidth ` | Relative X-ray bandwidth FWHM (e.g. `0.01` for a 1% DMM); default from file or 0 (monochromatic) | diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 3f86adae..ad878983 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -130,7 +130,7 @@ void print_usage() { std::cout << " --integration-high-resolution High resolution limit for prediction/integration. If omitted (or 0), integration extends as far as the detector reaches" << std::endl; std::cout << " --max-hkl Predict reflections with |h|,|k|,|l| <= n. Default: derived per crystal from the refined cell (ceil(longest axis / d_min) + 1), which is the exact bound - set it only to override that" << std::endl; std::cout << " --background-clip Monochromatic (rotation + still): high-side clip of the background ring at mean + n*sqrt(mean) (default 4; 0 = off). This is the default background estimator - it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Broadband data always clip, at 3 sigma; ignored by --integrator boxsum" << std::endl; - std::cout << " --background-radial[=on|off|auto] Correct the background ring for the CURVATURE of the radial background (default auto). The signal disk and the background ring are concentric, so a background linear in position cancels between them and only curvature survives - which on a smooth ice ring reaches +26 counts on a single reflection. Auto applies it per image where that image's ice score shows a smooth powder ring, which is where a radius-only background model holds; on ice made of discrete crystallite spots there is no such ring and the correction makes the bias worse. Costs one short dot product per reflection and no extra pixel reads" << std::endl; + std::cout << " --background-radial[=on|off|auto] Correct the background ring for the CURVATURE of the radial background (default off). The signal disk and the background ring are concentric, so a background linear in position cancels between them and only curvature survives - which on a smooth ice ring reaches +26 counts on a single reflection. =auto applies it per image where that image's ice score shows a smooth powder ring, which is where a radius-only background model holds; on ice made of discrete crystallite spots there is no such ring and the correction makes the bias worse. Costs one short dot product per reflection and no extra pixel reads" << std::endl; std::cout << " --background-trim Use the old symmetric trimmed mean for the background ring instead of the clip (0<=f<0.5; 0.10 was the former default). Switches --background-clip off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; 0 = plain ring mean" << std::endl; std::cout << " --integrator Spot integrator boxsum|gaussian|empirical (default: gaussian profile-fit; boxsum is the classical fallback)" << std::endl; std::cout << " --simple-stills stills: treat every reflection as a full (p=1, single-pass scale/merge); disables the default physical partiality post-refinement" << std::endl;