Radial background: one mechanism for stills and rotation, decided per run on empty apertures (D5)
Wherever the correction is set for the whole run its curve is the frame's peak-excluded ring background (the adaptive spot finder's, whole frame) and the ring's clip follows it - as stills have done since integ-pal - now rotation too. One empty-aperture measurement (BackgroundAperture) decides it for both workflows: - The apertures' corrected difference is now taken as the integrator applies the correction: the clip follows the curve (each ring pixel's threshold raised by the curve's rise from the centre, in whole counts) as well as the kernel correction. Before, the tally clipped flat, so it measured a correction the engine never applies. - Rotation measures the stencils the spot width can give it (r1 = 4, 5, 6 with the ring-area rule, RotationStencil) and decides on the one it gets, and separately on the pre-adaptive stencil the geometry pre-pass and the starvation guard return to. - Stills: on wherever the corrected ring predicts the empty disk better (unchanged rule). Rotation: the frame's curve on every image only where it takes >= 10% of the apertures' mean square error away (ROTATION_RADIAL_MIN_GAIN); otherwise the existing per-image ice gate with the reflections' curve is left exactly as it was. Why a margin on rotation only: a still has no other correction, a sweep has one. Measured over 18 rotation sets (forced on every image): photon-counting sweeps gain 0.1-5% on the apertures (z 3-4) and lose ISa/anomalous signal when corrected (insulin, lysozyme, cytochrome c; 9yzk R-free +0.009 at 1.2%), as the earlier stills-rule trial found; the home-source CCD and smooth-ice runs gain 13-31%. Result (CPU A/B vs 87684012f, 18 rotation sets): 12 .hkl byte-identical (every photon-counting control without ice), 5uth R-free 0.210 -> 0.201 / ISa 10.4 -> 10.8, the ice set neutral; stills: lcls242_s turns the correction on (the clip-following tally), R-free 0.2399 -> 0.2406. No integration kernel changed; the tally is host code. Tests: BackgroundAperture_CorrectedClipFollowsTheCurve, RadialHelps margins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dd3f3f2da39f5338e7dda1874bbdd535229bce98)
This commit is contained in:
1 parent
134fc56f70
commit
f199f23c54
8 files changed
+195
-49
No files matched your search
+1
-1
@@ -1335,7 +1335,7 @@ class Rugnux {
|
||||
std::vector<FootprintSpot> footprint;
|
||||
std::vector<float> spot_q;
|
||||
size_t width_images = 0;
|
||||
background_aperture::Tally apertures;
|
||||
std::vector<background_aperture::Tally> apertures; // one per candidate stencil
|
||||
};
|
||||
struct Defects {
|
||||
size_t pixels = 0;
|
||||
|
||||
+89
-25
@@ -88,6 +88,13 @@
|
||||
using namespace rugnux_internal;
|
||||
|
||||
namespace {
|
||||
// The rotation stencil the spot width gives a signal radius r1: r2 follows it, and r3 keeps the
|
||||
// r2..r3 background ring at the area it has at the shipped 4/6/13.
|
||||
background_aperture::Stencil RotationStencil(float r1) {
|
||||
const float r2 = r1 + 2.0f;
|
||||
return {r1, r2, {std::sqrt(r2 * r2 + 133.0f)}};
|
||||
}
|
||||
|
||||
// A detector-plane displacement resolved ACROSS the spindle - the only component the tolerance
|
||||
// above is about. Along the spindle the error translates the derotated cloud rigidly and the FFT
|
||||
// amplitude cannot see it at all; across it the peaks smear and the amplitude falls. (The lattice
|
||||
@@ -364,21 +371,29 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
return w;
|
||||
};
|
||||
|
||||
// Stills: how far the background ring should reach, measured on empty apertures of the same frames
|
||||
// (BackgroundAperture.h) at the signal disk and gap the run integrates with. The candidates run up
|
||||
// to the ring the run starts from, so where the noise wins that ring is what is kept. The ring is
|
||||
// Empty apertures of the same frames (BackgroundAperture.h): whether the radial background correction
|
||||
// makes the ring predict the background under the disk better, and on stills also how far the ring
|
||||
// should reach. Stills measure at the signal disk and gap the run integrates with, the candidate rings
|
||||
// running up to the one the run starts from, so where the noise wins that ring is what is kept.
|
||||
// Rotation does not know its stencil yet - the spot width decides it from these same frames - so each
|
||||
// stencil it can end up with is measured, and the one it does is read (RotationStencil). The ring is
|
||||
// clipped at the gain the run integrates at.
|
||||
const bool stills = !experiment_.IsRotationIndexing();
|
||||
const BraggIntegrationSettings aperture_bis = prescan_x.GetBraggIntegrationSettings();
|
||||
const std::vector<float> aperture_r3 = stills
|
||||
? background_aperture::CandidateR3(aperture_bis.GetR2(), aperture_bis.GetR3()) : std::vector<float>();
|
||||
std::vector<background_aperture::Stencil> aperture_stencils;
|
||||
if (stills)
|
||||
aperture_stencils.push_back({aperture_bis.GetR1(), aperture_bis.GetR2(),
|
||||
background_aperture::CandidateR3(aperture_bis.GetR2(), aperture_bis.GetR3())});
|
||||
else
|
||||
for (const float r1 : {4.0f, 5.0f, 6.0f}) // the radii spot_width::R1ForWidth chooses from
|
||||
aperture_stencils.push_back(RotationStencil(r1));
|
||||
const float aperture_gain = aperture_bis.GetPixelGain();
|
||||
|
||||
// The spots of one image, and what is taken off them.
|
||||
const auto find_spots = [&](PreScanWorker &w, const CompressedImage &image, int image_idx,
|
||||
bool for_width, std::vector<spot_width::FluxCurve> &curves,
|
||||
std::vector<float> &spot_q, std::vector<FootprintSpot> &footprint,
|
||||
background_aperture::Tally &apertures) {
|
||||
std::vector<background_aperture::Tally> &apertures) {
|
||||
try {
|
||||
// As in the image loops: a frame the device cannot decode goes to the host decoder.
|
||||
ImageStatistics stats;
|
||||
@@ -422,7 +437,7 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
MeasureSpotsOnTiles(*w.preprocessed, static_cast<int>(prescan_x.GetXPixelsNumConv()),
|
||||
static_cast<int>(prescan_x.GetYPixelsNumConv()), prescan_x.GetDiffractionGeometry(),
|
||||
prescan_x.GetModules(), spots, curves, footprint, aperture_gain);
|
||||
if (for_width && !aperture_r3.empty()) {
|
||||
if (for_width) {
|
||||
// The frame's ring background, as the integration of this frame will read it - from the
|
||||
// adaptive finder, which has it whole-frame and peak-excluded.
|
||||
const std::vector<float> no_rings;
|
||||
@@ -431,11 +446,14 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
prescan_mapping->GetQBinCount(),
|
||||
prescan_mapping->Settings().GetLowQ_recipA(),
|
||||
prescan_mapping->Settings().GetQSpacing_recipA());
|
||||
background_aperture::MeasureFrame(*w.preprocessed, static_cast<int>(prescan_x.GetXPixelsNumConv()),
|
||||
static_cast<int>(prescan_x.GetYPixelsNumConv()), spots,
|
||||
aperture_bis.GetR1(), aperture_bis.GetR2(), aperture_r3,
|
||||
aperture_bis.GetBackgroundClipNSigma(), aperture_gain, rings,
|
||||
prescan_x.GetDiffractionGeometry(), prescan_x.GetModules(), apertures);
|
||||
apertures.resize(aperture_stencils.size());
|
||||
for (size_t k = 0; k < aperture_stencils.size(); ++k)
|
||||
background_aperture::MeasureFrame(*w.preprocessed, static_cast<int>(prescan_x.GetXPixelsNumConv()),
|
||||
static_cast<int>(prescan_x.GetYPixelsNumConv()), spots,
|
||||
aperture_stencils[k].r1, aperture_stencils[k].r2,
|
||||
aperture_stencils[k].r3, aperture_bis.GetBackgroundClipNSigma(),
|
||||
aperture_gain, rings, prescan_x.GetDiffractionGeometry(),
|
||||
prescan_x.GetModules(), apertures[k]);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -446,8 +464,8 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
std::vector<spot_width::FluxCurve> shape_curves;
|
||||
// Every spot's measured widths, for the spot footprint (SpotFootprint.h).
|
||||
std::vector<FootprintSpot> footprint_spots;
|
||||
// The empty apertures of the frames the width is measured on (stills).
|
||||
background_aperture::Tally aperture_tally;
|
||||
// The empty apertures of the frames the width is measured on, one tally per aperture_stencils entry.
|
||||
std::vector<background_aperture::Tally> aperture_tally(aperture_stencils.size());
|
||||
// Images the width ended up being measured on, for the log: the tiers below stop as soon as the
|
||||
// answer has settled, so this is a property of the crystal and worth reporting.
|
||||
size_t width_images = 0;
|
||||
@@ -565,7 +583,7 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
std::vector<std::vector<spot_width::FluxCurve>> curves_of(ordinals.size());
|
||||
std::vector<std::vector<float>> spot_q_of(ordinals.size());
|
||||
std::vector<std::vector<FootprintSpot>> footprint_of(ordinals.size());
|
||||
std::vector<background_aperture::Tally> apertures_of(ordinals.size());
|
||||
std::vector<std::vector<background_aperture::Tally>> apertures_of(ordinals.size());
|
||||
// Frames read for the bandwidth alone, past the tier the width settled on. Their spots stay
|
||||
// out of the powder measurement, which is taken on the frames it always was.
|
||||
std::vector<char> shape_only(ordinals.size(), 0);
|
||||
@@ -683,7 +701,8 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
// In sample order, over the frames the width was taken on.
|
||||
for (size_t i = 0; i < ordinals.size(); i++)
|
||||
if (position[i] < width_images)
|
||||
aperture_tally.Add(apertures_of[i]);
|
||||
for (size_t k = 0; k < apertures_of[i].size(); ++k)
|
||||
aperture_tally[k].Add(apertures_of[i][k]);
|
||||
// In sample order, so the table does not depend on how the workers interleaved.
|
||||
for (const auto &f : footprint_of)
|
||||
footprint_spots.insert(footprint_spots.end(), f.begin(), f.end());
|
||||
@@ -733,13 +752,15 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
// The ring is chosen on the plain differences, so its reach does not rest on the radial
|
||||
// model; the correction is then kept, where it was left to auto, if it makes that ring predict
|
||||
// the disk better. Read only where every frame gave a ring background.
|
||||
const std::vector<double> plain = aperture_tally.MeanSquare(false);
|
||||
const std::vector<double> radial = aperture_tally.apertures_radial == aperture_tally.apertures
|
||||
? aperture_tally.MeanSquare(true) : std::vector<double>();
|
||||
const background_aperture::Tally &tally = aperture_tally.front();
|
||||
const std::vector<float> &aperture_r3 = aperture_stencils.front().r3;
|
||||
const std::vector<double> plain = tally.MeanSquare(false);
|
||||
const std::vector<double> radial = tally.apertures_radial == tally.apertures
|
||||
? tally.MeanSquare(true) : std::vector<double>();
|
||||
const int best = background_aperture::Best(plain);
|
||||
const bool use_radial = radial_setting.has_value()
|
||||
? *radial_setting
|
||||
: best >= 0 && !radial.empty() && radial[best] < plain[best];
|
||||
: best >= 0 && background_aperture::RadialHelps(tally, best);
|
||||
if (best < 0) {
|
||||
logger.Info("Background ring: no empty aperture measured on {} images, keeping r3={:.1f}",
|
||||
width_images, bis.GetR3());
|
||||
@@ -757,10 +778,10 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
logger.Info("Background ring: on {} empty apertures from {} images the r{:.0f}..r3 ring predicts "
|
||||
"the r{:.0f} disk with a mean square error of (r3:value){}, and with the radial "
|
||||
"curvature correction{} => integration radii r1={:.1f} r2={:.1f} r3={:.1f}, radial "
|
||||
"correction {} (z = {:.1f} at that ring)", aperture_tally.apertures, width_images, bis.GetR2(), bis.GetR1(),
|
||||
"correction {} (z = {:.1f} at that ring)", tally.apertures, width_images, bis.GetR2(), bis.GetR1(),
|
||||
table(plain), radial.empty() ? std::string(" (no ring background)") : table(radial),
|
||||
bis.GetR1(), bis.GetR2(), bis.GetR3(), use_radial ? "on" : "off",
|
||||
aperture_tally.RadialZ(best));
|
||||
tally.RadialZ(best));
|
||||
Note(fmt::format("background ring {:.0f}..{:.0f} px, radial correction {}, measured on empty "
|
||||
"apertures", bis.GetR2(), bis.GetR3(), use_radial ? "on" : "off"));
|
||||
}
|
||||
@@ -773,9 +794,8 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
"integration radius at r1={:.1f}", width_curves.size(), width_images,
|
||||
experiment_.GetBraggIntegrationSettings().GetR1());
|
||||
} else {
|
||||
const float r1 = spot_width::R1ForWidth(*r80);
|
||||
const float r2 = r1 + 2.0f;
|
||||
const float r3 = std::sqrt(r2 * r2 + 133.0f);
|
||||
const background_aperture::Stencil st = RotationStencil(spot_width::R1ForWidth(*r80));
|
||||
const float r1 = st.r1, r2 = st.r2, r3 = st.r3.front();
|
||||
BraggIntegrationSettings bis = experiment_.GetBraggIntegrationSettings();
|
||||
// Remembered where the APERTURE moved, so the starvation guard in RunAllPasses has
|
||||
// something to give back and cannot announce a fallback that changes nothing.
|
||||
@@ -799,6 +819,50 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
|
||||
*r80, r1, r2, r3));
|
||||
}
|
||||
|
||||
// The radial background correction, where it was left to auto, decided on the empty apertures of
|
||||
// each stencil the run may integrate with: the one the width gave it, and the one it started
|
||||
// from, which the geometry pre-pass and the starvation guard go back to. Rotation's own default
|
||||
// is the per-image correction of the frames the ice score marks (MXAnalysisWithoutFPGA), so the
|
||||
// frame's curve replaces it on every frame only where the apertures show it taking a real part
|
||||
// of the background error away (background_aperture::ROTATION_RADIAL_MIN_GAIN).
|
||||
const auto decide_radial = [&](BraggIntegrationSettings &bis, const char *which) {
|
||||
if (bis.GetBackgroundRadialCorrection().has_value())
|
||||
return;
|
||||
size_t k = 0;
|
||||
while (k < aperture_stencils.size()
|
||||
&& !(aperture_stencils[k].r1 == bis.GetR1() && aperture_stencils[k].r2 == bis.GetR2()
|
||||
&& aperture_stencils[k].r3.front() == bis.GetR3()))
|
||||
++k;
|
||||
if (k == aperture_stencils.size() || aperture_tally[k].apertures == 0) {
|
||||
logger.Info("Background ring ({}): r1={:.1f} r2={:.1f} r3={:.2f} not measured on empty "
|
||||
"apertures; radial correction per image", which, bis.GetR1(), bis.GetR2(),
|
||||
bis.GetR3());
|
||||
return;
|
||||
}
|
||||
const background_aperture::Tally &tally = aperture_tally[k];
|
||||
const std::vector<double> plain = tally.MeanSquare(false), radial = tally.MeanSquare(true);
|
||||
const bool on = background_aperture::RadialHelps(tally, 0,
|
||||
background_aperture::ROTATION_RADIAL_MIN_GAIN);
|
||||
if (on)
|
||||
bis.BackgroundRadialCorrection(true);
|
||||
logger.Info("Background ring ({}): on {} empty apertures from {} images the r{:.0f}..r{:.0f} ring "
|
||||
"predicts the r{:.0f} disk with a mean square error of {:.4g}, and {} with the radial "
|
||||
"curvature correction from the frame's curve (z = {:.1f}) => radial correction {}",
|
||||
which, tally.apertures, width_images, bis.GetR2(), bis.GetR3(), bis.GetR1(), plain[0],
|
||||
radial.empty() ? std::string("none (no ring background)")
|
||||
: fmt::format("{:.4g}", radial[0]),
|
||||
tally.RadialZ(0), on ? "on every image" : "per image, where the ice score sees a ring");
|
||||
};
|
||||
{
|
||||
BraggIntegrationSettings bis = experiment_.GetBraggIntegrationSettings();
|
||||
decide_radial(bis, "canonical");
|
||||
experiment_.ImportBraggIntegrationSettings(bis);
|
||||
if (bragg_before_adaptive_)
|
||||
decide_radial(*bragg_before_adaptive_, "pre-adaptive");
|
||||
if (bis.GetBackgroundRadialCorrection() == std::optional<bool>(true))
|
||||
Note("radial background correction on, measured on empty apertures");
|
||||
}
|
||||
|
||||
// How far the spots reach away from the beam, where the radius chosen above at 5 A no
|
||||
// longer holds them (SpotFootprint.h). It changes the integration only where a spot
|
||||
// outgrows the r1 disk.
|
||||
|
||||
@@ -255,7 +255,7 @@ void print_usage() {
|
||||
std::cout << " --integration-high-resolution <num> High resolution limit for prediction/integration. If omitted (or 0), integration extends as far as the detector reaches" << std::endl;
|
||||
std::cout << " --max-hkl <n> Predict reflections with |h|,|k|,|l| <= n. Default: derived per crystal from the refined cell (ceil(longest axis / d_min) + 1), which is the exact bound - set it only to override that" << std::endl;
|
||||
std::cout << " --background-clip <n> High-side clip of the background ring at mean + n*sqrt(mean) (default 4; 0 = off). This is the default background estimator - it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Ignored by --integrator boxsum" << std::endl;
|
||||
std::cout << " --background-radial[=on|off|auto] Correct the background ring for the CURVATURE of the radial background (default auto). The signal disk and the background ring are concentric, so a background linear in position cancels between them and only curvature survives - which on a smooth ice ring reaches +26 counts on a single reflection. =auto applies it per image where that image's ice score shows a smooth powder ring, which is where a radius-only background model holds; on ice made of discrete crystallite spots there is no such ring and the correction makes the bias worse. On stills the curve is taken from the spot finder's peak-excluded background of the whole frame, and =auto applies it to every image of a run where it makes the background ring predict the background of empty apertures better (measured in the pre-scan): a smooth solvent or carrier hump curves the background as much as ice does. Costs one short dot product per reflection and no extra pixel reads" << std::endl;
|
||||
std::cout << " --background-radial[=on|off|auto] Correct the background ring for the CURVATURE of the radial background (default auto). The signal disk and the background ring are concentric, so a background linear in position cancels between them and only curvature survives - which on a smooth ice ring reaches +26 counts on a single reflection. Where it is on, the curve is the spot finder's peak-excluded background of the whole frame, and the ring's outlier clip follows it, so a background falling steeply from the beam is not cut as outliers. =auto decides it per run in the pre-scan, on empty apertures of the pre-scan frames: on stills it is applied to every image where it makes the background ring predict the background of empty apertures better at all (a smooth solvent or carrier hump curves the background as much as ice does); on rotation data it is applied to every image where it takes at least 10% of that error away, and otherwise only to the images whose ice score shows a smooth powder ring, with the curve from the reflections' own rings (ice made of discrete crystallite spots leaves no such ring, and the correction makes the bias worse). Costs one short dot product per reflection and no extra pixel reads" << std::endl;
|
||||
std::cout << " --background-trim <f> Use the old symmetric trimmed mean for the background ring instead of the clip (0<=f<0.5; 0.10 was the former default). Switches --background-clip off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; 0 = plain ring mean" << std::endl;
|
||||
std::cout << " --overlap <txt> What to do where two predicted reflections share signal pixels: off|reject|exclude (default exclude). A pixel inside two signal disks belongs to the nearer centre; before this, nothing kept a neighbour's flux out of a reflection's own disk, so on a dense pattern a crowded reflection read high. exclude drops the shared PIXELS from the profile fit, which renormalises itself, so the reflection is kept; reject instead drops the whole reflection when less than --overlap-minpk of its expected profile is cleanly its own (what XDS calls MINPK). --integrator boxsum has no profile to renormalise with, so exclude does nothing there and only reject acts" << std::endl;
|
||||
std::cout << " --overlap-minpk <f> Least fraction of a reflection's expected profile that must be usable for the reflection to be kept (default 0.75, XDS MINPK). Governs both ways part of a profile is lost: the fraction that must be READABLE - not masked, untrusted, in a detector gap or overloaded - in every profile mode, and, under --overlap reject, the fraction that must be cleanly the reflection's own. --integrator boxsum has no profile to renormalise with, so there a signal disk with any unreadable pixel is still discarded outright and the reject fraction is by disk AREA, which cuts harder" << std::endl;
|
||||
@@ -3439,7 +3439,7 @@ static int RunRugnux(int argc, char **argv, bool no_gpu) {
|
||||
logger.Info("Background ring: radial curvature correction {}",
|
||||
radial.has_value() ? (*radial ? "on" : "off")
|
||||
: rotation_indexing
|
||||
? "auto (per image, where the ice score sees a smooth ring)"
|
||||
? "auto (decided in the pre-scan on empty apertures; otherwise per image, where the ice score sees a smooth ring)"
|
||||
: "auto (decided in the pre-scan on empty apertures; the curve from the whole frame)");
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user