From 9b7a91d52102ae6b777dd3f351a831a6818cd929 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 23 Sep 2026 19:59:17 +0200 Subject: [PATCH 01/21] battery: depositor-data comparison column; REFMAC check reads partial-ANISOU models depdata_check.py compares each open-arm merge with the depositor's own data, per resolution shell, using the deposited model as the common yardstick: the rank correlation of our IMEAN with |Fc|^2 minus that of the deposited intensities (or F^2), on the reflections both carry (d < 4 A, eight equal-count shells), with the data carried into the model's setting by model_check's change-of-basis search. New row keys dep_cc_delta_all, dep_cc_delta_outer (two outermost common shells), dep_beyond_cc / dep_beyond_d (our correlation with |Fc|^2 past the deposited data's limit), dep_kind, dep_d_min, dep_n_common, dep_status, dep_reason. gemmi only, no CCP4; runs on every open-arm set with a model, and `report` fills it in for older runs (four processes). Reported in its own report section and the per-set table, never scored. Rank rather than Pearson correlation: on the phase-1 run a handful of gross outliers (I/sigma above 1000 at 2.4 A, or ~1000x the shell median at 1.41 A) decided the outer-shell Pearson CC of several merges on their own. On the phase-1 run (38 of 40 open-arm sets compared; the two without are the lattice failures) the outer-shell delta has median -0.012; 22 s for the whole report fill-in. model_check.py: REFMAC's mmCIF reader stopped with "rdaniso_cif: Atom symbol mismatch" on 12 of 35 entries whose atoms carry ANISOU only in part. The model is now written in the PDB format where it fits (under 100000 atoms, chain names of at most two characters), where each ANISOU follows its own ATOM line; all 12 then score. R-factors of the entries that worked before move by < 0.001. The runner also records model_check's baseline note (e.g. an intensity-only deposition) in refmac_reason instead of leaving it empty. Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/battery/README.md | 25 ++++ tools/battery/battery.py | 9 +- tools/battery/depdata_check.py | 220 +++++++++++++++++++++++++++++++++ tools/battery/model_check.py | 11 +- tools/battery/report.py | 58 ++++++++- 5 files changed, 318 insertions(+), 5 deletions(-) create mode 100644 tools/battery/depdata_check.py diff --git a/tools/battery/README.md b/tools/battery/README.md index a9b8eeffc..d236c01e6 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -237,6 +237,28 @@ structure factors with a free set. The runner records `refmac_rfree`, `refmac_rw `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status` and `refmac_reason`. Small-molecule sets are skipped. +`deposited structure factors carry no amplitudes` (intensity-only depositions) leaves the ratio +null: the check does not convert intensities to amplitudes itself, because a conversion that +differed from the depositor's would bias the baseline. + +### The depositor-data comparison (open arm, always on) + +`tools/battery/depdata_check.py` compares our merge with the depositor's own data without CCP4: +per resolution shell, the rank correlation of our IMEAN with |Fc|^2 minus that of the deposited +intensities (or amplitudes squared), on the reflections both carry, with |Fc| from the deposited +model as it is (gemmi, no bulk solvent, no refinement), the data carried into the model's setting +as the REFMAC check does it. The model was refined against the depositor's data, so the comparison +has a home advantage for the deposition; around -0.01 is typical, and a set far below that merged +worse than the deposition at those resolutions. It runs on every open-arm set with a model +(seconds to a minute, gemmi only; the `-sf.cif` is cached in `pdb_cache` beside the coordinates), +and `report` computes it for an older run that lacks it. It is reported, never scored. Row keys: +`dep_cc_delta_all` (all common reflections at d < 4 A), `dep_cc_delta_outer` (mean of the two +outermost of eight equal-count shells), `dep_beyond_cc` and `dep_beyond_d` (the correlation of our +reflections with |Fc|^2 in the outermost of up to three shells past the deposited data's limit, +and that shell's limit: the model never saw these, so clearly above zero is signal), plus +`dep_kind` (`I` or `F`), `dep_d_min` (the deposited data's limit), `dep_n_common`, `dep_status` +and `dep_reason`. + `model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The open arm's command does the same inside the battery. @@ -308,6 +330,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. | `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited | | `rmodel`, `rmodel_shell_scaled`, `radial_misfit` | open arm with a model: R_MODEL, R_MODEL_SHELL_SCALED and MODEL_RADIAL_MISFIT - R over all the reflections, with and without one free scale per resolution shell, and how big that rescale was. `rmodel_shell_scaled` is the R that compares between runs (below) | | `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm) | +| `dep_status`, `dep_reason`, `dep_kind`, `dep_d_min`, `dep_n_common`, `dep_cc_delta_all`, `dep_cc_delta_outer`, `dep_beyond_cc`, `dep_beyond_d` | open arm with a model: our merge against the depositor's data per shell (`depdata_check.py`, above) | | `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line | ## The report @@ -324,6 +347,8 @@ SVG, no external files): - plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference) for each arm; on the XDS arms the reference-range ISa, R_meas, lowest-shell R_meas and CC1/2 noise over XDS's; on the open arm R_free / published R_free; +- on the open arm, **against the depositor's data**: the per-shell CC difference and the CC past + the deposited limit, one row per set, sorted by the outer-shell difference; - on the XDS arms the **like-for-like table**: the reference-range numbers beside XDS's, with the rows where rugnux's cut is coarser marked as coverage; - the **failures**, and **one row per set** with all the numbers; diff --git a/tools/battery/battery.py b/tools/battery/battery.py index 84c75a0d7..330168574 100644 --- a/tools/battery/battery.py +++ b/tools/battery/battery.py @@ -59,6 +59,7 @@ import report # noqa: E402 import score # noqa: E402 import model_check # noqa: E402 the deposited model (--model); REFMAC check (--model-check) +import depdata_check # noqa: E402 our merge vs the depositor's data, shell by shell PDB_ID = re.compile(r"^[0-9][A-Za-z0-9]{3}$") @@ -226,7 +227,7 @@ def run_one(run_dir, binary, e, opts): row = {"set": e["id"], "arm": e["arm"], "tags": e.get("tags", []), "input": e["input_path"], "gpu_others": gpu_others(), "model": None, "model_note": None, "rfree_deposited": None, "rfree_ratio": None, "cmd": None, "exit_code": None, "elapsed_s": None, "wall_s": None} - row.update({k: None for k in REFMAC_KEYS}) + row.update({k: None for k in REFMAC_KEYS + depdata_check.KEYS}) if not os.path.exists(e["input_path"]): row.update(score.judge(e, {}, "no input")) return row @@ -256,6 +257,8 @@ def run_one(run_dir, binary, e, opts): row["wall_s"] = row["rugnux_wall_s"] if row["rugnux_wall_s"] is not None else row["elapsed_s"] if row["rfree"] and row["rfree_deposited"]: row["rfree_ratio"] = round(row["rfree"] / row["rfree_deposited"], 4) + if row["model"] and os.path.exists(os.path.join(wd, "p.mtz")): + row.update(depdata_check.check_set(wd, e["id"], opts["pdb_cache"])) if opts["model_check"] and e["arm"] == "open": row.update(check_model(e, wd)) return row @@ -280,7 +283,8 @@ def check_model(e, wd): except Exception as ex: # a failing check must not lose the set's processing result return dict(out, refmac_status="error", refmac_reason=f"{type(ex).__name__}: {ex}"[:300]) out.update({"refmac_" + k: res.get(k) for k in ("rfree", "rwork", "rfree_depflags", "rfree_depdata")}, - refmac_status=res.get("status"), refmac_reason=res.get("reason")) + refmac_status=res.get("status"), + refmac_reason=res.get("reason") or res.get("baseline_note")) # The fair ratio: our merge and the depositor's structure factors, both scored on the # depositor's free set by the same protocol. Our own free set was mostly work reflections # for the depositor, so R_free on it reads low. @@ -663,6 +667,7 @@ def main(): a = ap.parse_args() site = load_site(a.site) report.ALIASES = load_aliases(site) + report.PDB_CACHE = site.get("pdb_cache") or model_check.DEFAULT_CACHE report.MANIFESTS = {arm: {e["id"]: e for e in load_sets(site, arm)} for arm, cfg in site["arms"].items() if os.path.exists(cfg["manifest"])} {"run": cmd_run, "list": cmd_list, "compare": cmd_compare, "report": cmd_report, diff --git a/tools/battery/depdata_check.py b/tools/battery/depdata_check.py new file mode 100644 index 000000000..0a1759c6c --- /dev/null +++ b/tools/battery/depdata_check.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 +"""depdata_check.py -- Rugnux's merged intensities against the depositor's, shell by shell. + +A second, REFMAC-free quality check for the open arm, run AFTER Rugnux has written its MTZ. The +deposited model is the common yardstick: per resolution shell, the rank correlation of Rugnux's +merged intensities with |Fc|^2 and that of the depositor's own data are computed on the SAME +reflections, and the difference is reported. |Fc| comes from the deposited model as it is (gemmi, +no bulk solvent, no refinement), so the model was refined against the depositor's data and the +comparison carries a home advantage for the deposition; a difference near zero means our merge is +as good as the one the model was built from. + + * Rugnux's merge (IMEAN) is carried into the model's setting exactly as model_check.py does it + (every change of basis between the two lattices, the one whose data correlate best with |Fc|^2 + at 3-6 A kept) and reduced to the deposited group's asymmetric unit; + * the depositor's data are the first merged reflection block of the entry's -sf.cif (intensities + where given, else amplitudes squared; with several wavelengths the one nearest ours), in the + model's cell and group; + * the reflections both carry, at d < 4 A (all of them where that leaves fewer than 400), are cut + into SHELLS shells of equal count in 1/d^2. + +Columns (report-only, never scored): + dep_cc_delta_all CC(ours) - CC(deposited) over the common reflections + dep_cc_delta_outer the same, mean of the two outermost common shells + dep_beyond_cc CC(ours, |Fc|^2) in the outermost of up to three equal-count shells PAST the + deposited data's limit (null when we do not reach past it); the model was + never refined against these, so a clearly positive value is signal + dep_beyond_d that shell's high-resolution limit + + depdata_check.py rugnux.mtz 1abc [--cache DIR] +""" +import argparse +import json +import os + +import gemmi +import numpy as np + +import model_check +import score + +SHELLS = 8 +D_COMMON = 4.0 # low resolution left out: unmodelled bulk solvent dominates there +KEYS = ("dep_status", "dep_reason", "dep_kind", "dep_d_min", "dep_n_common", "dep_cc_delta_all", + "dep_cc_delta_outer", "dep_beyond_cc", "dep_beyond_d") + + +def cc(a, b): + """Rank correlation: a handful of gross outliers (seen in merges of both kinds) would decide a + shell's Pearson CC on their own.""" + if len(a) < 20: + return None + return float(np.corrcoef(np.argsort(np.argsort(a)), np.argsort(np.argsort(b)))[0, 1]) + + +def shells(d, n): + """Shell index (0 = lowest resolution) of each reflection, n shells of equal count in 1/d^2.""" + s2 = 1.0 / d ** 2 + edges = np.quantile(s2, np.linspace(0, 1, n + 1)) + return np.clip(np.searchsorted(edges, s2, side="right") - 1, 0, n - 1) + + +def rugnux_intensities(mtz, t, cell_m, sg_m): + """IMEAN of Rugnux's merge reindexed by t and reduced to the model group's asymmetric unit.""" + i = np.array(mtz.column_with_label("IMEAN"), copy=False).astype(float) + s = np.array(mtz.column_with_label("SIGIMEAN"), copy=False).astype(float) + good = np.isfinite(i) & np.isfinite(s) & (s > 0) + hkl, src = model_check.expand_p1(mtz) + hm = hkl @ t + integral = np.all(np.abs(hm - np.round(hm)) < 1e-3, axis=1) & good[src] + hm, src = np.round(hm[integral]).astype(int), src[integral] + hkl, i, _, _ = model_check.to_group(hm, i[src], s[src], np.zeros(len(src), int), cell_m, sg_m, src) + return hkl, i + + +def deposited_intensities(path, wavelength, cell_m, sg_m): + """(hkl, I, kind) of the entry's merged data in the model's asymmetric unit, or None, why.""" + blocks = [] + for rb in gemmi.as_refln_blocks(gemmi.cif.read(path)): + labels = rb.column_labels() + if rb.is_unmerged(): + continue + for kind, cols in (("I", ("intensity_meas", "intensity_sigma")), + ("F", ("F_meas_au", "F_meas_sigma_au")), + ("I", ("pdbx_I_plus", "pdbx_I_minus")), + ("F", ("pdbx_F_plus", "pdbx_F_minus"))): + if all(c in labels for c in cols): + blocks.append((rb, kind, cols)) + break + if not blocks: + return None, "no measured data in the deposited structure factors" + rb, kind, cols = blocks[0] + if len(blocks) > 1 and wavelength: + near = [abs(b[0].wavelength - wavelength) if b[0].wavelength else 9.0 for b in blocks] + if min(near) < 0.02: + rb, kind, cols = blocks[int(np.argmin(near))] + if not np.allclose(rb.cell.parameters, cell_m.parameters, rtol=0.02, atol=0.5): + return None, "deposited structure factors in another cell" + a = np.array(rb.make_float_array(cols[0])) + if cols[0].startswith("pdbx_"): # anomalous pair: the mean of the two hands + b = np.array(rb.make_float_array(cols[1])) + a = np.where(np.isfinite(a) & np.isfinite(b), (a + b) / 2, np.where(np.isfinite(a), a, b)) + x = a * a if kind == "F" else a + ok = np.isfinite(x) + if "status" in rb.column_labels(): # '-', 'x', '<': not measured + st = [v.strip("'\"") for v in rb.block.find_values("_refln.status")] + if len(st) == len(x): + ok &= np.array([v not in ("-", "x", "<") for v in st]) + hkl = np.array(rb.make_miller_array())[ok] + x = x[ok] + d_min = float(rb.cell.calculate_d_array(hkl).min()) + hkl, x, _, _ = model_check.to_group(hkl, x, np.ones(len(x)), np.zeros(len(x), int), cell_m, sg_m, + np.arange(len(x))) + return (hkl, x, kind, d_min), None + + +def lookup(keys, hkl): + """Index into `keys` (sorted) of each reflection of hkl, and which of them were found.""" + k = model_check.hkl_key(hkl) + pos = np.clip(np.searchsorted(keys, k), 0, len(keys) - 1) + return pos, keys[pos] == k + + +def check(mtz_path, pdb_id, cache_dir=model_check.DEFAULT_CACHE, wavelength=None): + res = {k: None for k in KEYS} + res["dep_status"] = "failed" + pdb_id = pdb_id.lower() + if not model_check.PDB_ID.match(pdb_id): + return dict(res, dep_status="n/a", dep_reason="not a PDB entry") + try: + meta = model_check.deposition(pdb_id, cache_dir) + if meta is None: + return dict(res, dep_reason="entry could not be downloaded") + sf = model_check.fetch(model_check.RCSB + pdb_id + "-sf.cif.gz", + os.path.join(cache_dir, pdb_id + "-sf.cif.gz")) + if sf is None: + return dict(res, dep_status="n/a", dep_reason="no deposited structure factors") + st = gemmi.read_structure(meta["xyz"]) + sg_m, cell_m = st.find_spacegroup(), st.cell + if sg_m is None: + return dict(res, dep_reason=f"model space group '{st.spacegroup_hm}' not recognised") + dep, why = deposited_intensities(sf, wavelength, cell_m, sg_m) + if dep is None: + return dict(res, dep_status="n/a", dep_reason=why) + hkl_d, i_d, res["dep_kind"], d_dep = dep + res["dep_d_min"] = round(d_dep, 3) + + mtz = gemmi.read_mtz_file(mtz_path) + fc_hkl, fc = model_check.fcalc(st, mtz.resolution_high() - 0.005) + order = np.argsort(model_check.hkl_key(fc_hkl)) + fc_keys, fc_hkl, fc2 = model_check.hkl_key(fc_hkl)[order], fc_hkl[order], fc[order] ** 2 + fc_d = cell_m.calculate_d_array(fc_hkl) + + best = None # the setting: best CC(I, |Fc|^2) at 3-6 A (everything if that is too few) + for t in model_check.basis_changes(mtz.cell, mtz.spacegroup, cell_m, sg_m): + hkl, i = rugnux_intensities(mtz, t, cell_m, sg_m) + pos, found = lookup(fc_keys, hkl) + sel = found.copy() + sel[found] &= (fc_d[pos[found]] >= 3) & (fc_d[pos[found]] <= 6) + if sel.sum() < 200: + sel = found + c = cc(i[sel], fc2[pos[sel]]) + if c is not None and (best is None or c > best[0]): + best = (c, hkl, i) + if best is None or best[0] < 0.1: # a model-only |Fc| correlates weakly at 4-6 A + return dict(res, dep_reason="the merge does not match the model in any setting") + _, hkl_r, i_r = best + pos, found = lookup(fc_keys, hkl_r) + i_r, f_r, d_r = i_r[found], fc2[pos[found]], fc_d[pos[found]] + k_r = fc_keys[pos[found]] + + _, a, b = np.intersect1d(k_r, model_check.hkl_key(hkl_d), return_indices=True) + ir, idp, fcc, d = i_r[a], i_d[b], f_r[a], d_r[a] + sel = d < D_COMMON + if sel.sum() < SHELLS * 50: + sel = np.ones(len(d), bool) + ir, idp, fcc, d = ir[sel], idp[sel], fcc[sel], d[sel] + res["dep_n_common"] = int(len(d)) + if len(d) < SHELLS * 20: + return dict(res, dep_reason=f"only {len(d)} reflections in common") + res["dep_cc_delta_all"] = round(cc(ir, fcc) - cc(idp, fcc), 4) + sh = shells(d, SHELLS) + res["dep_cc_delta_outer"] = round(float(np.mean( + [cc(ir[sh == k], fcc[sh == k]) - cc(idp[sh == k], fcc[sh == k]) for k in (SHELLS - 1, SHELLS - 2)])), 4) + + past = d_r < d_dep - 1e-3 # ours only, past the deposited data's limit + if past.sum() >= 60: + n = max(1, min(3, int(past.sum() // 300))) + sh = shells(d_r[past], n) + out = sh == n - 1 + res["dep_beyond_cc"] = round(cc(i_r[past][out], f_r[past][out]), 4) + res["dep_beyond_d"] = round(float(d_r[past][out].min()), 3) + res["dep_status"] = "ok" + return res + except Exception as e: # one bad entry must not stop a battery + if os.environ.get("MODEL_CHECK_RAISE"): + raise + return dict(res, dep_reason=f"{type(e).__name__}: {e}"[:300]) + + +def check_set(wd, set_id, cache_dir=model_check.DEFAULT_CACHE): + """check() on a battery work directory (p.mtz, and the wavelength from p_report.txt).""" + try: + wl = float(score.read_report(os.path.join(wd, "p_report.txt")).get("WAVELENGTH")) + except (TypeError, ValueError): + wl = None + return check(os.path.join(wd, "p.mtz"), set_id.split("_")[0], cache_dir, wl) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("mtz", help="Rugnux merged MTZ (needs IMEAN, SIGIMEAN)") + ap.add_argument("pdb_id") + ap.add_argument("--cache", default=model_check.DEFAULT_CACHE) + ap.add_argument("--wavelength", type=float, help="picks the deposited block (multi-wavelength entries)") + a = ap.parse_args() + print(json.dumps(check(a.mtz, a.pdb_id, a.cache, a.wavelength), indent=1)) + + +if __name__ == "__main__": + main() diff --git a/tools/battery/model_check.py b/tools/battery/model_check.py index 85c0b71cd..e43c597ae 100644 --- a/tools/battery/model_check.py +++ b/tools/battery/model_check.py @@ -428,9 +428,16 @@ def check(mtz_path, pdb_id, workdir, cache_dir=DEFAULT_CACHE, baseline=True): dep_flags[dep_flags < 0] = 1 data_mtz = os.path.join(workdir, "data.mtz") write_mtz(data_mtz, hkl, f, s, fl, cell_m, sg_m, dep_flags) - xyz = os.path.join(workdir, "model.cif") + # REFMAC's mmCIF reader rejects many models whose atoms carry ANISOU only in part + # ("rdaniso_cif: Atom symbol mismatch", 12 of 35 entries on first use); in the PDB format + # each ANISOU follows its own ATOM line. mmCIF only for what the PDB format cannot hold. st.setup_entities() - st.make_mmcif_document().write_file(xyz) + if natoms < 100000 and all(len(ch.name) <= 2 for ch in st[0]): + xyz = os.path.join(workdir, "model.pdb") + st.write_pdb(xyz) + else: + xyz = os.path.join(workdir, "model.cif") + st.make_mmcif_document().write_file(xyz) res["rwork"], res["rfree"] = refmac(workdir, "rugnux", data_mtz, xyz, d_used) if dep is not None: diff --git a/tools/battery/report.py b/tools/battery/report.py index c6188e068..4ba1811c7 100644 --- a/tools/battery/report.py +++ b/tools/battery/report.py @@ -1,10 +1,12 @@ """The per-run report and the run-to-run comparison.""" import json +import multiprocessing import os import socket import statistics import sys +import depdata_check import score from render import Doc, ratio_dots, verdict_bars @@ -78,6 +80,26 @@ def rescore(run_dir, man, res): r.update(score.judge(dict(e, arm=r["arm"]), rep, "")) +def _depdata(job): + os.nice(10) + return depdata_check.check_set(*job) + + +def fill_depdata(run_dir, res): + """The depositor-data comparison (depdata_check.py) for open-arm rows of a run made before it + existed, computed from the saved merge, four sets at a time.""" + todo = [r for r in res if r["arm"] == "open" and "dep_status" not in r and r.get("model") + and os.path.exists(os.path.join(run_dir, "work", "open", r["set"], "p.mtz"))] + if not todo: + return + print(f"depositor-data comparison for {len(todo)} sets of an older run ...", file=sys.stderr) + jobs = [(os.path.join(run_dir, "work", "open", r["set"]), r["set"], + PDB_CACHE or depdata_check.model_check.DEFAULT_CACHE) for r in todo] + with multiprocessing.Pool(4) as pool: + for r, out in zip(todo, pool.map(_depdata, jobs, chunksize=1)): + r.update(out) + + def schema1(man, res): """Read a run from before the variants (results schema 1) as one: it ran each set once, plain on the open arm and with XDS's settings on the XDS arms - unless --unforced, which left the @@ -139,6 +161,9 @@ ALIASES = {} # {arm: {set id: manifest row}} of today's manifests (battery.py sets it); rescore() scores with them MANIFESTS = {} +# the site's cache of deposited models and structure factors (battery.py sets it) +PDB_CACHE = None + def key(r): return (r["arm"], ALIASES.get(r["arm"], {}).get(r["set"], r["set"])) @@ -274,6 +299,7 @@ def build(run_dir, baseline=None, allow_incomplete=False): man, res = load_run(run_dir, allow_incomplete) if baseline: check_baseline(baseline, man.get("private")) + fill_depdata(run_dir, res) b = man["binary"] doc = Doc(f"Rugnux battery - {os.path.basename(os.path.normpath(run_dir))}") if man.get("private"): @@ -354,7 +380,12 @@ def build(run_dir, baseline=None, allow_incomplete=False): ("rmodel_shell_scaled", "R_model, shell-scaled", "{:.3f}"), ("rfree", "R_free (placement, within-run only)", "{:.3f}"), ("radial_misfit", "radial misfit", "{:.2f}"), - ("rfree_ratio", "R_free ratio", "{:.3f}"), ("wall_s", "time s", "{:.0f}")): + ("rfree_ratio", "R_free ratio", "{:.3f}"), + ("refmac_rfree_ratio", "REFMAC R_free ratio", "{:.3f}"), + ("dep_cc_delta_all", "dCC vs depositor, all", "{:+.3f}"), + ("dep_cc_delta_outer", "dCC vs depositor, outer 2 shells", "{:+.3f}"), + ("dep_beyond_cc", "CC(Fc^2) past the deposited limit", "{:.3f}"), + ("wall_s", "time s", "{:.0f}")): q = quartiles(r.get(name) for r in rs) if q: cells = [pct(x) if fm is None else f(x, fm) for x in q] @@ -404,6 +435,30 @@ def build(run_dir, baseline=None, allow_incomplete=False): "the deposited structure factors, both on the depositor's free set with the same protocol, " "one point per set, sorted; below 1 = our data fit the model better") + dep = [r for r in res if r.get("dep_status") == "ok"] + if dep: + doc.h(3, "Against the depositor's data") + doc.p("Per resolution shell, the rank correlation of our merged intensities with |Fc|^2 minus that of the depositor's own " + "data, on the reflections both carry (d < 4 A, eight shells of equal count), |Fc| from " + "the deposited model as it is (no bulk solvent, no refinement; depdata_check.py). The " + "model was refined against the depositor's data, so zero is already a good result: " + "across a corpus the median sits near -0.01. 'Past the limit' is CC(I, |Fc|^2) of our " + "reflections in the outermost of up to three shells beyond the deposited data's limit, " + "which the model never saw: clearly above zero there is signal. Reported, never scored.") + rows = [] + for r in sorted(dep, key=lambda r: r["dep_cc_delta_outer"]): + rows.append([r["set"], r["dep_kind"], f(r.get("d_min")), f(r.get("dep_d_min")), + r["dep_n_common"], f(r["dep_cc_delta_all"], "{:+.3f}"), + f(r["dep_cc_delta_outer"], "{:+.3f}"), f(r.get("dep_beyond_cc"), "{:.3f}"), + f(r.get("dep_beyond_d")), f(r.get("refmac_rfree_ratio"), "{:.3f}"), + f(r.get("isa"), "{:.1f}"), ", ".join(r.get("tags", []))]) + doc.table(["set", "dep. data", "our d_min", "dep. d_min", "common refl.", "dCC all", + "dCC outer 2", "CC past limit", "to d A", "REFMAC ratio", "ISa", "tags"], + rows, num=range(2, 11)) + other = [r for r in res if r.get("dep_status") not in (None, "ok")] + if other: + doc.p("No comparison: " + "; ".join(f"{r['set']} ({r['dep_reason']})" for r in other)) + xds = [r for r in res if r.get("refres_range")] if xds: doc.h(3, "Like for like with XDS: the reference-range table") @@ -484,6 +539,7 @@ def build(run_dir, baseline=None, allow_incomplete=False): ("R_free ratio", "rfree_ratio")] if any(r.get("rfree") for r in res) else [] model += [("REFMAC R_free", "refmac_rfree_depflags"), ("REFMAC dep data", "refmac_rfree_depdata"), ("REFMAC ratio", "refmac_rfree_ratio")] if any(r.get("refmac_status") for r in res) else [] + model += [("dCC dep. outer", "dep_cc_delta_outer")] if any(r.get("dep_status") for r in res) else [] head = ["set", "arm", "verdict", "space group", "ref", "cell dev %", "V ratio", "d_min", "ref d_min", "gain", "compl %", "mult", "R_meas", "low-res R_meas", "CC1/2", "ISa", "ref ISa", "idx"] + \ [h for h, _ in model] + (["model fit"] if model else []) + ["time s", "note"] From beb97b62d3e2d425146f814b9bfb9b0a30425aa7 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 23 Sep 2026 18:50:47 +0200 Subject: [PATCH 02/21] rugnux: mask defective pixels measured on the pre-scan's own frames Unmasked persistent hot pixels are integrated into whichever reflection's box they fall in; under rotation one pixel collects a different reflection on every frame that reaches it, and the merge carries intensities hundreds to thousands of times their shell mean on one or two observations. HotPixelFinder (rugnux/HotPixels.{h,cpp}) reads the pre-scan sample a second time, once the beam-stop projection has measured where the background puts the beam, and on each frame calls a pixel lit when it exceeds its 2 px iso-2theta ring level (max of the ring and 1/16-sector medians) by 3.3 sigma (sqrt(level) or the ring's MAD) + 2. A pixel lit on at least max(k1, kB) frames is persistent: k1 = 1 + ceil((osc + 5 deg)/|zeta| / frame spacing) is more than one reflection can light, kB the binomial bound (0.01 family-wise over the detector) from the ring's own lit rate. A persistent pixel is masked, as the new PixelMask bit 10, only if it stands alone (component of persistent pixels <= 2), reads on average >= 10x its ring and its mean excess is above the Poisson bound; pixels holding the error value on most frames are masked with them. Counting sensors (thickness > 0) and rotation data only; a CCD is left alone. One log line reports the counts. Drawing the rings about the file's centre, as a first version did, masked pixels along the background fall-off on a sweep whose file centre is 171 px from the background's and cost it 14% ISa; about the measured centre that sweep is within 1%. Masking the detector's outermost row and column unconditionally was tried and dropped: the persistence test already catches the hot pixels there, and the whole lines bought nothing measurable. Numbers below are from the looser first criterion (no isolation / 10x gate), against rc173-final on the same base: merged reflections > 30x their shell mean gone on the sets with proven hot pixels (7brr 21 -> 0, 9ih9 15 -> 0, 8xte 10 -> 0, 6z8o worst 1706x -> 40x); 6z8o CC1/2 0.50 -> 0.995, ISa 8.6 -> 12.8, CC to model 0.82 -> 0.90; 8xte ISa 8.0 -> 10.8; 6u7g ISa 9.8 -> 12.0; controls (lyso_x06da_ref, marCCD) unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- common/PixelMask.cpp | 8 ++ common/PixelMask.h | 4 + docs/CPU_DATA_ANALYSIS_IMAGE.md | 27 ++++ docs/PIXEL_MASK.md | 4 + docs/RUGNUX_OVERVIEW.md | 3 +- rugnux/CMakeLists.txt | 2 + rugnux/HotPixels.cpp | 231 ++++++++++++++++++++++++++++++++ rugnux/HotPixels.h | 111 +++++++++++++++ rugnux/Rugnux.cpp | 73 +++++++++- rugnux/Rugnux.h | 7 + tests/CMakeLists.txt | 1 + tests/HotPixelFinderTest.cpp | 65 +++++++++ 12 files changed, 533 insertions(+), 3 deletions(-) create mode 100644 rugnux/HotPixels.cpp create mode 100644 rugnux/HotPixels.h create mode 100644 tests/HotPixelFinderTest.cpp diff --git a/common/PixelMask.cpp b/common/PixelMask.cpp index c59ef2dc2..a5338f4ee 100644 --- a/common/PixelMask.cpp +++ b/common/PixelMask.cpp @@ -247,6 +247,14 @@ void PixelMask::LoadBeamStopMask(const DiffractionExperiment& experiment, const UpdateDerived(experiment); } +void PixelMask::LoadHotPixelMask(const DiffractionExperiment& experiment, const std::vector &in_mask) { + if (in_mask.size() != mask.size()) + throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, + "Size of input hot pixel mask invalid"); + LoadMask(in_mask, HotPixelBit); + UpdateDerived(experiment); +} + void PixelMask::ClearBeamStopMask(const DiffractionExperiment& experiment) { for (auto &i: mask) i &= ~(1u << BeamStopPixelBit); diff --git a/common/PixelMask.h b/common/PixelMask.h index eb7b2534d..5408b0789 100644 --- a/common/PixelMask.h +++ b/common/PixelMask.h @@ -41,6 +41,9 @@ public: constexpr static const uint8_t NoisyPixelBit = 4; constexpr static const uint8_t UserMaskedPixelBit = 8; constexpr static const uint8_t BeamStopPixelBit = 9; + // Found bad on the run's own frames (rugnux pre-scan, HotPixelFinder): lit above its ring on too + // many frames, or holding the error value on most of them. + constexpr static const uint8_t HotPixelBit = 10; constexpr static const uint8_t ChipGapPixelBit = 31; constexpr static const uint8_t ModuleEdgePixelBit = 30; @@ -56,6 +59,7 @@ public: // The beam-stop shadow belongs to the run that found it, not to the dataset, so a mask read back // from a file that carries one starts clear. The user mask (bit 8) is deliberately left alone. void ClearBeamStopMask(const DiffractionExperiment& experiment); + void LoadHotPixelMask(const DiffractionExperiment& experiment, const std::vector& mask); void LoadDECTRISBadPixelMask(const std::vector& mask); void LoadDarkBadPixelMask(const DiffractionExperiment& experiment, const std::vector& mask); void LoadDetectorBadPixelMask(const DiffractionExperiment& experiment, const JFCalibration *calib); diff --git a/docs/CPU_DATA_ANALYSIS_IMAGE.md b/docs/CPU_DATA_ANALYSIS_IMAGE.md index 7e1ce00d9..4175042f4 100644 --- a/docs/CPU_DATA_ANALYSIS_IMAGE.md +++ b/docs/CPU_DATA_ANALYSIS_IMAGE.md @@ -334,6 +334,33 @@ reflection — a maximum over the frames well above background, in a cluster, so zinger does not count — is given back, because a beam stop cannot have blocked a reflection that was measured. +### 1.6 Defective pixels + +A pixel that reads high frame after frame, whatever the crystal does, is integrated into whichever +reflection's box it falls in; under rotation that is one pixel collecting a different reflection on +every frame that reaches it, and a merged intensity hundreds or thousands of times its shell mean, +carried by one or two observations the merge's outlier test cannot judge. The file's mask misses +such pixels on many detectors, so on rotation data from a counting sensor Rugnux measures them on +the pre-scan's own sample of frames (`HotPixelFinder`, `rugnux/HotPixels.h`) and masks them as bit 10. +The frames are read a second time for it, once the beam-stop projection has measured where the +scattered background puts the beam: the rings have to be drawn about the true beam, and a file's +centre can be far enough out that a ring crosses the background's radial fall-off. + +On each frame a pixel is **lit** when it exceeds $b + 3.3\,\max(\sqrt{b}, 1.4826\,\mathrm{MAD}) + 2$, +with $b$ the larger of the median of its 2 px iso-$2\theta$ ring and of that ring's sixteenth in +azimuth, so ice and powder rings, the polarization dip and partial shadows set their own level. It +is **persistent** when it is lit on at least $\max(k_1, k_B)$ of the sampled frames: one reflection +stays on a pixel for $(\Delta\phi + w)/|\zeta|$ of rotation ($w = 5°$, a generous rocking width), +which covers fewer than $k_1 = 1 + \lceil (\Delta\phi + w)/(|\zeta|\,\delta) \rceil$ frames sampled +$\delta$ apart; and $k_B$ is the binomial bound, from the lit rate of the ring's other pixels, that +fewer than 0.01 pixels of the whole detector reach by chance. A persistent pixel is masked if it +stands alone (by itself or in a pair — a larger patch is a feature of the scattering, not a +defect), reads on average at least ten times its ring, and its mean excess is above the Poisson +bound; a weaker one cannot make an outlier. Pixels holding the detector's error +value on most frames (already invalid on every frame, but not in the static mask) are masked with +them. A CCD (no sensor depth: read with an offset, not Poisson) is left alone. One log line says how +many pixels were masked and why. + --- ## 2. Azimuthal integration (radial profiles) diff --git a/docs/PIXEL_MASK.md b/docs/PIXEL_MASK.md index e1e6fae5c..22f5200a4 100644 --- a/docs/PIXEL_MASK.md +++ b/docs/PIXEL_MASK.md @@ -21,6 +21,10 @@ Unlike the other bits this one belongs to the run that found it, not to the data at the start of every run, so a mask read back from a file that carries one starts clear. The user mask (bit 8) is left alone. +Bit 10 - defective pixel found on the run's own frames (rotation data from a counting sensor; see +[CPU data analysis §1.6](CPU_DATA_ANALYSIS_IMAGE.md)): lit above its resolution ring on more frames +than one reflection or chance explains, or holding the detector's error value on most frames. + Bit 30 - module edge (only for PSI systems) Bit 31 - chip edge interpolated pixel (multipixel) diff --git a/docs/RUGNUX_OVERVIEW.md b/docs/RUGNUX_OVERVIEW.md index 902a24bee..87274be27 100644 --- a/docs/RUGNUX_OVERVIEW.md +++ b/docs/RUGNUX_OVERVIEW.md @@ -9,7 +9,8 @@ rotation run with the defaults; stills differences are at the end. serial stills — nothing is asked of the user. **Pre-scan.** A projection of the first frames (60 by default) finds the beam-stop shadow and masks -it ([§1.5](CPU_DATA_ANALYSIS_IMAGE.md)), measures the beam centre from the isotropy of the +it ([§1.5](CPU_DATA_ANALYSIS_IMAGE.md)), masks pixels its frames show to be defective +([§1.6](CPU_DATA_ANALYSIS_IMAGE.md)), measures the beam centre from the isotropy of the scattered background and compares it with the file's ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)), and reads how wide this crystal's spots are, which sets the integration radius ([§9.5](CPU_DATA_ANALYSIS_INTEGRATION.md)). diff --git a/rugnux/CMakeLists.txt b/rugnux/CMakeLists.txt index a496e1088..0f7b1cac8 100644 --- a/rugnux/CMakeLists.txt +++ b/rugnux/CMakeLists.txt @@ -23,6 +23,8 @@ ADD_LIBRARY(Rugnux STATIC ReportDocument.h SpotWidth.cpp SpotWidth.h + HotPixels.cpp + HotPixels.h DiagnosticOutput.cpp DiagnosticOutput.h SpindleCuspLoss.cpp diff --git a/rugnux/HotPixels.cpp b/rugnux/HotPixels.cpp new file mode 100644 index 000000000..d8eb8b67d --- /dev/null +++ b/rugnux/HotPixels.cpp @@ -0,0 +1,231 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "HotPixels.h" + +#include +#include +#include +#include + +#include "../common/JFJochMath.h" +#include "../common/ParallelFor.h" + +namespace { + +// The lower median of v[0..n), reordering it. +int32_t median_of(int32_t *v, size_t n) { + std::nth_element(v, v + n / 2, v + n); + return v[n / 2]; +} + +// The smallest k with npix * P(Binomial(n, q) >= k) below the family-wise rate, n + 1 if none. +int binomial_bound(int n, double q, double npix, double rate) { + double tail = 0.0; + int k = n + 1; + for (int j = n; j >= 0; j--) { + tail += std::exp(std::lgamma(n + 1.0) - std::lgamma(j + 1.0) - std::lgamma(n - j + 1.0) + + j * std::log(q) + (n - j) * std::log1p(-q)); + if (npix * tail >= rate) break; + k = j; + } + return k; +} + +} // namespace + +HotPixelFinder::HotPixelFinder(const DiffractionExperiment &experiment, const PixelMask &mask, size_t nthreads) + : geometry(experiment.GetDiffractionGeometry()), + axis(experiment.GetGoniometer() ? experiment.GetGoniometer()->GetAxis().Normalize() : Coord(0, 0, 0)), + width(experiment.GetXPixelsNumConv()), height(experiment.GetYPixelsNumConv()), + key(width * height, -1), + n_valid(width * height, 0), n_lit(width * height, 0), n_error(width * height, 0), + sum_value(width * height, 0), sum_level(width * height, 0) { + const auto &m = mask.GetMask(); + // Rings of equal 2theta, RING_WIDTH_PX wide where the beam meets the detector. + const float ring_rad = RING_WIDTH_PX * geometry.GetPixelSize_mm() / geometry.GetDetectorDistance_mm(); + ParallelChunks(static_cast(height), nthreads, [&](int y0, int y1) { + for (size_t y = y0; y < static_cast(y1); y++) + for (size_t x = 0; x < width; x++) { + const size_t i = y * width + x; + if (m[i] != 0) continue; + const float fx = static_cast(x), fy = static_cast(y); + const int ring = static_cast(geometry.TwoTheta_rad(fx, fy) / ring_rad); + const int sector = std::min(SECTORS - 1, static_cast(geometry.Phi_rad(fx, fy) + / static_cast(2.0 * PI) * SECTORS)); + key[i] = ring * SECTORS + sector; + } + }); + for (const int32_t k : key) + if (k >= 0) { + nrings = std::max(nrings, k / SECTORS + 1); + unmasked++; + } + + key_begin.assign(static_cast(nrings) * SECTORS + 1, 0); + for (const int32_t k : key) + if (k >= 0) key_begin[k + 1]++; + for (size_t k = 1; k < key_begin.size(); k++) + key_begin[k] += key_begin[k - 1]; +} + +void HotPixelFinder::AddImage(const int32_t *image, std::vector &scratch) { + const size_t npix = width * height; + const size_t nkeys = static_cast(nrings) * SECTORS; + + // The frame's valid counts laid out by ring and sector. A saturated pixel has no count to add. + scratch.resize(npix); + std::vector count(nkeys, 0); + for (size_t i = 0; i < npix; i++) { + const int32_t k = key[i], v = image[i]; + if (k < 0 || v == INT32_MIN || v == INT32_MAX) continue; + scratch[key_begin[k] + count[k]++] = v; + } + + std::vector sector_level(nkeys, 0); + for (size_t k = 0; k < nkeys; k++) + if (count[k] >= MIN_SECTOR_PIXELS) + sector_level[k] = median_of(scratch.data() + key_begin[k], count[k]); + + // Each ring's sectors packed together - forward into the space the ring owns, so nothing is + // overwritten before it is moved - then its median and its robust spread about that median. + std::vector ring_level(nrings, 0); + std::vector ring_spread(nrings, 0.0f); + std::vector ring_ok(nrings, 0); + for (int r = 0; r < nrings; r++) { + int32_t *ring = scratch.data() + key_begin[r * SECTORS]; + size_t n = 0; + for (int s = 0; s < SECTORS; s++) { + const size_t k = r * SECTORS + s; + std::memmove(ring + n, scratch.data() + key_begin[k], count[k] * sizeof(int32_t)); + n += count[k]; + } + if (n < MIN_RING_PIXELS) continue; + ring_ok[r] = 1; + ring_level[r] = median_of(ring, n); + for (size_t j = 0; j < n; j++) + ring[j] = std::abs(ring[j] - ring_level[r]); + ring_spread[r] = 1.4826f * static_cast(median_of(ring, n)); + } + + std::vector level(nkeys, 0); + std::vector threshold(nkeys, 0.0f); + for (size_t k = 0; k < nkeys; k++) { + const int r = static_cast(k / SECTORS); + level[k] = std::max(ring_level[r], sector_level[k]); + const float noise = std::max(std::sqrt(static_cast(std::max(level[k], 0))), ring_spread[r]); + threshold[k] = static_cast(level[k]) + LIT_NSIGMA * noise + LIT_OFFSET; + } + + // Every sum is an integer, so the result does not depend on the order the frames arrive in. + { + std::lock_guard lock(m); + frames++; + } + const size_t rows_per_band = (height + BANDS - 1) / BANDS; + const size_t first = next_band.fetch_add(1); + for (size_t b = 0; b < BANDS; b++) { + const size_t band = (first + b) % BANDS; + const size_t begin = std::min(npix, band * rows_per_band * width); + const size_t end = std::min(npix, (band + 1) * rows_per_band * width); + std::lock_guard lock(band_mutex[band]); + for (size_t i = begin; i < end; i++) { + const int32_t k = key[i], v = image[i]; + if (k < 0) continue; + if (v == INT32_MIN) { + n_error[i]++; + continue; + } + if (!ring_ok[k / SECTORS]) continue; + n_valid[i]++; + sum_value[i] += v; + sum_level[i] += level[k]; + if (v == INT32_MAX || static_cast(v) > threshold[k]) + n_lit[i]++; + } + } +} + +HotPixelFinder::Result HotPixelFinder::GetMask(double oscillation_deg, double spacing_deg, + size_t nthreads) const { + std::lock_guard lock(m); + Result ret; + ret.frames = frames; + ret.mask.assign(width * height, 0); + const int n = static_cast(frames); + + // The chance rate per ring, from the pixels lit on no more than half of their frames: whatever + // lights those - reflections, zingers, noise above the bound - lights a defect-free pixel too. + std::vector lit(nrings, 0.0), seen(nrings, 0.0); + for (size_t i = 0; i < key.size(); i++) + if (key[i] >= 0 && n_valid[i] > 0 && 2 * n_lit[i] <= n_valid[i]) { + lit[key[i] / SECTORS] += n_lit[i]; + seen[key[i] / SECTORS] += n_valid[i]; + } + std::vector k_chance(nrings, n + 1); + for (int r = 0; r < nrings; r++) + if (seen[r] > 0.0) + k_chance[r] = binomial_bound(n, std::clamp(lit[r] / seen[r], 1e-6, 0.999), + static_cast(unmasked), FAMILY_WISE_RATE); + + // Persistent: lit on more frames than one reflection or chance explains. + const int min_valid = std::max(10, n / 2); + std::vector persistent(width * height, 0); + ParallelChunks(static_cast(height), nthreads, [&](int y0, int y1) { + for (size_t y = y0; y < static_cast(y1); y++) + for (size_t x = 0; x < width; x++) { + const size_t i = y * width + x; + if (key[i] < 0 || n_valid[i] < min_valid || n_lit[i] == 0 || !(spacing_deg > 0.0)) continue; + // One reflection's stay on this pixel, in sampled frames: 1 / |zeta| turns the rocking + // width into rotation, and zeta = |axis . (s1 x s0)| with s0 along z. + const Coord s1 = geometry.LabCoord(static_cast(x), static_cast(y)).Normalize(); + const double zeta = std::max(1e-4, static_cast(std::fabs(axis * (s1 % Coord(0, 0, 1))))); + const double stay = (oscillation_deg + ROCKING_WIDTH_DEG) / zeta; + const double k_bragg = 1.0 + std::ceil(stay / spacing_deg); + const double k = std::min(std::max(k_bragg, k_chance[key[i] / SECTORS]), + n_valid[i]); + persistent[i] = n_lit[i] >= k; + } + }); + + // Masked: a persistent pixel standing alone - on its own or as one of a pair; a larger patch is + // a feature of the scattering, not a defect - whose mean is STRONG_RATIO times its ring's and + // whose excess is above the Poisson bound. A weaker one cannot make an outlier. + const auto persistent_neighbours = [&](size_t x, size_t y) { + int count = 0; + for (size_t yy = (y > 0 ? y - 1 : 0); yy <= std::min(height - 1, y + 1); yy++) + for (size_t xx = (x > 0 ? x - 1 : 0); xx <= std::min(width - 1, x + 1); xx++) + if ((xx != x || yy != y) && persistent[yy * width + xx]) count++; + return count; + }; + ParallelChunks(static_cast(height), nthreads, [&](int y0, int y1) { + for (size_t y = y0; y < static_cast(y1); y++) + for (size_t x = 0; x < width; x++) { + const size_t i = y * width + x; + if (key[i] < 0) continue; + if (2 * n_error[i] > n) { + ret.mask[i] = 2; + continue; + } + if (!persistent[i]) continue; + const int neighbours = persistent_neighbours(x, y); + bool isolated = neighbours == 0; + if (neighbours == 1) + for (size_t yy = (y > 0 ? y - 1 : 0); yy <= std::min(height - 1, y + 1); yy++) + for (size_t xx = (x > 0 ? x - 1 : 0); xx <= std::min(width - 1, x + 1); xx++) + if ((xx != x || yy != y) && persistent[yy * width + xx]) + isolated = persistent_neighbours(xx, yy) == 1; + const double mean_level = static_cast(sum_level[i]) / n_valid[i]; + const double excess = static_cast(sum_value[i] - sum_level[i]) / n_valid[i]; + if (isolated + && static_cast(sum_value[i]) >= STRONG_RATIO * static_cast(sum_level[i]) + && excess >= LIT_NSIGMA * std::sqrt(std::max(mean_level, 0.0)) + LIT_OFFSET) + ret.mask[i] = 1; + } + }); + for (const uint32_t v : ret.mask) { + ret.hot += v == 1; + ret.error += v == 2; + } + return ret; +} diff --git a/rugnux/HotPixels.h b/rugnux/HotPixels.h new file mode 100644 index 000000000..03dfc64e0 --- /dev/null +++ b/rugnux/HotPixels.h @@ -0,0 +1,111 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +// ============================================================================= +// HotPixelFinder - pixels the pre-scan's own frames show to be defective +// ============================================================================= +// +// A pixel that reads high on frame after frame, whatever the crystal is doing, is not recording +// diffraction. Unmasked, it is integrated as part of whichever reflection's box it falls in, and +// under rotation that is one pixel at one resolution collecting a different reflection on every +// frame that reaches it - a merged intensity hundreds or thousands of times its shell mean, +// supported by one or two observations the merge's outlier test cannot judge. The file's own +// mask misses such pixels on many detectors, and which pixels are hot moves with the threshold +// setting, so they are measured per dataset, on the pre-scan's sample of frames. +// +// On each frame a pixel is LIT when it is Poisson-significantly above the level of its own +// resolution ring - the larger of the median of the 2 px iso-2theta ring and of the ring's +// sixteenth in azimuth, so an ice or powder ring, the polarization dip and a partly shadowed ring +// all set their own level. It is PERSISTENT when it is lit on more of the sampled frames than +// either explanation that is not a defect allows: +// +// * one Bragg reflection: it stays on a pixel for (oscillation + rocking width) / |zeta| of +// rotation, i.e. on at most 1 + ceil(that / frame spacing) of frames sampled that far apart; +// * chance: independent reflections and noise light a pixel of this ring on a fraction q of the +// frames (measured, on the ring's pixels that are not candidates), and the number of lit frames +// that fewer than 0.01 of all the detector's pixels would reach by chance is the binomial bound. +// +// A persistent pixel is masked only if it stands alone (by itself or in a pair - a larger patch is a +// feature of the scattering, not a defect), reads on average at least ten times its ring, and its +// mean excess is above the Poisson bound: a weaker one cannot make an outlier. A pixel holding the +// detector's error value on most frames is masked with it: every frame already treats it as invalid, +// but the static mask did not know. The detector's outermost row and column get no rule of their +// own - a hot pixel there is caught like any other. +// +// None of this holds for a CCD, whose pixels carry a read-out offset and are not Poisson; the caller +// runs it for counting sensors only. +// ============================================================================= + +#include +#include +#include +#include + +#include "../common/DiffractionExperiment.h" +#include "../common/PixelMask.h" + +class HotPixelFinder { +public: + // Sixteen sectors of the ring; the level is the larger of the two medians. + static constexpr int SECTORS = 16; + // Ring width in 2theta: this many pixels where the beam meets the detector, wider further out. + static constexpr float RING_WIDTH_PX = 2.0f; + // A ring needs this many valid pixels to have a level, a sector this many to have one of its own. + static constexpr int MIN_RING_PIXELS = 16; + static constexpr int MIN_SECTOR_PIXELS = 8; + // Lit: above level + LIT_NSIGMA * noise + LIT_OFFSET (noise = the larger of sqrt(level) and the + // ring's robust spread; the offset keeps a pixel in a ring with no background from being lit by + // one or two photons). + static constexpr float LIT_NSIGMA = 3.3f; + static constexpr float LIT_OFFSET = 2.0f; + // How long one reflection can stay on a pixel, degrees of rotation beyond the oscillation: +-3 + // sigma of a rocking curve 2 degrees FWHM, wider than the crystals this is meant for. + static constexpr float ROCKING_WIDTH_DEG = 5.0f; + // Pixels expected to be called persistent by chance on the whole detector. + static constexpr double FAMILY_WISE_RATE = 0.01; + // A persistent pixel is masked only where it reads this many times its ring on average: the + // strength at which the pixels behind the merged outliers were found, and well clear of a pixel + // that is merely a little over-responding. + static constexpr double STRONG_RATIO = 10.0; + + struct Result { + std::vector mask; // non-zero = masked (1 hot, 2 error value), converted geometry + size_t hot = 0; // persistent above the ring + size_t error = 0; // holding the error value on most frames + uint32_t frames = 0; + }; + + HotPixelFinder(const DiffractionExperiment &experiment, const PixelMask &mask, size_t nthreads); + + // One preprocessed frame (INT32_MIN = masked or error value, INT32_MAX = saturated). Thread + // safe; `scratch` is the caller's own, reused across its calls. + void AddImage(const int32_t *image, std::vector &scratch); + + // The mask, from the frames added so far. oscillation_deg is the rotation per image and + // spacing_deg the rotation between two consecutive sampled frames. + [[nodiscard]] Result GetMask(double oscillation_deg, double spacing_deg, size_t nthreads) const; + +private: + const DiffractionGeometry geometry; + const Coord axis; // rotation axis; zero where there is none + const size_t width, height; + int nrings = 0; + + // Ring * SECTORS + sector of every pixel, -1 where the pixel is already masked. + std::vector key; + // Where each ring-sector's pixels start in a scratch buffer laid out by key. + std::vector key_begin; + size_t unmasked = 0; + + // The per-pixel sums are guarded by bands of rows, so workers adding frames at the same time + // meet only when they reach the same band. + static constexpr size_t BANDS = 64; + mutable std::mutex band_mutex[BANDS]; + std::atomic next_band{0}; + mutable std::mutex m; + uint32_t frames = 0; + std::vector n_valid, n_lit, n_error; + std::vector sum_value, sum_level; +}; diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index c4ce5ed0e..cd419d874 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -7,6 +7,7 @@ #include "WriteModel.h" #include "SpindleCuspLoss.h" #include "SpotWidth.h" +#include "HotPixels.h" #include "DiagnosticOutput.h" #include @@ -820,7 +821,15 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru && config_.two_pass_rotation && experiment_.IsRotationIndexing(); const bool want_width = config_.adaptive_integration_radius && !spot_width_measured_ && !second_pass_follows; - if (!want_shadow && !want_beam_center && !want_width) + // Defective pixels, for the same reason on the same pass: each pixel is compared against its + // resolution ring. Only where the counts are photons from a counting sensor - a CCD (no sensor + // depth) is read with an offset and is not Poisson - and only under rotation, where a reflection + // moves off a pixel and a defect does not. + const auto goniometer = experiment_.GetGoniometer(); + const bool want_hot = !hot_pixels_measured_ && !second_pass_follows + && experiment_.IsRotationIndexing() && goniometer && goniometer->IsScanning() + && experiment_.GetDetectorSetup().GetSensorThickness_um() > 0.0f; + if (!want_shadow && !want_beam_center && !want_width && !want_hot) return; // The two consumers want different frames and each gets its own set. The shadow is built from @@ -833,7 +842,6 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru // [0, images_to_process) always picks the very first and the very last image, and those are the // ones a shutter out of step spoils - the last carries under 90 % of the sample's median counts // on 12 of 38 regression sweeps and is essentially unexposed on 5. - const auto goniometer = experiment_.GetGoniometer(); const int margin = std::min(PRESCAN_END_MARGIN_IMAGES, images_to_process / 10); auto shadow_sample = select_equally_spaced_image_ordinals(images_to_process - 2 * margin, frame_count); @@ -1174,6 +1182,8 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru return; } + // Where the scattered background puts the beam, when the projection exists to say so. + std::optional> ring_centre; if (want_shadow) { // Every pixel is compared against the ring it sits on, so the rings have to be drawn about // the beam. A centre far enough out draws them across the background's own radial fall-off @@ -1189,6 +1199,7 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru if (const auto ring_center = FindBeamCenterFromBackground(experiment_, pixel_mask_, projection)) { finder.BeamCenter(ring_center->beam_x_pxl, ring_center->beam_y_pxl); + ring_centre = std::make_pair(ring_center->beam_x_pxl, ring_center->beam_y_pxl); logger.Info("Beam stop: comparing each pixel against its ring about the measured " "centre ({:.2f},{:.2f}) +- {:.2f} px, {:.2f} px from the file's", ring_center->beam_x_pxl, ring_center->beam_y_pxl, ring_center->sigma_pxl, @@ -1240,6 +1251,9 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru // projecting every frame for this alone is the largest per-frame cost of the pre-scan. On the // second pass of a two-pass run the shadow is already in the mask, so this does not run again - // by then the centre has been post-refined and the file's value is no longer what is in it. + if (want_hot) + MaskDefectivePixels(start_image, shadow_sample, ring_centre); + if (want_shadow && config_.beam_center_check) { const auto t0 = std::chrono::steady_clock::now(); BeamCenterFFTResult capture; @@ -1479,6 +1493,61 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru experiment_.BeamX_pxl(estimate->beam_x_pxl).BeamY_pxl(estimate->beam_y_pxl); } +void Rugnux::MaskDefectivePixels(int start_image, const std::vector &sample, + const std::optional> &ring_centre) { + Logger logger("Rugnux"); + hot_pixels_measured_ = true; + + // The frames are read a second time, because the rings each pixel is compared against have to + // be drawn about the beam, and the centre the file states can be far enough out that a ring + // crosses the background's own radial fall-off - the pixels on its bright side then read high on + // every frame. The centre the beam-stop projection has just measured does not exist until every + // frame of the first read is in. + DiffractionExperiment about = experiment_; + if (ring_centre) + about.BeamX_pxl(ring_centre->first).BeamY_pxl(ring_centre->second); + const size_t nthreads = static_cast(std::max(config_.nthreads, 1)); + HotPixelFinder finder(about, pixel_mask_, nthreads); + + std::atomic next{0}; + std::vector> futures; + const size_t nworkers = std::min(nthreads, std::min(PRESCAN_MAX_WORKERS, sample.size())); + for (size_t t = 0; t < nworkers; t++) + futures.emplace_back(std::async(std::launch::async, [&] { + ImagePreprocessorCPU preprocessor(experiment_, pixel_mask_); + ImagePreprocessorBuffer image(experiment_.GetPixelsNum()); + JFJochReaderRawImage raw; + std::vector buffer; + std::vector scratch; + for (size_t i = next.fetch_add(1); i < sample.size(); i = next.fetch_add(1)) { + const int image_idx = start_image + sample[i] * config_.stride; + try { + if (!reader_.ReadRawImage(image_idx, raw)) continue; + preprocessor.Analyze(image, raw.image.GetUncompressedPtr(buffer), raw.image.GetMode()); + } catch (const std::exception &e) { + if (IsFatalResourceError(e)) throw; + logger.Warning("Defective pixels: failed to read image {}: {}", image_idx, e.what()); + continue; + } + finder.AddImage(image.data(), scratch); + } + })); + for (auto &f : futures) f.get(); + + const auto goniometer = experiment_.GetGoniometer(); + const double oscillation = std::abs(goniometer->GetIncrement_deg()); + const double spacing = sample.size() > 1 + ? oscillation * (sample.back() - sample.front()) / static_cast(sample.size() - 1) + : 0.0; + const auto hot = finder.GetMask(oscillation, spacing, nthreads); + if (hot.frames == 0) + return; + pixel_mask_.LoadHotPixelMask(experiment_, hot.mask); + logger.Info("Defective pixels: {} masked from {} pre-scan frames - {} lit above their ring on more " + "frames than one reflection or chance explains, {} holding the detector's error value", + hot.hot + hot.error, hot.frames, hot.hot, hot.error); +} + void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_process, RugnuxObserver *observer) { Logger logger("Rugnux"); diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 1ed2bee04..c74278625 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -554,6 +554,9 @@ class Rugnux { // The recorded spot width the pre-scan measured (config_.adaptive_integration_radius), kept so the // two-pass rotation run measures it once and both passes integrate at the same radius. bool spot_width_measured_ = false; + // The pixels the pre-scan's own frames show to be defective (HotPixelFinder) are measured once, + // on the pre-scan that also measures the width, and stay in pixel_mask_ for every pass after it. + bool hot_pixels_measured_ = false; // The integration radii in force before the measured spot width widened them, so a widening that // turns out to cost data can be given back exactly rather than to a written-down default. std::optional bragg_before_adaptive_; @@ -600,6 +603,10 @@ class Rugnux { // width that sets the integration radius (config_.adaptive_integration_radius). Any of the three // may be asked for without the others. void PreScan(int start_image, int images_to_process, int frame_count, RugnuxObserver *observer); + // The pre-scan's defective-pixel mask (HotPixelFinder), read on `sample` with the rings drawn + // about `ring_centre` where the background measured one. + void MaskDefectivePixels(int start_image, const std::vector &sample, + const std::optional> &ring_centre); // Powder calibration, second pass: re-integrate the same images into a profile binned with the // geometry the first fit produced, and fit that. Nothing but the binning changes, which is the // point - which pixel went into which (q, phi) bin was decided by the geometry the run started diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index c0e10383c..2992aa624 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -52,6 +52,7 @@ ADD_EXECUTABLE(jfjoch_test JFJochReaderTest.cpp SweepLayoutTest.cpp ShadowFinderTest.cpp + HotPixelFinderTest.cpp ParallelForTest.cpp RugnuxTest.cpp ResultReportTest.cpp diff --git a/tests/HotPixelFinderTest.cpp b/tests/HotPixelFinderTest.cpp new file mode 100644 index 000000000..6fc72fcb1 --- /dev/null +++ b/tests/HotPixelFinderTest.cpp @@ -0,0 +1,65 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include + +#include +#include +#include + +#include "../common/DetectorSetup.h" +#include "../common/DiffractionExperiment.h" +#include "../common/PixelMask.h" +#include "../rugnux/HotPixels.h" + +namespace { + constexpr int W = 257, H = 257, C = 128; + constexpr int NFRAMES = 60; + constexpr double OSC_DEG = 0.1, SPACING_DEG = 6.0; // 60 frames spread over a full turn + + constexpr size_t I(int x, int y) { return static_cast(y) * W + x; } +} + +// A Poisson background with a powder ring, and on it four kinds of pixel: one that reads 60 counts +// high on every frame, one only 20 high - persistent, but too weak to make an outlier - one that holds +// the error value on every frame, and one crossed by a genuine reflection. The last sits close to the rotation axis, where one reflection stays on a pixel for a +// long stretch of rotation - here nine consecutive sampled frames, more than the chance bound allows +// but fewer than the one-reflection bound at that zeta. Only the first and the third are masked. +TEST_CASE("HotPixelFinder_PersistentPixelNotBragg", "[HotPixelFinder]") { + DiffractionExperiment x(DetDECTRIS(W, H, "Test detector", "")); + x.IncidentEnergy_keV(WVL_1A_IN_KEV).DetectorDistance_mm(10.0f); + x.BeamX_pxl(static_cast(C)).BeamY_pxl(static_cast(C)); + x.Goniometer(GoniometerAxis("omega", 0.0f, static_cast(OSC_DEG), Coord(1, 0, 0), std::nullopt)); + const PixelMask pixel_mask(x); + HotPixelFinder finder(x, pixel_mask, 4); + + constexpr int HOT_X = 200, HOT_Y = 200, WARM_X = 190, WARM_Y = 60, ERR_X = 60, ERR_Y = 190; + constexpr int BRAGG_X = 40, BRAGG_Y = 115; + std::mt19937 rng(1); + // The powder ring is a smooth radial profile, as a real one is: 45 counts over the background at + // 82 px, 4 px sigma. + std::vector frame(static_cast(W) * H), scratch; + for (int f = 0; f < NFRAMES; f++) { + for (int y = 0; y < H; y++) + for (int x_ = 0; x_ < W; x_++) { + const double r = std::hypot(x_ - C, y - C); + const double mean = 5.0 + 45.0 * std::exp(-0.5 * (r - 82.0) * (r - 82.0) / 16.0); + frame[I(x_, y)] = std::poisson_distribution(mean)(rng); + } + frame[I(HOT_X, HOT_Y)] += 60; + frame[I(WARM_X, WARM_Y)] += 20; + frame[I(ERR_X, ERR_Y)] = INT32_MIN; + if (f >= 20 && f < 29) + frame[I(BRAGG_X, BRAGG_Y)] += 500; + finder.AddImage(frame.data(), scratch); + } + + const auto result = finder.GetMask(OSC_DEG, SPACING_DEG, 4); + CHECK(result.frames == NFRAMES); + CHECK(result.mask[I(HOT_X, HOT_Y)] == 1); // 1 = hot, 2 = error value + CHECK(result.mask[I(ERR_X, ERR_Y)] == 2); + CHECK(result.mask[I(WARM_X, WARM_Y)] == 0); + CHECK(result.mask[I(BRAGG_X, BRAGG_Y)] == 0); + CHECK(result.hot == 1); + CHECK(result.error == 1); +} From 19a850cd20681b6d0a2d0762f4900c74fb45b57c Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 23 Sep 2026 20:42:42 +0200 Subject: [PATCH 03/21] rugnux: measure the X-ray bandwidth from the pre-scan's spot shapes (reported only) Per isolated strong spot of the spot-width sample, second moments along and across its radius, with the exact per-spot Jacobians (pixels per radian of 2theta and of the angle across the scattering plane, from the geometry) and the sensor parallax fixed from physics (conversion depth exponential with length L cos(psi), truncated at the thickness, smeared z tan(psi) along the ray's in-plane direction). y = (m_rad - 1/12 - par_u) - (jr/jt)^2 (m_tan - 1/12 - par_v) is linear in (2 jr tan theta)^2 with slope sigma^2; fitted over eight equal-count 20%-trimmed bins, error from 200 seeded bootstrap re-draws inflated by the reduced chi^2, significant at z > 3. The pre-scan logs the FWHM, its standard error, z and chi2 next to the bandwidth the run uses; nothing consumes it, so output is unchanged. The truncated-exponential depth variance moves into SensorAbsorption.h (ConversionDepthVariance_um2), shared with the integrator's parallax_var_px2 (same arithmetic). Catch2: a synthetic spot population painted with 0.45% FWHM bandwidth and 450 um Si parallax recovers 0.449% (z 82); the same spots without bandwidth give 0.06% at z 0.7. On the battery sets (67d49f base, estimate as logged): MicroMAX pink lyso 0.41% z 7.3, thau 0.42% z 6.9, CHESS 7B2 9q41 0.36% z 14.6, ALS 8.2.1 8u0i 0.27% z 5.6; mono controls lyso_micromax_mono, lyso_x06da_ref, 5mln and near misses 9z44, 7orr, lyso_x06da_5keV all below z 3. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 16 +- docs/RUGNUX_OVERVIEW.md | 4 +- image_analysis/SensorAbsorption.h | 15 ++ .../BraggIntegrationEngine.cpp | 10 +- rugnux/Rugnux.cpp | 20 ++ rugnux/SpotWidth.cpp | 234 ++++++++++++++++++ rugnux/SpotWidth.h | 62 ++++- tests/BandwidthEstimateTest.cpp | 98 ++++++++ tests/CMakeLists.txt | 1 + 9 files changed, 445 insertions(+), 15 deletions(-) create mode 100644 tests/BandwidthEstimateTest.cpp diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 8d7942dad..37d85b306 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -161,7 +161,7 @@ The integrator is selected by `--integrator boxsum|gaussian|empirical` (default ### 9.4 The prescaling correction -The deterministic per-reflection corrections are carried as three multiplicative factors. `prescaling_corr` holds the reciprocal Lorentz factor (rotation only — a still's Lorentz factor is one) times the reciprocal polarization factor from the geometry-based term (§2.2), and nothing else: it is Lorentz x polarization, which is what `LP` means in every format the field reads. Beside it sit the two terms that describe what happened to the photon between leaving the sample and being counted, and that are kept apart from `LP` because detector response and beam/crystal geometry are different things: `qe_corr`, the sensor's angle-dependent efficiency, and `flight_corr`, the medium in the flight path (§9.6). The total deterministic correction on a reflection is the product `prescaling_corr * qe_corr * flight_corr`, and every site that corrects an intensity — the integrator, the scaling fits, the merge ingest, the anisotropy analysis and the unmerged export — multiplies all three. None of them is a scale: the fitted per-image scale and the partiality are separate. The three reach the unmerged MTZ as its `LP`, `QE` and `FLIGHT` columns unchanged (`QE` and `FLIGHT` as divisors normalised to 1 at normal incidence), so raw counts are `I / LP * QE * FLIGHT`. +The deterministic per-reflection corrections are carried as three multiplicative factors. `prescaling_corr` holds the reciprocal Lorentz factor (rotation only — a still's Lorentz factor is one) times the reciprocal polarization factor from the geometry-based term (§2.2), and nothing else: it is Lorentz x polarization, which is what `LP` means in every format the field reads. Beside it sit the two terms that describe what happened to the photon between leaving the sample and being counted, and that are kept apart from `LP` because detector response and beam/crystal geometry are different things: `qe_corr`, the sensor's angle-dependent efficiency, and `flight_corr`, the medium in the flight path (§9.7). The total deterministic correction on a reflection is the product `prescaling_corr * qe_corr * flight_corr`, and every site that corrects an intensity — the integrator, the scaling fits, the merge ingest, the anisotropy analysis and the unmerged export — multiplies all three. None of them is a scale: the fitted per-image scale and the partiality are separate. The three reach the unmerged MTZ as its `LP`, `QE` and `FLIGHT` columns unchanged (`QE` and `FLIGHT` as divisors normalised to 1 at normal incidence), so raw counts are `I / LP * QE * FLIGHT`. One term is deliberately **not** in it. The **per-pixel solid angle**, which the azimuthal profile does divide out (§2.2), is correctly absent here: a Bragg integration sums all the photons in a @@ -195,7 +195,19 @@ Every integration pass, adaptive or not, now logs the radii it used together wit --- -### 9.6 The flight path +### 9.6 Measuring the bandwidth (rotation, reported only) + +A finite energy spread $\sigma$ ($\Delta\lambda/\lambda$, rms) smears a reflection along its own radius by $2\tan\theta\,\sigma$ radians of $2\theta$ and not at all across it. Most files do not state it — a multilayer monochromator is a beamline option, not a header field — so Rugnux reads it off the spots, on the same isolated strong spots the width of §9.5 is measured on (`spot_width::EstimateBandwidth`), and **reports** it in the log; the run itself uses the file's value or `--bandwidth`, and 0 otherwise. For each spot, with $u$ along the radius and $v$ across it, the second moments about its centroid are + +$$m_u = \tfrac1{12} + p_u + j_r^2\big(s^2 + 4\tan^2\theta\,\sigma^2\big),\qquad m_v = \tfrac1{12} + p_v + j_t^2 s^2,$$ + +with $j_r$, $j_t$ the exact pixels per radian of $2\theta$ and of the angle across the scattering plane at that spot, $s$ everything isotropic in angle (divergence, crystal size, mosaic spread), and $p_u$, $p_v$ the sensor parallax, fixed from the sensor's physics: a photon converting at depth $z$ (exponential with length $L\cos\psi$ at angle $\psi$ to the normal, truncated at the thickness) lands $z\tan\psi$ along the ray's in-plane direction. Then + +$$y = (m_u - \tfrac1{12} - p_u) - (j_r/j_t)^2\,(m_v - \tfrac1{12} - p_v) = a + \sigma^2\,(2 j_r\tan\theta)^2$$ + +is a straight line whose slope is the bandwidth. It is fitted over eight equal-count bins of the abscissa, each a 20 %-trimmed mean weighted by its own scatter; the slope's error is the spread of 200 bootstrap re-draws of the spots, inflated by the reduced $\chi^2$ of the binned fit where the line fits worse than the scatter says, and the log calls the estimate significant at $z>3$. It is a **lower bound** — mosaic spread seen along the radius subtracts — and a spread of cell edges is exactly degenerate with it, so what it measures is the effective radial broadening. + +### 9.7 The flight path A reflection leaving the sample at incidence angle $\alpha$ to the detector normal reaches its pixel after $D/\cos\alpha$ of flight rather than $D$, so it crosses more of whatever fills the flight path diff --git a/docs/RUGNUX_OVERVIEW.md b/docs/RUGNUX_OVERVIEW.md index 87274be27..68f8028ba 100644 --- a/docs/RUGNUX_OVERVIEW.md +++ b/docs/RUGNUX_OVERVIEW.md @@ -13,7 +13,9 @@ it ([§1.5](CPU_DATA_ANALYSIS_IMAGE.md)), masks pixels its frames show to be def ([§1.6](CPU_DATA_ANALYSIS_IMAGE.md)), measures the beam centre from the isotropy of the scattered background and compares it with the file's ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)), and reads how wide this crystal's spots are, which sets the integration radius -([§9.5](CPU_DATA_ANALYSIS_INTEGRATION.md)). +([§9.5](CPU_DATA_ANALYSIS_INTEGRATION.md)), and how much longer they are along their radius than +across it, which is the beam's bandwidth - reported in the log, not yet used +([§9.6](CPU_DATA_ANALYSIS_INTEGRATION.md)). **Spots.** Every image is decoded — on the GPU straight from the compressed chunk ([§0](CPU_DATA_ANALYSIS_IMAGE.md)) — and one fused pass computes the azimuthal profile and finds diff --git a/image_analysis/SensorAbsorption.h b/image_analysis/SensorAbsorption.h index 378d0105e..7ff580b7b 100644 --- a/image_analysis/SensorAbsorption.h +++ b/image_analysis/SensorAbsorption.h @@ -4,6 +4,7 @@ #ifndef JUNGFRAUJOCH_SENSORABSORPTION_H #define JUNGFRAUJOCH_SENSORABSORPTION_H +#include #include #include @@ -123,6 +124,20 @@ inline double AttenuationLength_um(const std::string &material, double lambda_A) return mu_cm > 0.0 ? 1e4 / mu_cm : 0.0; } +// Variance [um^2] of the depth at which a photon converts: exponential with length L_um, truncated +// at the sensor thickness. A photon arriving at angle psi to the normal travels z/cos(psi) to reach +// depth z, so its depth length is L*cos(psi) - pass that for an oblique ray. Zero without a sensor. +inline double ConversionDepthVariance_um2(double L_um, double thickness_um) { + if (!(thickness_um > 0.0) || !(L_um > 0.0)) + return 0.0; + const double a = thickness_um / L_um, e = std::exp(-a); + if (1.0 - e <= 0.0) + return 0.0; + const double mean = L_um * (1.0 - (1.0 + a) * e) / (1.0 - e); + const double ez2 = L_um * L_um * (2.0 - (a * a + 2.0 * a + 2.0) * e) / (1.0 - e); + return std::max(0.0, ez2 - mean * mean); +} + // Everything the per-reflection correction needs, reduced to the two numbers that are constant for // a dataset. Built once on the host; the GPU predictor takes the two floats. struct SensorQE { diff --git a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp index ced5d0caf..af9ad59ac 100644 --- a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp +++ b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp @@ -27,15 +27,7 @@ double parallax_var_px2(const std::string &material, double thickness_um, double if (!(thickness_um > 0.0) || !(pixel_um > 0.0) || !(lambda_A > 0.0)) return 0.0; const double L = sensor_absorption::AttenuationLength_um(material, lambda_A); - if (!(L > 0.0)) - return 0.0; - const double a = thickness_um / L, e = std::exp(-a); - if (1.0 - e <= 0.0) - return 0.0; - const double mean = L * (1.0 - (1.0 + a) * e) / (1.0 - e); - const double ez2 = L * L * (2.0 - (a * a + 2.0 * a + 2.0) * e) / (1.0 - e); - const double var = std::max(0.0, ez2 - mean * mean); // um^2 - return var / (pixel_um * pixel_um); + return sensor_absorption::ConversionDepthVariance_um2(L, thickness_um) / (pixel_um * pixel_um); } // The radial-offset kernels below are a pure function of these six numbers, and one engine is built diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index cd419d874..6c5fdcc3f 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -8,6 +8,7 @@ #include "SpindleCuspLoss.h" #include "SpotWidth.h" #include "HotPixels.h" +#include "../image_analysis/SensorAbsorption.h" #include "DiagnosticOutput.h" #include @@ -1146,6 +1147,25 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru } } + // The X-ray bandwidth, read off the shapes of the same spots (spot_width::EstimateBandwidth). + // Reported only: the run uses what the file or --bandwidth states, 0 otherwise, as it always has. + if (want_width) { + const auto &det = experiment_.GetDetectorSetup(); + const double lambda = experiment_.GetWavelength_A(); + const auto estimate = spot_width::EstimateBandwidth( + width_curves, sensor_absorption::AttenuationLength_um(det.GetSensorMaterial(), lambda), + det.GetSensorThickness_um(), experiment_.GetPixelSize_mm() * 1000.0); + const float used = experiment_.GetBandwidthFWHM().value_or(0.0f); + if (!estimate) + logger.Info("Bandwidth: not measurable on {} spots; the run uses {:.4f}", width_curves.size(), + used); + else + logger.Info("Bandwidth: spot shapes give FWHM {:.4f} from {} spots (one standard error is " + "FWHM {:.4f}; z = {:.1f}, chi2 {:.1f}) - {}; the run uses {:.4f}", estimate->fwhm, + estimate->spots, estimate->fwhm_floor, estimate->z, estimate->chi2, + estimate->z > spot_width::BANDWIDTH_Z ? "significant" : "not significant", used); + } + // Powder contamination, measured on every run that finds spots here. A crystalline phase other // than the crystal - hexagonal ice, a shower of microcrystals, salt out of the cryoprotectant - // diffracts as rings, and its spots are handed to the indexer and to scaling as if they were this diff --git a/rugnux/SpotWidth.cpp b/rugnux/SpotWidth.cpp index 9095f6dbf..e933a0573 100644 --- a/rugnux/SpotWidth.cpp +++ b/rugnux/SpotWidth.cpp @@ -8,8 +8,12 @@ #include #include #include +#include #include +#include "../common/JFJochMath.h" +#include "../image_analysis/SensorAbsorption.h" + using namespace spot_width; namespace { @@ -94,6 +98,113 @@ float interpolate_radius(double frac, const std::array &prof) { return static_cast(R_MAX); } +// The disk the second moments are taken over, px. +constexpr int R_MOMENT = 9; + +// The spot's second moments along and across its radius, about its own flux-weighted centroid. +// (cx, cy) is the pixel the spot is centred on, (mx, my) the core centroid relative to it, `ring` +// the background ring's counts. The background is the ring's mean with anything 5 sigma above it +// clipped: a median is biased low on sparse Poisson counts, and a bias here adds the same flux at +// every radius of the disk - an extra second moment that does not cancel between the two directions +// once they are weighted by the obliquity. +SpotShape shape_of(const int32_t *centre_px, int width, int cx, int cy, double mx, double my, + const std::vector &ring, const DiffractionGeometry &geometry) { + SpotShape shape; + double sum = 0.0; + for (const int32_t v : ring) sum += v; + const double clip = sum / static_cast(ring.size()) + + 5.0 * std::sqrt(std::abs(sum / static_cast(ring.size())) + 0.5) + 2.0; + sum = 0.0; + size_t n = 0; + for (const int32_t v : ring) + if (v < clip) { sum += v; n++; } + if (n == 0) return shape; + const double bkg = sum / static_cast(n); + + // Centroid over the moment disk, three times, then the moments about it. + constexpr double R2 = static_cast(R_MOMENT) * R_MOMENT; + double flux = 0.0, sxx = 0.0, syy = 0.0, sxy = 0.0; + for (int iter = 0; iter < 4; iter++) { + double f = 0.0, sx = 0.0, sy = 0.0, qxx = 0.0, qyy = 0.0, qxy = 0.0; + for (int dy = -R_MOMENT - 2; dy <= R_MOMENT + 2; dy++) { + const int32_t *row = centre_px + static_cast(dy) * width; + for (int dx = -R_MOMENT - 2; dx <= R_MOMENT + 2; dx++) { + const double ddx = dx - mx, ddy = dy - my; + if (ddx * ddx + ddy * ddy >= R2) continue; + const double w = row[dx] - bkg; + f += w; + sx += w * ddx; + sy += w * ddy; + qxx += w * ddx * ddx; + qyy += w * ddy * ddy; + qxy += w * ddx * ddy; + } + } + if (!(f > 0.0)) return shape; + if (iter < 3) { + mx += sx / f; + my += sy / f; + if (std::abs(mx) > 2.0 || std::abs(my) > 2.0) return shape; + } else { + flux = f; + // About the centroid of this last pass, not the disk's centre. + const double ox = sx / f, oy = sy / f; + sxx = qxx / f - ox * ox; + syy = qyy / f - oy * oy; + sxy = qxy / f - ox * oy; + } + } + if (!(flux > 0.0)) return shape; + + // Where one radian of 2theta, and of the angle across the scattering plane, moves the spot. + const float x = static_cast(cx + mx), y = static_cast(cy + my); + const Coord s0(0, 0, 1); + const Coord s1 = geometry.LabCoord(x, y).Normalize(); + const float two_theta = std::acos(std::clamp(s1 * s0, -1.0f, 1.0f)); + if (!(two_theta > 1.0f * static_cast(PI) / 180.0f)) return shape; + const Coord across = (s0 % s1).Normalize(); + const Coord along = across % s1; // in the scattering plane, away from the beam + const float lambda = geometry.GetWavelength_A(); + constexpr float EPS = 1e-3f; + const auto hit = [&](const Coord &dir) { + return geometry.RecipToDetector(dir / lambda - s0 / lambda); + }; + const auto p0 = hit(s1); + const auto pr = hit(s1 * std::cos(EPS) + along * std::sin(EPS)); + const auto pt = hit(s1 * std::cos(EPS) + across * std::sin(EPS)); + const double jrx = (pr.first - p0.first) / EPS, jry = (pr.second - p0.second) / EPS; + const double jr = std::hypot(jrx, jry); + if (!(jr > 0.0)) return shape; + const double ux = jrx / jr, uy = jry / jr, vx = -uy, vy = ux; + const double jt = std::abs((pt.first - p0.first) / EPS * vx + (pt.second - p0.second) / EPS * vy); + if (!(jt > 0.0)) return shape; + + // Parallax: the ray's in-plane direction on the sensor, and tan^2 of its angle to the normal. + const Coord normal = geometry.GetNormalAxis(); + const float cos_psi = std::abs(s1 * normal); + const Coord in_plane = s1 - normal * (s1 * normal); + const double e_len = in_plane.Length(); + double par_u = 0.0, par_v = 0.0; + if (e_len > 1e-6 && cos_psi > 1e-3f) { + const double ex = in_plane * geometry.GetFastAxis() / e_len; + const double ey = in_plane * geometry.GetSlowAxis() / e_len; + const double tan2 = e_len * e_len / (static_cast(cos_psi) * cos_psi); + par_u = tan2 * (ex * ux + ey * uy) * (ex * ux + ey * uy); + par_v = tan2 * (ex * vx + ey * vy) * (ex * vx + ey * vy); + } + + shape.valid = true; + shape.two_theta = two_theta; + shape.jr = static_cast(jr); + shape.jt = static_cast(jt); + shape.m_rad = static_cast(ux * ux * sxx + 2.0 * ux * uy * sxy + uy * uy * syy); + shape.m_tan = static_cast(vx * vx * sxx + 2.0 * vx * vy * sxy + vy * vy * syy); + shape.cos_psi = cos_psi; + shape.par_u = static_cast(par_u); + shape.par_v = static_cast(par_v); + return shape; +} + template double median_of(std::vector &v) { if (v.empty()) return 0.0; @@ -267,6 +378,7 @@ void MeasureSpotFluxCurves(const ImagePreprocessorBuffer &image, int width, int if (!(curve.c[R_NORM - 1] > 0.0f) || !(curve.c[R_MAX - 1] > 0.0f)) continue; const float norm = curve.c[R_NORM - 1]; for (float &v : curve.c) v /= norm; + curve.shape = shape_of(centre_px, width, cx, cy, mx, my, ring, geometry); out.push_back(curve); } } @@ -352,3 +464,125 @@ bool spot_width::WidthSettled(float r80, float r80_before) { && std::abs(r80 - 2.25f) > SWITCH_CLEARANCE_PX && std::abs(r80 - 2.75f) > SWITCH_CLEARANCE_PX; } + +namespace { + +struct BandwidthPoint { double x, y; }; + +// numpy's default (linear) quantile of sorted values. +double quantile_sorted(const std::vector &v, double p) { + const double pos = p * static_cast(v.size() - 1); + const size_t lo = static_cast(std::floor(pos)); + const size_t hi = std::min(lo + 1, v.size() - 1); + return v[lo] + (pos - static_cast(lo)) * (v[hi] - v[lo]); +} + +// The line y = a + s2 x through eight equal-count bins of x, each a 20 %-trimmed mean weighted by its +// own scatter; returns {s2, reduced chi^2}. Nothing where fewer than two bins could be formed. +std::optional> fit_bandwidth_line(std::vector points) { + constexpr size_t N_BIN = 8; + constexpr size_t MIN_PER_BIN = 10; + constexpr double TRIM = 0.2; + std::stable_sort(points.begin(), points.end(), + [](const BandwidthPoint &a, const BandwidthPoint &b) { return a.x < b.x; }); + std::vector bx, by, bw, ys; + size_t begin = 0; + for (size_t b = 0; b < N_BIN; b++) { + const size_t len = points.size() / N_BIN + (b < points.size() % N_BIN ? 1 : 0); + const size_t end = begin + len; + if (len >= MIN_PER_BIN) { + ys.clear(); + for (size_t i = begin; i < end; i++) ys.push_back(points[i].y); + std::sort(ys.begin(), ys.end()); + const double lo = quantile_sorted(ys, TRIM), hi = quantile_sorted(ys, 1.0 - TRIM); + double n = 0.0, sx = 0.0, sy = 0.0, syy = 0.0; + for (size_t i = begin; i < end; i++) + if (points[i].y >= lo && points[i].y <= hi) { + n += 1.0; + sx += points[i].x; + sy += points[i].y; + syy += points[i].y * points[i].y; + } + const double mean = sy / n, var = syy / n - mean * mean; + bx.push_back(sx / n); + by.push_back(mean); + bw.push_back(n / std::max(var, 1e-6)); + } + begin = end; + } + if (bx.size() < 2) return std::nullopt; + + double sw = 0.0, sx = 0.0, sy = 0.0, sxx = 0.0, sxy = 0.0; + for (size_t i = 0; i < bx.size(); i++) { + sw += bw[i]; + sx += bw[i] * bx[i]; + sy += bw[i] * by[i]; + sxx += bw[i] * bx[i] * bx[i]; + sxy += bw[i] * bx[i] * by[i]; + } + const double det = sw * sxx - sx * sx; + if (!(std::abs(det) > 0.0)) return std::nullopt; + const double slope = (sw * sxy - sx * sy) / det; + const double intercept = (sy - slope * sx) / sw; + double chi2 = 0.0; + for (size_t i = 0; i < bx.size(); i++) { + const double r = by[i] - intercept - slope * bx[i]; + chi2 += bw[i] * r * r; + } + chi2 /= std::max(1.0, static_cast(bx.size()) - 2.0); + return std::make_pair(slope, chi2); +} + +} // namespace + +std::optional spot_width::EstimateBandwidth(const std::vector &curves, + double attenuation_um, + double thickness_um, + double pixel_um) { + constexpr double PIXEL_VAR = 1.0 / 12.0; + constexpr double FWHM_PER_SIGMA = 2.3548; + constexpr int N_BOOTSTRAP = 200; + + std::vector points; + for (const auto &c : curves) { + const SpotShape &s = c.shape; + if (!s.valid) continue; + // The conversion depth's variance for this ray: its depth length is L cos(psi). + const double depth_var_px2 = sensor_absorption::ConversionDepthVariance_um2( + attenuation_um * s.cos_psi, thickness_um) / (pixel_um * pixel_um); + const double obliquity = (static_cast(s.jr) / s.jt) * (static_cast(s.jr) / s.jt); + const double x = 2.0 * s.jr * std::tan(0.5 * s.two_theta); + points.push_back({x * x, (s.m_rad - PIXEL_VAR - depth_var_px2 * s.par_u) + - obliquity * (s.m_tan - PIXEL_VAR - depth_var_px2 * s.par_v)}); + } + if (points.size() < BANDWIDTH_MIN_SPOTS) return std::nullopt; + + const auto fit = fit_bandwidth_line(points); + if (!fit) return std::nullopt; + + // The slope's spread over re-draws of the spots. The generator is seeded, so the answer is the + // same on every run of the same data. + std::mt19937 rng(1); + std::vector redraw(points.size()); + double sum = 0.0, sum2 = 0.0; + int n = 0; + for (int b = 0; b < N_BOOTSTRAP; b++) { + for (auto &p : redraw) p = points[rng() % points.size()]; + if (const auto f = fit_bandwidth_line(redraw)) { + sum += f->first; + sum2 += f->first * f->first; + n++; + } + } + const double mean = n > 0 ? sum / n : 0.0; + const double sd = n > 1 ? std::sqrt(std::max(0.0, sum2 / n - mean * mean)) : 0.0; + const double se = sd * std::sqrt(std::max(1.0, fit->second)); + + BandwidthEstimate e; + e.spots = points.size(); + e.chi2 = fit->second; + e.fwhm = (fit->first < 0.0 ? -1.0 : 1.0) * FWHM_PER_SIGMA * std::sqrt(std::abs(fit->first)); + e.fwhm_floor = FWHM_PER_SIGMA * std::sqrt(se); + e.z = se > 0.0 ? fit->first / se : 0.0; + return e; +} diff --git a/rugnux/SpotWidth.h b/rugnux/SpotWidth.h index 00fdb3d60..29a1d7898 100644 --- a/rugnux/SpotWidth.h +++ b/rugnux/SpotWidth.h @@ -45,11 +45,25 @@ constexpr int R_BKG_OUT = R_MAX + 6; // Resolution the widths are compared at. Inside the measured range of essentially every crystal. constexpr float D_REF_A = 5.0f; +// One spot's second moments along and across its own radius, and the exact geometry that turns +// them into a bandwidth (EstimateBandwidth). jr and jt are the pixels one radian of 2theta, and of +// the angle across the scattering plane, moves the spot by; cos_psi is the angle of the ray to the +// sensor normal, and par_u/par_v are tan^2(psi) times the square of the in-plane ray direction's +// component along the radius and across it - the conversion depth's lateral smear, per um^2 of depth +// variance, in the two directions. +struct SpotShape { + bool valid = false; + float two_theta = 0.0f, jr = 0.0f, jt = 0.0f; + float m_rad = 0.0f, m_tan = 0.0f; // px^2, about the flux-weighted centroid + float cos_psi = 1.0f, par_u = 0.0f, par_v = 0.0f; +}; + // One spot's encircled flux C(1), C(2), ... C(R_MAX), divided by C(R_NORM), with the resolution it -// was recorded at. +// was recorded at, and its shape. struct FluxCurve { float d_A = 0.0f; std::array c{}; + SpotShape shape; }; // The fitted width law: r80(d) = c0 + c1/d over resolution bands, weighted by the spots in each, so @@ -120,10 +134,52 @@ constexpr float SETTLED_STEP_PX = 0.40f; // does not move with r1, and would put four of the rule's six wins within a factor of two of the // loss. Splitting the two apart takes the separation from 2x to 13x. constexpr double BKG_STARVED_MAX_FRACTION = 0.0113; + +// ----------------------------------------------------------------------------------------------- +// The X-ray bandwidth, read off the same spots. +// +// A spread dlambda/lambda of sigma smears a reflection along its own radius by 2 tan(theta) sigma +// radians of 2theta and not at all across it. Per spot, with u along the radius and v across it, +// +// m_rad = 1/12 + par_u + jr^2 (s_ang^2 + 4 tan^2(theta) sigma^2) +// m_tan = 1/12 + par_v + jt^2 s_ang^2 +// +// (1/12: the pixel; par: the sensor parallax, fixed from the sensor's physics; s_ang: everything +// isotropic in angle - divergence, crystal size, mosaic spread seen edge-on). So +// +// y = (m_rad - 1/12 - par_u) - (jr/jt)^2 (m_tan - 1/12 - par_v) = a + sigma^2 (2 jr tan theta)^2 +// +// is a straight line whose slope is the bandwidth. It is fitted over eight equal-count bins of the +// abscissa, each a 20 %-trimmed mean, weighted by its own scatter, and the slope's error is the +// spread of 200 bootstrap re-draws of the spots, inflated by the fit's reduced chi^2 where the line +// fits worse than the scatter says. The slope is significant at BANDWIDTH_Z standard errors - over +// the whole battery that separates the multilayer beamlines from every monochromatic one. +// +// It is a LOWER bound: mosaic spread seen along the radius subtracts, and a spread of cell edges +// (strain) is exactly degenerate with it. What it measures is the effective radial broadening. The +// run reports it and does not use it (yet): the bandwidth it integrates with is the file's or +// --bandwidth's. +constexpr size_t BANDWIDTH_MIN_SPOTS = 80; +constexpr double BANDWIDTH_Z = 3.0; + +struct BandwidthEstimate { + double fwhm = 0.0; // relative FWHM of dlambda/lambda; negative where the slope is + double fwhm_floor = 0.0; // the FWHM one standard error of sigma^2 amounts to + double z = 0.0; // slope / its standard error + double chi2 = 0.0; // reduced chi^2 of the binned line + size_t spots = 0; +}; + +// The bandwidth from the shapes of a pool of spots, or nothing where too few were measured. +// attenuation_um is the sensor's attenuation length at this wavelength, thickness_um its thickness +// (0 = no depth, no parallax), pixel_um the pixel size. +[[nodiscard]] std::optional EstimateBandwidth(const std::vector &curves, + double attenuation_um, double thickness_um, + double pixel_um); } // namespace spot_width -// Every isolated, strong, fully readable spot of one image, as a normalised encircled-flux curve, -// appended to `out`. Called on one worker's own vector, so the pool is independent of the order the +// Every isolated, strong, fully readable spot of one image, as a normalised encircled-flux curve +// with its shape, appended to `out`. Called on one worker's own vector, so the pool is independent of the order the // workers finish in. void MeasureSpotFluxCurves(const ImagePreprocessorBuffer &image, int width, int height, const DiffractionGeometry &geometry, diff --git a/tests/BandwidthEstimateTest.cpp b/tests/BandwidthEstimateTest.cpp new file mode 100644 index 000000000..a5aa8691c --- /dev/null +++ b/tests/BandwidthEstimateTest.cpp @@ -0,0 +1,98 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include + +#include +#include +#include + +#include "../common/DiffractionGeometry.h" +#include "../image_analysis/SensorAbsorption.h" +#include "../rugnux/SpotWidth.h" + +namespace { + constexpr int W = 3000, H = 3000, C = 1500; + constexpr double DIST_MM = 100.0, PIXEL_MM = 0.075, LAMBDA_A = 1.0; + constexpr double THICKNESS_UM = 450.0; + constexpr double ANGULAR_SIGMA = 1.5e-4; // rad: divergence and crystal, the same in every direction + + // Spots on an untilted detector, each an elliptical Gaussian with its long axis along the radius: + // the isotropic angular width, the bandwidth's radial streak 2 tan(theta) sigma, and the sensor + // parallax - a conversion depth z moves the photon by z tan(psi) along the radius, and on an + // untilted detector psi = 2theta. Worked out here from the flat-detector geometry, independently + // of the estimator's own Jacobians: R = D tan(2theta), so one radian of 2theta is D/cos^2(2theta) + // along the radius and one radian across the scattering plane D/cos(2theta) across it. Painted + // with 4x4 sub-pixel sampling so each pixel carries the box it integrates, and Poisson noise. + void PaintSpots(std::vector &image, std::vector &spots, double bw_fwhm, + int offset, std::mt19937 &rng) { + const double sigma_bw = bw_fwhm / 2.3548; + const double L_um = sensor_absorption::AttenuationLength_um("Si", LAMBDA_A); + std::vector mean(image.size(), 3.0); + for (int cy = 40 + offset; cy < H - 40; cy += 60) + for (int cx = 40 + offset; cx < W - 40; cx += 60) { + const double rx = (cx - C) * PIXEL_MM, ry = (cy - C) * PIXEL_MM, r = std::hypot(rx, ry); + if (r < 8.0) continue; + const double tt = std::atan2(r, DIST_MM), c2 = std::cos(tt); + const double jr = DIST_MM / (PIXEL_MM * c2 * c2), jt = DIST_MM / (PIXEL_MM * c2); + const double depth_var = sensor_absorption::ConversionDepthVariance_um2(L_um * c2, THICKNESS_UM); + const double par = depth_var * std::tan(tt) * std::tan(tt) / (PIXEL_MM * 1000.0 * PIXEL_MM * 1000.0); + const double streak = 2.0 * std::tan(tt / 2.0) * sigma_bw; + const double var_u = jr * jr * (ANGULAR_SIGMA * ANGULAR_SIGMA + streak * streak) + par; + const double var_v = jt * jt * ANGULAR_SIGMA * ANGULAR_SIGMA; + const double ux = rx / r, uy = ry / r; + const double total = 20000.0; + const double norm = total / (2.0 * M_PI * std::sqrt(var_u * var_v) * 16.0); + for (int dy = -14; dy <= 14; dy++) + for (int dx = -14; dx <= 14; dx++) { + double v = 0.0; + for (int sy = 0; sy < 4; sy++) + for (int sx = 0; sx < 4; sx++) { + const double px = dx - 0.375 + 0.25 * sx, py = dy - 0.375 + 0.25 * sy; + const double u = px * ux + py * uy, t = -px * uy + py * ux; + v += std::exp(-0.5 * (u * u / var_u + t * t / var_v)); + } + mean[static_cast(cy + dy) * W + cx + dx] += norm * v; + } + spots.emplace_back(static_cast(cx), static_cast(cy), + static_cast(total)); + } + for (size_t i = 0; i < image.size(); i++) + image[i] = std::poisson_distribution(mean[i])(rng); + } + + std::optional Estimate(double bw_fwhm) { + DiffractionGeometry geometry; + geometry.BeamX_pxl(C).BeamY_pxl(C).DetectorDistance_mm(DIST_MM).PixelSize_mm(PIXEL_MM) + .Wavelength_A(LAMBDA_A); + std::mt19937 rng(7); + std::vector curves; + for (int offset : {0, 20, 40}) { + ImagePreprocessorBuffer image(static_cast(W) * H); + std::vector spots; + PaintSpots(image.getBuffer(), spots, bw_fwhm, offset, rng); + MeasureSpotFluxCurves(image, W, H, geometry, spots, curves); + } + return spot_width::EstimateBandwidth(curves, sensor_absorption::AttenuationLength_um("Si", LAMBDA_A), + THICKNESS_UM, PIXEL_MM * 1000.0); + } +} + +// A 0.45 % FWHM bandwidth - a multilayer's - on a thick silicon sensor, whose parallax elongates the +// spots along the radius as well and grows with angle much as the bandwidth does. The estimator has +// to take the parallax out from the sensor's physics and return the bandwidth, significantly. +TEST_CASE("BandwidthEstimate_RecoversBandwidthPastParallax", "[SpotWidth]") { + const auto e = Estimate(0.0045); + REQUIRE(e.has_value()); + CHECK(e->spots >= spot_width::BANDWIDTH_MIN_SPOTS); + CHECK(e->fwhm == Catch::Approx(0.0045).margin(0.0006)); + CHECK(e->z > spot_width::BANDWIDTH_Z); +} + +// The same spots from a monochromatic beam: the parallax alone must not read as a bandwidth. +TEST_CASE("BandwidthEstimate_MonochromaticIsNotSignificant", "[SpotWidth]") { + const auto e = Estimate(0.0); + REQUIRE(e.has_value()); + CHECK(e->z < spot_width::BANDWIDTH_Z); + CHECK(std::abs(e->fwhm) < 0.0015); +} diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 2992aa624..ae752e42e 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -53,6 +53,7 @@ ADD_EXECUTABLE(jfjoch_test SweepLayoutTest.cpp ShadowFinderTest.cpp HotPixelFinderTest.cpp + BandwidthEstimateTest.cpp ParallelForTest.cpp RugnuxTest.cpp ResultReportTest.cpp From 23e1976e5540566a1007409da99444a2cb316d8d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 02:13:27 +0200 Subject: [PATCH 04/21] Surfaces: compare held-out half-set CC on Fisher's z; fit modulation, time, goniometer frame in that order The correction-surface gate averaged the per-shell change of the held-out half-set CC over 20 equal-occupancy shells and adopted a surface on its sign. On the shells where a multiplicative error shows, CC1/2 sits at 0.99-0.9996 and a large reduction of the error moves it in the fourth decimal; the shells beyond the data's reach (up to 12 of 20 at CC ~ 0) each add +-0.01, so the sign was set by noise (measured spread of the mean +-0.001-0.003 against effects of -0.0003..-0.0008). It refused the 24x24 detector surface on four EIGER2 16M sweeps. The change is now averaged on atanh(CC) - the candidate from the hq-bisect investigation. The order of the three overlapping surfaces also decides what is adopted. With the gate fixed, goniometer-frame first took a 5 keV insulin sweep from ISa 31.8 to 20.1 (the time surface then refused; R_meas 7.8 -> 8.9%, CC to the model 0.8307 -> 0.8282) and cost a 0.1 deg-sliced thaumatin sweep R_meas 18.2 -> 19.6%; time x detector first refused the detector surface on the 16 keV thaumatin sweep (ISa 33 instead of 40). Modulation, time, goniometer frame avoids both. Bare runs, --report-resolution at XDS's range, against rc173 (0252880e0): ISa / lowest-shell R_meas thaumatin 16 keV 32.9/.030 -> 39.8/.028, lysozyme 90 deg sweeps 10.1/.059 -> 14.1/.048 and 10.1/.060 -> 13.8/.049, cytochrome c 19.2 -> 20.4 and 17.3 -> 19.3, thaumatin 3.8 keV 13.3 -> 14.8; CC to a deposited model (standard-protein models on 14 in-house sets, the deposition on 13 open sets) within +-0.0045, R_model_shell_scaled within +-0.002 except where d_min moved. 46 sets, one regression: a half-image lysozyme test set, CC1/2 0.878 -> 0.862. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/ACKNOWLEDGEMENT.md | 7 ++++ docs/CPU_DATA_ANALYSIS.md | 1 + docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 6 ++-- .../scale_merge/RotationScaleMerge.cpp | 33 +++++++++++++++---- 4 files changed, 38 insertions(+), 9 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index f01a37203..e685dd9ed 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -211,6 +211,13 @@ K. Diederichs, "Linking crystallographic model and data quality" (2012), Science P. A. Karplus, "Better models by discarding data?" (2013), Acta Cryst. D69, 1215-1222 [doi:10.1107/S0907444913001121](https://doi.org/10.1107/S0907444913001121). +**Fisher's z-transformation** — the cross-validation of the scaling correction surfaces averages the +change of the half-set CC1/2 over resolution shells on atanh(CC), so that the shells near CC = 1, where +a multiplicative error shows, are not outweighed by the noise of the shells without signal. R. A. +Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an +indefinitely large population" (1915), Biometrika 10, 507-521 +[doi:10.2307/2331838](https://doi.org/10.2307/2331838). + **The frame disposition** - which stretches of a rotation sweep are kept, carried at reduced weight or dropped from the merge - decides every exclusion on delta-CC1/2, the change in the overall CC1/2 when a group of images is left out, measured in the sigma-tau form so that no random half-dataset split is diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 16fedef2c..3237dcb38 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -52,6 +52,7 @@ The methods draw on, and in places reimplement, solutions from: - A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc). - S. French & K. Wilson, "On the treatment of negative intensity observations", *Acta Cryst.* **A34** (1978), 517-525 (Bayesian amplitude estimation from intensities). - A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free cross-validation). +- R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", *Biometrika* **10** (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared). - M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (model / structure-factor / map machinery used in §14). - J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and [github.com/jonwright/bslz4decoders](https://github.com/jonwright/bslz4decoders) (device-side decoding of bitshuffle+LZ4 images, §0). - A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", *J. Appl. Cryst.* **44** (2011), 1285-1287 (anomalous difference density read at the model's sites). diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 8d7942dad..344c508dd 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -370,11 +370,13 @@ After scale-fulls, four **correction surfaces** are fitted on the combined fulls - **Decay.** Radiation damage weakens later frames more at higher resolution — a resolution×time (Debye–Waller) systematic the resolution-flat per-image scale cannot capture. A single global relative-$B$ rate is fitted, $\ln(I_\mathrm{ref}/I_\mathrm{obs}) = 2\,(\mathrm{d}B/\mathrm{d}n)\,(n-\bar n)\,s^2$ (frame $n$, $s^2 = 1/4d^2$), and folded into the scale. It engages only when the total relative-$B$ over the run exceeds a physical floor (2 Ų); below that the decay is negligible and "correcting" it only spreads symmetry equivalents (same $s^2$, different frames). An optional **per-batch relative-$B$** (`--relative-b[=deg]`, off unless requested; 10°-of-rotation batches by default) extends the single global rate to a smooth $B(n)$ curve — the same $s^2$-weighted decay fit solved independently over short frame batches, curvature-penalized so it cannot over-fit and cross-validated like the surfaces below — for crystals whose decay is non-linear in dose. Its cross-validation splits on **ASU-group parity**, not the frame parity the surfaces below use: a per-batch parameter owns whole frames and so cannot be scored on a held-out frame, whereas splitting the symmetry equivalents tests whether a batch's $B$ generalises to reflections it was not fitted on. - **Absorption.** A smooth multiplicative factor over the diffracted-beam direction expressed in the goniometer (crystal) frame: each full's predicted detector position gives the lab diffracted direction, de-rotated by the spindle so a fixed crystal-frame direction is sampled at many rotation angles and its grid cell is well-determined. Negligible at hard X-rays / thin crystals; it matters at low photon energy. - **Modulation** (detector-plane flat-field). A smooth multiplicative factor over where each reflection lands on the detector (predicted $x,y$): symmetry-equivalents land at different positions as the crystal rotates, over-determining the surface. It absorbs detector-response and geometric systematics that inflate $R_\mathrm{meas}$. -- **Time-dependent absorption.** The same surface as *Absorption*, but over (rotation angle × detector position) instead of the crystal-frame direction alone. The two agree while the illuminated volume stays put — the incident path then depends only on the spindle angle, which the per-image scale already takes, and the exit path is fixed in the crystal frame. Once the diffracting volume drifts through the beam the exit path becomes a function of the spindle angle as well, and nothing time-independent describes it. Fitted last, on 12 rotation bins × a 10×10 equal-occupancy detector grid, so the two time-independent surfaces get first claim on what they can explain. +- **Time-dependent absorption.** The same surface as *Absorption*, but over (rotation angle × detector position) instead of the crystal-frame direction alone. The two agree while the illuminated volume stays put — the incident path then depends only on the spindle angle, which the per-image scale already takes, and the exit path is fixed in the crystal frame. Once the diffracting volume drifts through the beam the exit path becomes a function of the spindle angle as well, and nothing time-independent describes it. Fitted on 12 rotation bins × a 10×10 equal-occupancy detector grid. + +The surfaces overlap, so the order decides what is adopted: modulation first (every frame measures the static detector pattern, so its fine grid is the best determined), then time-dependent absorption, then the goniometer-frame absorption, whose cells collect directions from the whole sweep and the whole resolution range and which, fitted first, takes up part of what the other two describe. Each cell's factor is fitted under a **prior pull to 1** (a Gaussian prior of width 0.1 on its logarithm): a cell moves off 1 in proportion to the information its observations carry, so a cell with little signal stays near 1 instead of being fitted to its noise. Negative observations enter the fit as measured. On the detector-plane surfaces (modulation, time-dependent absorption) the component that is a function of resolution alone is projected out within resolution shells, since the symmetry equivalents of a reflection share one resolution and such a factor cannot be determined from them. -Each surface is **cross-validated** on the half-set agreement it is meant to improve: fitted on even-numbered frames and used to merge the odd ones, fitted on the odd frames and used to merge the even ones, and kept only if the correlation between the two half-set means, taken within resolution shells, rises above the same two halves merged with no surface. Each half is corrected by a surface it did not help to fit, so a surface fitted to noise lowers the correlation, and a correlation within a shell is blind to the resolution-dependent scale the surface cannot determine. +Each surface is **cross-validated** on the half-set agreement it is meant to improve: fitted on even-numbered frames and used to merge the odd ones, fitted on the odd frames and used to merge the even ones, and kept only if the correlation between the two half-set means, taken within resolution shells, rises above the same two halves merged with no surface. Each half is corrected by a surface it did not help to fit, so a surface fitted to noise lowers the correlation, and a correlation within a shell is blind to the resolution-dependent scale the surface cannot determine. The change is averaged over the shells on Fisher's $z = \operatorname{atanh}(CC)$, not on $CC$: the strong shells, where a multiplicative error matters, sit at $CC_{1/2} \approx 0.999$, where even a large reduction of the error moves $CC$ in the fourth decimal, and averaged on $CC$ itself the shells of pure noise beyond the reach of the data decide the sign. **Radiation-damage report (rotation, report-only).** Independently of whether any decay correction is applied, Rugnux measures and reports the relative Debye–Waller $B$ across the sweep: the per-image scale's correlation to the merge and the per-image mosaicity versus frame (dose), together with a per-batch relative-$B$ curve whose first→last change is a single headline number (measured before any decay correction, against the least-damaged early wedge). It is written to the log and to the merged mmCIF as a data-quality-vs-dose diagnostic and **never** alters the merged intensities — distinct from the decay correction above, which does fold into the scale. diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 5cf81b4f9..629f216be 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -3463,16 +3463,25 @@ void RotationScaleMerge::ApplyCellSurface(const std::vector &cell, int const std::vector A_even = fit_surface(0), A_odd = fit_surface(1); const std::vector odd0 = half_means(1, ident), even0 = half_means(0, ident); const std::vector odd1 = half_means(1, A_even), even1 = half_means(0, A_odd); + // Each shell's change is taken on Fisher's z = atanh(CC), not on CC itself. CC1/2 is S / (S + E), + // and on the shells a multiplicative error matters in - the strong ones - it sits at 0.99-0.9996, + // where removing a third of E moves it by 0.0003: averaged as it stands, every shell counting alike, + // the shells of pure noise past the reach of the data (each +-0.01) set the sign. That refused the + // detector surface on EIGER2 16M sets where it improved 16 of 20 shells - mean change in CC -0.0003 + // to -0.0008 - and adopting it raised ISa 33 -> 40 and 10 -> 14 and lowered the lowest-shell R_meas + // 0.030 -> 0.028 and 0.059 -> 0.048. On z a change of E by a given fraction is worth the same at any + // CC, and a shell of noise still only adds its own +-0.01. + // Following Fisher (1915) Biometrika 10, 507-521 double gain = 0.0; int n_cc = 0; for (int sh = 0; sh < nsh_cc; ++sh) { const double c0 = shell_cc(odd0, even0, sh), c1 = shell_cc(odd1, even1, sh); - if (std::isfinite(c0) && std::isfinite(c1)) { gain += c1 - c0; ++n_cc; } + if (std::isfinite(c0) && std::isfinite(c1)) { gain += std::atanh(c1) - std::atanh(c0); ++n_cc; } } if (n_cc > 0) gain /= n_cc; if (!(gain > 0.0)) { - logger.Info("{} correction: not cross-validated (held-out half-set CC {:+.4f} per shell, skipped)", - name, gain); + logger.Info("{} correction: not cross-validated (held-out half-set CC {:+.4f} per shell on Fisher's z, " + "skipped)", name, gain); return; } const std::vector A = fit_surface(-1); @@ -3481,7 +3490,7 @@ void RotationScaleMerge::ApplyCellSurface(const std::vector &cell, int if (cell[i] >= 0) fulls[i].corr = static_cast(fulls[i].corr * A[cell[i]]); }); - logger.Info("{} correction: cross-validated, held-out half-set CC {:+.4f} per shell; {} in {} round(s); " + logger.Info("{} correction: cross-validated, held-out half-set CC {:+.4f} per shell on Fisher's z; {} in {} round(s); " "resolution gauge removed a {:.2f}x ramp; {} of {} cells on the [0.25, 4] clamp", name, gain, settled ? "settled" : "NOT settled", rounds_used, gauge_ramp, n_clamped, ncell); @@ -5672,12 +5681,22 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st RefineDecay(n_groups); if (refine_surfaces && relative_b_deg > 0.0) RefineRelativeB(n_groups); // per-batch relative-B on top of the single decay slope - if (refine_surfaces && absorption_iter > 0) - RefineAbsorption(absorption_iter, n_groups); + // The three surfaces overlap, so the one fitted first gets the first claim and the others are tested + // on what it left - the order decides what is adopted. Detector first: every frame of the sweep + // measures the static detector pattern, so its fine grid is the best-determined of the three. Then + // the time x detector surface, which takes what changes through the sweep. The goniometer-frame + // surface last: its cells collect directions from the whole sweep and the whole resolution range, + // so fitted first it soaks up part of both others and fits it where it is not. Measured with the gate + // below: goniometer frame first, a 5 keV sweep then refused the time surface and fell from ISa 31.8 + // to 20.1 (R_meas 7.8 -> 8.9%, CC to the model 0.8307 -> 0.8282), and a finely sliced sweep with a + // large drift went from R_meas 18.2 to 19.6%; time surface first, the detector surface was refused + // after it on a 16 keV sweep that it takes from ISa 33 to 40. In this order neither happens. if (refine_surfaces && modulation_iter > 0) RefineModulation(modulation_iter, n_groups); if (refine_surfaces && absorption_iter > 0) - RefineAbsorptionTime(absorption_iter, n_groups); // last: the static surfaces get first claim + RefineAbsorptionTime(absorption_iter, n_groups); + if (refine_surfaces && absorption_iter > 0) + RefineAbsorption(absorption_iter, n_groups); // --- 4c. The disposition. What keeping each batch of the sweep costs the merged intensities // (delta-CC1/2), which frames cost too much, and what became of every frame of the run. From 257686c6fde42a39256f49f033e154032ac3db5e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 23 Sep 2026 19:58:32 +0200 Subject: [PATCH 05/21] Merge: Wilson outlier test The median outlier test needs three observations, so a reflection measured once or twice (one good observation and one artefact, the usual case after Friedel merging) was never tested, and where there were more, a precise-looking artefact (a hot pixel, thousands of counts) could outweigh mates from weak frames and become the weighted median itself. Every observation of the written merge is now also judged against Wilson statistics beyond 4 A: E^2 = I / (epsilon _shell), centric and acentric laws, a bound set by alpha = 0.01 expected false rejections per dataset (ln(2N/alpha); twice that for centrics), the lower confidence limit I - z*sigma (own sigma, z from the same budget) required to exceed it, shells whose is not established at that significance not judged, and the bound widened by the dataset's measured tail scale (peaks over threshold on the well-measured shells; 1 for a Wilson crystal, larger under tNCS/anisotropy). An improbable singleton is rejected; one with company only when most of its reflection's other observations are probable and it disagrees with them. The reflection is the group, or the Friedel pair under -A. Search merges are not tested (epsilon is 1 in P1, and the screw rows are the reflections a wrong epsilon would call improbable). The flags are handed to the device merge as pre-rejections, so CPU and GPU agree. Report: OBSERVATIONS_REJECTED_WILSON= (part of OBSERVATIONS_REJECTED=), and the developer report lists each rejected observation (hkl, d, E^2, image, x, y). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/ACKNOWLEDGEMENT.md | 8 + docs/CPU_DATA_ANALYSIS_DECISIONS.md | 2 + image_analysis/scale_merge/CMakeLists.txt | 4 +- image_analysis/scale_merge/Merge.h | 12 ++ .../scale_merge/RotationScaleMerge.cpp | 70 +++++++- .../scale_merge/RotationScaleMergeGPU.cu | 34 ++-- .../scale_merge/RotationScaleMergeGPU.h | 5 +- image_analysis/scale_merge/WilsonOutliers.cpp | 168 ++++++++++++++++++ image_analysis/scale_merge/WilsonOutliers.h | 60 +++++++ rugnux/ResultReport.cpp | 17 ++ tests/CMakeLists.txt | 1 + tests/WilsonOutliersTest.cpp | 125 +++++++++++++ 12 files changed, 484 insertions(+), 22 deletions(-) create mode 100644 image_analysis/scale_merge/WilsonOutliers.cpp create mode 100644 image_analysis/scale_merge/WilsonOutliers.h create mode 100644 tests/WilsonOutliersTest.cpp diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index e685dd9ed..c4bbf6fc1 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -186,6 +186,14 @@ package: algorithms and new approaches for multi-crystal scaling" (2020), Acta C G. Winter, D. G. Waterman et al., "Robust background modelling in DIALS" (2016), J. Appl. Cryst. 49, 1912-1921 [doi:10.1107/S1600576716013595](https://doi.org/10.1107/S1600576716013595). +**Wilson outlier test** — judging an observation that has no symmetry mates against the acentric and +centric intensity distributions of its resolution shell, with the symmetry enhancement factor, is +Wilson's statistics; rejecting only observations that are also significant, and keeping a reflection +whose observations are all large, follows AIMLESS's EMAX test. A. J. C. Wilson, "The probability +distribution of X-ray intensities" (1949), Acta Cryst. 2, 318-321 (DOI not verified, so omitted); +P. Evans, "Scaling and assessment of data quality" (2006), Acta Cryst. D62, 72-82 +[doi:10.1107/S0907444905036693](https://doi.org/10.1107/S0907444905036693). + **Diffraction anisotropy** — the description of the overall fall-off by a single anisotropic displacement tensor, its symmetry constraints, and the fact that only its deviatoric part is determined (the isotropic part being degenerate with the overall scale) are Sheriff and Hendrickson's. diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index f248d736d..3a8b7fe74 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -64,6 +64,8 @@ Beside them, on rotation data, the run reports **twin-immune zone evidence** for Merging applies an optional per-observation median-based $N\sigma$ cut (`--reject-outliers`, default 6σ for `rot3d`, off otherwise). The same $N\sigma$ cut is fed back into the error model: after an initial $a,b$ fit the parameters are re-fit once on the reflections that survive rejection (dropping any whose squared deviation exceeds $N^2\,[a\,\sigma^2 + (b\,\langle I\rangle)^2]$), so the calibrated errors describe the reflections that actually enter the merge rather than the pre-rejection pool. +The median needs three observations, so a reflection measured once or twice — typically one good observation and one artefact (a hot pixel, a zinger) after Friedel merging — is never tested by it, and an artefact that looks precise (thousands of counts, a small relative $\sigma$) can outweigh mates from weak frames and become the median itself. Every observation of the written merge is therefore also judged by **Wilson statistics**. Beyond 4 Å its $E^2 = I/(\varepsilon\,\langle I/\varepsilon\rangle_\mathrm{shell})$ is compared with a bound set by a budget of $\alpha = 0.01$ expected false rejections per dataset: with $N$ observations tested, $\ln(2N/\alpha)$ for acentrics and twice that for centrics, and the observation's lower confidence limit $I - z\sigma$ (with its own $\sigma$, and $z$ from the same budget) must exceed it, so noise in weak shells does not trigger it; a shell whose $\langle I\rangle$ is not itself established at that significance is not judged. The bound is widened by the tail scale of the dataset's own intensity distribution, measured on its well-measured observations (1 for a Wilson crystal; larger under a pseudo-translation or anisotropy). An improbable singleton is dropped; an improbable observation with company only when most of the reflection's other observations are probable and it disagrees with their mean beyond the errors — several large observations confirm each other. The count is `OBSERVATIONS_REJECTED_WILSON=` in the report (included in `OBSERVATIONS_REJECTED=`), and the developer report lists each observation with its image and detector position. + ### 13.4 Automatic resolution cutoff By default the reported/written high-resolution limit is trimmed where $\mathrm{CC}_{1/2}$ falls off: a logistic is fitted to $\mathrm{CC}_{1/2}(s)$, and the limit is set **one reported-shell width past** the point where the fit crosses 0.30 — deliberately "one shell too far", so weak-but-real data below the crossing are kept rather than discarded. The extension is measured over the range that is actually kept, not the full measured range, so a detector reaching far past where the crystal diffracts cannot inflate it. `--scaling-high-resolution` overrides the limit and `--resolution-cutoff off` disables it. diff --git a/image_analysis/scale_merge/CMakeLists.txt b/image_analysis/scale_merge/CMakeLists.txt index 94617388c..985ce4692 100644 --- a/image_analysis/scale_merge/CMakeLists.txt +++ b/image_analysis/scale_merge/CMakeLists.txt @@ -26,7 +26,9 @@ ADD_LIBRARY(JFJochScaleMerge TranslationalNCS.cpp TranslationalNCS.h ScalingResult.h - ScalingResult.cpp) + ScalingResult.cpp + WilsonOutliers.cpp + WilsonOutliers.h) TARGET_LINK_LIBRARIES(JFJochScaleMerge Ceres::ceres Eigen3::Eigen JFJochCommon fftw3f) IF (JFJOCH_CUDA_AVAILABLE) diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index bd80bf972..adb81c7ce 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -139,6 +139,16 @@ struct AnomalousMergeStatistics { bool measured = false; }; +// An observation the Wilson outlier test removed from the merge (see WilsonOutliers.h): its own +// indices, its normalised intensity, and where it was recorded, so a recurring detector position shows. +struct WilsonRejectedObservation { + int32_t h = 0, k = 0, l = 0; + float d = NAN; + float e2 = NAN; + float image = NAN; // fractional image number + float x = NAN, y = NAN; // predicted detector position (pixels) +}; + struct MergeStatistics { std::vector shells; MergeStatisticsShell overall; @@ -163,6 +173,8 @@ struct MergeStatistics { // BETTER by every number it reports, so the count has to be visible or the failure is silent. // Zero when rejection is off. size_t n_observations_rejected = 0; + // The part of those the Wilson test removed (see WilsonOutliers.h), each listed, in the written range. + std::vector wilson_rejected; double radiation_damage_delta_b = NAN; std::vector radiation_damage_b_batch; double radiation_damage_batch_deg = 0.0; diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 629f216be..d5f25838a 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -23,6 +23,7 @@ #include "RfreeFlags.h" #include "FrenchWilson.h" #include "ResolutionCutoff.h" +#include "WilsonOutliers.h" #include "../../common/CorrelationCoefficient.h" #include "../../common/CrystalLattice.h" #include "../../common/Definitions.h" @@ -4202,6 +4203,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool fit_ab(kept); } }; + // Friedel pair of each group when the hands are merged separately (-A); filled with the medians below. + std::vector pair_of_group; { std::vector cnt(n_groups, 0); // per-group usable count (both paths; feeds reject-median) bool did_gpu = false; @@ -4291,7 +4294,6 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // the merge and took a whole CC1/2 bin, and the cut with it, from 0.80 to 0.18. A hand with // three of its own keeps its own median, as before. Median set g is group g's own; median // set n_groups + p is Friedel pair p's, filled only where a hand of p needs it. - std::vector pair_of_group; std::vector pair_needed; if (!merge_friedel) { const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, x.GetSpaceGroupOrP1()); @@ -4361,9 +4363,9 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool fit_error_model(samples); } - auto corrected_sigma = [&](const Obs &o, float I_corr, float sigma_corr) -> float { + // The full's sigma under the error model, with the variance evaluated at intensity I_for_b. + auto model_sigma = [&](const Obs &o, double I_for_b, float sigma_corr) -> float { if (!error_model_active) return sigma_corr; - const double I_for_b = std::isfinite(em_mean[o.group]) ? em_mean[o.group] : I_corr; // A full's own sigma carries its own Poisson fluctuation: a full that came out low has a // smaller sigma, so 1/sigma^2 weights it up and the merged mean drifts below . Rebuild the // variance at the reflection's EXPECTED intensity - var_bkg + var_per_I*, the linear model @@ -4378,6 +4380,9 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool + (error_model_b * I_for_b) * (error_model_b * I_for_b); return v > 0.0 ? static_cast(std::sqrt(v)) : sigma_corr; }; + auto corrected_sigma = [&](const Obs &o, float I_corr, float sigma_corr) -> float { + return model_sigma(o, std::isfinite(em_mean[o.group]) ? em_mean[o.group] : I_corr, sigma_corr); + }; // ---- What the pooled outlier test's tolerance gains from the shell's own Bijvoet variance. ---- // An observation of one hand sits dI/2 from the pooled pair's mean, so its share of the pair's // anomalous variance is dI^2/4 and the cut becomes n * sqrt(sigma^2 + v_anom). That is the plain @@ -4480,6 +4485,33 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool std::vector merged_I; size_t reject_count = 0; std::vector rejected_obs; // per-full outlier-rejected flag (both paths) + + // ---- The Wilson test (WilsonOutliers.h): every observation against the intensity distribution of + // its shell, beside the median test. It reaches what the median cannot - a reflection measured + // once or twice, and an artefact that outweighs its mates and becomes the median - and its + // reflection is the group, or with the hands merged separately the Friedel pair, which is + // what the median pools too. On the written merges only: the + // search merges run in P1 and its subgroups, where an axial reflection has epsilon 1 and the + // rows the screw-axis test reads are the ones a wrong epsilon would call improbable. ---- + const bool wilson_test = reject_outliers && !for_search; + std::vector group_epsilon; + std::vector group_centric; + if (wilson_test) { + const gemmi::GroupOps gops = x.GetSpaceGroupOrP1().operations(); + group_epsilon.resize(n_groups); + group_centric.resize(n_groups); + ParallelChunks(n_groups, ThreadsForWork(n_groups, nthreads), [&](int lo, int hi) { + for (int g = lo; g < hi; ++g) { + const gemmi::Op::Miller hkl{group_h[g], group_k[g], group_l[g]}; + group_epsilon[g] = static_cast(std::max(1, gops.epsilon_factor_without_centering(hkl))); + group_centric[g] = gops.is_reflection_centric(hkl) ? 1 : 0; + } + }); + } + // Expected false rejections per dataset. + constexpr double WILSON_ALPHA = 0.01; + WilsonOutlierResult wilson; + std::vector wilson_full; // the full behind each Wilson observation auto run_merge = [&]() { // The device path's unpack below writes every field of every group - MergeAccumKernel fills the // empty ones too - so there this only has to be the right size. The host path accumulates into @@ -4494,6 +4526,24 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool merged_I.assign(n_groups, NAN); reject_count = 0; rejected_obs.assign(fulls.size(), 0); + if (wilson_test) { + // Each observation with its OWN sigma, the error model evaluated at its own intensity: the + // reflection's mean, which the merge weights use, is exactly what an artefact inflates. + std::vector wobs; + wilson_full.clear(); + for (int i = 0; i < n_full; ++i) { + const int g = mf.group[i]; + if (g < 0) continue; + const Obs &o = fulls[i]; + const float I_corr = o.I * o.corr; + wobs.push_back({I_corr, model_sigma(o, I_corr, o.sigma * o.corr), o.d, group_epsilon[g], + group_centric[g] != 0, merge_friedel ? g : pair_of_group[g]}); + wilson_full.push_back(i); + } + wilson = WilsonOutliers(wobs, WILSON_ALPHA); + for (size_t j = 0; j < wobs.size(); ++j) + if (wilson.rejected[j]) rejected_obs[wilson_full[j]] = 1; + } bool did_gpu_acc = false; #ifdef JFJOCH_USE_CUDA if (use_gpu_merge) { @@ -4539,6 +4589,10 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool if (!did_gpu_acc) for (const auto &o : fulls) { if (!usable_merge(o)) continue; + if (rejected_obs[&o - fulls.data()]) { // by the Wilson test above + ++reject_count; + continue; + } const float I_corr = o.I * o.corr; float sigma_corr = o.sigma * o.corr; sigma_corr = corrected_sigma(o, I_corr, sigma_corr); @@ -4769,6 +4823,10 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool if (reject_count > 0) logger.Info("Merge outlier rejection: dropped {} observations", reject_count); + if (wilson_test && wilson.n_tested > 0) + logger.Info("Wilson outlier test: {} observations tested, " + "tail scale {:.2f}, bound E^2 > {:.1f} (acentric), {} rejected", wilson.n_tested, + wilson.tail_scale, wilson.bound, wilson.n_rejected); // ---- Statistics (report_shell_count shells): completeness, multiplicity, , R_meas, CC1/2, // SigAno and CCanom, over a shell grid. A lambda because the same merged reflections are // binned a second time, over the reference range, when --report-resolution asked for that @@ -5180,6 +5238,12 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // Radiation-damage monitor (measured before any correction by MeasureRadiationDamageB): carry the // first->last relative-B change and the per-batch curve into the reported statistics / mmCIF. out.n_observations_rejected = reject_count; + for (size_t j = 0; j < wilson.rejected.size(); ++j) { + if (!wilson.rejected[j]) continue; + const Obs &o = fulls[wilson_full[j]]; + if (effective_d_min && o.d < *effective_d_min) continue; + out.wilson_rejected.push_back({o.h, o.k, o.l, o.d, wilson.e2[j], o.image_number, o.px, o.py}); + } out.radiation_damage_delta_b = rad_damage_delta_b; out.radiation_damage_b_batch = rad_damage_b_batch; out.radiation_damage_batch_deg = rad_damage_batch_deg; diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.cu b/image_analysis/scale_merge/RotationScaleMergeGPU.cu index 302017845..4f4317e6e 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.cu +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.cu @@ -528,6 +528,18 @@ namespace { } } + // The full's sigma under the error model with its variance at intensity I_for_b - the host model_sigma. + // At the reflection's expected intensity the merge weight cannot know this full's own fluctuation. + __device__ __forceinline__ float ModelSigma(int i, const MergeParams &p, double I_for_b, float sigma_raw) { + const double bi = p.error_model_b * I_for_b; + const double c = p.corr[i]; + double a_var = double(sigma_raw) * sigma_raw; + const double base = c * c * p.var_bkg[i] + c * p.var_per_I[i] * Dmax(0.0, I_for_b); + if (base > 0.0) a_var = base; + const double v = p.error_model_a * a_var + bi * bi; + return v > 0.0 ? float(sqrt(v)) : sigma_raw; + } + // One thread per group: the merge accumulators (inv-var sums + deterministic half-sets), with the // error-model-corrected sigma. Mirrors MergeAndStats' merge loop (reject path stays on the host). __global__ void MergeAccumKernel(MergeParams p) { @@ -541,20 +553,12 @@ namespace { for (int q = lo; q < hi; ++q) { const int i = p.gperm[q]; if (!MergeUsable(i, p)) continue; + if (p.rejected_obs[i]) { ++rejected; continue; } // by the host's Wilson test const float I_corr = p.I[i] * p.corr[i]; - float sigma_corr = p.sigma[i] * p.corr[i]; - if (p.error_model_active) { - const double I_for_b = isfinite(p.em_mean[g]) ? p.em_mean[g] : double(I_corr); - const double bi = p.error_model_b * I_for_b; - // The full's variance rebuilt at the reflection's expected intensity, so the merge - // weight cannot know this full's own fluctuation (see the host corrected_sigma). - const double c = p.corr[i]; - double a_var = double(sigma_corr) * sigma_corr; - const double base = c * c * p.var_bkg[i] + c * p.var_per_I[i] * Dmax(0.0, I_for_b); - if (base > 0.0) a_var = base; - const double v = p.error_model_a * a_var + bi * bi; - if (v > 0.0) sigma_corr = float(sqrt(v)); - } + const float sigma_raw = p.sigma[i] * p.corr[i]; + const float sigma_corr = p.error_model_active + ? ModelSigma(i, p, isfinite(p.em_mean[g]) ? p.em_mean[g] : double(I_corr), sigma_raw) + : sigma_raw; if (p.reject_outliers && p.error_model_active && isfinite(rmed)) { const double tol = p.reject_nsigma * sqrt(double(sigma_corr) * sigma_corr + v_add); if (fabs(double(I_corr) - rmed) > tol) { @@ -934,9 +938,7 @@ void RotationScaleMergeGPU::MergeAccum(double error_model_a, double error_model_ d.a_nh0 = CudaDevicePtr(std::max(1, ng)); d.a_nh1 = CudaDevicePtr(std::max(1, ng)); d.a_d = CudaDevicePtr(std::max(1, ng)); d.a_rejected = CudaDevicePtr(std::max(1, ng)); d.a_on_ice = CudaDevicePtr(std::max(1, ng)); - d.m_rejected = CudaDevicePtr(std::max(1, d.n_fulls)); - CudaCheck(cudaMemset(d.m_rejected.get(), 0, size_t(std::max(1, d.n_fulls)) * sizeof(uint8_t)), - "zero m_rejected"); + Upload(d.m_rejected, rejected_obs, d.n_fulls); Upload(d.reject_median, reject_median, ng); if (reject_var_add && ng > 0) Upload(d.reject_var_add, reject_var_add, ng); else d.reject_var_add = CudaDevicePtr(); diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.h b/image_analysis/scale_merge/RotationScaleMergeGPU.h index bc96945a8..1f33d2e18 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.h +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.h @@ -95,8 +95,9 @@ public: // from a/b). `half` is the per-full CC1/2 half-set (length n_fulls) assigned on the host, so this // kernel and the host merge loop cannot disagree about it. // The per-group results stay on the device for MergeAccumRange to download; reject_median is - // uploaded (NAN where none). rejected_obs is the per-full flag (length n_fulls): the host needs it - // for the reductions it still does itself, above all the anomalous I(+)/I(-) split. + // uploaded (NAN where none). rejected_obs is the per-full flag (length n_fulls): on entry the + // observations the host already rejected (the Wilson test), on return those plus the median test's; + // the host needs it for the reductions it still does itself, above all the anomalous I(+)/I(-) split. // frame_cc_factor (length n_frames) is each frame's (G_ref/G)^2 floored at 1; swh_typ0/1 are the // half-set weights multiplied by it. Requires MergeEmSamples first (em_mean resident). // reject_var_add (n_groups) widens the pooled cut by the shell's own measured Bijvoet diff --git a/image_analysis/scale_merge/WilsonOutliers.cpp b/image_analysis/scale_merge/WilsonOutliers.cpp new file mode 100644 index 000000000..b98ca4814 --- /dev/null +++ b/image_analysis/scale_merge/WilsonOutliers.cpp @@ -0,0 +1,168 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "WilsonOutliers.h" + +#include +#include +#include + +WilsonOutlierResult WilsonOutliers(const std::vector &obs, double alpha) { + WilsonOutlierResult out; + out.rejected.assign(obs.size(), 0); + out.e2.assign(obs.size(), NAN); + + std::vector idx; + for (size_t i = 0; i < obs.size(); ++i) { + const auto &o = obs[i]; + if (o.unit >= 0 && o.d > 0.0f && o.d < WILSON_OUTLIER_D_MAX && std::isfinite(o.I) + && o.sigma > 0.0f && std::isfinite(o.sigma)) + idx.push_back(i); + } + if (idx.empty()) + return out; + + // alpha / N per observation, half of it to each tail. Wilson's acentric law is P(E^2 > t) = exp(-t) + // and the centric one P(E^2 > t) = erfc(sqrt(t/2)) <= exp(-t/2), so t and 2t; the Gaussian tail is + // P(> z sigma) <= exp(-z^2/2) / 2. Both bounds are the conservative side of the exact quantile. + // Following Wilson (1949) Acta Cryst. 2, 318-321; the significance guard as in Aimless, Evans (2006) + // Acta Cryst. D62, 72-82. + const double p = alpha / (2.0 * static_cast(idx.size())); + const double t = std::log(1.0 / p); + const double z = std::sqrt(2.0 * std::log(1.0 / (2.0 * p))); + + // in shells of equal observation count, low to high resolution. Two thousand observations + // put the shell mean at 2% (an exponential's sd equals its mean); narrow shells keep the fall-off + // across one from reading as spread. + constexpr size_t OBS_PER_SHELL = 2000; + std::sort(idx.begin(), idx.end(), [&](size_t a, size_t b) { + return obs[a].d != obs[b].d ? obs[a].d > obs[b].d : a < b; + }); + const size_t n_shells = std::max(1, idx.size() / OBS_PER_SHELL); + const auto shell_of = [&](size_t j) { return j * n_shells / idx.size(); }; + const auto lower_e2 = [&](const WilsonObservation &o, double mu) { + return (o.I - z * o.sigma) / (o.epsilon * mu); + }; + const auto wilson_bound = [&](const WilsonObservation &o) { return o.centric ? 2.0 * t : t; }; + // A shell is judged only where its mean is itself established at the same significance: with no + // measured , nothing in the shell can be called improbable against it. + std::vector mu(n_shells, 0.0); + std::vector shell_valid(n_shells, 0); + { + std::vector sum(n_shells, 0.0); + std::vector cnt(n_shells, 0); + for (size_t j = 0; j < idx.size(); ++j) { + sum[shell_of(j)] += obs[idx[j]].I / obs[idx[j]].epsilon; + ++cnt[shell_of(j)]; + } + // Once more without the observations the plain Wilson bound already calls improbable: a single + // artefact thousands of times would otherwise raise the mean of its own shell several-fold + // and hide the smaller artefacts sharing it. + std::vector sum_kept(n_shells, 0.0), sum2_kept(n_shells, 0.0); + std::vector cnt_kept(n_shells, 0); + for (size_t j = 0; j < idx.size(); ++j) { + const auto &o = obs[idx[j]]; + const size_t s = shell_of(j); + if (sum[s] > 0.0 && lower_e2(o, sum[s] / cnt[s]) > wilson_bound(o)) + continue; + const double x = o.I / o.epsilon; + sum_kept[s] += x; sum2_kept[s] += x * x; ++cnt_kept[s]; + } + for (size_t s = 0; s < n_shells; ++s) { + if (cnt_kept[s] < 2) continue; + const double n = static_cast(cnt_kept[s]); + mu[s] = sum_kept[s] / n; + const double var = std::max(0.0, (sum2_kept[s] - n * mu[s] * mu[s]) / (n - 1.0)); + shell_valid[s] = mu[s] > 0.0 && mu[s] > z * std::sqrt(var / n); + } + } + + // The tail's scale, peaks over threshold: above any threshold an exponential's excess is the same + // exponential, so the median excess of the acentric observations over a common threshold u is + // k ln 2. u is where Wilson leaves a hundred acentric observations above it. Only shells whose + // typical observation is significant at u take part: where sigma is comparable to u , what + // exceeds u is the noise tail, and it would read as a heavy intensity tail. + size_t n_acentric = 0; + for (size_t j = 0; j < idx.size(); ++j) + if (!obs[idx[j]].centric && shell_valid[shell_of(j)]) ++n_acentric; + const double u = std::max(0.0, std::log(static_cast(n_acentric) / 100.0)); + std::vector shell_measures_tail(n_shells, 0); + { + std::vector> noise(n_shells); // sigma / (epsilon ) + for (size_t j = 0; j < idx.size(); ++j) { + const auto &o = obs[idx[j]]; + if (shell_valid[shell_of(j)]) noise[shell_of(j)].push_back(o.sigma / (o.epsilon * mu[shell_of(j)])); + } + for (size_t s = 0; s < n_shells; ++s) { + auto &v = noise[s]; + if (v.empty()) continue; + const size_t mid = v.size() / 2; + std::nth_element(v.begin(), v.begin() + mid, v.end()); + shell_measures_tail[s] = z * v[mid] <= u; + } + } + std::vector excess; + for (size_t j = 0; j < idx.size(); ++j) { + const auto &o = obs[idx[j]]; + if (o.centric || !shell_measures_tail[shell_of(j)]) continue; + const double e2 = o.I / (o.epsilon * mu[shell_of(j)]); + if (e2 > u) excess.push_back(e2 - u); + } + if (!excess.empty()) { + const size_t mid = excess.size() / 2; + std::nth_element(excess.begin(), excess.begin() + mid, excess.end()); + out.tail_scale = std::max(1.0, excess[mid] / std::log(2.0)); + } + out.bound = out.tail_scale * t; + + // Which observations are improbable, and the observations of each reflection (in valid shells). + std::vector improbable(obs.size(), 0); + std::vector tested; + for (size_t j = 0; j < idx.size(); ++j) { + const size_t i = idx[j]; + if (!shell_valid[shell_of(j)]) continue; + const auto &o = obs[i]; + const double m = mu[shell_of(j)]; + out.e2[i] = static_cast(o.I / (o.epsilon * m)); + improbable[i] = lower_e2(o, m) > out.tail_scale * wilson_bound(o); + tested.push_back(i); + } + out.n_tested = tested.size(); + std::sort(tested.begin(), tested.end(), [&](size_t a, size_t b) { + return obs[a].unit != obs[b].unit ? obs[a].unit < obs[b].unit : a < b; + }); + + // An improbable observation measured once is rejected. One with company is rejected when most of + // its reflection's other observations are probable and it disagrees with their mean beyond the + // errors - for a pair, the other member. Where most are large, the reflection is (Aimless's "keep + // if most observations are large"). + for (size_t lo = 0; lo < tested.size();) { + size_t hi = lo; + while (hi < tested.size() && obs[tested[hi]].unit == obs[tested[lo]].unit) ++hi; + for (size_t q = lo; q < hi; ++q) { + const size_t i = tested[q]; + if (!improbable[i]) continue; + size_t n_probable = 0; + double sw = 0.0, swI = 0.0; + for (size_t r = lo; r < hi; ++r) { + const auto &o = obs[tested[r]]; + if (r == q || improbable[tested[r]]) continue; + const double w = 1.0 / (static_cast(o.sigma) * o.sigma); + sw += w; swI += w * o.I; + ++n_probable; + } + const size_t n_others = hi - lo - 1; + bool reject = n_others == 0; + if (2 * n_probable > n_others) { + const auto &o = obs[i]; + reject = std::fabs(o.I - swI / sw) > z * std::sqrt(static_cast(o.sigma) * o.sigma + 1.0 / sw); + } + if (reject) { + out.rejected[i] = 1; + ++out.n_rejected; + } + } + lo = hi; + } + return out; +} diff --git a/image_analysis/scale_merge/WilsonOutliers.h b/image_analysis/scale_merge/WilsonOutliers.h new file mode 100644 index 000000000..8fd908f4f --- /dev/null +++ b/image_analysis/scale_merge/WilsonOutliers.h @@ -0,0 +1,60 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include +#include + +// An outlier test that judges an observation against the intensity DISTRIBUTION rather than against its +// symmetry mates. The median test in the merge needs three observations of a reflection, so a +// singleton is never tested and a PAIR - one good observation and one artefact, the usual case after +// Friedel merging - is never tested either; and where there are three or more, an artefact that looks +// precise (a hot pixel reads thousands of counts with a small relative sigma) can outweigh mates from +// weak frames and become the weighted median itself. Wilson statistics say how large a genuine +// intensity can plausibly be in its resolution shell, given the symmetry enhancement factor epsilon +// and whether the reflection is centric - independent of the other observations. +// +// The bound is set by the number of false rejections the dataset can afford, alpha, not tuned: with N +// observations tested, each may be falsely called improbable with probability alpha / N. That budget +// is split between the two ways a genuine observation can land high - a true intensity far up the +// Wilson tail, and a measurement error far up its Gaussian tail - so an observation is improbable only +// when even its lower confidence bound I - z*sigma lies beyond the Wilson bound. That is what keeps weak +// shells, where noise alone reaches many times , from misfiring, with no separate I/sigma threshold; +// and a shell whose own is not established at that significance is not judged at all. +// +// Real crystals are not all Wilson crystals - tNCS modulates class by class, anisotropy does +// direction by direction - and every such departure makes the distribution a MIXTURE of exponentials, +// whose tail is heavier than Wilson's (never lighter). The tail's scale is therefore measured on the +// dataset's own well-measured observations and the bound widened by it: a Wilson crystal measures 1 +// and is tested at Wilson's bound. +// +// Improbable is not yet rejected. An improbable observation measured once is rejected; one with +// company only when most of its reflection's other observations are probable and it disagrees with +// them beyond the errors. Several large observations of one reflection confirm each other and are +// kept, which is how a genuinely strong reflection survives. +struct WilsonObservation { + float I; // scaled intensity + float sigma; // its own standard uncertainty, at its own intensity + float d; // resolution (A) + float epsilon; // symmetry enhancement factor of the reflection (without centring) + bool centric; + int32_t unit; // the reflection the observation measures; < 0 = not part of the test +}; + +struct WilsonOutlierResult { + std::vector rejected; // per observation + std::vector e2; // I / (epsilon _shell); NaN outside the tested range + double tail_scale = 1.0; // the measured tail scale (1 = Wilson) + double bound = 0.0; // acentric bound on E^2 of the lower confidence limit; centric 2x + size_t n_tested = 0; // observations in shells with an established , beyond 4 A + size_t n_rejected = 0; +}; + +// Wilson statistics hold where the scattering of the unit cell is effectively random - beyond about +// 4 A. At lower resolution the solvent and the secondary structure make the distribution a mixture of +// its own. +constexpr float WILSON_OUTLIER_D_MAX = 4.0f; + +WilsonOutlierResult WilsonOutliers(const std::vector &obs, double alpha); diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 7e787b1fe..5c4537331 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -770,6 +770,23 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, // Outlier rejection drops observations from the merge AND from R_meas and the CC1/2 // half-sets, so a run that rejects too much scores BETTER on every other number here. Add(s, KeyInt("OBSERVATIONS_REJECTED", ms.n_observations_rejected)); + // The part of those removed as improbable under Wilson statistics (an observation with no + // symmetry mates to be judged against, or out-voting them); each is listed in the developer + // report. + Add(s, KeyInt("OBSERVATIONS_REJECTED_WILSON", static_cast(ms.wilson_rejected.size()))); + if (!ms.wilson_rejected.empty()) { + std::string list = " Observations rejected by the Wilson test:\n" + " h k l d E^2 image x y\n"; + constexpr size_t MAX_ROWS = 100; + for (size_t i = 0; i < std::min(ms.wilson_rejected.size(), MAX_ROWS); ++i) { + const auto &w = ms.wilson_rejected[i]; + list += fmt::format(" {:6d} {:4d} {:4d} {:6.3f} {:7.1f} {:8.1f} {:8.1f} {:8.1f}\n", + w.h, w.k, w.l, w.d, w.e2, w.image, w.x, w.y); + } + if (ms.wilson_rejected.size() > MAX_ROWS) + list += fmt::format(" ... and {} more\n", ms.wilson_rejected.size() - MAX_ROWS); + Add(s, Prose(list, true)); + } // One rule for every quantity: a run that did not measure it writes NO key, rather than // the word "nan" or a zero that reads as a measured absence. if (o.possible_unique_reflections > 0) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index ae752e42e..64cd2092d 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -107,6 +107,7 @@ ADD_EXECUTABLE(jfjoch_test TranslationalNCSTest.cpp RfreeFlagsTest.cpp FrenchWilsonTest.cpp + WilsonOutliersTest.cpp ReindexAmbiguityTest.cpp LoadReferenceMtzFreeFlagsTest.cpp WriteReflectionsTest.cpp diff --git a/tests/WilsonOutliersTest.cpp b/tests/WilsonOutliersTest.cpp new file mode 100644 index 000000000..e5de6494c --- /dev/null +++ b/tests/WilsonOutliersTest.cpp @@ -0,0 +1,125 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include + +#include +#include +#include + +#include "../image_analysis/scale_merge/WilsonOutliers.h" + +namespace { + // Acentric Wilson intensities of mean `mean` between 3.9 and 2.0 A, measured `per_unit` times each + // with Poisson-like noise of variance I + bkg_var. + std::vector WilsonPopulation(int n_units, int per_unit, double mean, double bkg_var, + uint32_t seed) { + std::mt19937 rng(seed); + std::exponential_distribution wilson(1.0 / mean); + std::normal_distribution gauss(0.0, 1.0); + std::vector v; + for (int u = 0; u < n_units; ++u) { + const double I_true = wilson(rng); + const float d = 3.9f - 1.9f * static_cast(u) / n_units; + for (int m = 0; m < per_unit; ++m) { + const double sigma = std::sqrt(I_true + bkg_var); + v.push_back({static_cast(I_true + sigma * gauss(rng)), static_cast(sigma), d, + 1.0f, false, u}); + } + } + return v; + } +} + +TEST_CASE("WilsonOutliers: the artefact member of a discordant pair is dropped", "[wilson_outliers]") { + auto obs = WilsonPopulation(10000, 2, 1000.0, 100.0, 1); + // A pair of one ordinary observation and one two hundred times the shell mean. + const int32_t unit = 10000; + obs.push_back({800.0f, 30.0f, 2.5f, 1.0f, false, unit}); + obs.push_back({200000.0f, 450.0f, 2.5f, 1.0f, false, unit}); + + const auto r = WilsonOutliers(obs, 0.01); + CHECK(r.n_tested == obs.size()); + CHECK(r.tail_scale == Catch::Approx(1.0).margin(0.2)); + CHECK(r.rejected[obs.size() - 1] == 1); + CHECK(r.rejected[obs.size() - 2] == 0); + CHECK(r.n_rejected == 1); +} + +TEST_CASE("WilsonOutliers: two large observations of one reflection confirm each other", "[wilson_outliers]") { + auto obs = WilsonPopulation(10000, 2, 1000.0, 100.0, 2); + const int32_t unit = 10000; + obs.push_back({60000.0f, 300.0f, 2.5f, 1.0f, false, unit}); + obs.push_back({61000.0f, 300.0f, 2.5f, 1.0f, false, unit}); + + const auto r = WilsonOutliers(obs, 0.01); + CHECK(r.n_rejected == 0); +} + +TEST_CASE("WilsonOutliers: the symmetry enhancement factor keeps an axial reflection", "[wilson_outliers]") { + auto obs = WilsonPopulation(10000, 2, 1000.0, 100.0, 3); + // Measured once, 12x at epsilon 4: 48x the shell mean, but only E^2 = 12. + obs.push_back({48000.0f, 250.0f, 2.5f, 4.0f, false, 10000}); + auto r = WilsonOutliers(obs, 0.01); + CHECK(r.rejected.back() == 0); + CHECK(r.e2.back() == Catch::Approx(12.0).epsilon(0.1)); + CHECK(r.n_rejected == 0); + + // The same observation on a general reflection is improbable. + obs.back().epsilon = 1.0f; + r = WilsonOutliers(obs, 0.01); + CHECK(r.rejected.back() == 1); +} + +TEST_CASE("WilsonOutliers: a weak shell does not misfire", "[wilson_outliers]") { + // = 20 under a background of sd 100: noise alone reaches twenty times , while the shell mean + // is still established. + const auto obs = WilsonPopulation(20000, 1, 20.0, 10000.0, 4); + const auto r = WilsonOutliers(obs, 0.01); + CHECK(r.n_tested == obs.size()); + CHECK(r.n_rejected == 0); +} + +TEST_CASE("WilsonOutliers: an artefact among several mates is dropped", "[wilson_outliers]") { + auto obs = WilsonPopulation(5000, 4, 1000.0, 100.0, 5); + // One precise-looking artefact beside three ordinary mates: the mates out-vote it. + obs[0].I = 500000.0f; + obs[0].sigma = 700.0f; + const auto r = WilsonOutliers(obs, 0.01); + CHECK(r.n_tested == obs.size()); + CHECK(r.rejected[0] == 1); + CHECK(r.n_rejected == 1); +} + +TEST_CASE("WilsonOutliers: a reflection whose observations are mostly large is kept", "[wilson_outliers]") { + auto obs = WilsonPopulation(5000, 3, 1000.0, 100.0, 8); + // Two of three observations large, the third low (a partial that caught little, say). + obs[0].I = 60000.0f; obs[0].sigma = 300.0f; + obs[1].I = 62000.0f; obs[1].sigma = 300.0f; + obs[2].I = 500.0f; obs[2].sigma = 30.0f; + const auto r = WilsonOutliers(obs, 0.01); + CHECK(r.n_rejected == 0); +} + +TEST_CASE("WilsonOutliers: a shell without a measured mean is not judged", "[wilson_outliers]") { + // Pure noise: = 0 within its error, so nothing can be improbable against it. + auto obs = WilsonPopulation(10000, 1, 1e-6, 10000.0, 9); + obs[0].I = 5000.0f; obs[0].sigma = 100.0f; + const auto r = WilsonOutliers(obs, 0.01); + CHECK(r.n_tested == 0); + CHECK(r.n_rejected == 0); +} + +TEST_CASE("WilsonOutliers: a heavier-tailed population widens the bound", "[wilson_outliers]") { + // Half the reflections at twice the mean, half at a tenth - the intensity classes of a strong + // pseudo-translation. Wilson's single exponential would call the top of the strong class improbable. + auto strong = WilsonPopulation(10000, 1, 2000.0, 100.0, 6); + const auto weak = WilsonPopulation(10000, 1, 100.0, 100.0, 7); + for (auto o : weak) { + o.unit += 10000; + strong.push_back(o); + } + const auto r = WilsonOutliers(strong, 0.01); + CHECK(r.tail_scale > 1.5); + CHECK(r.n_rejected == 0); +} From b1327033aed235061e258e46638c303317d855da Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 23 Sep 2026 19:36:53 +0200 Subject: [PATCH 06/21] rugnux: --polarization is a polarization degree; the default is an undulator value The help text claimed 0.99 is right for every synchrotron in the corpus. Bending-magnet, superbend and wiggler sources are plausibly 0.8-0.95, no file format read states the value reliably, and it cannot be fitted from a single dataset (it correlates with spindle-symmetric absorption). Say what the number is (degree p; XDS FRACTION_OF_POLARIZATION = (1+p)/2) and that a known beamline value can be passed. Documentation only; the default stays 0.99. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CPU_DATA_ANALYSIS_IMAGE.md | 5 ++++- rugnux/rugnux_cli.cpp | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_IMAGE.md b/docs/CPU_DATA_ANALYSIS_IMAGE.md index 4175042f4..dc00bb54f 100644 --- a/docs/CPU_DATA_ANALYSIS_IMAGE.md +++ b/docs/CPU_DATA_ANALYSIS_IMAGE.md @@ -407,7 +407,10 @@ The **polarization plane** is taken to contain the lab $x$ axis — a horizontal A vertically polarized one is expressed by a **negative** $P$: flipping the sign of the $\cos 2\phi$ term is exactly a $90^\circ$ rotation of the plane. The plane is **not** autodetected and is not read from the file — no format Rugnux reads declares one — so a vertical beamline must say so with -`--polarization`. The default $P = 0.99$ is applied to every Rugnux run. +`--polarization`. $P$ is the polarization *degree* (XDS's `FRACTION_OF_POLARIZATION` is $(1+P)/2$). +The default $P = 0.99$, an undulator value, is applied to every Rugnux run; it is not fitted, because +a single dataset does not determine it. Bending-magnet and wiggler beamlines are typically lower +(0.8–0.95), and a known value for one should be passed with `--polarization`. ### 2.3 Background estimate for profiles diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 55f38c0c0..2e0f81ac0 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -219,7 +219,7 @@ void print_usage() { std::cout << " --rot3 PONI rotation 3, about the beam (rad)" << std::endl; std::cout << " --detector-mirror-y Stored image is mirrored in Y vs the detector" << std::endl; std::cout << " --detector-quarter-turns <0-3> Stored image is turned by this many 90 deg about the beam" << std::endl; - std::cout << " --polarization Degree of polarization, signed: positive for a horizontally polarized source, negative for a vertically polarized one (the sign rotates the polarization plane by 90 degrees). Default 0.99, which is right for every synchrotron in the corpus; no file format rugnux reads declares a polarization plane, so a vertical beamline has to say so here" << std::endl; + std::cout << " --polarization Degree of polarization p (XDS FRACTION_OF_POLARIZATION = (1+p)/2), signed: positive for a horizontally polarized source, negative for a vertically polarized one (the sign rotates the polarization plane by 90 degrees). Default 0.99, an undulator value; bending-magnet and wiggler sources are typically lower (0.8-0.95), and no file format rugnux reads states it reliably, so pass a known beamline value here" << std::endl; } enum { From a2c281055175c619e9ac53a2ae86e3153cc21d58 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 23 Sep 2026 20:51:52 +0200 Subject: [PATCH 07/21] Merge: a clipped observation is no witness in the Wilson test The pair rule drops the improbable (higher) member of a discordant pair. If the lower member lost its core to saturation or the mask, its profile estimate of what remained may be low, and a genuinely strong reflection would be replaced by the clipped value - the failure Aimless guards against. Integration now marks a reflection whose signal disk was not fully readable (Reflection::clipped, from the engines' existing full-disk test); the flag is carried through the partials to the fulls (OR over an event's partials, CPU and GPU combine alike), and the Wilson test never counts a clipped observation as a probable witness against a larger one. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- common/Reflection.h | 4 +++ docs/CPU_DATA_ANALYSIS_DECISIONS.md | 2 +- .../BraggIntegrationEngine.cpp | 2 ++ .../scale_merge/RotationScaleMerge.cpp | 20 +++++++++----- .../scale_merge/RotationScaleMerge.h | 5 ++-- .../scale_merge/RotationScaleMergeGPU.cu | 26 +++++++++++++------ .../scale_merge/RotationScaleMergeGPU.h | 4 ++- image_analysis/scale_merge/WilsonOutliers.cpp | 8 +++--- image_analysis/scale_merge/WilsonOutliers.h | 5 +++- tests/WilsonOutliersTest.cpp | 22 +++++++++++----- 10 files changed, 69 insertions(+), 29 deletions(-) diff --git a/common/Reflection.h b/common/Reflection.h index 2e304cd79..b5ae95f65 100644 --- a/common/Reflection.h +++ b/common/Reflection.h @@ -54,6 +54,10 @@ struct Reflection { float image_scale_corr; // I_true = image_scale_corr * I; = prescaling_corr * qe_corr * flight_corr / (partiality * image_scale) bool observed = false; bool on_ice_ring = false; // sits on a hexagonal-ice powder ring: excluded from scaling, kept for merging + // The signal disk lost pixels to the mask or to saturation, so the intensity is the profile's + // estimate over what remained. Such a measurement may be low, and the merge does not let it + // testify against a larger observation of the same reflection (WilsonOutliers.h). + bool clipped = false; }; struct MergedReflection { diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index 3a8b7fe74..f2a303428 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -64,7 +64,7 @@ Beside them, on rotation data, the run reports **twin-immune zone evidence** for Merging applies an optional per-observation median-based $N\sigma$ cut (`--reject-outliers`, default 6σ for `rot3d`, off otherwise). The same $N\sigma$ cut is fed back into the error model: after an initial $a,b$ fit the parameters are re-fit once on the reflections that survive rejection (dropping any whose squared deviation exceeds $N^2\,[a\,\sigma^2 + (b\,\langle I\rangle)^2]$), so the calibrated errors describe the reflections that actually enter the merge rather than the pre-rejection pool. -The median needs three observations, so a reflection measured once or twice — typically one good observation and one artefact (a hot pixel, a zinger) after Friedel merging — is never tested by it, and an artefact that looks precise (thousands of counts, a small relative $\sigma$) can outweigh mates from weak frames and become the median itself. Every observation of the written merge is therefore also judged by **Wilson statistics**. Beyond 4 Å its $E^2 = I/(\varepsilon\,\langle I/\varepsilon\rangle_\mathrm{shell})$ is compared with a bound set by a budget of $\alpha = 0.01$ expected false rejections per dataset: with $N$ observations tested, $\ln(2N/\alpha)$ for acentrics and twice that for centrics, and the observation's lower confidence limit $I - z\sigma$ (with its own $\sigma$, and $z$ from the same budget) must exceed it, so noise in weak shells does not trigger it; a shell whose $\langle I\rangle$ is not itself established at that significance is not judged. The bound is widened by the tail scale of the dataset's own intensity distribution, measured on its well-measured observations (1 for a Wilson crystal; larger under a pseudo-translation or anisotropy). An improbable singleton is dropped; an improbable observation with company only when most of the reflection's other observations are probable and it disagrees with their mean beyond the errors — several large observations confirm each other. The count is `OBSERVATIONS_REJECTED_WILSON=` in the report (included in `OBSERVATIONS_REJECTED=`), and the developer report lists each observation with its image and detector position. +The median needs three observations, so a reflection measured once or twice — typically one good observation and one artefact (a hot pixel, a zinger) after Friedel merging — is never tested by it, and an artefact that looks precise (thousands of counts, a small relative $\sigma$) can outweigh mates from weak frames and become the median itself. Every observation of the written merge is therefore also judged by **Wilson statistics**. Beyond 4 Å its $E^2 = I/(\varepsilon\,\langle I/\varepsilon\rangle_\mathrm{shell})$ is compared with a bound set by a budget of $\alpha = 0.01$ expected false rejections per dataset: with $N$ observations tested, $\ln(2N/\alpha)$ for acentrics and twice that for centrics, and the observation's lower confidence limit $I - z\sigma$ (with its own $\sigma$, and $z$ from the same budget) must exceed it, so noise in weak shells does not trigger it; a shell whose $\langle I\rangle$ is not itself established at that significance is not judged. The bound is widened by the tail scale of the dataset's own intensity distribution, measured on its well-measured observations (1 for a Wilson crystal; larger under a pseudo-translation or anisotropy). An improbable singleton is dropped; an improbable observation with company only when most of the reflection's other observations are probable and it disagrees with their mean beyond the errors — several large observations confirm each other. An observation whose signal disk lost pixels to the mask or to saturation is never counted as such a witness, since its profile estimate may be low: a strong reflection is not replaced by its clipped mate. The count is `OBSERVATIONS_REJECTED_WILSON=` in the report (included in `OBSERVATIONS_REJECTED=`), and the developer report lists each observation with its image and detector position. ### 13.4 Automatic resolution cutoff diff --git a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp index af9ad59ac..a0a8b263d 100644 --- a/image_analysis/bragg_integration/BraggIntegrationEngine.cpp +++ b/image_analysis/bragg_integration/BraggIntegrationEngine.cpp @@ -228,6 +228,8 @@ std::vector BraggIntegrationEngine::Finalize(const std::vector fbuf[NFLOAT]; for (auto &b : fbuf) b.reset(new float[slice]); std::unique_ptr frm(new int32_t[slice]); - std::unique_ptr onice(new uint8_t[slice]); + std::unique_ptr onice(new uint8_t[slice]), clip(new uint8_t[slice]); for (int at = 0; at < n; at += slice) { const int cnt = std::min(n - at, slice); ParallelChunks(cnt, nthreads, [&](int lo, int hi) { @@ -563,7 +563,7 @@ void RotationScaleMerge::Ingest() { fbuf[3][i] = o.partiality; fbuf[4][i] = o.zeta; fbuf[5][i] = o.corr; fbuf[6][i] = o.bkg; fbuf[7][i] = o.var_bkg; fbuf[8][i] = o.image_number; fbuf[9][i] = o.d; fbuf[10][i] = o.px; fbuf[11][i] = o.py; - frm[i] = o.frame; onice[i] = o.on_ice; + frm[i] = o.frame; onice[i] = o.on_ice; clip[i] = o.clipped; } }); constexpr F fields[NFLOAT] = {F::I, F::Sigma, F::PrescalingCorr, F::Partiality, @@ -573,6 +573,7 @@ void RotationScaleMerge::Ingest() { gpu_->SetObsField(fields[k], at, cnt, fbuf[k].get()); gpu_->SetObsFrame(at, cnt, frm.get()); gpu_->SetObsOnIce(at, cnt, onice.get()); + gpu_->SetObsClipped(at, cnt, clip.get()); } } if (gpu_active_) { @@ -753,7 +754,7 @@ void RotationScaleMerge::BuildInRangeObservations(std::vector &keys) { std::unique_ptr fbuf[NFLOAT]; for (auto &b : fbuf) b.reset(new float[bufsize]); const std::unique_ptr frm(new int32_t[bufsize]); - const std::unique_ptr onice(new uint8_t[bufsize]); + const std::unique_ptr onice(new uint8_t[bufsize]), clip(new uint8_t[bufsize]); int f0 = 0; while (f0 < n_frames) { int f1 = f0 + 1; @@ -783,6 +784,7 @@ void RotationScaleMerge::BuildInRangeObservations(std::vector &keys) { fbuf[10][at] = r.predicted_y; frm[at] = o; onice[at] = r.on_ice_ring ? 1 : 0; + clip[at] = r.clipped ? 1 : 0; ++at; } }); @@ -792,6 +794,7 @@ void RotationScaleMerge::BuildInRangeObservations(std::vector &keys) { gpu_->SetObsField(fields[k], base, cnt, fbuf[k].get()); gpu_->SetObsFrame(base, cnt, frm.get()); gpu_->SetObsOnIce(base, cnt, onice.get()); + gpu_->SetObsClipped(base, cnt, clip.get()); } f0 = f1; } @@ -818,6 +821,7 @@ void RotationScaleMerge::BuildInRangeObservations(std::vector &keys) { obs.image_number = r.image_number; obs.frame = o; obs.on_ice = r.on_ice_ring ? 1 : 0; + obs.clipped = r.clipped ? 1 : 0; obs.corr = r.image_scale_corr; obs.group = -1; finite_ok[at] = (std::isfinite(obs.I) && std::isfinite(obs.prescaling_corr) && obs.prescaling_corr != 0.0f @@ -3728,6 +3732,8 @@ void RotationScaleMerge::Combine() { float peak_px = partials[ev[i]].px, peak_py = partials[ev[i]].py; float peak_partiality = -1.0f; const bool on_ice = partials[ev[i]].on_ice; + bool clipped = false; + for (size_t m = i; m < kk; ++m) clipped = clipped || partials[ev[m]].clipped; for (size_t m = i; m < kk; ++m) { const auto &r2 = partials[ev[m]]; const double sigma_corr = static_cast(r2.sigma) * r2.corr; @@ -3798,6 +3804,7 @@ void RotationScaleMerge::Combine() { full.px = peak_px; full.py = peak_py; full.frame = peak_outcome; full.on_ice = on_ice ? 1 : 0; + full.clipped = clipped ? 1 : 0; full.group = group; // the raw hkl's ASU group for the current space group (<0 = absent) out.push_back(full); @@ -4537,7 +4544,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool const Obs &o = fulls[i]; const float I_corr = o.I * o.corr; wobs.push_back({I_corr, model_sigma(o, I_corr, o.sigma * o.corr), o.d, group_epsilon[g], - group_centric[g] != 0, merge_friedel ? g : pair_of_group[g]}); + group_centric[g] != 0, o.clipped != 0, merge_friedel ? g : pair_of_group[g]}); wilson_full.push_back(i); } wilson = WilsonOutliers(wobs, WILSON_ALPHA); @@ -5622,7 +5629,8 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st FullsStaging &st = fulls_staging; st.Resize(nf); gpu_->GetFulls(st.h.data(), st.k.data(), st.l.data(), st.I.data(), st.sigma.data(), st.d.data(), - st.image_number.data(), st.frame.data(), st.on_ice.data(), st.group.data()); + st.image_number.data(), st.frame.data(), st.on_ice.data(), st.clipped.data(), + st.group.data()); gpu_->GetFullsPxPy(st.px.data(), st.py.data()); gpu_->GetFullsVariance(st.var_bkg.data(), st.var_per_I.data()); if (scaled_fulls_on_gpu) gpu_->GetFullsCorr(st.corr.data()); @@ -5636,7 +5644,7 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search, bool full_st o.corr = scaled_fulls_on_gpu ? st.corr[i] : 1.0f; o.image_number = st.image_number[i]; o.frame = st.frame[i]; o.px = st.px[i]; o.py = st.py[i]; - o.on_ice = st.on_ice[i]; o.group = st.group[i]; + o.on_ice = st.on_ice[i]; o.clipped = st.clipped[i]; o.group = st.group[i]; // A full has no rocking geometry of its own and carries no background; the combine on // the device does not produce these, and a reused buffer would keep the last chain's. o.zeta = 0.0f; o.delta_phi = 0.0f; o.bkg = 0.0f; diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index 03a4e75fe..5f5e7a6e0 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -139,6 +139,7 @@ private: float image_number; // fractional frame position (for 3D-combine contiguity) int32_t frame; // index of the outcome whose per-frame scale G applies to this obs uint8_t on_ice; + uint8_t clipped; // Reflection::clipped; a full is clipped if any of its partials is float corr; // image_scale_corr (working; updated by scaling) int32_t group; // dense ASU-group id for the current space group; <0 = never mergeable }; @@ -318,12 +319,12 @@ private: struct FullsStaging { std::vector h, k, l, frame, group; std::vector I, sigma, d, image_number, corr, px, py, var_bkg, var_per_I; - std::vector on_ice; + std::vector on_ice, clipped; void Resize(int n) { h.resize(n); k.resize(n); l.resize(n); frame.resize(n); group.resize(n); I.resize(n); sigma.resize(n); d.resize(n); image_number.resize(n); corr.resize(n); px.resize(n); py.resize(n); var_bkg.resize(n); var_per_I.resize(n); - on_ice.resize(n); + on_ice.resize(n); clipped.resize(n); } }; FullsStaging fulls_staging; diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.cu b/image_analysis/scale_merge/RotationScaleMergeGPU.cu index 4f4317e6e..ccce57c9d 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.cu +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.cu @@ -265,7 +265,7 @@ namespace { *__restrict__ partiality, *__restrict__ bkg, *__restrict__ var_bkg, *__restrict__ image_number, *__restrict__ d, *__restrict__ px, *__restrict__ py; const int32_t *__restrict__ frame; - const uint8_t *__restrict__ on_ice; + const uint8_t *__restrict__ on_ice, *__restrict__ clipped; const int32_t *__restrict__ perm, *__restrict__ rr_start, *__restrict__ rr_count, *__restrict__ rr_h, *__restrict__ rr_k, *__restrict__ rr_l, *__restrict__ rr_group; @@ -273,7 +273,7 @@ namespace { const int32_t *rr_offset; // emit pass: per-run base offset into the fulls arrays int32_t *f_h, *f_k, *f_l, *f_frame, *f_group; float *f_I, *f_sigma, *f_d, *f_img, *f_px, *f_py, *f_var_bkg, *f_var_per_I; - uint8_t *f_on_ice; + uint8_t *f_on_ice, *f_clipped; }; // One thread's raw-hkl run: split its usable partials (already in image_number order within the run) @@ -334,6 +334,8 @@ namespace { float peak_px = p.px[first], peak_py = p.py[first]; float peak_partiality = -1.0f; const bool on_ice = p.on_ice[first]; + bool clipped = false; + for (int m = ev_start; m <= ev_end; ++m) clipped = clipped || p.clipped[p.perm[m]]; for (int m = ev_start; m <= ev_end; ++m) { const int i = p.perm[m]; if (!CombineUsable(i, p.I, p.sigma, p.corr)) continue; @@ -405,6 +407,7 @@ namespace { p.f_px[o] = peak_px; p.f_py[o] = peak_py; p.f_frame[o] = peak_outcome; p.f_on_ice[o] = on_ice ? 1 : 0; + p.f_clipped[o] = clipped ? 1 : 0; p.f_group[o] = group; } ++n_emit; @@ -632,7 +635,7 @@ struct RotationScaleMergeGPU::Impl { // immutable per-obs CudaDevicePtr I, sigma, prescaling_corr, partiality, zeta; - CudaDevicePtr on_ice; + CudaDevicePtr on_ice, clipped; CudaDevicePtr frame; CudaDevicePtr corr; // mutable, resident across iterations CudaDevicePtr frame_start, frame_count; @@ -681,7 +684,7 @@ struct RotationScaleMergeGPU::Impl { int n_fulls = 0; CudaDevicePtr f_h, f_k, f_l, f_frame, f_group; CudaDevicePtr f_I, f_sigma, f_d, f_img, f_px, f_py, f_var_bkg, f_var_per_I; - CudaDevicePtr f_on_ice; + CudaDevicePtr f_on_ice, f_clipped; // scale-fulls (Unity model, kept resident): all-ones partiality/prescaling_corr/zeta so the shared scaling kernels // yield coeff=mean, plus the working corr, the per-obs scale scratch, and the fulls frame/group CSRs // (built on the host from the small f_frame/f_group key arrays, over the emit-ordered fulls). @@ -749,6 +752,7 @@ void RotationScaleMergeGPU::SetPartialsLayout(int n_obs, int n_frames, d.px_obs = CudaDevicePtr(n); d.py_obs = CudaDevicePtr(n); d.frame = CudaDevicePtr(n); d.on_ice = CudaDevicePtr(n); + d.clipped = CudaDevicePtr(n); d.g = CudaDevicePtr(n_frames); d.scaled = CudaDevicePtr(n_frames); d.inv_sigma = CudaDevicePtr(n_obs); @@ -796,6 +800,11 @@ void RotationScaleMergeGPU::SetObsOnIce(int offset, int count, const uint8_t *on UploadChunk(impl_->on_ice, offset, count, on_ice); } +void RotationScaleMergeGPU::SetObsClipped(int offset, int count, const uint8_t *clipped) { + DeviceGuard guard(impl_->device, impl_->available); + UploadChunk(impl_->clipped, offset, count, clipped); +} + void RotationScaleMergeGPU::SetGroups(int n_groups, const int32_t *group, const int32_t *group_perm, int n_group_perm, const int32_t *group_start, const int32_t *group_count) { @@ -1124,7 +1133,7 @@ int RotationScaleMergeGPU::Combine(const int32_t *rawrun_group, double min_parti p.I = d.I.get(); p.sigma = d.sigma.get(); p.corr = d.corr.get(); p.partiality = d.partiality.get(); p.bkg = d.bkg.get(); p.var_bkg = d.var_bkg.get(); p.image_number = d.image_number.get(); p.d = d.d_obs.get(); p.px = d.px_obs.get(); p.py = d.py_obs.get(); - p.frame = d.frame.get(); p.on_ice = d.on_ice.get(); + p.frame = d.frame.get(); p.on_ice = d.on_ice.get(); p.clipped = d.clipped.get(); p.perm = d.perm.get(); p.rr_start = d.rr_start.get(); p.rr_count = d.rr_count.get(); p.rr_h = d.rr_h.get(); p.rr_k = d.rr_k.get(); p.rr_l = d.rr_l.get(); p.rr_group = d.rr_group.get(); p.rr_nevents = d.rr_nevents.get(); @@ -1152,7 +1161,7 @@ int RotationScaleMergeGPU::Combine(const int32_t *rawrun_group, double min_parti d.f_d = CudaDevicePtr(nf); d.f_img = CudaDevicePtr(nf); d.f_px = CudaDevicePtr(nf); d.f_py = CudaDevicePtr(nf); d.f_var_bkg = CudaDevicePtr(nf); d.f_var_per_I = CudaDevicePtr(nf); - d.f_on_ice = CudaDevicePtr(nf); + d.f_on_ice = CudaDevicePtr(nf); d.f_clipped = CudaDevicePtr(nf); d.f_corr = CudaDevicePtr(nf); d.f_partiality = CudaDevicePtr(nf); d.f_rlp = CudaDevicePtr(nf); d.f_zeta = CudaDevicePtr(nf); d.f_inv_sigma = CudaDevicePtr(nf); @@ -1166,7 +1175,7 @@ int RotationScaleMergeGPU::Combine(const int32_t *rawrun_group, double min_parti p.f_I = d.f_I.get(); p.f_sigma = d.f_sigma.get(); p.f_d = d.f_d.get(); p.f_img = d.f_img.get(); p.f_px = d.f_px.get(); p.f_py = d.f_py.get(); p.f_var_bkg = d.f_var_bkg.get(); p.f_var_per_I = d.f_var_per_I.get(); - p.f_on_ice = d.f_on_ice.get(); + p.f_on_ice = d.f_on_ice.get(); p.f_clipped = d.f_clipped.get(); if (d.n_fulls > 0) { CombineKernel<<>>(p); CudaCheck(cudaGetLastError(), "combine emit launch"); @@ -1177,7 +1186,7 @@ int RotationScaleMergeGPU::Combine(const int32_t *rawrun_group, double min_parti void RotationScaleMergeGPU::GetFulls(int32_t *h, int32_t *k, int32_t *l, float *I, float *sigma, float *d, float *image_number, int32_t *frame, uint8_t *on_ice, - int32_t *group) const { + uint8_t *clipped, int32_t *group) const { DeviceGuard guard(impl_->device, impl_->available); const auto &dd = *impl_; const size_t n = static_cast(dd.n_fulls); @@ -1191,6 +1200,7 @@ void RotationScaleMergeGPU::GetFulls(int32_t *h, int32_t *k, int32_t *l, float * dl(I, dd.f_I.get(), n * sizeof(float)); dl(sigma, dd.f_sigma.get(), n * sizeof(float)); dl(d, dd.f_d.get(), n * sizeof(float)); dl(image_number, dd.f_img.get(), n * sizeof(float)); dl(on_ice, dd.f_on_ice.get(), n * sizeof(uint8_t)); + dl(clipped, dd.f_clipped.get(), n * sizeof(uint8_t)); } void RotationScaleMergeGPU::GetFullsKeys(int32_t *frame, int32_t *group) const { diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.h b/image_analysis/scale_merge/RotationScaleMergeGPU.h index 1f33d2e18..4aad3fce8 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.h +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.h @@ -44,6 +44,7 @@ public: void SetObsField(ObsField f, int offset, int count, const float *v); void SetObsFrame(int offset, int count, const int32_t *frame); void SetObsOnIce(int offset, int count, const uint8_t *on_ice); + void SetObsClipped(int offset, int count, const uint8_t *clipped); // Per space group: the dense ASU-group id per obs, and a group-ordered permutation of the obs whose // group >= 0 (group_perm), with its CSR (group_start/group_count, length n_groups) - so each group's @@ -144,7 +145,8 @@ public: // Download the combined fulls SoA (length = Combine()'s return). The working corr is downloaded // separately by GetFullsCorr (it is only meaningful after ScaleFulls; otherwise the caller sets it). void GetFulls(int32_t *h, int32_t *k, int32_t *l, float *I, float *sigma, float *d, - float *image_number, int32_t *frame, uint8_t *on_ice, int32_t *group) const; + float *image_number, int32_t *frame, uint8_t *on_ice, uint8_t *clipped, + int32_t *group) const; // Download the fulls' predicted detector position (peak partial's px/py), for the host absorption // surface. Length = n_fulls. diff --git a/image_analysis/scale_merge/WilsonOutliers.cpp b/image_analysis/scale_merge/WilsonOutliers.cpp index b98ca4814..c1bd0ae6f 100644 --- a/image_analysis/scale_merge/WilsonOutliers.cpp +++ b/image_analysis/scale_merge/WilsonOutliers.cpp @@ -133,9 +133,9 @@ WilsonOutlierResult WilsonOutliers(const std::vector &obs, do }); // An improbable observation measured once is rejected. One with company is rejected when most of - // its reflection's other observations are probable and it disagrees with their mean beyond the - // errors - for a pair, the other member. Where most are large, the reflection is (Aimless's "keep - // if most observations are large"). + // its reflection's other observations are probable, unclipped witnesses and it disagrees with their + // mean beyond the errors - for a pair, the other member. Where most are large, the reflection is + // (Aimless's "keep if most observations are large"). for (size_t lo = 0; lo < tested.size();) { size_t hi = lo; while (hi < tested.size() && obs[tested[hi]].unit == obs[tested[lo]].unit) ++hi; @@ -146,7 +146,7 @@ WilsonOutlierResult WilsonOutliers(const std::vector &obs, do double sw = 0.0, swI = 0.0; for (size_t r = lo; r < hi; ++r) { const auto &o = obs[tested[r]]; - if (r == q || improbable[tested[r]]) continue; + if (r == q || improbable[tested[r]] || o.clipped) continue; const double w = 1.0 / (static_cast(o.sigma) * o.sigma); sw += w; swI += w * o.I; ++n_probable; diff --git a/image_analysis/scale_merge/WilsonOutliers.h b/image_analysis/scale_merge/WilsonOutliers.h index 8fd908f4f..20ff9506e 100644 --- a/image_analysis/scale_merge/WilsonOutliers.h +++ b/image_analysis/scale_merge/WilsonOutliers.h @@ -33,13 +33,16 @@ // Improbable is not yet rejected. An improbable observation measured once is rejected; one with // company only when most of its reflection's other observations are probable and it disagrees with // them beyond the errors. Several large observations of one reflection confirm each other and are -// kept, which is how a genuinely strong reflection survives. +// kept, which is how a genuinely strong reflection survives. An observation whose signal disk lost +// pixels to the mask or to saturation is not counted as a probable witness: its profile estimate of +// what remained may be low, and a strong reflection must not be replaced by its clipped mate. struct WilsonObservation { float I; // scaled intensity float sigma; // its own standard uncertainty, at its own intensity float d; // resolution (A) float epsilon; // symmetry enhancement factor of the reflection (without centring) bool centric; + bool clipped; // lost pixels to the mask or to saturation: may be low, so never a witness int32_t unit; // the reflection the observation measures; < 0 = not part of the test }; diff --git a/tests/WilsonOutliersTest.cpp b/tests/WilsonOutliersTest.cpp index e5de6494c..ae901362d 100644 --- a/tests/WilsonOutliersTest.cpp +++ b/tests/WilsonOutliersTest.cpp @@ -24,7 +24,7 @@ namespace { for (int m = 0; m < per_unit; ++m) { const double sigma = std::sqrt(I_true + bkg_var); v.push_back({static_cast(I_true + sigma * gauss(rng)), static_cast(sigma), d, - 1.0f, false, u}); + 1.0f, false, false, u}); } } return v; @@ -35,8 +35,8 @@ TEST_CASE("WilsonOutliers: the artefact member of a discordant pair is dropped", auto obs = WilsonPopulation(10000, 2, 1000.0, 100.0, 1); // A pair of one ordinary observation and one two hundred times the shell mean. const int32_t unit = 10000; - obs.push_back({800.0f, 30.0f, 2.5f, 1.0f, false, unit}); - obs.push_back({200000.0f, 450.0f, 2.5f, 1.0f, false, unit}); + obs.push_back({800.0f, 30.0f, 2.5f, 1.0f, false, false, unit}); + obs.push_back({200000.0f, 450.0f, 2.5f, 1.0f, false, false, unit}); const auto r = WilsonOutliers(obs, 0.01); CHECK(r.n_tested == obs.size()); @@ -46,11 +46,21 @@ TEST_CASE("WilsonOutliers: the artefact member of a discordant pair is dropped", CHECK(r.n_rejected == 1); } +TEST_CASE("WilsonOutliers: a clipped mate does not testify against a strong observation", "[wilson_outliers]") { + auto obs = WilsonPopulation(10000, 2, 1000.0, 100.0, 10); + // The strong member is real; the low one lost its saturated core to the mask. + const int32_t unit = 10000; + obs.push_back({800.0f, 30.0f, 2.5f, 1.0f, false, true, unit}); + obs.push_back({200000.0f, 450.0f, 2.5f, 1.0f, false, false, unit}); + const auto r = WilsonOutliers(obs, 0.01); + CHECK(r.n_rejected == 0); +} + TEST_CASE("WilsonOutliers: two large observations of one reflection confirm each other", "[wilson_outliers]") { auto obs = WilsonPopulation(10000, 2, 1000.0, 100.0, 2); const int32_t unit = 10000; - obs.push_back({60000.0f, 300.0f, 2.5f, 1.0f, false, unit}); - obs.push_back({61000.0f, 300.0f, 2.5f, 1.0f, false, unit}); + obs.push_back({60000.0f, 300.0f, 2.5f, 1.0f, false, false, unit}); + obs.push_back({61000.0f, 300.0f, 2.5f, 1.0f, false, false, unit}); const auto r = WilsonOutliers(obs, 0.01); CHECK(r.n_rejected == 0); @@ -59,7 +69,7 @@ TEST_CASE("WilsonOutliers: two large observations of one reflection confirm each TEST_CASE("WilsonOutliers: the symmetry enhancement factor keeps an axial reflection", "[wilson_outliers]") { auto obs = WilsonPopulation(10000, 2, 1000.0, 100.0, 3); // Measured once, 12x at epsilon 4: 48x the shell mean, but only E^2 = 12. - obs.push_back({48000.0f, 250.0f, 2.5f, 4.0f, false, 10000}); + obs.push_back({48000.0f, 250.0f, 2.5f, 4.0f, false, false, 10000}); auto r = WilsonOutliers(obs, 0.01); CHECK(r.rejected.back() == 0); CHECK(r.e2.back() == Catch::Approx(12.0).epsilon(0.1)); From 9b6736dbda57c5fb617e3435f158955e3957d20e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 07:27:11 +0200 Subject: [PATCH 08/21] Changelog: hot-pixel mask, Wilson outlier test, bandwidth report, surface gate, polarization docs Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index aae489881..a2928f583 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,6 +1,14 @@ # Changelog ## 1.0.0 +### 1.0.0-rc.173 + +* Rugnux masks detector pixels that stay hot on every frame but are missing from the file's pixel mask, found on the frames it already reads before integration. +* Rugnux rejects single observations that Wilson statistics make implausible (typically a hot pixel, zinger or ice spot), including pairs and single measurements that the equivalents test cannot judge; the count is reported as `OBSERVATIONS_REJECTED_WILSON`. +* Rugnux reports the X-ray bandwidth it measures from spot shapes (for multilayer and pink beams); the value is reported only and does not change processing. +* Rugnux accepts a detector-modulation or absorption correction surface when held-out data support it on Fisher's z, and fits them in the order modulation, time, goniometer frame; this recovers corrections that were wrongly refused. +* `rugnux --polarization` is documented as the polarization degree (XDS `FRACTION_OF_POLARIZATION` = (1 + p)/2); the default 0.99 suits undulators. + ### 1.0.0-rc.172 * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. From 032bf9fe2bbf0e03f55b11a82dd0dcef2c63466a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 11:34:06 +0200 Subject: [PATCH 09/21] Rugnux: walk the goniometer rotation scale to its fixed point, decided on the whole sweep The pass-1 post-refinement fits the rotation scale k only on the frames the stored angles still track, and a rate error is exactly what stops them tracking the rest: on a sweep whose stage turned ~3 % slow the fit read 0.979 over 94 deg, failed its leave-a-fifth-out test and was thrown away, leaving half the frames unscaled. The pass no longer decides. Between the passes, at the pass-2 detector geometry, the lattice is indexed (index-only probe) under the stored angles and under the fitted k and scored on the validation frames of the whole sweep: share of the spots on the lattice beyond the wrong-spindle null. k is adopted only where it scores higher by more than the binomial noise of the two (ValidationEvidencePrefers - the test the beam-centre arms already used, now one function); the run then integrates and post-refines at k (post-refine-only probe), fits again on top of it and repeats until the next k no longer scores better (WalkRotationScale). The stored angles are the first hypothesis. Measured: 1.000 25.1 %, 0.97874 58.9 %, 0.97041 90.0 %, 0.97006 90.7 % (not significant) -> 0.97041 adopted. Probes restore the experiment, the pass-2 geometry, pass-1 mosaicity and the beam-centre-search flag; a probe opens no beam-centre search. Forced pass-1 results get their axis scaled; the header revert drops the scale. GONIOMETER_ROTATION_SCALE reports the adopted k (SUSPECT = adopted). The leave-a-fifth-out figure stays in the log only. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CPU_DATA_ANALYSIS_INDEXING.md | 2 +- image_analysis/geom_refinement/PostRefine.cpp | 53 +---- image_analysis/geom_refinement/PostRefine.h | 4 +- rugnux/Rugnux.cpp | 212 +++++++++++++----- rugnux/Rugnux.h | 48 +++- tests/CMakeLists.txt | 1 + tests/RotationScaleWalkTest.cpp | 94 ++++++++ 7 files changed, 316 insertions(+), 98 deletions(-) create mode 100644 tests/RotationScaleWalkTest.cpp diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index 8f7e5cc7f..d219691ea 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -335,7 +335,7 @@ The space group is determined **after** pass 2, on the geometry the run refined, Only pass 2 is written, as the canonical `_*` output. Pass 1's merge exists to give the guard something to judge pass 2 against, so it stops short of the parts of the merge that only fill in a file — the correction surfaces, the twinning and radiation-damage analyses, the R-free flags and the amplitudes — and writes no merged files of its own. -**Goniometer rotation scale (report only).** A stage that turns further than it was commanded to leaves no trace in the file, because the stored $\omega$ values *are* the commanded ones; the excess then presents as the crystal drifting, in this program and in others. The excitation residual already measures it without a new degree of freedom: it rotates by $-\phi\,\mathbf{u}$ with $\mathbf{u}$ an **unnormalised** 3-vector, so $|\mathbf{u}|$ is the factor by which the stage actually turned, and normalising the axis throws it away. It is reported, and warned about beyond 0.5 %, under its own leave-a-fifth-of-the-sweep-out check — a fold that merely soaked up noise cannot raise the flag. It is a detector, not a calibration: nothing corrects the data, and it **under-reads** the true magnitude, because the fit only sees reflections that indexed at the nominal angle and per-frame orientation refinement has already absorbed part of the error. +**Goniometer rotation scale.** A stage that turns further than it was commanded to leaves no trace in the file, because the stored $\omega$ values *are* the commanded ones; the excess then presents as the crystal drifting, in this program and in others. The excitation residual already measures it without a new degree of freedom: it rotates by $-\phi\,\mathbf{u}$ with $\mathbf{u}$ an **unnormalised** 3-vector, so $|\mathbf{u}|$ is the factor $k$ by which the stage actually turned, and normalising the axis throws it away. Pass 1 fits $k$ as a single parameter on its rocking events, with the crystal and the axis direction held at their committed values and the angle measured from the centre of the sweep. That fit **under-reads** a real error: it only sees the frames the stored angles still track, and a rate error is exactly what stops them tracking the rest. So it is not acted on directly. Between the passes, at the detector geometry pass 2 runs at, the lattice is indexed under the stored angles and under the fitted $k$, and each is scored on the validation frames spread over the whole sweep, as the share of their spots it puts on the lattice beyond what it puts there at a wrong spindle angle. The fitted $k$ is adopted only where it scores higher by more than the binomial noise of the two scores (z = 3.29); the run then integrates and post-refines at it, fits $k$ again on top of it, and repeats until the next $k$ no longer scores better - the fixed point of the fit. Otherwise the stored angles stand. The adopted $k$ drives every later pass (prediction, integration, scaling and the reported oscillation) and is reported as `GONIOMETER_ROTATION_SCALE`, with `GONIOMETER_ROTATION_SCALE_SUSPECT= TRUE`. `--rotation-scale ` asserts a calibration and skips all of this. ### 7.6 Detector geometry from powder rings diff --git a/image_analysis/geom_refinement/PostRefine.cpp b/image_analysis/geom_refinement/PostRefine.cpp index a2fc800f0..4acd73472 100644 --- a/image_analysis/geom_refinement/PostRefine.cpp +++ b/image_analysis/geom_refinement/PostRefine.cpp @@ -1080,50 +1080,19 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, const double k_fit = solve_scale(-1); result.rotation_scale = k_fit; - // ----- Whether to COMMIT it. A stage fault is rare - 36 of 37 rotation datasets sit at 1.0000 - // on a direct scan - and a 1 % angle correction applied to a healthy dataset would damage it - // silently, so every test below has to pass. - // Preconditions: below these the fit is reported but never acted on. Under ~30 deg of sweep k - // entangles with the axis direction and 10-20 deg truncations of a perfect dataset wander by - // +-0.6 %; a screening wedge must not trigger a correction. - constexpr int MIN_SCALE_EVENTS = 5000; - constexpr double MIN_SCALE_SWEEP_DEG = 30.0; - // T1 significance: 0.5 % is 18 sigma on the between-dataset scatter of healthy stages - // (robust sd 2.8e-4) and still 3.5x below the one measured fault. - constexpr double ROTATION_SCALE_TOL = 0.005; - // T2 relevance: the misorientation the error produces at each end of the sweep. A large k over - // a short sweep moves nothing and is not worth correcting. - constexpr double MIN_SCALE_END_ERROR_DEG = 0.5; - // T3 uniformity: a stage error is a ramp present in EVERY part of the sweep, so dropping any - // fifth of it must leave the same k. A second lattice that dominates ONE END of the sweep - - // exactly what happens where the primary stops indexing - fakes a k indistinguishable from a - // real fault on T1 and T2, and is the reason this test is not optional. It replaces the - // hkl-hash split used elsewhere here, which cannot see it: both halves of that split sit at - // the same angles, so anything structured in phi survives in both folds. - constexpr double MIN_SCALE_JACKKNIFE_FRAC = 0.5; - const double end_error_deg = std::fabs(k_fit - 1.0) * sweep_deg / 2.0; - const bool enough_data = static_cast(n_events) >= MIN_SCALE_EVENTS - && sweep_deg >= MIN_SCALE_SWEEP_DEG; - const bool big_enough = enough_data && std::fabs(k_fit - 1.0) >= ROTATION_SCALE_TOL - && end_error_deg >= MIN_SCALE_END_ERROR_DEG; + // Whether the same k comes back with any fifth of the sweep left out, as the smallest share + // of the fitted excess the folds keep: a stage error is a ramp present in EVERY part of the + // sweep. Reported, not acted on - the fit only sees the frames the angles it was measured at + // still track, and a rate error is exactly what stops them tracking the rest, so the part + // it sees can be too short to agree with itself. What acts on k is the caller, which walks + // it to its fixed point and decides it on the whole sweep (rugnux WalkRotationScale). double jackknife = 1.0; - if (big_enough) - for (int f = 0; f < 5; ++f) - jackknife = std::min(jackknife, (solve_scale(f) - 1.0) / (k_fit - 1.0)); - result.rotation_scale_suspect = big_enough && jackknife >= MIN_SCALE_JACKKNIFE_FRAC; + for (int f = 0; f < 5; ++f) + jackknife = std::min(jackknife, (solve_scale(f) - 1.0) / (k_fit - 1.0)); logger.Info("Post-refine rotation SCALE: k = {:.5f} over {:.0f} deg of sweep centred on {:.1f} " - "deg ({} events): end error {:.2f} deg, leave-a-fifth-out {:.2f} => {}", - k_fit, sweep_deg, phi_c * 180.0 / PI, n_events, end_error_deg, jackknife, - result.rotation_scale_suspect ? "COMMIT" - : !enough_data ? "report only (too little sweep or too few events)" - : "reject (kept the stored angles)"); - if (result.rotation_scale_suspect) - logger.Warning("Goniometer rotation scale looks off by {:+.2f} % (fitted {:.5f}): the stage " - "appears to have turned {} than the angles stored in the file, which are the " - "COMMANDED values. This is a hardware calibration fault, not a data problem - " - "left uncorrected it inflates mosaicity, biases the cell and loses " - "high-resolution reflections", - 100.0 * (k_fit - 1.0), k_fit, k_fit > 1.0 ? "further" : "less far"); + "deg ({} events): end error {:.2f} deg, leave-a-fifth-out {:.2f}", + k_fit, sweep_deg, phi_c * 180.0 / PI, n_events, + std::fabs(k_fit - 1.0) * sweep_deg / 2.0, jackknife); // Assemble the committed geometry. result.distance_after_mm = dist[0]; diff --git a/image_analysis/geom_refinement/PostRefine.h b/image_analysis/geom_refinement/PostRefine.h index b5dcf079b..6ed478587 100644 --- a/image_analysis/geom_refinement/PostRefine.h +++ b/image_analysis/geom_refinement/PostRefine.h @@ -73,8 +73,8 @@ struct PostRefineResult { // the header). Fitted after the joint fit as a single free parameter, with the crystal and the axis // direction held at their committed values. Always the fitted value; 1.0 = header and stage agree. double rotation_scale = 1.0; - // Whether the fit passed every test needed to ACT on it: enough sweep and events, a significant and - // physically relevant size, and the same k from every fifth of the sweep. Only then is it applied. + // Whether the run ACTED on it: set by the caller where the scale, walked to its fixed point, was + // adopted on the evidence of the whole sweep (rugnux WalkRotationScale) - never by the fit itself. bool rotation_scale_suspect = false; }; diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 6c5fdcc3f..05d9296ec 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -75,6 +75,48 @@ #include #include +bool ValidationEvidencePrefers(const ValidationSpotEvidence ¤t, const ValidationSpotEvidence &candidate) { + const auto excess = [](const ValidationSpotEvidence &e) { + return static_cast(e.on_lattice - e.by_chance) / static_cast(std::max(1, e.spots)); + }; + const auto rate_var = [](const ValidationSpotEvidence &e) { + const double n = static_cast(std::max(1, e.spots)); + const double p = static_cast(e.on_lattice) / n; + return p * (1.0 - p) / n; + }; + return excess(candidate) - excess(current) + > SPOT_BUDGET_SIGNIFICANCE_Z * std::sqrt(rate_var(candidate) + rate_var(current)); +} + +RotationScaleWalk WalkRotationScale(double first_fit, + const std::function &index_at, + const std::function(float)> &refit_at, + int max_rounds) { + RotationScaleWalk walk; + auto next = static_cast(first_fit); + if (next == walk.scale) + return walk; + const auto score = [](const ValidationSpotEvidence &e) { + return 100.0 * static_cast(e.on_lattice - e.by_chance) + / static_cast(std::max(1, e.spots)); + }; + walk.evidence = index_at(walk.scale); + walk.trail = fmt::format("{:.5f}: {:.1f}%", walk.scale, score(walk.evidence)); + for (int round = 0; round < max_rounds && next != walk.scale; ++round) { + const ValidationSpotEvidence e = index_at(next); + walk.trail += fmt::format(", {:.5f}: {:.1f}%", next, score(e)); + if (!ValidationEvidencePrefers(walk.evidence, e)) + break; + walk.scale = next; + walk.evidence = e; + const auto fit = refit_at(walk.scale); + if (!fit) + break; + next = static_cast(walk.scale * *fit); + } + return walk; +} + double MetricViolation(const UnitCell &uc, const gemmi::SpaceGroup &sg) { const double a = uc.a, b = uc.b, c = uc.c; const double d2r = PI / 180.0; @@ -2024,7 +2066,6 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { const DiffractionExperiment file_experiment = experiment_; const auto file_mosaicity = prepass_mosaicity_; const auto file_geometry = prepass_detector_geometry_; - const auto file_scale = prepass_rotation_scale_; const auto file_result = prepass_result_; experiment_ = experiment_before_first_pass; @@ -2036,7 +2077,6 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { experiment_.BeamX_pxl(alt_center[0]).BeamY_pxl(alt_center[1]); prepass_mosaicity_.clear(); prepass_detector_geometry_.reset(); - prepass_rotation_scale_.reset(); prepass_result_.reset(); ProcessResult alt; @@ -2113,7 +2153,6 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { experiment_ = file_experiment; prepass_mosaicity_ = file_mosaicity; prepass_detector_geometry_ = file_geometry; - prepass_rotation_scale_ = file_scale; prepass_result_ = file_result; } // The arm's own pass asks the same question again at its own centre; the run has already @@ -2152,10 +2191,90 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { experiment_.BeamX_pxl(g[0]).BeamY_pxl(g[1]).DetectorDistance_mm(g[2]) .PoniRot1_rad(g[3]).PoniRot2_rad(g[4]); } - // ... and the goniometer rotation scale, on the same measure-then-re-integrate footing: the angles - // in the file are the commanded ones, so a stage that ran fast is a geometry error like any other. + // The probe passes below and the lattice arms after them exist ONLY to measure - see there - so + // the count of passes the run made has to include them even though they write nothing. + int arm_passes = 0; + + // The goniometer rotation scale: the angles in the file are the commanded ones, so a stage that + // turned at the wrong rate is a geometry error like any other - but one pass 1's fit reads only + // in part (see WalkRotationScale), so it is walked to the fit's fixed point here and adopted + // only where the validation frames of the whole sweep prefer it. The walk runs at the detector + // geometry the second pass will, so every hypothesis, the stored angles included, is scored + // there, and at that geometry only: a probe whose angles lose the lattice must score what it + // scores, not start the beam-centre search that a poor first pass otherwise opens. Its passes + // only measure - an indexing probe stops once the lattice is scored, a refit stops once the + // post-refinement has measured - and they leave nothing behind: the experiment, the geometry + // the second pass is to run at, pass 1's mosaicity (a width in degrees fitted against the + // stored angles) and whether the run has searched for the beam centre are put back when the + // walk ends. + std::string rotation_scale_walk; + if (!cancelled_ && gonio_snapshot && pass1.post_refine && !config_.rotation_scale) { + const DiffractionExperiment before_walk = experiment_; + const auto geometry_before_walk = prepass_detector_geometry_; + const auto mosaicity_before_walk = prepass_mosaicity_; + const bool searched_before_walk = beam_center_searched_; + beam_center_searched_ = true; + const auto probe = [&](float k, bool index_only) { + experiment_ = before_walk; + experiment_.Goniometer(ScaleRotation(*before_walk.GetGoniometer(), k)); + prepass_rotation_scale_ = k; + prepass_mosaicity_.clear(); + indexing_probe_only_ = index_only; + postrefine_probe_ = postrefine_probe_only_ = !index_only; + ProcessResult r; + try { + r = RunPipeline(observer, /*write_output=*/false, /*geometry_prepass=*/false); + } catch (const std::exception &e) { + if (IsFatalResourceError(e)) throw; + // Angles under which nothing indexes have scored nothing, which is what an empty + // result reads as. + logger.Info("Rotation scale {:.5f}: the probe pass did not complete ({})", k, e.what()); + } + indexing_probe_only_ = postrefine_probe_ = postrefine_probe_only_ = false; + ++arm_passes; + return r; + }; + const auto index_at = [&](float k) { return probe(k, true).validation_evidence; }; + const auto refit_at = [&](float k) -> std::optional { + const ProcessResult r = probe(k, false); + if (!r.post_refine) + return std::nullopt; + logger.Info("Rotation scale {:.5f}: the post-refinement there fits {:.5f} on top of it, " + "held-out residual {:.3e}", k, r.post_refine->rotation_scale, + r.post_refine->held_out_before); + return r.post_refine->rotation_scale; + }; + constexpr int MAX_ROTATION_SCALE_ROUNDS = 8; + const RotationScaleWalk walk = WalkRotationScale(pass1.post_refine->rotation_scale, index_at, + refit_at, MAX_ROTATION_SCALE_ROUNDS); + experiment_ = before_walk; + prepass_detector_geometry_ = geometry_before_walk; + prepass_mosaicity_ = mosaicity_before_walk; + beam_center_searched_ = searched_before_walk; + prepass_rotation_scale_.reset(); + if (!walk.trail.empty()) { + rotation_scale_walk = fmt::format( + "goniometer rotation scale walked from the pass-1 fit {:.5f}, validation spots on " + "the lattice beyond chance by scale: {} - {}", pass1.post_refine->rotation_scale, + walk.trail, walk.scale == 1.0f ? "the stored angles stand" + : fmt::format("{:.5f} adopted", walk.scale)); + logger.Info("Two-pass: {}", rotation_scale_walk); + } + if (walk.scale != 1.0f) { + prepass_rotation_scale_ = walk.scale; + pass1.post_refine->rotation_scale = walk.scale; + pass1.post_refine->rotation_scale_suspect = true; + logger.Warning("Goniometer rotation scale {:.5f} ({:+.2f} %): the stage turned {} than the " + "angles stored in the file, which are the COMMANDED values. The second pass " + "integrates at the corrected angles; the fault is in the hardware and should " + "be fixed there", walk.scale, 100.0 * (walk.scale - 1.0), + walk.scale > 1.0f ? "further" : "less far"); + } + } + + // ... and apply the adopted scale, on the same measure-then-re-integrate footing as the geometry. if (prepass_rotation_scale_ && gonio_snapshot) { - experiment_.Goniometer(ScaleRotation(*gonio_snapshot, *prepass_rotation_scale_)); + experiment_.Goniometer(ScaleRotation(*experiment_.GetGoniometer(), *prepass_rotation_scale_)); // The pre-pass mosaicity is a width in degrees fitted against the angles the second pass has // just stopped using, and the override can only ever raise the second pass's own estimate (it // takes the larger of the two). Carrying it over would hold the second pass at the rocking @@ -2197,10 +2316,6 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { .PoniRot1_rad(g[3]).PoniRot2_rad(g[4]); }; - // The lattice arms below run passes that exist ONLY to measure - see there - so the - // count of passes the run made has to include them even though they write nothing. - int arm_passes = 0; - // The metric symmetry, asked of the spots rather than of the tolerance that admitted it. The // Bravais class is chosen by a walk that reads two axes as EQUAL when they agree to a fixed // relative tolerance, and the class carrying that equality is then imposed everywhere below: @@ -2548,6 +2663,8 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { if (const auto current = experiment_.GetGoniometer()) restored.Axis(current->GetAxis()); experiment_.Goniometer(restored); + prepass_rotation_scale_.reset(); + pass1.post_refine->rotation_scale_suspect = false; } config_.output_prefix = base_prefix; // This pass is the answer whatever it measures - the guard has had its one chance - @@ -2573,6 +2690,8 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { : "the post-refinement committed no geometry change, so this pass reproduces the first"; if (!lattice_arm.empty()) pass2.pass_decision = lattice_arm + "; " + pass2.pass_decision; + if (!rotation_scale_walk.empty()) + pass2.pass_decision = rotation_scale_walk + "; " + pass2.pass_decision; pass2.geometry_not_converged = geometry_not_converged; return pass2; } @@ -3234,11 +3353,7 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // The pooled evidence for one candidate lattice: the validation frames' non-ice spots, how many // of them lie on the candidate, and how many a wrong spindle angle still puts there. - struct PooledEvidence { - int64_t spots = 0; - int64_t on_lattice = 0; - int64_t by_chance = 0; - }; + using PooledEvidence = ValidationSpotEvidence; // Displacements are a fixed fraction of the sweep, so the null is a function of the data alone // and the same file gives the same verdict every time. Sevenths: no crystallographic rotation @@ -3274,19 +3389,10 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b return static_cast(e.on_lattice - e.by_chance) > SPOT_BUDGET_SIGNIFICANCE_Z * sigma; }; - // The same quantity for COMPARING two lattices: the share of the pooled spots each one puts on - // itself over and above what a wrong spindle angle puts there. Each is scored against its own - // null, so a denser lattice is not credited for the spots it catches by accident - which is - // what makes the comparison fair between a cell and its axis harmonic. - auto excess = [](const PooledEvidence &e) { - return static_cast(e.on_lattice - e.by_chance) - / static_cast(std::max(1, e.spots)); - }; - auto rate_var = [](const PooledEvidence &e) { - const double n = static_cast(std::max(1, e.spots)); - const double p = static_cast(e.on_lattice) / n; - return p * (1.0 - p) / n; - }; + // Two lattices are COMPARED on the same quantity, ValidationEvidencePrefers: the share of the + // pooled spots each one puts on itself over and above what a wrong spindle angle puts there. + // Each is scored against its own null, which is what makes the comparison fair between a cell + // and its axis harmonic. // How deep into an image's intensity-ordered spot list this lattice is still being seen - the // measured spot budget. Same frames, same per-image path as count_indexed above, but scoring the @@ -4033,10 +4139,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b restore_beam_center(header_x, header_y); if (best.result.has_value()) at_header = pooled_evidence(*indexer, *best.result); - const double margin = excess(at_measured) - excess(at_header); - const double margin_sigma = std::sqrt(rate_var(at_measured) + rate_var(at_header)); if (alt.result.has_value() && beats_chance(at_measured) - && margin > SPOT_BUDGET_SIGNIFICANCE_Z * margin_sigma) { + && ValidationEvidencePrefers(at_header, at_measured)) { try_beam_center(measured_x, measured_y); logger.Warning("Beam centre from the background: neither centre indexes a " "validation frame on its own, but the pooled spots put " @@ -4179,11 +4283,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b const PooledEvidence at_header = pooled_evidence(*indexer, *best.result); try_beam_center(measured_x, measured_y); const PooledEvidence at_measured = pooled_evidence(*indexer, *alt.result); - const double margin = excess(at_measured) - excess(at_header); - const double margin_sigma = std::sqrt(rate_var(at_measured) - + rate_var(at_header)); adopted_measured = beats_chance(at_measured) - && margin > SPOT_BUDGET_SIGNIFICANCE_Z * margin_sigma; + && ValidationEvidencePrefers(at_header, at_measured); if (adopted_measured) { logger.Warning("Beam centre check: the larger cell is the measured " "centre's, and it is the one the spots are on - " @@ -4639,6 +4740,10 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b gemmi::crystal_system_str(prepass_result_->search_result.system), pc.a, pc.b, pc.c, pc.alpha, pc.beta, pc.gamma); best.result = *prepass_result_; + // Pass 1's result carries pass 1's goniometer; forcing it whole would put the + // uncorrected angles back (the same repair as the supercell re-run in RunAllPasses). + if (prepass_rotation_scale_ && best.result->axis) + best.result->axis = ScaleRotation(*best.result->axis, *prepass_rotation_scale_); // Pass 1's lattice was FITTED at pass 1's detector distance, and this pass // integrates at the post-refined one. Re-scoring it here is not the same as // re-fitting it: a real-space cell is measured against the distance the spots were @@ -4685,6 +4790,16 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b / static_cast(std::max(1, evidence.spots)); const double pooled_chance = static_cast(evidence.by_chance) / static_cast(std::max(1, evidence.spots)); + result.validation_evidence = evidence; + // A pass run only to score a lattice has its score, whatever it is: a lattice that does not + // beat chance is a result for the comparison that asked, not a reason to stop the run. + if (indexing_probe_only_) { + logger.Info("Indexing probe: scheme '{}', {}/{} validation frames, {}/{} validation spots " + "= {:.1f}% against {:.1f}% at a wrong spindle angle", best.name, best.score, + static_cast(validation.size()), evidence.on_lattice, evidence.spots, + 100.0 * pooled, 100.0 * pooled_chance); + return result; + } if (!beats_chance(evidence)) { { // Name the cell and Bravais class that was rejected. The commonest cause is a metric @@ -4821,7 +4936,10 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b "pass 1 ({}-centred, {:.0f} A^3) - integrating with pass-1's lattice instead", best.result->search_result.centering, v2, prepass_result_->search_result.centering, v1); - indexer->ForceRotationIndexerResult(*prepass_result_); + RotationIndexerResult forced = *prepass_result_; + if (prepass_rotation_scale_ && forced.axis) + forced.axis = ScaleRotation(*forced.axis, *prepass_rotation_scale_); + indexer->ForceRotationIndexerResult(forced); } } } @@ -5529,19 +5647,11 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b logger.Info("Two-pass: geometry post-refine committed no detector change{}", geometry_prepass ? " - the second pass reproduces the first" : ""); } - // DECISION POINT for the goniometer rotation scale. It does not ride on pr.ok - the - // scale is its own cross-validated fit and the crystals that have a stage fault are - // exactly the ones whose cell and detector steps do NOT pass, because the angle error - // is what their residual is made of. A calibration fault is rare (36 of 37 rotation - // datasets sit at 1.0000) and applying a 1 % angle correction to a healthy dataset - // would silently damage it, so the asymmetry is deliberate: committed only when the - // fit is both cross-validated and outside the tolerance. A manual --rotation-scale is - // already on the goniometer and is left alone. - // Only the pre-pass fits the rotation scale. It is applied from the goniometer - // the file came with, so a scale measured again on angles that have already been - // corrected once is not a correction that can be applied on top of that one. - if (pr.rotation_scale_suspect && !config_.rotation_scale.has_value() && geometry_prepass) - prepass_rotation_scale_ = static_cast(pr.rotation_scale); + // The goniometer rotation scale in pr is only a fit: RunAllPasses walks it to its + // fixed point and decides it on the validation frames of the whole sweep + // (WalkRotationScale). It does not ride on pr.ok - the crystals that have a stage + // fault are exactly the ones whose cell and detector steps do NOT pass, because the + // angle error is what their residual is made of. } } }; @@ -5559,8 +5669,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b if (postrefine_probe_only_) { result.processing_time_s = std::chrono::duration( std::chrono::steady_clock::now() - start_time).count(); - logger.Info("Lattice-arm probe: {} images integrated and post-refined in {:.2f} s " - "(the arm reads the held-out residual only, so this pass does not merge)", + logger.Info("Probe pass: {} images integrated and post-refined in {:.2f} s " + "(the comparison reads the post-refinement only, so this pass does not merge)", result.images_processed, result.processing_time_s); return result; } diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index c74278625..d6375cd58 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -186,6 +187,42 @@ struct ProcessConfig { bool finalist_ledger = false; // --finalist-ledger; report-only symmetry evidence table }; +// A rotation lattice's score on the validation frames spread over the whole sweep: their spots (off +// the ice rings unless those are indexed), how many lie on the lattice, and how many still do at a +// wrong spindle angle - the median over displaced angles, which is what chance and the per-frame +// orientation polish give for free. +struct ValidationSpotEvidence { + int64_t spots = 0; + int64_t on_lattice = 0; + int64_t by_chance = 0; +}; + +// Whether `candidate` puts a larger share of its validation spots on its lattice, over and above what +// a wrong spindle angle puts there, than `current` does - by more than the binomial noise of the two +// shares (SPOT_BUDGET_SIGNIFICANCE_Z). Each is measured against its own null, so a denser lattice is +// not credited for the spots it catches by accident. +bool ValidationEvidencePrefers(const ValidationSpotEvidence ¤t, const ValidationSpotEvidence &candidate); + +// The goniometer rotation scale - the factor by which the stage turned relative to the angles stored +// in the file - walked to the fit's fixed point and decided on the whole sweep. The fit only sees the +// frames the angles it was measured at still track, and a stage at the wrong rate is exactly what +// stops them tracking the rest, so one fit reads only part of the error. So: index the lattice under +// the fitted scale and under the angles in hand; where the fitted one scores better on the validation +// frames (ValidationEvidencePrefers), adopt it, fit again there and repeat. The stored angles (k = 1) +// are the first hypothesis, and stand unless the evidence moves the run off them. +// index_at(k): the validation evidence of the lattice indexed with the stored angles scaled by k +// refit_at(k): the scale the post-refinement fits on reflections integrated at k, relative to k; +// empty where it fitted none +struct RotationScaleWalk { + float scale = 1.0f; // adopted; 1 = the stored angles stand + ValidationSpotEvidence evidence; // at the adopted scale + std::string trail; // every scale tried, with its validation score +}; +RotationScaleWalk WalkRotationScale(double first_fit, + const std::function &index_at, + const std::function(float)> &refit_at, + int max_rounds); + struct ProcessResult { bool cancelled = false; uint64_t images_processed = 0; @@ -233,6 +270,8 @@ struct ProcessResult { // cell - so two cells can only be compared for volume once this has brought both to primitive. std::optional consensus_centering; bool rotation_lattice_found = false; + // The rotation lattice's score on the validation frames; all zero where no lattice was scored. + ValidationSpotEvidence validation_evidence; // The metric symmetry the rotation lattice was classified in - the class GeometryRefiner holds the // cell to and the space-group search enumerates under. Empty when no rotation lattice was found. std::optional rotation_lattice_type; @@ -484,8 +523,9 @@ class Rugnux { // describes neither geometry. The two travel together or not at all. std::optional> prepass_detector_geometry_; - // Two-pass geometry pre-pass: the goniometer rotation scale the post-refine fitted and flagged as a - // stage fault, applied by Run() to the second pass's goniometer. Empty when the fit found nothing. + // The goniometer rotation scale the run adopted (WalkRotationScale, in RunAllPasses), applied to the + // second pass's goniometer - and, while the walk runs, the scale its current probe pass is at. + // Empty where the stored angles stand. std::optional prepass_rotation_scale_; // Two-pass geometry pre-pass: pass-1's FULL indexing result (correct lattice + orientation + refined @@ -591,6 +631,10 @@ class Rugnux { // Everything below that point - the merges, the space-group search, the correction surfaces, the // reports - is work no comparison looks at, and on a long sweep it is four fifths of the pass. bool postrefine_probe_only_ = false; + // Whether a pass exists only to index: it stops as soon as its first-pass indexing has scored the + // lattice on the validation frames (ProcessResult::validation_evidence), before a single image is + // integrated. The rotation-scale walk (see RunAllPasses) compares angle models on that score. + bool indexing_probe_only_ = false; // The file's detector distance, from before the first pass of the rotation two-pass: a pass whose // distance is still this one asks the post-refinement to test "the header distance is right" // (PostRefineSettings::distance_at_header); a pass that has walked off it does not. diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 64cd2092d..24f86cd3e 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -59,6 +59,7 @@ ADD_EXECUTABLE(jfjoch_test ResultReportTest.cpp DiagnosticOutputTest.cpp PostRefineTest.cpp + RotationScaleWalkTest.cpp RugnuxLargeTest.cpp TestData.h MovingAverageTest.cpp diff --git a/tests/RotationScaleWalkTest.cpp b/tests/RotationScaleWalkTest.cpp new file mode 100644 index 000000000..562d46a24 --- /dev/null +++ b/tests/RotationScaleWalkTest.cpp @@ -0,0 +1,94 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include +#include + +#include "../rugnux/Rugnux.h" + +namespace { + // A synthetic sweep whose stage turned `true_scale` times the stored angles. Scored at a scale k, + // the validation spots stay on the lattice as long as the angles track the rotation, and the share + // that does falls off with the relative rate error; a wrong spindle angle keeps half a percent. + struct SyntheticSweep { + double true_scale; + int64_t spots = 35000; + int index_calls = 0; + int refit_calls = 0; + + ValidationSpotEvidence IndexAt(float k) { + ++index_calls; + const double error = std::fabs(true_scale / k - 1.0); + const double on = 0.9 * std::max(0.0, 1.0 - 30.0 * error); + return ValidationSpotEvidence{spots, std::llround(on * spots), std::llround(0.005 * spots)}; + } + + // The post-refinement at k, relative to k. It reads only 70 % of the error that is left: it + // sees only the frames the angles at k still track. + std::optional RefitAt(float k) { + ++refit_calls; + return 1.0 + 0.7 * (true_scale / k - 1.0); + } + + RotationScaleWalk Walk(double first_fit) { + return WalkRotationScale(first_fit, [this](float k) { return IndexAt(k); }, + [this](float k) { return RefitAt(k); }, 8); + } + }; +} + +TEST_CASE("ValidationEvidencePrefers", "[RotationScale]") { + const ValidationSpotEvidence base{10000, 3000, 50}; + // 1 % more of the spots beyond chance is under the noise of two 30 % shares over 10000 spots + // (sqrt(2 * 0.3 * 0.7 / 10000) = 0.65 %, times 3.29); 5 % is well over it. + CHECK_FALSE(ValidationEvidencePrefers(base, {10000, 3100, 50})); + CHECK(ValidationEvidencePrefers(base, {10000, 3500, 50})); + // A candidate is judged against its own null: more spots on the lattice bought by a null that + // rose just as much is no gain. + CHECK_FALSE(ValidationEvidencePrefers(base, {10000, 3500, 550})); + // Never against itself, and nothing that scored nothing wins. + CHECK_FALSE(ValidationEvidencePrefers(base, base)); + CHECK_FALSE(ValidationEvidencePrefers(base, {})); + CHECK(ValidationEvidencePrefers({}, base)); +} + +TEST_CASE("WalkRotationScale_ReachesTheFixedPoint", "[RotationScale]") { + // A stage 3 % slow. The first fit reads 70 % of that; each refit at the adopted scale reads 70 % + // of what is left, and the walk goes on as long as the validation frames prefer the new scale. + SyntheticSweep sweep{0.97}; + const auto walk = sweep.Walk(sweep.RefitAt(1.0f).value()); + CHECK(walk.scale == Catch::Approx(0.97).margin(0.001)); + CHECK(walk.scale != 1.0f); + CHECK(walk.evidence.on_lattice > sweep.IndexAt(1.0f).on_lattice); + CHECK(sweep.refit_calls > 2); + CHECK_FALSE(walk.trail.empty()); +} + +TEST_CASE("WalkRotationScale_StoredAnglesStand", "[RotationScale]") { + SECTION("A healthy stage: a fit off by noise scores no better than the stored angles") { + SyntheticSweep sweep{1.0}; + const auto walk = sweep.Walk(1.0002); + CHECK(walk.scale == 1.0f); + CHECK(sweep.index_calls == 2); // the stored angles and the fit, nothing more + CHECK(sweep.refit_calls == 0); + } + SECTION("A real but small error the spots cannot resolve beyond their noise") { + SyntheticSweep sweep{0.999}; + sweep.spots = 400; + const auto walk = sweep.Walk(0.9993); + CHECK(walk.scale == 1.0f); + } + SECTION("A fit that tracks something other than the rotation scores worse, and is refused") { + SyntheticSweep sweep{1.0}; + const auto walk = sweep.Walk(0.98); + CHECK(walk.scale == 1.0f); + CHECK(sweep.refit_calls == 0); + } + SECTION("A fit of exactly one asks for no probe at all") { + SyntheticSweep sweep{1.0}; + const auto walk = sweep.Walk(1.0); + CHECK(walk.scale == 1.0f); + CHECK(walk.trail.empty()); + CHECK(sweep.index_calls == 0); + } +} From 5c28f35f85f425ac10332ded26dc5261467560cc Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 11:37:38 +0200 Subject: [PATCH 10/21] Rugnux: report the oscillation the run integrated at; test the rotation-scale fit on a simulated sweep OSCILLATION_RANGE was read off the caller's experiment, so a run that adopted a goniometer rotation scale still reported the file's oscillation; it is now scaled by the adopted k (the starting angle is the stage's and stays). The post-refinement logs the range of k over the leave-a-fifth-out folds instead of a ratio to k - 1, which is meaningless near k = 1. PostRefine_RotationScale: partials of a 180 deg sweep simulated at a known stage rate are fitted back to it (0.97 and 1.0, to 1e-3). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- image_analysis/geom_refinement/PostRefine.cpp | 25 ++++--- rugnux/ResultReport.cpp | 6 +- tests/PostRefineTest.cpp | 74 +++++++++++++++++++ tests/ResultReportTest.cpp | 25 +++++++ 4 files changed, 118 insertions(+), 12 deletions(-) diff --git a/image_analysis/geom_refinement/PostRefine.cpp b/image_analysis/geom_refinement/PostRefine.cpp index 4acd73472..c713826ea 100644 --- a/image_analysis/geom_refinement/PostRefine.cpp +++ b/image_analysis/geom_refinement/PostRefine.cpp @@ -1080,19 +1080,22 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, const double k_fit = solve_scale(-1); result.rotation_scale = k_fit; - // Whether the same k comes back with any fifth of the sweep left out, as the smallest share - // of the fitted excess the folds keep: a stage error is a ramp present in EVERY part of the - // sweep. Reported, not acted on - the fit only sees the frames the angles it was measured at - // still track, and a rate error is exactly what stops them tracking the rest, so the part - // it sees can be too short to agree with itself. What acts on k is the caller, which walks - // it to its fixed point and decides it on the whole sweep (rugnux WalkRotationScale). - double jackknife = 1.0; - for (int f = 0; f < 5; ++f) - jackknife = std::min(jackknife, (solve_scale(f) - 1.0) / (k_fit - 1.0)); + // What k comes back with each fifth of the sweep left out: a stage error is a ramp present + // in EVERY part of the sweep, so the folds should agree. Reported, not acted on - the fit + // only sees the frames the angles it was measured at still track, and a rate error is + // exactly what stops them tracking the rest, so the part it sees can be too short to agree + // with itself. What acts on k is the caller, which walks it to its fixed point and decides + // it on the whole sweep (rugnux WalkRotationScale). + double fold_lo = k_fit, fold_hi = k_fit; + for (int f = 0; f < 5; ++f) { + const double k_fold = solve_scale(f); + fold_lo = std::min(fold_lo, k_fold); + fold_hi = std::max(fold_hi, k_fold); + } logger.Info("Post-refine rotation SCALE: k = {:.5f} over {:.0f} deg of sweep centred on {:.1f} " - "deg ({} events): end error {:.2f} deg, leave-a-fifth-out {:.2f}", + "deg ({} events): end error {:.2f} deg, leave-a-fifth-out {:.5f} to {:.5f}", k_fit, sweep_deg, phi_c * 180.0 / PI, n_events, - std::fabs(k_fit - 1.0) * sweep_deg / 2.0, jackknife); + std::fabs(k_fit - 1.0) * sweep_deg / 2.0, fold_lo, fold_hi); // Assemble the committed geometry. result.distance_after_mm = dist[0]; diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 5c4537331..5c2b47175 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -345,7 +345,11 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, Add(s, KeyInt("IMAGES_PROCESSED", static_cast(result.images_processed))); Add(s, KeyReal("WAVELENGTH", experiment.GetWavelength_A(), "{:.5f}")); if (const auto gonio = experiment.GetGoniometer()) { - Add(s, KeyReal("OSCILLATION_RANGE", gonio->GetIncrement_deg(), "{:.4f}")); + // The oscillation the run integrated at: the file's, scaled where the run adopted a + // goniometer rotation scale (the starting angle is the stage's, and stays). + const double scale = result.post_refine && result.post_refine->rotation_scale_suspect + ? result.post_refine->rotation_scale : 1.0; + Add(s, KeyReal("OSCILLATION_RANGE", gonio->GetIncrement_deg() * scale, "{:.4f}")); Add(s, KeyReal("STARTING_ANGLE", gonio->GetStart_deg(), "{:.3f}")); const auto ax = gonio->GetAxis(); Add(s, KeyText("ROTATION_AXIS", fmt::format("{:.6f} {:.6f} {:.6f}", ax.x, ax.y, ax.z))); diff --git a/tests/PostRefineTest.cpp b/tests/PostRefineTest.cpp index eec9cb593..e0479e9b4 100644 --- a/tests/PostRefineTest.cpp +++ b/tests/PostRefineTest.cpp @@ -4,6 +4,8 @@ #include #include "../image_analysis/geom_refinement/PostRefine.h" +#include "../common/DiffractionExperiment.h" +#include "../common/Logger.h" namespace { PostRefineResult Measured(double residual, double se) { @@ -24,6 +26,71 @@ namespace { UnitCell Cell(float a, float b, float c) { return UnitCell{.a = a, .b = b, .c = c, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; } + + // The integrated partials of a sweep whose stage turned `true_scale` times the stored angles: the + // crystal is at rotation true_scale * phi when the file says phi. Each reflection crosses the + // Ewald sphere where its excitation, p_z + lambda |p|^2 / 2 for p rotated by minus the angle + // about the spindle, is zero, and is recorded on the frames about that stored angle with a + // Gaussian rocking curve 0.1 deg wide. No spot positions: the scale is fitted on the angles. + std::vector SimulatedSweep(const CrystalLattice &latt, const GoniometerAxis &axis, + int images, double lambda, double true_scale) { + constexpr double D_MIN = 3.0, ROCKING_DEG = 0.1; + const Coord u = axis.GetAxis().Normalize(); + const double first = axis.GetAngle_deg(0.0f), last = axis.GetAngle_deg(static_cast(images)); + const double inc = axis.GetIncrement_deg(); + std::vector out(images); + for (int h = -20; h <= 20; ++h) + for (int k = -20; k <= 20; ++k) + for (int l = -25; l <= 25; ++l) { + const Coord p = latt.Astar() * h + latt.Bstar() * k + latt.Cstar() * l; + const double p2 = p * p; + if (p2 == 0.0 || p2 > 1.0 / (D_MIN * D_MIN)) + continue; + const double up = u * p; + const double A = p.z - u.z * up; + const double B = u.x * p.y - u.y * p.x; + const double C = u.z * up + 0.5 * lambda * p2; + const double R = std::hypot(A, B); + if (std::fabs(C) >= R) + continue; + for (const double sign : {-1.0, 1.0}) { + // A cos(psi) - B sin(psi) + C = 0, i.e. R cos(psi + atan2(B, A)) = -C + double psi = (sign * std::acos(-C / R) - std::atan2(B, A)) * 180.0 / PI; + const double phi = std::remainder(psi, 360.0) / true_scale; + if (phi < first + 1.0 || phi > last - 1.0) + continue; + for (int i = 0; i < images; ++i) { + const double centre = axis.GetAngle_deg(static_cast(i)) + inc / 2.0; + const double x = (centre - phi) / ROCKING_DEG; + if (std::fabs(x) > 3.0) + continue; + Reflection r{}; + r.h = h; r.k = k; r.l = l; + r.image_number = static_cast(i); + r.I = static_cast(1000.0 * std::exp(-0.5 * x * x)); + r.sigma = std::sqrt(r.I) + 1.0f; + r.observed_x = r.observed_y = NAN; + out[i].reflections.push_back(r); + } + } + } + return out; + } + + double FittedRotationScale(double true_scale) { + DiffractionExperiment x(DetJF(1)); + x.IncidentEnergy_keV(12.4); + const GoniometerAxis axis("omega", -90.0f, 0.1f, Coord(1, 0, 0), {}); + const CrystalLattice latt(40.0f, 50.0f, 60.0f, 90.0f, 90.0f, 90.0f); + auto outcomes = SimulatedSweep(latt, axis, 1800, x.GetWavelength_A(), true_scale); + Logger logger("PostRefineTest"); + PostRefineSettings settings; + settings.refine_geometry = true; + settings.num_threads = 4; + return PostRefineRotationGeometry(GatherPostRefineObservations(outcomes, 4, true, logger), axis, + x.GetDiffractionGeometry(), latt, settings, logger) + .rotation_scale; + } } TEST_CASE("PostRefine_HeldOutResidualFell", "[PostRefine]") { @@ -51,3 +118,10 @@ TEST_CASE("PostRefine_ReindexPushesCellBack", "[PostRefine]") { CHECK(ReindexPushesCellBack(refused, Fit(Cell(96.7f, 107.6f, 112.6f), Cell(96.6f, 106.4f, 112.8f))) == 0); CHECK(ReindexPushesCellBack(fit, PostRefineResult{}) == 0); } + +TEST_CASE("PostRefine_RotationScale", "[PostRefine]") { + // Where every frame of the sweep is on the lattice the fit reads the stage's rate itself, and a + // healthy stage reads as one. + CHECK(FittedRotationScale(0.97) == Catch::Approx(0.97).margin(0.001)); + CHECK(FittedRotationScale(1.0) == Catch::Approx(1.0).margin(0.001)); +} diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index 037a92118..c9f18d039 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -841,3 +841,28 @@ TEST_CASE("ResultReport_GeometryNotConverged", "[Diagnostics]") { CHECK(text.substr(flags, text.find('\n', flags + 1) - flags).find("GEOMETRY_NOT_CONVERGED") != std::string::npos); } + +TEST_CASE("ResultReport_RotationScale", "[Diagnostics]") { + // A run that adopted a goniometer rotation scale integrated at the scaled oscillation, so that + // is the oscillation the report gives; the starting angle is the stage's own and stays. + DiffractionExperiment x(DetJF(1)); + x.Goniometer(GoniometerAxis("omega", 10.0f, 0.1f, Coord(1, 0, 0), {})); + ProcessResult result; + result.post_refine = PostRefineResult{}; + result.post_refine->rotation_scale = 0.97; + result.post_refine->rotation_scale_suspect = true; + + RunProvenance dev; + dev.developer = true; + const auto text = RenderResultReport("prefix", "in.h5", x, result, dev); + CHECK(text.find("\nOSCILLATION_RANGE= 0.0970\n") != std::string::npos); + CHECK(text.find("\nSTARTING_ANGLE= 10.000\n") != std::string::npos); + CHECK(text.find("\nGONIOMETER_ROTATION_SCALE= 0.97000\n") != std::string::npos); + CHECK(text.find("\nGONIOMETER_ROTATION_SCALE_SUSPECT= TRUE\n") != std::string::npos); + + // A fit the run did not act on is reported as a fit, and the file's oscillation stands. + result.post_refine->rotation_scale_suspect = false; + const auto kept = RenderResultReport("prefix", "in.h5", x, result, dev); + CHECK(kept.find("\nOSCILLATION_RANGE= 0.1000\n") != std::string::npos); + CHECK(kept.find("\nGONIOMETER_ROTATION_SCALE_SUSPECT= FALSE\n") != std::string::npos); +} From 1a307d9c362d54c8f03074e9ee01b89b3640429f Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 12:17:03 +0200 Subject: [PATCH 11/21] rugnux: use the pre-scan's measured bandwidth when the file states none The pre-scan's spot-shape estimate (spot_width::EstimateBandwidth) becomes the run's bandwidth where it is significant (z > 3) and neither the file nor --bandwidth states one; an explicit value, including --bandwidth 0, still wins. Everything that already reads GetBandwidthFWHM() - prediction, partiality, the profile's radial width, the stencil growth, scaling - takes it from there. Below z 3 the bandwidth stays unset, so monochromatic runs are bit-identical (12 mono sets checked on p.hkl md5). The width tiers stop as soon as r80 settles, often at 15 frames and a few hundred spots, which is too few for the slope. The pre-scan now keeps measuring spot shapes for the bandwidth alone until the pool holds 1000 spots or the sample ends. Those extra frames feed neither r80 nor the powder measurement, so r1..r3, powder and the spot-resolution quantiles are unchanged. Over the 68-set battery plus the known mono controls (47 sets re-probed) no monochromatic set crosses z 3 (highest lyso_x06da_5keV 2.0, insu 5-6 keV 1.7); the one new positive is 9z44 (ALS 8.2.1, multilayer beamline, 0.38% at z 3.1 from 382 spots instead of z 1.0 from 94). Two hidden b > 0 mode switches go, so the bandwidth acts continuously: - the background clip default dropped from 4 to 3 sigma when a bandwidth was set. With the measured bandwidth, clip 3 vs 4 on the MicroMAX pink sets: REFRES R_meas 0.0507/0.0507 (lyso), 0.0842/0.0844 (thau), REFRES ISa 28.83/28.77, 14.12/14.12 - no bandwidth-specific effect. - the "--integration-stencil has no effect without --bandwidth" message, which is decided before the pre-scan and would now be wrong. The one set where clip 3 helps (9z44, 7 A: R_meas 0.252 -> 0.201) gains the same at zero bandwidth, so that is a property of the clip, not of the beam. Effect of the measured bandwidth (stencil 0, clip 4), against hq-integ 9b6736dbd at the same resolution range (REFRES; open sets rerun with --report-resolution at the base d_min): R_meas up on all six consumers - MicroMAX pink lyso 0.0499 -> 0.0507, thau 0.0824 -> 0.0844, 9q41 0.250 -> 0.265, 8u0i 0.0796 -> 0.0816, 9sl0 0.0758 -> 0.0773, 9z44 0.252 -> 0.255; CC1/2 unchanged to 1e-4; CC_MODEL overall +0.001/+0.0004/+0.0005/-0.0025 (9q41/8u0i/9sl0/9z44); no space-group change. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 10 ++-- docs/RUGNUX_OVERVIEW.md | 2 +- rugnux/Rugnux.cpp | 66 +++++++++++++++++++-------- rugnux/SpotWidth.h | 10 ++-- rugnux/rugnux_cli.cpp | 24 ++-------- 5 files changed, 64 insertions(+), 48 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index ad468f553..bc49a734e 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -121,7 +121,7 @@ $ $ with a Poisson-like uncertainty $\sigma(\hat{I})=\max\!\big(1,\ r_\sigma\hat{I},\ \sqrt{S + n_S^2\,\mathrm{var}(\hat{b})}\big)$, i.e. $\sqrt{S}$ floored both at 1 count (pixel values are photon counts) and at a small fraction $r_\sigma$ of the intensity. The second term under the root is the **uncertainty of the background estimate itself**: $\hat b$ is measured from a finite number of ring pixels, $\mathrm{var}(\hat b)=\hat b/n_B$, and it is subtracted $n_S$ times over, so it enters squared. Omitting it understates the **variance** by $1+n_S/n_B$ — 1.11 with the shipped circular stencil ($n_S = 45$, $n_B = 408$) — and so understates $\sigma$ by up to $\sqrt{1+n_S/n_B} \approx 1.05$, a bound attained on background-limited (weak) reflections and falling towards 1 on strong ones, where $S$ dominates; with an elongated ring $n_B$ grows with resolution, so the factor is no longer one number for a run. The same term is carried into the profile fit (§9.3), where it adds $\big(\sum P/v \,\big/ \sum P^2/v\big)^2\,\mathrm{var}(\hat b)$ — the square of $\partial I/\partial\hat b$ for that fit; $n_B$ is the count of pixels behind the *final* background value, so a clip or trim that discards ring pixels raises it. A box sum is accepted as “observed” only if all signal pixels were valid and $n_B$ exceeds a minimum — it measures what is in the disk with no model of what should be there, so it cannot renormalise a disk it has lost pixels out of. The profile modes can, and do (§9.3). This box sum is the classical estimator; it is used directly with `--integrator boxsum`, and otherwise seeds the profile fit below, where $S$ and $n_S$ then count only the pixels that were actually read. -**High-side clipped background (default on).** Because $\hat{I}=S-n_S\hat{b}$ is a small difference of large numbers for weak reflections, a per-pixel background bias $\delta\hat{b}$ becomes a *fractional* intensity bias $\approx n_S\,\delta\hat{b}/\hat{I}$ that grows as $\hat{I}$ shrinks — worst at the resolution edge. A plain ring mean reads high there, because neighbour-spot wings that survive the signal-disk mask, tails and zingers are one-sided (positive) contaminants. The ring mean is therefore made robust: pixels above $\hat{b}+n\sqrt{\hat{b}}$ are rejected and the mean recomputed, with $n=4$ (`--background-clip`; $n=0$ disables), lowered by `rugnux` to $n=3$ on broadband (non-zero bandwidth: pink-beam / DMM) data, where a bandwidth-streaked high-resolution spot leaks into the ring more readily. That is only a default — the flag sets $n$ whatever the bandwidth is. A clean Poisson ring is essentially unchanged by the cut (measured false-rejection rate 0.04–0.39 % at $4\sigma$), while a 40-pixel neighbour core at $+100$ counts shifts the estimate by $+0.009$ ct/px. +**High-side clipped background (default on).** Because $\hat{I}=S-n_S\hat{b}$ is a small difference of large numbers for weak reflections, a per-pixel background bias $\delta\hat{b}$ becomes a *fractional* intensity bias $\approx n_S\,\delta\hat{b}/\hat{I}$ that grows as $\hat{I}$ shrinks — worst at the resolution edge. A plain ring mean reads high there, because neighbour-spot wings that survive the signal-disk mask, tails and zingers are one-sided (positive) contaminants. The ring mean is therefore made robust: pixels above $\hat{b}+n\sqrt{\hat{b}}$ are rejected and the mean recomputed, with $n=4$ (`--background-clip`; $n=0$ disables), whatever the bandwidth. A clean Poisson ring is essentially unchanged by the cut (measured false-rejection rate 0.04–0.39 % at $4\sigma$), while a 40-pixel neighbour core at $+100$ counts shifts the estimate by $+0.009$ ct/px. The clip cuts only the high tail, which matters: the **symmetric** trimmed mean it replaced (drop the lowest and highest fraction $f$ of ring pixels, $f=0.10$; still reachable with `--background-trim`, which switches the clip off) is *not* a consistent estimator of the mean of a right-skewed Poisson sample. It sits $\approx0.1$ ct/px **below** the true mean at every level, and with $n_S = 45$ signal pixels in the $r_1$ disk that under-estimate adds $\approx4.5$ counts to **every** partial — negligible at low resolution, but a large fraction of a partial in the outermost shell. The trim also collapses once contamination exceeds $\approx10\,\%$ of the ring, where the clip does not. Note that removing a positive background bias *lowers* $\langle I/\sigma\rangle$ and *raises* edge $R_\text{meas}$, because both are inflated by information-free counts — so neither may be read as evidence against the change. @@ -195,9 +195,9 @@ Every integration pass, adaptive or not, now logs the radii it used together wit --- -### 9.6 Measuring the bandwidth (rotation, reported only) +### 9.6 Measuring the bandwidth -A finite energy spread $\sigma$ ($\Delta\lambda/\lambda$, rms) smears a reflection along its own radius by $2\tan\theta\,\sigma$ radians of $2\theta$ and not at all across it. Most files do not state it — a multilayer monochromator is a beamline option, not a header field — so Rugnux reads it off the spots, on the same isolated strong spots the width of §9.5 is measured on (`spot_width::EstimateBandwidth`), and **reports** it in the log; the run itself uses the file's value or `--bandwidth`, and 0 otherwise. For each spot, with $u$ along the radius and $v$ across it, the second moments about its centroid are +A finite energy spread $\sigma$ ($\Delta\lambda/\lambda$, rms) smears a reflection along its own radius by $2\tan\theta\,\sigma$ radians of $2\theta$ and not at all across it. Most files do not state it — a multilayer monochromator is a beamline option, not a header field — so Rugnux reads it off the spots, on the same isolated strong spots the width of §9.5 is measured on (`spot_width::EstimateBandwidth`). The width settles on fewer spots than this slope needs, so the pre-scan keeps measuring spot shapes for the bandwidth alone until it holds 1000 of them or its sample runs out. For each spot, with $u$ along the radius and $v$ across it, the second moments about its centroid are $$m_u = \tfrac1{12} + p_u + j_r^2\big(s^2 + 4\tan^2\theta\,\sigma^2\big),\qquad m_v = \tfrac1{12} + p_v + j_t^2 s^2,$$ @@ -205,7 +205,9 @@ with $j_r$, $j_t$ the exact pixels per radian of $2\theta$ and of the angle acro $$y = (m_u - \tfrac1{12} - p_u) - (j_r/j_t)^2\,(m_v - \tfrac1{12} - p_v) = a + \sigma^2\,(2 j_r\tan\theta)^2$$ -is a straight line whose slope is the bandwidth. It is fitted over eight equal-count bins of the abscissa, each a 20 %-trimmed mean weighted by its own scatter; the slope's error is the spread of 200 bootstrap re-draws of the spots, inflated by the reduced $\chi^2$ of the binned fit where the line fits worse than the scatter says, and the log calls the estimate significant at $z>3$. It is a **lower bound** — mosaic spread seen along the radius subtracts — and a spread of cell edges is exactly degenerate with it, so what it measures is the effective radial broadening. +is a straight line whose slope is the bandwidth. It is fitted over eight equal-count bins of the abscissa, each a 20 %-trimmed mean weighted by its own scatter; the slope's error is the spread of 200 bootstrap re-draws of the spots, inflated by the reduced $\chi^2$ of the binned fit where the line fits worse than the scatter says, and the log calls the estimate significant at $z>3$. It is a **lower bound** — mosaic spread seen along the radius subtracts — and a spread of cell edges is exactly degenerate with it, so what it measures is the effective radial broadening — which is what its consumers need. + +A significant estimate is the run's bandwidth, unless the file states one or `--bandwidth` is given (`--bandwidth 0` forces a monochromatic beam); anything short of $z>3$ leaves the beam monochromatic. From there it acts continuously, with no broadband mode: prediction and partiality (§8), the profile's radial width (§9.3) and the background ring's elongation (`--integration-stencil`, §9.1) all scale with it and are exactly what they were at zero bandwidth. ### 9.7 The flight path diff --git a/docs/RUGNUX_OVERVIEW.md b/docs/RUGNUX_OVERVIEW.md index 68f8028ba..64e9b6b10 100644 --- a/docs/RUGNUX_OVERVIEW.md +++ b/docs/RUGNUX_OVERVIEW.md @@ -14,7 +14,7 @@ it ([§1.5](CPU_DATA_ANALYSIS_IMAGE.md)), masks pixels its frames show to be def scattered background and compares it with the file's ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md)), and reads how wide this crystal's spots are, which sets the integration radius ([§9.5](CPU_DATA_ANALYSIS_INTEGRATION.md)), and how much longer they are along their radius than -across it, which is the beam's bandwidth - reported in the log, not yet used +across it, which is the beam's bandwidth where the file does not state one ([§9.6](CPU_DATA_ANALYSIS_INTEGRATION.md)). **Spots.** Every image is decoded — on the GPU straight from the compressed chunk diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 6c5fdcc3f..c20d4288f 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -961,6 +961,9 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru std::vector prescan_spot_q; std::vector beam_center_spots; std::vector width_curves; + // The spots the bandwidth is measured on: the width's, and more where those are too few + // (spot_width::BANDWIDTH_POOL_SPOTS). + std::vector shape_curves; // Images the width ended up being measured on, for the log: the tiers below stop as soon as the // answer has settled, so this is a property of the crystal and worth reporting. size_t width_images = 0; @@ -981,6 +984,9 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru std::vector> spot_q_of(ordinals.size()); // A frame joins the pool only if it could be read, as it did when this was a serial loop. std::vector spot_read(ordinals.size(), 0); + // Frames read for the bandwidth alone, past the tier the width settled on. Their spots stay + // out of the powder measurement, which is taken on the frames it always was. + std::vector shape_only(ordinals.size(), 0); const size_t nworkers = std::min(std::max(config_.nthreads, 1), std::min(PRESCAN_MAX_WORKERS, ordinals.size())); finder.SetShardCount(nworkers); @@ -1020,6 +1026,9 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru std::vector worker_built(nworkers, 0); bool width_settled = !want_width; + // The bandwidth reads the same spots, but its slope needs more of them than the width does: + // the tiers carry on past the width's for it alone, until the pool is large enough. + bool shapes_settled = !want_width; std::optional previous_r80; size_t phase_begin = 0; for (const size_t phase_end : tier_end) { @@ -1065,10 +1074,13 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru // so asking for a beam centre cannot move it either. const bool for_beam_center = spot_set.contains(ordinal); const bool for_width = !width_settled && shadow_set.contains(ordinal); + const bool for_shape = width_settled && !shapes_settled + && shadow_set.contains(ordinal); if (for_beam_center) spot_read[i] = 1; - if (for_beam_center || for_width) + if (for_shape && !for_beam_center) shape_only[i] = 1; + if (for_beam_center || for_width || for_shape) find_spots(w, msg.image, image_idx, for_beam_center, spots_of[i], - for_width, curves_of[i], spot_q_of[i]); + for_width || for_shape, curves_of[i], spot_q_of[i]); } })); for (auto &f : futures) f.get(); @@ -1078,20 +1090,27 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru // depend on how the workers interleaved. A tier too sparse to measure settles nothing: // the test is against what the smaller sample actually said, not against the radius it // fell back to. - if (width_settled) continue; - width_images = static_cast( - std::count_if(visit.begin(), visit.begin() + static_cast(phase_end), - [&](size_t i) { return shadow_set.contains(ordinals[i]); })); - width_curves.clear(); + if (shapes_settled) continue; + if (!width_settled) { + width_images = static_cast( + std::count_if(visit.begin(), visit.begin() + static_cast(phase_end), + [&](size_t i) { return shadow_set.contains(ordinals[i]); })); + width_curves.clear(); + for (const auto &c : curves_of) + width_curves.insert(width_curves.end(), c.begin(), c.end()); + const auto r80 = spot_width::R80AtReference(width_curves); + width_settled = r80 && previous_r80 && spot_width::WidthSettled(*r80, *previous_r80); + previous_r80 = r80; + } + shape_curves.clear(); for (const auto &c : curves_of) - width_curves.insert(width_curves.end(), c.begin(), c.end()); - const auto r80 = spot_width::R80AtReference(width_curves); - width_settled = r80 && previous_r80 && spot_width::WidthSettled(*r80, *previous_r80); - previous_r80 = r80; + shape_curves.insert(shape_curves.end(), c.begin(), c.end()); + shapes_settled = width_settled && shape_curves.size() >= spot_width::BANDWIDTH_POOL_SPOTS; } - for (const auto &q : spot_q_of) - prescan_spot_q.insert(prescan_spot_q.end(), q.begin(), q.end()); + for (size_t i = 0; i < ordinals.size(); i++) + if (!shape_only[i]) + prescan_spot_q.insert(prescan_spot_q.end(), spot_q_of[i].begin(), spot_q_of[i].end()); // Frame numbering follows the sample order, exactly as the serial read did. for (size_t i = 0; i < ordinals.size(); i++) { @@ -1147,23 +1166,30 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru } } - // The X-ray bandwidth, read off the shapes of the same spots (spot_width::EstimateBandwidth). - // Reported only: the run uses what the file or --bandwidth states, 0 otherwise, as it always has. + // The X-ray bandwidth, read off the shapes of the same spots (spot_width::EstimateBandwidth). A + // value the file states, or --bandwidth, wins; otherwise a significant estimate is the bandwidth + // of the run - prediction, partiality, the profile's radial width and the background stencil all + // take it - and anything short of significant leaves the beam monochromatic. if (want_width) { const auto &det = experiment_.GetDetectorSetup(); const double lambda = experiment_.GetWavelength_A(); const auto estimate = spot_width::EstimateBandwidth( - width_curves, sensor_absorption::AttenuationLength_um(det.GetSensorMaterial(), lambda), + shape_curves, sensor_absorption::AttenuationLength_um(det.GetSensorMaterial(), lambda), det.GetSensorThickness_um(), experiment_.GetPixelSize_mm() * 1000.0); + const bool stated = experiment_.GetBandwidthFWHM().has_value(); + const bool significant = estimate && estimate->z > spot_width::BANDWIDTH_Z; + if (!stated && significant) + experiment_.BandwidthFWHM(static_cast(estimate->fwhm)); const float used = experiment_.GetBandwidthFWHM().value_or(0.0f); + const char *why = stated ? "as stated" : significant ? "as measured" : "monochromatic"; if (!estimate) - logger.Info("Bandwidth: not measurable on {} spots; the run uses {:.4f}", width_curves.size(), - used); + logger.Info("Bandwidth: not measurable on {} spots; the run uses {:.4f} ({})", shape_curves.size(), + used, why); else logger.Info("Bandwidth: spot shapes give FWHM {:.4f} from {} spots (one standard error is " - "FWHM {:.4f}; z = {:.1f}, chi2 {:.1f}) - {}; the run uses {:.4f}", estimate->fwhm, + "FWHM {:.4f}; z = {:.1f}, chi2 {:.1f}) - {}; the run uses {:.4f} ({})", estimate->fwhm, estimate->spots, estimate->fwhm_floor, estimate->z, estimate->chi2, - estimate->z > spot_width::BANDWIDTH_Z ? "significant" : "not significant", used); + significant ? "significant" : "not significant", used, why); } // Powder contamination, measured on every run that finds spots here. A crystalline phase other diff --git a/rugnux/SpotWidth.h b/rugnux/SpotWidth.h index 29a1d7898..3e6796f3f 100644 --- a/rugnux/SpotWidth.h +++ b/rugnux/SpotWidth.h @@ -156,10 +156,14 @@ constexpr double BKG_STARVED_MAX_FRACTION = 0.0113; // the whole battery that separates the multilayer beamlines from every monochromatic one. // // It is a LOWER bound: mosaic spread seen along the radius subtracts, and a spread of cell edges -// (strain) is exactly degenerate with it. What it measures is the effective radial broadening. The -// run reports it and does not use it (yet): the bandwidth it integrates with is the file's or -// --bandwidth's. +// (strain) is exactly degenerate with it. What it measures is the effective radial broadening, and +// that is what the consumers of the bandwidth need. The run uses it where it is significant and +// neither the file nor --bandwidth states one. +// +// The width settles on fewer spots than the slope needs, so the pre-scan keeps measuring spots for the +// bandwidth alone until the pool holds BANDWIDTH_POOL_SPOTS or the sample runs out. constexpr size_t BANDWIDTH_MIN_SPOTS = 80; +constexpr size_t BANDWIDTH_POOL_SPOTS = 1000; constexpr double BANDWIDTH_Z = 3.0; struct BandwidthEstimate { diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 2e0f81ac0..081579f09 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -60,10 +60,6 @@ // in its favour and no measurable cost in wall clock. It is also what jfjoch_viewer already sends. constexpr int64_t RUGNUX_MAX_SPOT_COUNT = 1000; -// Default background-ring high-side clip for broadband (non-zero bandwidth) data, tighter than the -// monochromatic default because a pink-beam reflection is long and its wings reach into the ring. -constexpr float BROADBAND_BACKGROUND_CLIP_NSIGMA = 3.0f; - // Default rot3d per-frame scale-G smoothing range (XDS DELPHI-like), in degrees of rotation. constexpr double SMOOTH_G_DEFAULT_DEG = 5.0; @@ -187,15 +183,15 @@ void print_usage() { std::cout << std::endl; std::cout << " Integration" << std::endl; - std::cout << " --bandwidth Relative X-ray bandwidth FWHM (e.g. 0.01 for 1% DMM); default from file or 0" << std::endl; + std::cout << " --bandwidth Relative X-ray bandwidth FWHM (e.g. 0.01 for 1% DMM). Default: the file's value, else measured in the pre-scan from the radial elongation of strong spots and used where significant, else 0; --bandwidth 0 forces a monochromatic beam" << std::endl; std::cout << " --integration-radius Signal-box radius r1, or r1,r2,r3 (px). One value => r2=r1+2, r3=r1+4" << std::endl; std::cout << " --adaptive-integration-radius[=on|off] Set the signal radius r1 from how wide this crystal's spots actually are, measured in the pre-scan (default: on for rotation, off for stills). r1 is the aperture the integrator learns the profile WIDTH over, and a second moment over a disk of radius a saturates at a^2/4, so at the shipped r1=4 the learned sigma can never exceed 2 px and a broader spot is fitted with a profile the model cannot represent. r80 - the radius holding 80% of a spot's flux, at a common resolution - is read off isolated strong spots over a fixed aperture that owes nothing to r1, and r1 = clamp(round(2*r80), 4, 6); the background ring keeps the area it has at the default 4,6,13. Ignored when --integration-radius is given" << std::endl; - std::cout << " --integration-stencil Push the r2..r3 background ring out by k times the beam's radial streak (bandwidth*Rpx), per reflection (default 0 = a fixed circular ring). A fixed ring otherwise ends up on a streaked reflection's own tails at high resolution and measures them as background. Only the ring moves, and only radially - the r1 signal box stays a circle. Needs --bandwidth: on a monochromatic beam the streak is zero and this does nothing" << std::endl; + std::cout << " --integration-stencil Push the r2..r3 background ring out by k times the beam's radial streak (bandwidth*Rpx), per reflection (default 0 = a fixed circular ring). A fixed ring otherwise ends up on a streaked reflection's own tails at high resolution and measures them as background. Only the ring moves, and only radially - the r1 signal box stays a circle. On a monochromatic beam the streak is zero and this does nothing" << std::endl; std::cout << " --integration-high-resolution High resolution limit for prediction/integration. If omitted (or 0), integration extends as far as the detector reaches" << std::endl; std::cout << " --max-hkl Predict reflections with |h|,|k|,|l| <= n. Default: derived per crystal from the refined cell (ceil(longest axis / d_min) + 1), which is the exact bound - set it only to override that" << std::endl; - std::cout << " --background-clip High-side clip of the background ring at mean + n*sqrt(mean) (default 4, or 3 when --bandwidth is set; 0 = off). This is the default background estimator - it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Ignored by --integrator boxsum" << std::endl; + std::cout << " --background-clip High-side clip of the background ring at mean + n*sqrt(mean) (default 4; 0 = off). This is the default background estimator - it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Ignored by --integrator boxsum" << std::endl; std::cout << " --background-radial[=on|off|auto] Correct the background ring for the CURVATURE of the radial background (default off). The signal disk and the background ring are concentric, so a background linear in position cancels between them and only curvature survives - which on a smooth ice ring reaches +26 counts on a single reflection. =auto applies it per image where that image's ice score shows a smooth powder ring, which is where a radius-only background model holds; on ice made of discrete crystallite spots there is no such ring and the correction makes the bias worse. Costs one short dot product per reflection and no extra pixel reads" << std::endl; - std::cout << " --background-trim Use the old symmetric trimmed mean for the background ring instead of the clip (0<=f<0.5; 0.10 was the former default). Switches --background-clip off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; 0 = plain ring mean. Applies whatever --bandwidth is set to" << std::endl; + std::cout << " --background-trim Use the old symmetric trimmed mean for the background ring instead of the clip (0<=f<0.5; 0.10 was the former default). Switches --background-clip off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; 0 = plain ring mean" << std::endl; std::cout << " --overlap What to do where two predicted reflections share signal pixels: off|reject|exclude (default exclude). A pixel inside two signal disks belongs to the nearer centre; before this, nothing kept a neighbour's flux out of a reflection's own disk, so on a dense pattern a crowded reflection read high. exclude drops the shared PIXELS from the profile fit, which renormalises itself, so the reflection is kept; reject instead drops the whole reflection when less than --overlap-minpk of its expected profile is cleanly its own (what XDS calls MINPK). --integrator boxsum has no profile to renormalise with, so exclude does nothing there and only reject acts" << std::endl; std::cout << " --overlap-minpk Least fraction of a reflection's expected profile that must be usable for the reflection to be kept (default 0.75, XDS MINPK). Governs both ways part of a profile is lost: the fraction that must be READABLE - not masked, untrusted, in a detector gap or overloaded - in every profile mode, and, under --overlap reject, the fraction that must be cleanly the reflection's own. --integrator boxsum has no profile to renormalise with, so there a signal disk with any unreadable pixel is still discarded outright and the reject fraction is by disk AREA, which cuts harder" << std::endl; std::cout << " --integrator Spot integrator boxsum|gaussian|empirical (default: gaussian profile-fit; boxsum is the classical fallback)" << std::endl; @@ -2593,9 +2589,6 @@ static int RunRugnux(int argc, char **argv) { experiment.ImportBraggIntegrationSettings(bis); if (!(k > 0.0f)) logger.Info("Background ring left circular (--integration-stencil 0)"); - else if (!(experiment.GetBandwidthFWHM().value_or(0.0f) > 0.0f)) - logger.Info("--integration-stencil {:.2f} has no effect without --bandwidth: " - "the ring is elongated by the beam's radial streak, which is zero here", k); else logger.Info("Background ring elongated by {:.2f} times the beam's radial streak, per reflection", k); } @@ -2638,15 +2631,6 @@ static int RunRugnux(int argc, char **argv) { bis.BackgroundClipNSigma(static_cast(*background_clip_arg)); experiment.ImportBraggIntegrationSettings(bis); logger.Info("Background ring: high-side clip at {:.1f} sigma", *background_clip_arg); - } else if (!background_trim_arg && experiment.GetBandwidthFWHM().value_or(0.0f) > 0.0f) { - // Broadband (pink-beam / DMM) reflections are long and their wings reach into the background - // ring, so the default clip is tighter there. It is only a default - --background-clip and - // --background-trim override it as they do for any other data. - BraggIntegrationSettings bis = experiment.GetBraggIntegrationSettings(); - bis.BackgroundClipNSigma(BROADBAND_BACKGROUND_CLIP_NSIGMA); - experiment.ImportBraggIntegrationSettings(bis); - logger.Info("Background ring: high-side clip at {:.1f} sigma (broadband default)", - BROADBAND_BACKGROUND_CLIP_NSIGMA); } if (overlap_arg || overlap_minpk_arg) { From 81529a3086f68b7634581a113d65663f5c0bb618 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 13:44:52 +0200 Subject: [PATCH 12/21] Beam-centre check: carry both centres when only lean first passes index, decide on the merge Where neither centre indexes the whole spot list, the check ran the lean-rung ladder at the file's centre and at the background-measured one and kept whichever indexed more validation frames. On the same lattice those counts are two readings of one hypothesis, and the frame count is the statistic the check itself says must not arbitrate centres: on one rotation set the measured centre read 56/60 against the file's 50/60, then 48/60 against 50/60 once the hot-pixel mask took out ~3k defective pixels. The run then kept a centre 12.6 px out, the post-refinement moved it 0.15 px, and the merge lost it: 32% of frames rejected (17%), low-res R_meas 14.7% (8.1%), CC_MODEL 0.42 (0.89). The P1 setting change seen alongside it (alpha/beta exchanged) is an a~b Niggli-boundary flip of the same lattice and was not causal; model validation reindexes it. Now, when both ladders index a majority on the same lattice (class + primitive volume within 2%), the file's centre stays for the pass and the measured centre is carried into the existing two-arm arbitration in RunAllPasses. The measured arm starts from the depth (rings/seed/d_min) the check found it at - a fresh pass at that centre walked 1 px onto a 2x supercell instead - and the file arm's spot-finding settings are restored with the rest of its state when kept. The arbitration moves to MeasuredCentreWins(): arms on the same lattice are judged on the reflections at I/sigma >= 2 of their search merges (the quality guard's signal statistic), without the guard's 10% margin, which covers header-vs-refined passes integrated with different spot width and mosaicity - the two arms here integrate identically. Arms on different lattices keep the CC1/2 + 0.05 rule. On the set above: 10095 vs 9116 -> measured centre, CC_MODEL 0.866, R-free 0.333 (rc173 0.333), R_MODEL_SHELL_SCALED 0.316. 27 other sets bit-identical. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- rugnux/Rugnux.cpp | 154 ++++++++++++++++++++++++++++++------------ rugnux/Rugnux.h | 20 +++++- rugnux/rugnux_cli.cpp | 2 +- tests/RugnuxTest.cpp | 37 ++++++++++ 4 files changed, 168 insertions(+), 45 deletions(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 6c5fdcc3f..29a1d6e2f 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -1959,6 +1959,43 @@ namespace { } } +bool ArmsHoldSameLattice(const ProcessResult &a, const ProcessResult &b) { + if (!a.rotation_lattice_type || !b.rotation_lattice_type || !a.consensus_cell || !b.consensus_cell) + return false; + const auto primitive_volume = [](const ProcessResult &r) { + return std::abs(CrystalLattice(*r.consensus_cell) + .ToPrimitive(r.consensus_centering.value_or('P')).CalcVolume()); + }; + const double va = primitive_volume(a), vb = primitive_volume(b); + return a.rotation_lattice_type->crystal_system == b.rotation_lattice_type->crystal_system + && a.rotation_lattice_type->centering == b.rotation_lattice_type->centering + && std::max(va, vb) < 1.02 * std::min(va, vb); +} + +std::string MeasuredCentreWins(const ProcessResult &file_arm, const ProcessResult &measured_arm) { + if (!file_arm.has_merge_statistics || !measured_arm.has_merge_statistics) + return {}; + // Same lattice: which geometry is the more accurate, read off the signal each arm measured - the + // statistic the two-pass quality guard found to agree with an external arbiter where CC1/2 did not + // (RefinedPassIsWorse). Without its margin: that margin covers what the header and refined passes + // differ by for reasons that are not the geometry - a different spot width and mosaicity - and the + // two arms here integrate with the same settings, so the geometry is the only difference, and the + // file's centre has no standing the measured one lacks - neither indexed the whole spot list. + if (ArmsHoldSameLattice(file_arm, measured_arm)) { + if (measured_arm.search_merge_strong_reflections > file_arm.search_merge_strong_reflections) + return fmt::format("the measured centre measures more of the crystal - {} reflections at " + "I/sigma >= 2 before corrections against {}", + measured_arm.search_merge_strong_reflections, + file_arm.search_merge_strong_reflections); + return {}; + } + if (measured_arm.search_merge_cc_half > file_arm.search_merge_cc_half + MAX_CC_HALF_LOSS) + return fmt::format("the measured centre merges better - CC1/2 before corrections {:.3f} against " + "{:.3f} on the search merge", measured_arm.search_merge_cc_half, + file_arm.search_merge_cc_half); + return {}; +} + ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { // Only the refined passes of the rotation two-pass are judged against a header-geometry pass, and // only while this holds one; a second Run() on the same object must not inherit the last one's. @@ -2013,8 +2050,13 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { // question needs - on a crystal with an axis under the FFT floor the check runs before that // pass and compares supercells. // - // Adopting the alternative moves the run off the geometry it was given, so it has to be + // Adopting a different metric moves the run off the geometry it was given, so it has to be // decisively better rather than merely ahead. + // + // The same two arms also answer a second question the first pass cannot: where neither centre + // indexes the whole spot list and both index the same lattice once the pass reads less of each + // frame, which centre is right (see the beam-centre check). The measured arm then starts from + // the depth the check found it at. if (!cancelled_ && beam_center_alternative_) { const auto alt_center = *beam_center_alternative_; beam_center_alternative_.reset(); @@ -2022,6 +2064,7 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { "({:.2f},{:.2f}) and judging the two on their merges", alt_center[0], alt_center[1]); const DiffractionExperiment file_experiment = experiment_; + const auto file_spot_finding = config_.spot_finding; const auto file_mosaicity = prepass_mosaicity_; const auto file_geometry = prepass_detector_geometry_; const auto file_scale = prepass_rotation_scale_; @@ -2057,28 +2100,22 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { // FFT floor the check compared two supercells the short-axis pass then threw away. Where // both arms now hold the same lattice in the same class, the centre did not decide the // symmetry after all: the two merges are two integrations of ONE hypothesis, their CC1/2 - // difference is the noise between them, and there is nothing for it to arbitrate - so the - // file's centre stays, and the post-refinement moves it wherever the spots put it. Same - // lattice means what it means in the check: same class and primitive volumes within 2 %. - const auto primitive_volume = [](const ProcessResult &r) { - return std::abs(CrystalLattice(*r.consensus_cell) - .ToPrimitive(r.consensus_centering.value_or('P')).CalcVolume()); - }; - const bool arms_agree = alt.rotation_lattice_type && pass1.rotation_lattice_type - && alt.consensus_cell && pass1.consensus_cell - && alt.rotation_lattice_type->crystal_system - == pass1.rotation_lattice_type->crystal_system - && alt.rotation_lattice_type->centering - == pass1.rotation_lattice_type->centering - && std::max(primitive_volume(alt), primitive_volume(pass1)) - < 1.02 * std::min(primitive_volume(alt), primitive_volume(pass1)); + // difference is the noise between them, and it is not asked. Same lattice means what it + // means in the check: same class and primitive volumes within 2 %. + // + // What is left is which of the two GEOMETRIES is the more accurate, answered on the signal + // each arm measured (MeasuredCentreWins). The post-refinement cannot be left to settle it: + // it moves the beam by what the spots ask of a lattice already fitted at that centre, which + // on the crystal that showed it was 0.15 px of a 12.6 px error. + const bool arms_agree = ArmsHoldSameLattice(pass1, alt); if (arms_agree) logger.Info("Beam centre check: after their whole first passes both centres hold the " "same lattice - {}-centred {} - so the centre did not decide the metric " - "symmetry, and the merges (CC1/2 before corrections {:.3f} at the measured " - "centre against {:.3f}) are not asked to", pass1.rotation_lattice_type->centering, + "symmetry; the two are judged on the signal each measured, {} reflections at " + "I/sigma >= 2 before corrections at the measured centre against {} at the " + "file's", pass1.rotation_lattice_type->centering, gemmi::crystal_system_str(pass1.rotation_lattice_type->crystal_system), - alt.search_merge_cc_half, pass1.search_merge_cc_half); + alt.search_merge_strong_reflections, pass1.search_merge_strong_reflections); else if (alt.rotation_lattice_type && pass1.rotation_lattice_type) logger.Info("Beam centre check: after their whole first passes the arms still differ - " "{}-centred {} at the file's centre against {}-centred {} at the measured one", @@ -2086,11 +2123,8 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { gemmi::crystal_system_str(pass1.rotation_lattice_type->crystal_system), alt.rotation_lattice_type->centering, gemmi::crystal_system_str(alt.rotation_lattice_type->crystal_system)); - const bool alt_merges_better = !cancelled_ && !arms_agree && alt.has_merge_statistics - && pass1.has_merge_statistics - && alt.search_merge_cc_half - > pass1.search_merge_cc_half + MAX_CC_HALF_LOSS; - if (alt_merges_better) { + const std::string measured_wins = cancelled_ ? std::string() : MeasuredCentreWins(pass1, alt); + if (!measured_wins.empty()) { const auto cell_text = [](const ProcessResult &r) { if (!r.consensus_cell) return std::string("no cell"); @@ -2098,19 +2132,18 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { return fmt::format("{:.3f} {:.3f} {:.3f} {:.2f} {:.2f} {:.2f}", c.a, c.b, c.c, c.alpha, c.beta, c.gamma); }; - logger.Warning("Beam centre check: the measured centre merges better - CC1/2 before " - "corrections {:.3f} against {:.3f} on the search merge, on {} against " - "{} - so the run adopts it and the metric symmetry it finds", - alt.search_merge_cc_half, pass1.search_merge_cc_half, - cell_text(alt), cell_text(pass1)); + logger.Warning("Beam centre check: {}, on {} against {} - so the run adopts the measured " + "centre and the lattice it finds", measured_wins, cell_text(alt), + cell_text(pass1)); pass1 = std::move(alt); } else { - if (!arms_agree) - logger.Info("Beam centre check: the measured centre does not merge better (CC1/2 " - "before corrections {:.3f} against {:.3f} on the search merge) - the run " - "keeps the file's centre and the symmetry it found", - alt.search_merge_cc_half, pass1.search_merge_cc_half); + logger.Info("Beam centre check: the measured centre does not win ({} against " + "{} reflections at I/sigma >= 2, CC1/2 before corrections {:.3f} against " + "{:.3f} on the search merge) - the run keeps the file's centre", + alt.search_merge_strong_reflections, pass1.search_merge_strong_reflections, + alt.search_merge_cc_half, pass1.search_merge_cc_half); experiment_ = file_experiment; + config_.spot_finding = file_spot_finding; prepass_mosaicity_ = file_mosaicity; prepass_detector_geometry_ = file_geometry; prepass_rotation_scale_ = file_scale; @@ -2119,6 +2152,7 @@ ProcessResult Rugnux::RunAllPasses(RugnuxObserver *observer) { // The arm's own pass asks the same question again at its own centre; the run has already // decided it, so nothing is to consume a second answer. beam_center_alternative_.reset(); + beam_center_alternative_state_.reset(); if (cancelled_) { config_.output_prefix = base_prefix; return pass1; } } @@ -3595,7 +3629,6 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // tried and rejected leaves nothing behind - and a later pass that finds no rung restores what // an earlier pass adopted rather than resetting the run to the defaults. const auto first_pass_file_d_min = config_.spot_finding.high_resolution_limit; - struct FirstPassState { bool rings = false; size_t seed = 0; std::optional d_min; }; FirstPassState committed{false, 0, first_pass_file_d_min}; const auto apply_state = [&](const FirstPassState &st) { const std::vector r = st.rings ? powder_.rings_q_recipA : std::vector{}; @@ -3606,6 +3639,14 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b first_pass_spots_per_image = st.seed; spot_cache.clear(); // the per-image spot budget is spent again under these settings }; + // The measured-centre arm of a carried beam-centre hypothesis reads each frame as deep as the + // check found that centre indexing at: the arm is the hypothesis the check measured, not a new + // search at the same centre that may end somewhere else. + if (beam_center_alternative_state_) { + committed = *beam_center_alternative_state_; + beam_center_alternative_state_.reset(); + apply_state(committed); + } // min_score is an ABSOLUTE bar beside the gain. A caller that takes a rung only when it // indexes a majority must not be handed a state the ladder committed on a lesser rung: @@ -3978,19 +4019,46 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b apply_state(committed); try_beam_center(measured_x, measured_y); const auto away = first_pass_ladder(alt.score, 1, majority + 1); - if (home && (!away || home->score >= away->score)) { + const FirstPassState away_state = committed; + // Both centres index a majority, and on the same lattice. Then the two counts + // are two readings of one hypothesis at two geometries, and which is higher is + // the count's noise: measured on one crystal, the same two ladders gave the + // measured centre 56/60 against the file's 50/60, and 48/60 against 50/60 once + // a few thousand defective pixels were masked - and the run that then kept the + // file's centre, 12.6 px out, merged with twice the frames rejected. So the + // count decides nothing here: the file's centre stays for this pass, and the + // measured one is carried forward to be judged on the merges (RunAllPasses). + const bool same_lattice = + home && away + && home->result->search_result.system == away->result->search_result.system + && home->result->search_result.centering + == away->result->search_result.centering + && std::max(home->vol, away->vol) < 1.02 * std::min(home->vol, away->vol); + if (home && (!away || same_lattice || home->score >= away->score)) { committed = home_state; restore_beam_center(header_x, header_y); apply_state(committed); best = *home; deeper_at_home = true; - logger.Warning("Beam centre check: reading less of each frame takes the " - "file's centre to {}/{} validation frames against {}/{} at " - "the measured centre {:.2f} px away - it is how deep this " - "run was reading and not the centre, so the centre stays", - home->score, static_cast(validation.size()), - away ? away->score : 0, - static_cast(validation.size()), moved); + if (same_lattice && geometry_prepass) { + beam_center_alternative_ = std::array{measured_x, measured_y}; + beam_center_alternative_state_ = away_state; + logger.Warning("Beam centre check: reading less of each frame, the " + "file's centre indexes {}/{} validation frames and the " + "measured centre {:.2f} px away {}/{}, on the same " + "lattice - the counts cannot say which centre is right, " + "so the run carries BOTH centres forward and adopts " + "whichever measures more of the crystal", + home->score, static_cast(validation.size()), moved, + away->score, static_cast(validation.size())); + } else + logger.Warning("Beam centre check: reading less of each frame takes the " + "file's centre to {}/{} validation frames against {}/{} at " + "the measured centre {:.2f} px away - it is how deep this " + "run was reading and not the centre, so the centre stays", + home->score, static_cast(validation.size()), + away ? away->score : 0, + static_cast(validation.size()), moved); } else if (away) { alt = *away; measured_indexes = true; diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index c74278625..32c05d332 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -432,6 +432,16 @@ struct ProcessResult { // element of G, so it is dimensionless and comparable across cells. double MetricViolation(const UnitCell &uc, const gemmi::SpaceGroup &sg); +// Whether two passes hold the same lattice: the same Bravais class and primitive volumes within 2 %. +bool ArmsHoldSameLattice(const ProcessResult &a, const ProcessResult &b); + +// The beam-centre arbitration between two first passes, one at the file's centre and one at the +// centre the background measured (RunAllPasses). Returns why the measured centre wins, or an empty +// string when the file's centre stays. Where the two hold the same lattice the question is which +// geometry is the more accurate, answered on the signal each measured; where they differ it is which +// metric the crystal has, answered on the search merges. +std::string MeasuredCentreWins(const ProcessResult &file_arm, const ProcessResult &measured_arm); + // The metric never fits exactly - the cell is refined against the data, not constrained to the group - // so the test needs a tolerance, and it is used to REFUSE, so the tolerance has to sit above what a // correct answer reaches rather than below what a wrong one does. Measured over 113 corpus runs, every @@ -543,8 +553,16 @@ class Rugnux { // that symmetry (RunPipeline's beam-centre check). Set by the two-pass first pass and consumed by // RunAllPasses, which runs the first pass again there and keeps whichever arm merges better: the // promotion is a length equality read off an unrefined FFT candidate, both centres index the same - // frames, and only the intensities can say which metric the crystal actually has. + // frames, and only the intensities can say which metric the crystal actually has. Also kept when + // neither centre indexes the whole spot list but both index the same lattice once the pass reads + // less of each frame: the frame counts cannot say which centre is right, the merges can. std::optional> beam_center_alternative_; + // How deep into each frame's spot list a first pass reads: powder rings set aside, the strongest + // `seed` spots an image (0 = the indexer's default), spot finding stopped at `d_min`. + struct FirstPassState { bool rings = false; size_t seed = 0; std::optional d_min; }; + // Where the carried centre was found by reading less of each frame, the depth it was found at, + // for the measured-centre arm to start from. + std::optional beam_center_alternative_state_; // The same pre-scan measurement, kept for the whole run instead of being consumed by the first // pass. Post-refinement measures how far it may move the beam from the nearer of the nominal diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 2e0f81ac0..ded036b8e 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -113,7 +113,7 @@ void print_usage() { std::cout << " --detect-beam-stop[=N|off] Find the beam stop and its holder in a projection of N images and add them to the pixel mask (bit 9), so nothing shadowed by them is integrated. ON by default (60 images); =off disables. Reflections behind the stop are attenuated but not flagged, so they are integrated low with a plausible sigma and no existing rejection catches them" << std::endl; std::cout << " --estimate-beam-center Place the beam centre before anything is indexed, and use it in place of the header value when it is measured precisely enough. On a sweep that reaches half a turn it comes from the symmetry of the spot positions - the frames 180 deg apart are each other's mirror image, and every reflection is recorded twice - and where the sweep does not reach that far, from the isotropy of the scattered background, which needs only a few frames. It reads frames of its own, chosen as pairs half a turn apart, so it does not change the mask --detect-beam-stop finds. Ignored when --beam-x/--beam-y are given, and when a stills geometry refinement has already placed the centre from indexed spots" << std::endl; std::cout << " --beam-center-search[=N|off] After a first pass that indexes fewer than half the validation frames, try the beam centre a pixel at a time out to N px along each detector axis and keep the first one that indexes a majority. A beam-centre error is fixed in the lab frame, so it smears the accumulated reciprocal-space cloud and the FFT takes an axis harmonic instead of the true axis; nothing downstream repairs that. ON by default (12 px); =off disables. It runs only after a pass that has already failed, so a run that indexes never pays for it" << std::endl; - std::cout << " --beam-center-check[=off] Measure the beam centre from the isotropy of the scattered background on EVERY run, report how far the file's value is from it, and index a second first pass with it to see whether the two centres give the same lattice. The fit reads the projection --detect-beam-stop already builds, so it costs no extra frames. On a run that indexes, the measured centre is adopted in two cases only: where the file's centre indexes nothing and the measured one indexes a majority, and where the two centres return cells related by an integer volume factor and the measured centre's is the LARGER one AND carries materially more of the pooled spots - the axis harmonic a centre error along the spindle produces, which nothing downstream repairs. ON by default; =off disables" << std::endl; + std::cout << " --beam-center-check[=off] Measure the beam centre from the isotropy of the scattered background on EVERY run, report how far the file's value is from it, and index a second first pass with it to see whether the two centres give the same lattice. The fit reads the projection --detect-beam-stop already builds, so it costs no extra frames. On a run that indexes, the measured centre is adopted where the file's centre indexes nothing and the measured one indexes a majority, and where the two centres return cells related by an integer volume factor and the measured centre's is the LARGER one AND carries materially more of the pooled spots - the axis harmonic a centre error along the spindle produces, which nothing downstream repairs. Where the indexing cannot tell the two centres apart - the same cell with a different metric symmetry, or the same lattice found at both only by reading less of each frame - the first pass is run at both and the run keeps the one that merges better (on the same lattice: the one that measures more reflections at I/sigma >= 2). ON by default; =off disables" << std::endl; std::cout << " --no-fit-spindle Take the goniometer axis from the file rather than measuring the spindle's rotation about the beam from the spots. Measuring it is the DEFAULT and the file is never right: every master writes an exact lab axis and no goniometer is one. Both mirror lines of the beam-centre estimator turn with the spindle, so an axis a few tenths of a milliradian out smears the vote until a neighbouring tooth wins. Only has an effect with --estimate-beam-center" << std::endl; std::cout << std::endl; diff --git a/tests/RugnuxTest.cpp b/tests/RugnuxTest.cpp index 105705e36..5eefe879b 100644 --- a/tests/RugnuxTest.cpp +++ b/tests/RugnuxTest.cpp @@ -241,3 +241,40 @@ TEST_CASE("SpotWidth_Gaussian", "[process]") { CHECK(spot_width::R1ForWidth(3.947f) == 6.0f); CHECK(spot_width::R1ForWidth(9.0f) == 6.0f); } + +// The beam-centre arbitration between a first pass at the file's centre and one at the measured +// centre. Two arms on the same lattice are two geometries of one hypothesis: they are judged on the +// signal each measured, not on a CC1/2 that reads the same on both. +TEST_CASE("MeasuredCentreWins", "[process]") { + const auto arm = [](const UnitCell &cell, gemmi::CrystalSystem system, double cc_half, int64_t strong) { + ProcessResult r; + r.has_merge_statistics = true; + r.consensus_cell = cell; + r.consensus_centering = 'P'; + r.rotation_lattice_type = LatticeMessage{'P', 0, system}; + r.search_merge_cc_half = cc_half; + r.search_merge_strong_reflections = strong; + r.search_merge_reflections = 40000; + return r; + }; + const UnitCell triclinic{40.0f, 41.0f, 100.0f, 86.0f, 84.0f, 72.0f}; + // The same lattice in the setting with a and b exchanged, as a noisy a ~ b can come out. + const UnitCell swapped{41.02f, 39.98f, 100.1f, 84.0f, 86.0f, 72.0f}; + const auto tri = gemmi::CrystalSystem::Triclinic; + + const ProcessResult file = arm(triclinic, tri, 0.87, 9000); + REQUIRE(ArmsHoldSameLattice(file, arm(swapped, tri, 0.87, 9000))); + + // Same lattice, same CC1/2: the arm that measured more signal wins, whichever centre it is... + CHECK(!MeasuredCentreWins(file, arm(swapped, tri, 0.87, 9500)).empty()); + CHECK(MeasuredCentreWins(file, arm(swapped, tri, 0.87, 8500)).empty()); + // ...a tie keeps the file's centre, and a higher CC1/2 alone does not move it. + CHECK(MeasuredCentreWins(file, arm(swapped, tri, 0.99, 9000)).empty()); + + // Different lattices: the metric question, decided on the search merges. + const UnitCell monoclinic{57.0f, 42.0f, 100.0f, 90.0f, 95.0f, 90.0f}; + const auto mono = gemmi::CrystalSystem::Monoclinic; + REQUIRE(!ArmsHoldSameLattice(file, arm(monoclinic, mono, 0.87, 9000))); + CHECK(!MeasuredCentreWins(file, arm(monoclinic, mono, 0.95, 9000)).empty()); + CHECK(MeasuredCentreWins(file, arm(monoclinic, mono, 0.88, 20000)).empty()); +} From e7ef72ba407578cca7fa1c2178f243c2bc2c0b81 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 08:48:31 +0200 Subject: [PATCH 13/21] Rotation prediction: place each partial at its slice of the rocking curve A rotation frame records the part of a reflection's rocking curve inside its own oscillation, and while the crystal turns through the curve the spot walks along its Debye ring. The predictor put every partial at the exact diffracting condition, so a partial recorded on the curve's flank was integrated pixels away from where its flux landed. The walk is largest where the reflection moves nearly tangent to the Ewald sphere (low |zeta|), whose curves are widest. The prediction now turns S about the beam by the rotation's component along the ring times the flux-weighted centre of the frame's slice of the curve (a truncated-normal mean, RockingSlice.h), on the CPU and the GPU predictor alike. 2theta is unchanged; a frame that straddles the condition symmetrically gets no shift. Measured (observed r1 centroid minus prediction, tangential, I/sig > 10): against the slice model correlation 0.93-0.98 before, 0.0 after; residual rms 1.0-1.8 px -> 0.35-0.48 px at |zeta| < 0.3. Emulated capture of low-|zeta| partials at high angle 0.50 -> 0.94 of a 12 px aperture. CPU runs on four rotation sets (400-800 frames): R_meas -0.03..-0.09 pp, ISa +1.2..+1.8, rejected observations roughly halved, space groups unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- .../bragg_prediction/BraggPredictionRot.cpp | 12 +++ .../bragg_prediction/BraggPredictionRotGPU.cu | 18 +++++ .../bragg_prediction/RockingSlice.h | 42 ++++++++++ tests/CalcBraggPredictionTest.cpp | 76 +++++++++++++++++++ 4 files changed, 148 insertions(+) create mode 100644 image_analysis/bragg_prediction/RockingSlice.h diff --git a/image_analysis/bragg_prediction/BraggPredictionRot.cpp b/image_analysis/bragg_prediction/BraggPredictionRot.cpp index 1d4321d05..495a40a6c 100644 --- a/image_analysis/bragg_prediction/BraggPredictionRot.cpp +++ b/image_analysis/bragg_prediction/BraggPredictionRot.cpp @@ -5,6 +5,7 @@ #include "BraggPredictionRot.h" #include "../SensorAbsorption.h" #include "../bragg_integration/SystematicAbsence.h" +#include "RockingSlice.h" int BraggPredictionRot::Calc(const DiffractionExperiment &experiment, const CrystalLattice &lattice, @@ -47,6 +48,7 @@ int BraggPredictionRot::Calc(const DiffractionExperiment &experiment, const Crys const float m3_S0 = m3 * S0; int i = 0; + const Coord beam_dir = S0.Normalize(); const float mos_angle_rad = settings.mosaicity_deg * static_cast(PI) / 180.f; const float half_wedge_angle_rad = settings.wedge_deg * static_cast(PI) / 180.f / 2.0f ; @@ -163,6 +165,16 @@ int BraggPredictionRot::Calc(const DiffractionExperiment &experiment, const Crys const float partiality = (std::erf((phi + half_wedge_angle_rad) * c1) - std::erf((phi - half_wedge_angle_rad) * c1)) / 2.0f; + // Place the partial where this frame recorded it: at the centre of the frame's slice of + // the rocking curve, which is the exact-condition position walked along its Debye ring by + // the rotation between the two (RockingSlice.h). Walking along the ring is turning S + // about the beam, by the rotation's component along the ring over the ring's radius. + const float c_slice = RockingSliceCentroid_rad(phi, half_wedge_angle_rad, c1, partiality); + const Coord S_par = beam_dir * (beam_dir * S), S_perp = S - S_par; + const Coord S_turn = beam_dir % S; // |S_turn| = |S_perp| + const float psi = c_slice * ((m2 % p) * S_turn) / (S_perp * S_perp); + S = S_par + S_perp * std::cos(psi) + S_turn * std::sin(psi); + // Inlined RecipToDetector: the full transposed detector matrix, tilt and discrete orientation // Apply rotation matrix transpose float S_rot_x = rot[0] * S.x + rot[1] * S.y + rot[2] * S.z; diff --git a/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu b/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu index 59855cf14..07ce14209 100644 --- a/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu +++ b/image_analysis/bragg_prediction/BraggPredictionRotGPU.cu @@ -4,6 +4,7 @@ #include #include "../../common/JFJochMath.h" #include "BraggPredictionRotGPU.h" +#include "RockingSlice.h" #include "../SensorAbsorption.h" #ifdef JFJOCH_USE_CUDA @@ -156,6 +157,23 @@ namespace { - erff((phi - half_wedge) * c1)) / 2.0f; + // The partial sits at the centre of the frame's slice of its rocking curve, walked along + // the Debye ring from the exact condition: S turned about the beam (see the CPU predictor). + float kx = C.S0.x, ky = C.S0.y, kz = C.S0.z; + normalize3(kx, ky, kz); + const float kS = dot3(kx, ky, kz, Sx, Sy, Sz); + const float qx = Sx - kS * kx, qy = Sy - kS * ky, qz = Sz - kS * kz; // S_perp + float tx, ty, tz; // S_turn = k x S + cross3(kx, ky, kz, Sx, Sy, Sz, tx, ty, tz); + float wx, wy, wz; // m2 x p + cross3(C.m2.x, C.m2.y, C.m2.z, px, py, pz, wx, wy, wz); + const float c_slice = RockingSliceCentroid_rad(phi, half_wedge, c1, partiality); + const float psi = c_slice * dot3(wx, wy, wz, tx, ty, tz) / dot3(qx, qy, qz, qx, qy, qz); + const float cos_psi = cosf(psi), sin_psi = sinf(psi); + Sx = kS * kx + qx * cos_psi + tx * sin_psi; + Sy = kS * ky + qy * cos_psi + ty * sin_psi; + Sz = kS * kz + qz * cos_psi + tz * sin_psi; + // Use S (rotated) for projection float Srx = C.rot[0] * Sx + C.rot[1] * Sy + C.rot[2] * Sz; float Sry = C.rot[3] * Sx + C.rot[4] * Sy + C.rot[5] * Sz; diff --git a/image_analysis/bragg_prediction/RockingSlice.h b/image_analysis/bragg_prediction/RockingSlice.h new file mode 100644 index 000000000..6178f6bac --- /dev/null +++ b/image_analysis/bragg_prediction/RockingSlice.h @@ -0,0 +1,42 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +// Where on its rocking curve a rotation frame records a reflection. +// +// The rocking curve is a Gaussian in the rotation angle, centred on the exact diffracting condition, +// and a frame records the part of it inside its own oscillation: [phi - w/2, phi + w/2] in the +// convention of the rotation predictor, where phi is the frame's offset from that condition and +// c1 = zeta / (sqrt(2) sigma_M) is the curve's reciprocal width. The flux-weighted centre of that slice +// is the mean of a truncated normal, +// +// c = (exp(-a^2) - exp(-b^2)) / (2 sqrt(pi) c1 p), a = (phi - w/2) c1, b = (phi + w/2) c1, +// +// with p the slice's mass - the partiality. A frame whose oscillation straddles the condition +// symmetrically gets c = 0; one that only catches the curve's tail gets a c of up to several curve +// widths. +// +// It matters because the recorded spot is not fixed on the detector while the crystal rotates through +// the curve: it walks along its Debye ring, one detector velocity per radian of rotation, and a partial +// recorded at c sits c times that velocity from the exact-condition position. Measured on rotation +// data the walk follows this c with slope ~1 (0.9 at high angle, where the r1 centroid it was measured +// with is truncated least), and at |zeta| < 0.2 it is pixels - the reflection moves nearly tangent to +// the Ewald sphere, so its curve is wide and the spot travels far along the ring while it lasts. + +#include + +#ifdef __CUDACC__ +#define ROCKING_SLICE_HD __host__ __device__ inline +#else +#define ROCKING_SLICE_HD inline +#endif + +// Flux-weighted centre of the frame's slice of the rocking curve, relative to the exact diffracting +// condition, in radians of rotation. partiality is the slice's own mass (the caller has it already). +ROCKING_SLICE_HD float RockingSliceCentroid_rad(float phi, float half_wedge, float c1, float partiality) { + if (!(partiality > 0.0f) || !(c1 > 0.0f)) + return 0.0f; + const float a = (phi - half_wedge) * c1, b = (phi + half_wedge) * c1; + return (expf(-a * a) - expf(-b * b)) / (2.0f * sqrtf(3.14159265f) * c1 * partiality); +} diff --git a/tests/CalcBraggPredictionTest.cpp b/tests/CalcBraggPredictionTest.cpp index 752823cf6..9d94c9a90 100644 --- a/tests/CalcBraggPredictionTest.cpp +++ b/tests/CalcBraggPredictionTest.cpp @@ -6,6 +6,9 @@ #include "../image_analysis/bragg_prediction/BraggPrediction.h" #include #include "../image_analysis/SensorAbsorption.h" +#include "../image_analysis/bragg_prediction/BraggPredictionRot.h" +#include "../image_analysis/bragg_prediction/RockingSlice.h" +#include // The flight-path term is a zero-parameter prediction: a NIST attenuation coefficient, the stated // sample-to-detector distance, and Beer-Lambert. Nothing about it is fitted, so it can be checked @@ -282,6 +285,79 @@ TEST_CASE("BraggPrediction_systematic_absences") { } } +TEST_CASE("RockingSliceCentroid_TruncatedNormalMean", "[rocking_slice]") { + // The closed form against a direct quadrature of the slice, on a frame that holds the centre, one on + // the curve's flank and one on its far tail. + const float sigma = 0.004f, half_wedge = 0.0015f, c1 = 1.0f / (std::sqrt(2.0f) * sigma); + for (float phi : {0.0f, 0.001f, -0.006f, 0.012f}) { + double sw = 0.0, stw = 0.0; + for (int i = 0; i <= 20000; ++i) { + const double t = phi - half_wedge + 2.0 * half_wedge * i / 20000.0; + const double w = std::exp(-t * t / (2.0 * sigma * sigma)); + sw += w; stw += t * w; + } + const float partiality = (std::erf((phi + half_wedge) * c1) - std::erf((phi - half_wedge) * c1)) / 2.0f; + INFO("phi " << phi); + CHECK(RockingSliceCentroid_rad(phi, half_wedge, c1, partiality) == Catch::Approx(stw / sw).margin(2e-6)); + } + CHECK(RockingSliceCentroid_rad(0.0f, half_wedge, c1, 0.3f) == 0.0f); +} + +TEST_CASE("BraggPredictionRot_PartialWalksAlongItsRing", "[rocking_slice]") { + // Each rotation frame predicts a partial where the frame's slice of its rocking curve puts it: the + // exact-condition position walked along the Debye ring. Over the frames of one reflection the + // predicted positions therefore stay at one distance from the beam and move monotonically along + // the ring; a prediction at the exact condition would be the same point on every frame. + DiffractionExperiment experiment(DetJF4M()); + experiment.DetectorDistance_mm(100.0).BeamX_pxl(1100.0).BeamY_pxl(1000.0).IncidentEnergy_keV(12.4); + const GoniometerAxis axis("omega", 0.0f, 0.1f, Coord(-1, 0, 0), {}); + experiment.Goniometer(axis); + const CrystalLattice lattice(Coord{40, 0, 0}, Coord{0, 50, 0}, Coord{0, 0, 60}); + BraggPredictionSettings settings{.high_res_A = 2.5, .ewald_dist_cutoff = 0.0015, + .max_h = 20, .max_k = 25, .max_l = 30, + .wedge_deg = 0.1f, .mosaicity_deg = 0.1f}; + const float bx = 1100.0f, by = 1000.0f; + + std::map, std::vector>> track; // (radius, azimuth) + BraggPredictionRot pred; + for (int frame = 0; frame < 300; ++frame) { + const auto latt = lattice.Multiply(axis.GetTransformationAngle(frame * 0.1f)); + const int n = pred.Calc(experiment, latt, settings); + for (int i = 0; i < n; ++i) { + const auto &r = pred.GetReflections().at(i); + if (r.zeta > 0.3f) continue; + const float dx = r.predicted_x - bx, dy = r.predicted_y - by; + track[{r.h, r.k, r.l}].emplace_back(std::hypot(dx, dy), std::atan2(dy, dx)); + } + } + int tested = 0; + for (const auto &[hkl, pts] : track) { + if (pts.size() < 20) continue; + float rmin = pts[0].first, rmax = pts[0].first; + float along_first = 0.0f, along_last = 0.0f; + int sign_changes = 0; + float prev_step = 0.0f; + for (size_t j = 0; j < pts.size(); ++j) { + rmin = std::min(rmin, pts[j].first); + rmax = std::max(rmax, pts[j].first); + const float along = (pts[j].second - pts[0].second) * pts[0].first; // px along the ring + if (j == 0) along_first = along; + along_last = along; + if (j > 0) { + const float step = (pts[j].second - pts[j - 1].second); + if (step * prev_step < 0.0f) ++sign_changes; + if (step != 0.0f) prev_step = step; + } + } + INFO("hkl " << std::get<0>(hkl) << " " << std::get<1>(hkl) << " " << std::get<2>(hkl) << " frames " << pts.size()); + CHECK(rmax - rmin < 0.05f); // stays on its ring + CHECK(std::fabs(along_last - along_first) > 0.1f); // but walks along it + CHECK(sign_changes == 0); // in one direction + ++tested; + } + CHECK(tested > 20); +} + #ifdef JFJOCH_USE_CUDA #include "../image_analysis/bragg_prediction/BraggPredictionGPU.h" #include "../image_analysis/bragg_prediction/BraggPredictionRotGPU.h" From aa1f9393477cff236e70cc292fc957799cd61247 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 24 Sep 2026 14:32:03 +0200 Subject: [PATCH 14/21] Error model: calibrate each bin on the median NORMALISED deviation, binned by counting I/sigma The rotation merge fitted var = a*s2 + b^2*^2 from three separate medians (s2, I2, dev2) per bin of I2. A median of dev2 over observations whose variances differ is not 0.455 times their mean variance, so the ratio of medians read a too low and b too high: on the scaled fulls of 28 sets (in-house, open and private) the core of the normalised deviations scattered at up to 1.8x its stated variance in the weak and middle bins and at 0.1-0.7x in the strongest. Reproduced on synthetic samples with a known model (a 1.3 read as 1.12, ISa 33 read as 31). Now (ErrorModel.h/.cpp, host-only, so the GPU and CPU paths share it): - bins are equal counts in counting I/sigma (I2/s2), where b is identified; - each bin is calibrated on the median of dev2/var with var from the previous iteration, iterated to a fixed point - heterogeneity inside a bin no longer biases it, and the median keeps it robust to tails; - s2 is the counting variance the merge actually applies the model to (rebuilt at the reflection's mean), not the observation's own sigma^2. The separate 6-sigma misfit refit is gone: the median does not need it. A mean-based fit (misfits cut at z^2 > 2 ln N) was tried first: it calibrates the total variance best (median rms log chi2 over the 28 sets 0.14 vs 0.19 here) but on heavy-tailed data it sizes the sigmas on the tails, the merge's outlier test widens with them, and CC1/2 fell 0.80 -> 0.71 on a powder-contaminated set (0.83 with this fit). Rejected for that. Offline, 28 sets: rms log chi2 of the median normalised deviation over (counting I/sigma x resolution) 0.208 -> 0.139 (better on 23), of the mean 0.239 -> 0.191 (better on 22). Battery, 45 of 48 sets against the 9b6736 run (3 lost to CUDA OOM from GPU contention): space group unchanged on all; d_min unchanged except two poor multi-lattice sets (1.69 -> 1.56, 1.96 -> 1.90); ISa x1.13 (median), in-house ISa/XDS 0.73 -> 0.93; ISa*R_meas_lo/0.8 0.93 -> 1.05 (XDS ~1.2); CC1/2 over the XDS range +0.002 (mean; up 0.014-0.031 on the three poorest sets, else +-0.0001); CC_model +0.0011, R_model_shell_scaled -0.0007 (mean over 17 open sets); CC_anom +0.003 (mean). Six private sets: space group, d_min and CC1/2 unchanged, ISa up by 14-67% towards XDS's. Remaining misfit, not addressed: the excess variance grows slower than ^2 (the effective fractional error falls 2-2.5x from counting I/sigma 5 to 200), so the strongest reflections still scatter below their sigma on open sets. A third, linear term (as in Aimless) fits it better on most sets but leaves b unidentified on some (b -> 0 on 4 of 28); not landed. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- image_analysis/scale_merge/CMakeLists.txt | 4 +- image_analysis/scale_merge/ErrorModel.cpp | 140 ++++++++++++ image_analysis/scale_merge/ErrorModel.h | 48 +++++ .../scale_merge/RotationScaleMerge.cpp | 200 +++--------------- .../scale_merge/RotationScaleMerge.h | 15 +- tests/MergeScaleTest.cpp | 70 ++++++ 6 files changed, 303 insertions(+), 174 deletions(-) create mode 100644 image_analysis/scale_merge/ErrorModel.cpp create mode 100644 image_analysis/scale_merge/ErrorModel.h diff --git a/image_analysis/scale_merge/CMakeLists.txt b/image_analysis/scale_merge/CMakeLists.txt index 985ce4692..b54a9fa7d 100644 --- a/image_analysis/scale_merge/CMakeLists.txt +++ b/image_analysis/scale_merge/CMakeLists.txt @@ -28,7 +28,9 @@ ADD_LIBRARY(JFJochScaleMerge ScalingResult.h ScalingResult.cpp WilsonOutliers.cpp - WilsonOutliers.h) + WilsonOutliers.h + ErrorModel.cpp + ErrorModel.h) TARGET_LINK_LIBRARIES(JFJochScaleMerge Ceres::ceres Eigen3::Eigen JFJochCommon fftw3f) IF (JFJOCH_CUDA_AVAILABLE) diff --git a/image_analysis/scale_merge/ErrorModel.cpp b/image_analysis/scale_merge/ErrorModel.cpp new file mode 100644 index 000000000..4e68cdac6 --- /dev/null +++ b/image_analysis/scale_merge/ErrorModel.cpp @@ -0,0 +1,140 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "ErrorModel.h" + +#include +#include +#include +#include + +#include "../../common/ParallelFor.h" + +ErrorModelFit FitErrorModel(const std::vector &pool, std::vector &smp, + size_t nthreads) { + constexpr int n_bins = 16; + ErrorModelFit fit; + if (pool.size() < static_cast(8 * n_bins)) + return fit; + + // The rank key is computed once, here, and carried with the sample: a key recomputed inside the + // comparator can round differently at different inlined call sites, and nth_element then sees an + // inconsistent order. + smp.resize(pool.size()); + ParallelChunks(static_cast(pool.size()), ThreadsForWork(pool.size(), nthreads), [&](int lo, int hi) { + for (int i = lo; i < hi; ++i) + smp[i] = {pool[i].I2 / pool[i].s2, pool[i].s2, pool[i].I2, pool[i].dev2, 0.0}; + }); + + // Equal-count bins, each boundary put in place with nth_element; the split runs down the middle so + // each level halves the range it works on. Total order, so which of two equal keys lands in which bin + // is a property of the samples, not of the order they arrived in. The two halves of a split are + // disjoint ranges, so they can run side by side while the range is still worth a thread. + const size_t per = smp.size() / n_bins; + const auto by_snr = [](const ErrorModelBinned &a, const ErrorModelBinned &b) { + if (a.snr2 != b.snr2) return a.snr2 < b.snr2; + if (a.s2 != b.s2) return a.s2 < b.s2; + if (a.I2 != b.I2) return a.I2 < b.I2; + return a.dev2 < b.dev2; + }; + constexpr size_t SPLIT_MIN_PARALLEL = 1u << 17; + const std::function split = + [&](size_t lo, size_t hi, int b0, int b1, int depth) { + if (b0 >= b1) return; + const int mid = (b0 + b1) / 2; + const size_t k = static_cast(mid) * per; + std::nth_element(smp.begin() + lo, smp.begin() + k, smp.begin() + hi, by_snr); + if (depth > 0 && hi - lo > SPLIT_MIN_PARALLEL) { + auto left = std::async(std::launch::async, + [&, lo, k, b0, mid, depth] { split(lo, k, b0, mid, depth - 1); }); + split(k, hi, mid + 1, b1, depth - 1); + left.get(); + } else { + split(lo, k, b0, mid, 0); + split(k, hi, mid + 1, b1, 0); + } + }; + split(0, smp.size(), 1, n_bins, nthreads > 1 ? 3 : 0); + auto bin_lo = [&](int bin) { return static_cast(bin) * per; }; + auto bin_hi = [&](int bin) { return bin == n_bins - 1 ? smp.size() : static_cast(bin + 1) * per; }; + + // b is identified only if the strongest bin reaches where b* can outgrow the counting term: when + // fewer reflections are strong than one bin holds, that bin sits at I/sigma ~ 1 and a two-term fit + // hands b the bins' own noise instead (measured: b = 5.6, ISa 0.5, on a crystal XDS gives 6.1). Then + // fit a alone and leave b at zero. Over the rotation battery the two crystals this fires on sit at + // (I/sigma)^2 0.22 and 0.84 while the next is 31.7, so the threshold is not delicate. + constexpr double B_LEVER_MIN = 4.0; + { + const auto top_mid = smp.begin() + (bin_lo(n_bins - 1) + bin_hi(n_bins - 1)) / 2; + std::nth_element(smp.begin() + bin_lo(n_bins - 1), top_mid, smp.end(), by_snr); + fit.b_measured = top_mid->snr2 > B_LEVER_MIN; + } + + // The median of a chi^2 with one degree of freedom: a normalised squared deviation that is right + // on average has half of its values below this. + constexpr double CHI2_1_MEDIAN = 0.454936; + std::vector bs2(n_bins), bI2(n_bins), bd2(n_bins); + double a = 1.0, b2 = 0.0; + for (int iter = 0; iter < 30; ++iter) { + // Per bin: the terms of var = a*s2 + b^2*I2, each divided by the current model's variance, and + // the median normalised squared deviation, expressed as the sum it stands for. Reordering inside + // a bin does not change which samples it holds. One bin per task, so the result does not depend + // on threads. + ParallelChunks(n_bins, ThreadsForWork(smp.size(), nthreads, 8 * 32768), [&](int blo, int bhi) { + for (int bin = blo; bin < bhi; ++bin) { + double s = 0.0, I = 0.0; + for (size_t i = bin_lo(bin); i < bin_hi(bin); ++i) { + auto &x = smp[i]; + const double v = a * x.s2 + b2 * x.I2; + s += x.s2 / v; I += x.I2 / v; + x.z2 = x.dev2 / v; + } + const size_t n = bin_hi(bin) - bin_lo(bin); + const auto mid = smp.begin() + bin_lo(bin) + n / 2; + std::nth_element(smp.begin() + bin_lo(bin), mid, smp.begin() + bin_hi(bin), + [](const ErrorModelBinned &p, const ErrorModelBinned &q) { return p.z2 < q.z2; }); + bs2[bin] = s; bI2[bin] = I; bd2[bin] = mid->z2 / CHI2_1_MEDIAN * static_cast(n); + } + }); + + // Every bin weighs the same in RELATIVE terms, so the weak bins still constrain a. + double Ass = 0, AsI = 0, AII = 0, Bs = 0, BI = 0; + for (int bin = 0; bin < n_bins; ++bin) { + if (!(bd2[bin] > 0.0)) continue; + const double w = 1.0 / (bd2[bin] * bd2[bin]); + Ass += w * bs2[bin] * bs2[bin]; AsI += w * bs2[bin] * bI2[bin]; AII += w * bI2[bin] * bI2[bin]; + Bs += w * bs2[bin] * bd2[bin]; BI += w * bI2[bin] * bd2[bin]; + } + const double det = Ass * AII - AsI * AsI; + double a_new, b2_new; + if (!fit.b_measured) { + if (!(Ass > 0.0)) return fit; + a_new = std::clamp(Bs / Ass, 0.25, 100.0); + b2_new = 0.0; + } else { + if (!(det > 1e-10 * Ass * AII)) return fit; + const double a_fit = (Bs * AII - BI * AsI) / det; + const double b2_fit = (Ass * BI - AsI * Bs) / det; + a_new = std::clamp(a_fit, 0.25, 100.0); + b2_new = std::max(b2_fit, 0.0); + // Leverage is not enough: b^2 can have it and still come out at zero within its own error, + // and 1/b then reads the noise in b as an I/sigma. The standard error of b^2 comes from the + // bins' own scatter about the fitted line; this only decides what is printed. + double chi = 0.0; + for (int bin = 0; bin < n_bins; ++bin) { + if (!(bd2[bin] > 0.0)) continue; + const double r = (bd2[bin] - a_fit * bs2[bin] - b2_fit * bI2[bin]) / bd2[bin]; + chi += r * r; + } + fit.b_resolved = b2_fit >= 2.0 * std::sqrt(chi / (n_bins - 2) * Ass / det); + } + const bool settled = std::fabs(a_new - a) <= 1e-4 * a && std::fabs(b2_new - b2) <= 1e-4 * b2; + a = a_new; + b2 = b2_new; + fit.active = true; + fit.a = a; + fit.b2 = b2; + if (settled) break; + } + return fit; +} diff --git a/image_analysis/scale_merge/ErrorModel.h b/image_analysis/scale_merge/ErrorModel.h new file mode 100644 index 000000000..89716bf9f --- /dev/null +++ b/image_analysis/scale_merge/ErrorModel.h @@ -0,0 +1,48 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +#include +#include + +// The error model of the rotation merge: var = a * s2 + b^2 * ^2, fitted so that the scatter of +// symmetry equivalents matches it - in each bin of counting I/sigma, the median of dev2 / var is the +// median of a chi^2 with one degree of freedom. +// +// One sample per observation: s2 is its counting variance as the merge rebuilds it at the reflection's +// mean intensity, I2 that mean squared, dev2 its leverage-corrected squared deviation from the mean. +// +// Two choices, each against a measured failure of the fit this replaces, which took the medians of s2, +// I2 and dev2 separately over bins of I2: +// * Bins are equal counts in COUNTING I/sigma (I2 / s2), not in I2. b is identified by how far the +// bins reach into the regime where b* dominates the counting term, and that is what this ranks +// on; ranking on I2 mixes weak high-resolution reflections into the strong bins. +// * The median is taken of the NORMALISED deviation, each sample divided by the variance the previous +// iteration gave it. A median of dev2 over observations whose variances differ is not 0.455 times +// their mean variance, and the ratio of three medians read a too low and b too high: on most of 28 +// sets the core of the normalised deviations then scattered at up to 1.8 times its stated variance +// in the weak and middle bins and at 0.1-0.7 times it in the strongest. +// A median, not a mean: on heavy-tailed data (split and powder-contaminated crystals) a mean calibrates +// the sigmas on the tails, the merge's outlier test widens with them, and CC1/2 fell 0.80 -> 0.71 on one +// such set where this fit raises it to 0.83. +struct ErrorModelSample { + double s2, I2, dev2; + float d; +}; + +struct ErrorModelFit { + bool active = false; // the pool was large enough to fit + double a = 1.0, b2 = 0.0; + bool b_measured = true; // false: no bin reached where b could be seen; b is held at 0 + bool b_resolved = true; // b^2 stood clear of two of its standard errors +}; + +// Scratch: the partitioned copy of the pool (reused between calls to save the allocation), with the rank +// key and the current normalised deviation. +struct ErrorModelBinned { + double snr2, s2, I2, dev2, z2; +}; + +ErrorModelFit FitErrorModel(const std::vector &pool, std::vector &scratch, + size_t nthreads); diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index a558e74ee..b57d21126 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -4047,8 +4047,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool double error_model_a = 1.0, error_model_b = 0.0; bool error_model_active = false; // What the reported chi2 is taken from, below: the pool the LAST successful fit was handed, with - // that fit's own a and b^2. fit_ab works on a partitioned copy of its pool and a median does not - // depend on the order, so the copy is not the thing to keep; b^2 is the fit's own because + // that fit's own a and b^2. FitErrorModel works on a partitioned copy of its pool and a median does + // not depend on the order, so the copy is not the thing to keep; b^2 is the fit's own because // error_model_b is its square root and squaring it back up is not the same number. const std::vector *chi2_pool = nullptr; double chi2_a = 0.0, chi2_b2 = 0.0; @@ -4056,159 +4056,29 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool bool error_model_b_resolved = true; // b^2 stood clear of its own standard error std::vector &samples = em_samples; samples.clear(); - constexpr int n_bins = 16; - // Fit (a, b) from the intensity-binned median deviations of a pool of samples, then refit on that - // pool's misfit-free subset. A lambda because the pool changes once the cutoff below is known. - std::vector &fit_scratch = em_fit_pool; + // A full's counting variance at intensity I_for_b. Its own sigma carries its own Poisson fluctuation: + // a full that came out low has a smaller sigma, so 1/sigma^2 weights it up and the merged mean drifts + // below . Rebuild the variance at the reflection's EXPECTED intensity - var_bkg + var_per_I*, + // the linear model the combine measured - so the weight no longer knows this full's own fluctuation. + // Mirrors MergeOnTheFly::CorrectedSigma on the stills path. The error model is fitted on this same + // variance, so that its a scales what the merge applies it to. + auto counting_variance = [](const Obs &o, double I_for_b, double own_s2) { + const double base = static_cast(o.corr) * o.corr * o.var_bkg + + static_cast(o.corr) * o.var_per_I * std::max(0.0, I_for_b); + return base > 0.0 ? base : own_s2; + }; + // Fit (a, b) to a pool of samples (ErrorModel.h). A lambda because the pool changes once the cutoff + // below is known. + std::vector &fit_scratch = em_fit_pool; auto fit_error_model = [&](const std::vector &pool) { - // Works on a scratch copy, and the caller's pool keeps its order. That matters: the refit - // below builds its own pool by walking this one, and the partition is not a total order, so - // handing the refit a reordered pool moves which of two samples of equal I2 lands in which - // bin. One reused buffer rather than a fresh vector per call - the copy is a memcpy and costs - // a few milliseconds against the hundreds the sort it replaces used to. - auto fit_ab = [&](const std::vector &in) { - fit_scratch.assign(in.begin(), in.end()); - std::vector &smp = fit_scratch; - if (smp.size() < static_cast(8 * n_bins)) - return; - const size_t per = smp.size() / n_bins; - // The bins are equal COUNTS in I2 rank, and all that comes out of one is three medians - so - // the bin has to be the right SET, not a sorted one. Put each boundary in place with - // nth_element instead, splitting the boundaries down the middle so each level halves the - // range it works on: order n per level and four levels, against n log n for the sort. The - // medians are then taken off the bin's own span, by the field wanted, with no copy at all - - // the three vectors this used to build were the whole pool again, three times over, and - // median_of takes the lower median, which is what nth_element leaves at that index. - // Total, so which of two samples of equal I2 falls in which bin is a property of the - // samples rather than of the algorithm that partitioned them. The full sort this replaces - // was unstable, so it resolved those ties by whatever order it happened to receive - and - // the refit below is handed a different order from the first fit. Ordering on the other - // two fields costs nothing (they are already in the cache line) and settles it for good. - const auto by_I2 = [](const Sample &a, const Sample &b) { - if (a.I2 != b.I2) return a.I2 < b.I2; - if (a.s2 != b.s2) return a.s2 < b.s2; - if (a.dev2 != b.dev2) return a.dev2 < b.dev2; - return a.d < b.d; - }; - // The two halves a split leaves behind are disjoint ranges whose contents the nth_element - // above them has already fixed, so a child sees the same sequence whether it runs before, - // after or beside its sibling and the bins come out the same either way. Run them side by - // side while the range is still worth a thread; below that they stay on this one. - constexpr size_t SPLIT_MIN_PARALLEL = 1u << 17; - const std::function split = - [&](size_t lo, size_t hi, int b0, int b1, int depth) { - if (b0 >= b1) return; - const int mid = (b0 + b1) / 2; - const size_t k = static_cast(mid) * per; - std::nth_element(smp.begin() + lo, smp.begin() + k, smp.begin() + hi, by_I2); - if (depth > 0 && hi - lo > SPLIT_MIN_PARALLEL) { - auto left = std::async(std::launch::async, - [&, lo, k, b0, mid, depth] { split(lo, k, b0, mid, depth - 1); }); - split(k, hi, mid + 1, b1, depth - 1); - left.get(); - } else { - split(lo, k, b0, mid, 0); - split(k, hi, mid + 1, b1, 0); - } - }; - split(0, smp.size(), 1, n_bins, nthreads > 1 ? 3 : 0); - std::vector bs2(n_bins), bI2(n_bins), bd2(n_bins); - ParallelChunks(n_bins, ThreadsForWork(smp.size(), nthreads, 8 * 32768), [&](int blo, int bhi) { - for (int bin = blo; bin < bhi; ++bin) { - const size_t lo = static_cast(bin) * per; - const size_t hi = (bin == n_bins - 1) ? smp.size() : lo + per; - const auto mid = smp.begin() + lo + (hi - lo) / 2; - std::nth_element(smp.begin() + lo, mid, smp.begin() + hi, - [](const Sample &a, const Sample &b) { return a.s2 < b.s2; }); - bs2[bin] = mid->s2; - std::nth_element(smp.begin() + lo, mid, smp.begin() + hi, by_I2); - bI2[bin] = mid->I2; - std::nth_element(smp.begin() + lo, mid, smp.begin() + hi, - [](const Sample &a, const Sample &b) { return a.dev2 < b.dev2; }); - bd2[bin] = mid->dev2 / CHI2_1_MEDIAN; - } - }); - // `b` is identified ONLY by the spread of I^2/sigma^2 across the bins, and the bins hold equal - // COUNTS - so when fewer reflections are strong than one bin holds (1/16 of the pool), the top - // bin's median sits at an intensity where b cannot be measured, and the fit assigns it the - // bins' own noise-selection slope instead. Measured: a crystal with 2.2% of its fulls above - // I/sigma 2 reports b = 5.6, i.e. sigma -> 2*I at the strong end and ISa 0.50, while XDS gets - // 6.13 from the same images; synthetic data with b_true = 0 and 1.8% strong reproduces that to - // two digits, and recovers the truth as soon as the strong fraction passes 1/16. sigma^2 is - // then a*sigma^2 + (b*mean)^2 with a per-group constant, which caps merged |I/sigma| at - // sqrt(n)/b - reporting an impossible systematic error on data whose CC1/2 is 99%. - // - // So refuse to report what was not measured: if the strongest bin's own (I/sigma)^2 is small, - // fit `a` alone and leave b at zero. The threshold is not delicate - over the rotation battery - // the two crystals this fires on sit at 0.22 and 0.84 while the next is 31.7 and a healthy one - // is 342, so anything from 4 to 25 selects the same two and nothing else changes. - constexpr double B_LEVER_MIN = 4.0; - const bool b_measurable = bI2.back() > B_LEVER_MIN * bs2.back(); - std::vector bd2_sorted = bd2; - const double dev2_floor = std::max(1e-30, 1e-3 * median_of(bd2_sorted)); - double Ass = 0, AsI = 0, AII = 0, Bs = 0, BI = 0; - for (int bin = 0; bin < n_bins; ++bin) { - const double s2 = bs2[bin], I2 = bI2[bin], d2 = bd2[bin]; - const double d2w = std::max(d2, dev2_floor); - const double wgt = 1.0 / (d2w * d2w); - Ass += wgt * s2 * s2; AsI += wgt * s2 * I2; AII += wgt * I2 * I2; - Bs += wgt * s2 * d2; BI += wgt * I2 * d2; - } - const double det = Ass * AII - AsI * AsI; - if (!b_measurable) { - // One parameter, same weights: dev2 = a*sigma^2 alone. - if (Ass > 0.0) { - error_model_a = std::clamp(Bs / Ass, 0.25, 100.0); - error_model_b = 0.0; - error_model_b_unmeasured = true; - error_model_active = true; - chi2_pool = ∈ chi2_a = error_model_a; chi2_b2 = 0.0; - } - } else if (det > 1e-10 * Ass * AII) { - const double a_fit = (Bs * AII - BI * AsI) / det; - const double b2_fit = (Ass * BI - AsI * Bs) / det; - error_model_a = std::clamp(a_fit, 0.25, 100.0); - const double b2 = std::max(b2_fit, 0.0); - error_model_b = std::sqrt(b2); - error_model_b_unmeasured = false; - // Leverage is not enough: b^2 can have it and still come out at zero within its own error - - // the strong bins scatter no more than counting says, or there are too few samples for a - // bin median to mean anything. 1/b then reads the noise in b as an I/sigma, any value from - // ~30 to infinity between merges of the same data. The standard error of b^2 is taken from - // the bins' own scatter about the fitted line; the fit and the sigmas are left as they are, - // and so is the fitted ISa every decision reads - this only decides what is printed. - // Measured: z = 13.6 and 22 on two crystals whose b is plainly visible, 0.1-0.5 on a - // 3000-sample low-multiplicity set and -1.2 to -6 on one whose strongest bins scatter - // LESS than counting says (both printed ISa 96-130). - double chi = 0.0; - for (int bin = 0; bin < n_bins; ++bin) { - const double d2w = std::max(bd2[bin], dev2_floor); - const double r = bd2[bin] - a_fit * bs2[bin] - b2_fit * bI2[bin]; - chi += r * r / (d2w * d2w); - } - const double b2_se = std::sqrt(chi / (n_bins - 2) * Ass / det); - error_model_b_resolved = b2_fit >= 2.0 * b2_se; - error_model_active = true; - chi2_pool = ∈ chi2_a = error_model_a; chi2_b2 = b2; - } - }; - fit_ab(pool); - // Refit on a misfit-free pool: the merge drops symmetry outliers (|I - median| > reject_nsigma * - // sigma) from the merged intensity, so drop the equivalent samples (dev2 > reject_nsigma^2 * model - // variance) from the error-model fit too, keeping the fitted sigmas consistent with the reflections - // that actually survive. Operates on the shared samples, so CPU and GPU stay bit-identical. - if (reject_outliers && error_model_active) { - const double ns2 = reject_nsigma * reject_nsigma, b2 = error_model_b * error_model_b; - std::vector &kept = em_refit_pool; - kept.clear(); - kept.reserve(pool.size()); - for (const auto &s : pool) { - const double v = error_model_a * s.s2 + b2 * s.I2; - if (v > 0.0 && s.dev2 <= ns2 * v) kept.push_back(s); - } - if (kept.size() >= static_cast(8 * n_bins) && kept.size() < pool.size()) - fit_ab(kept); - } + const ErrorModelFit fit = FitErrorModel(pool, fit_scratch, nthreads); + if (!fit.active) return; + error_model_a = fit.a; + error_model_b = std::sqrt(fit.b2); + error_model_b_unmeasured = !fit.b_measured; + error_model_b_resolved = fit.b_resolved; + error_model_active = true; + chi2_pool = &pool; chi2_a = fit.a; chi2_b2 = fit.b2; }; // Friedel pair of each group when the hands are merged separately (-A); filled with the medians below. std::vector pair_of_group; @@ -4227,7 +4097,9 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool gs2.data(), gI2.data(), gdev2.data(), gvalid.data()); samples.reserve(nf); for (int i = 0; i < nf; ++i) - if (gvalid[i]) samples.push_back({gs2[i], gI2[i], gdev2[i], mf.d[i]}); + if (gvalid[i]) + samples.push_back({counting_variance(fulls[i], em_mean[fulls[i].group], gs2[i]), + gI2[i], gdev2[i], mf.d[i]}); did_gpu = true; } #endif @@ -4267,7 +4139,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool const double factor = 1.0 - w / (on_hand ? swh[s] : sw[o.group]); if (factor < 0.05) continue; const double resid = static_cast(o.I) * o.corr - mean; - samples.push_back({s2, mean * mean, resid * resid / factor, o.d}); + samples.push_back({counting_variance(o, em_mean[o.group], s2), mean * mean, + resid * resid / factor, o.d}); } } @@ -4373,16 +4246,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // The full's sigma under the error model, with the variance evaluated at intensity I_for_b. auto model_sigma = [&](const Obs &o, double I_for_b, float sigma_corr) -> float { if (!error_model_active) return sigma_corr; - // A full's own sigma carries its own Poisson fluctuation: a full that came out low has a - // smaller sigma, so 1/sigma^2 weights it up and the merged mean drifts below . Rebuild the - // variance at the reflection's EXPECTED intensity - var_bkg + var_per_I*, the linear model - // the combine measured - so the weight no longer knows this full's own fluctuation. Mirrors - // MergeOnTheFly::CorrectedSigma on the stills path. - double a_var = static_cast(sigma_corr) * sigma_corr; - const double base = static_cast(o.corr) * o.corr * o.var_bkg - + static_cast(o.corr) * o.var_per_I * std::max(0.0, I_for_b); - if (base > 0.0) - a_var = base; + const double a_var = counting_variance(o, I_for_b, static_cast(sigma_corr) * sigma_corr); const double v = error_model_a * a_var + (error_model_b * I_for_b) * (error_model_b * I_for_b); return v > 0.0 ? static_cast(std::sqrt(v)) : sigma_corr; @@ -4715,7 +4579,9 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool const double I_corr = static_cast(mf.I[i]) * mf.corr[i]; const double sigma_corr = static_cast(mf.sigma[i]) * mf.corr[i]; auto &g = gs[gi]; - g.sum += I_corr; g.sum_sq += I_corr * I_corr; g.sum_var += sigma_corr * sigma_corr; ++g.n; + g.sum += I_corr; g.sum_sq += I_corr * I_corr; + g.sum_var += counting_variance(fulls[i], em_mean[gi], sigma_corr * sigma_corr); + ++g.n; } // Per-group counting-subtracted fractional systematic variance, paired with the group's I/sigma. // Two conventions decide whether this measures anything: diff --git a/image_analysis/scale_merge/RotationScaleMerge.h b/image_analysis/scale_merge/RotationScaleMerge.h index 5f5e7a6e0..116378579 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.h +++ b/image_analysis/scale_merge/RotationScaleMerge.h @@ -16,6 +16,7 @@ #include "../../common/UnitCell.h" #include "../IntegrationOutcome.h" +#include "ErrorModel.h" #include "Merge.h" // MergedReflection, MergeStatistics #ifdef JFJOCH_USE_CUDA #include @@ -144,10 +145,11 @@ private: int32_t group; // dense ASU-group id for the current space group; <0 = never mergeable }; - // One leverage-corrected error-model sample per usable full: its raw variance, its group's mean - // intensity, its squared deviation from that mean - and the resolution it sits at, because the fit is - // re-run over the samples that survive the automatic resolution cutoff. See MergeAndStats. - struct Sample { double s2, I2, dev2; float d; }; + // One leverage-corrected error-model sample per usable full: its counting variance at the group's + // mean, that mean squared, its squared deviation from the mean - and the resolution it sits at, + // because the fit is re-run over the samples that survive the automatic resolution cutoff. See + // MergeAndStats and ErrorModel.h. + using Sample = ErrorModelSample; // The narrow per-observation record the ingest sort orders: the raw hkl the runs are cut on, the // frame position that breaks a tie inside one, the observation's own index (which makes the order @@ -355,11 +357,12 @@ private: std::vector asymptote_scatter; // The error model's working pools (see MergeAndStats): the samples themselves, the scratch copy each - // fit partitions, the misfit-free subset the refit uses, the subset inside the resolution cutoff, and + // fit partitions, the subset inside the resolution cutoff, and // the per-sample chi2 the reported number is the median of. Members for the same reason as FullsStaging - one is 32 bytes per full and // there are two dozen fits per run, so as locals this is gigabytes of pages faulted in and handed // straight back. Every one of them is cleared and refilled before it is read. - std::vector em_samples, em_fit_pool, em_refit_pool, em_cut_pool; + std::vector em_samples, em_cut_pool; + std::vector em_fit_pool; std::vector em_chi2; // Set by FitPerFrameG: which frames were fitted this call (so corr/G is updated only there). diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index 4794b6a7e..7be51a64e 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -4,6 +4,7 @@ #include #include +#include "../image_analysis/scale_merge/ErrorModel.h" #include "../image_analysis/scale_merge/HKLKey.h" #include "../image_analysis/scale_merge/Merge.h" #include "../image_analysis/scale_merge/ResolutionCutoff.h" @@ -406,3 +407,72 @@ TEST_CASE("ResolutionCutoff_RaggedFallOffIsReadOffTheBins") { REQUIRE(rc.d_cut); CHECK(*rc.d_cut > 2.30); // coarser than the band that correlates again } + +namespace { + // Samples of var = a*s2 + b^2*I2 over four decades of counting I/sigma, with counting variances + // spread over a factor of 100 at every intensity, and a fraction of gross outliers. + std::vector SyntheticErrorModelSamples(double a, double b, double max_snr, + double outlier_fraction) { + std::mt19937 rng(7); + std::uniform_real_distribution u(0.0, 1.0); + std::normal_distribution z(0.0, 1.0); + std::vector out; + for (int i = 0; i < 200000; ++i) { + const double s2 = std::pow(10.0, 2.0 * u(rng)); + const double snr = max_snr * std::pow(10.0, -4.0 * u(rng)); + const double I2 = snr * snr * s2; + const double e = z(rng); + double dev2 = (a * s2 + b * b * I2) * e * e; + if (u(rng) < outlier_fraction) dev2 *= 400.0; + out.push_back({s2, I2, dev2, 2.0f}); + } + return out; + } +} + +// The fit recovers a and b from observations whose counting variances differ widely inside every +// intensity bin (where a ratio of bin medians does not), with 0.3% gross outliers in the pool. +TEST_CASE("ErrorModel_RecoversAAndBThroughOutliers") { + const auto pool = SyntheticErrorModelSamples(1.3, 0.03, 300.0, 0.003); + std::vector scratch; + const auto fit = FitErrorModel(pool, scratch, 4); + REQUIRE(fit.active); + CHECK(fit.b_measured); + CHECK(fit.b_resolved); + CHECK(fit.a == Catch::Approx(1.3).epsilon(0.03)); + CHECK(1.0 / std::sqrt(fit.b2) == Catch::Approx(1.0 / 0.03).epsilon(0.05)); +} + +// A few observations whose counting variance is hugely overstated - their scatter is ordinary - do not +// set the fit: each sample counts by its deviation relative to its own variance. +TEST_CASE("ErrorModel_OverstatedCountingVarianceDoesNotSetTheFit") { + auto pool = SyntheticErrorModelSamples(1.3, 0.03, 300.0, 0.0); + for (size_t i = 0; i < pool.size(); i += 200) + pool[i].s2 *= 1e4; + std::vector scratch; + const auto fit = FitErrorModel(pool, scratch, 4); + REQUIRE(fit.active); + CHECK(fit.a == Catch::Approx(1.3).epsilon(0.03)); + CHECK(1.0 / std::sqrt(fit.b2) == Catch::Approx(1.0 / 0.03).epsilon(0.05)); +} + +// The same pool on one thread and on many gives the same numbers. +TEST_CASE("ErrorModel_SameOnAnyNumberOfThreads") { + const auto pool = SyntheticErrorModelSamples(1.1, 0.05, 100.0, 0.001); + std::vector scratch; + const auto one = FitErrorModel(pool, scratch, 1); + const auto many = FitErrorModel(pool, scratch, 16); + CHECK(one.a == many.a); + CHECK(one.b2 == many.b2); +} + +// Data that never reach where b could be seen: a is fitted alone and b held at 0. +TEST_CASE("ErrorModel_BNotMeasuredOnWeakData") { + const auto pool = SyntheticErrorModelSamples(0.9, 0.05, 1.5, 0.0); + std::vector scratch; + const auto fit = FitErrorModel(pool, scratch, 4); + REQUIRE(fit.active); + CHECK(!fit.b_measured); + CHECK(fit.b2 == 0.0); + CHECK(fit.a == Catch::Approx(0.9).epsilon(0.03)); +} From 6dcef006bb2e37a5d3897d2f2f9a3761b995c9ec Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 25 Sep 2026 02:40:25 +0200 Subject: [PATCH 15/21] Two-pass: lost-centring guard only for the same primitive metric A pass 2 that re-indexes pass 1's axis-multiple supercell to the true cell reports it primitive-centred; the guard read that as a dropped centring and forced pass 1's supercell back (completeness 99.7% -> 65% on a 7x case). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- rugnux/Rugnux.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 805a7369b..6289bfda4 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -5023,8 +5023,11 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // centring implies, and the pass then merges in P1 (measured on a C-centred monoclinic // crystal). Only this direction - a centring pass 2 finds where pass 1 had none is the // refined geometry doing its job, and the intensity test in the search confirms it. + // Same primitive metric only: a pass 2 that lands on a primitive cell several times + // smaller is a different lattice - pass 1's supercell corrected - not a dropped centring. const bool lost_centring = prepass_result_->search_result.centering != 'P' - && best.result->search_result.centering == 'P'; + && best.result->search_result.centering == 'P' + && v1 > 1.0 && std::abs(v2 / v1 - 1.0) < 0.1; if (supercell || lost_centring) { logger.Info("Two-pass: that lattice ({}-centred, volume {:.0f} A^3) disagrees with " "pass 1 ({}-centred, {:.0f} A^3) - integrating with pass-1's lattice instead", From 94ed6aec045b1147064496a0a053c27cc2369b90 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 25 Sep 2026 04:44:50 +0200 Subject: [PATCH 16/21] Scaling: a frame below the credible floor does not hold the loop's step open ComputeSmoothGWindow leaves a frame below MIN_CREDIBLE_SCALE_RATIO of the median out of every window, and DropCollapsedScales drops it after the loop, but RunScalingLoop still counted its step. Such a frame has lost its vote in its own references and keeps moving, so the loop was stopped as "unsettled". On 8xtf (hq-pool battery) the first-pass partials loop stopped after 16 rounds at rms dlogG 1.8e-01 (base 9.5e-04); with the frame left out of the step it settles in 23 rounds (8.0e-04), and every other loop of that run that had stopped unsettled (pass 2 at 1.6e-03) now settles too. Every loop of that sweep that stopped unsettled was one that went on to drop a frame. The final 8xtf merge is bit-identical; the false SCALING_NOT_CONVERGED warning goes. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CHANGELOG.md | 1 + image_analysis/scale_merge/RotationScaleMerge.cpp | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a2928f583..9504b1a74 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,6 +7,7 @@ * Rugnux rejects single observations that Wilson statistics make implausible (typically a hot pixel, zinger or ice spot), including pairs and single measurements that the equivalents test cannot judge; the count is reported as `OBSERVATIONS_REJECTED_WILSON`. * Rugnux reports the X-ray bandwidth it measures from spot shapes (for multilayer and pink beams); the value is reported only and does not change processing. * Rugnux accepts a detector-modulation or absorption correction surface when held-out data support it on Fisher's z, and fits them in the order modulation, time, goniometer frame; this recovers corrections that were wrongly refused. +* Rugnux no longer stops per-frame scaling as unsettled because of a single frame that is about to be dropped as blank. * `rugnux --polarization` is documented as the polarization degree (XDS `FRACTION_OF_POLARIZATION` = (1 + p)/2); the default 0.99 suits undulators. ### 1.0.0-rc.172 diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index b57d21126..e480d2aee 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -1568,10 +1568,23 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( } rescale(scaled_any, ratio); if (it > 0) { + // A frame already below the credible floor is not part of the fit's answer: the window + // leaves it out (ComputeSmoothGWindow) and the guard after the loop drops it. Left in the + // step, a single such frame held the rms at 0.18 and the loop was stopped as unsettled after + // 16 rounds; with it left out the same loop settles in 23. On that sweep every loop that + // stopped unsettled was one that went on to drop a frame. + std::vector fitted; + for (int f = 0; f < n_frames; ++f) + if (scaled_any[f] && g[f] > 0.0) fitted.push_back(g[f]); + double g_floor = 0.0; + if (!fitted.empty()) { + std::nth_element(fitted.begin(), fitted.begin() + fitted.size() / 2, fitted.end()); + g_floor = fitted[fitted.size() / 2] * MIN_CREDIBLE_SCALE_RATIO; + } double ss = 0.0, sw = 0.0; int n = 0; for (int f = 0; f < n_frames; ++f) - if (scaled_any[f] && g_prev[f] > 0.0 && g[f] > 0.0) { + if (scaled_any[f] && g_prev[f] > 0.0 && g[f] > 0.0 && g[f] >= g_floor) { const double d = std::log(g[f] / g_prev[f]); ss += frame_obs_count[f] * d * d; sw += frame_obs_count[f]; From 5c5eb00904468c9e912a53db6f3788e4253d3d5b Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 25 Sep 2026 04:52:18 +0200 Subject: [PATCH 17/21] Merge statistics: R_meas weighted as the merge weights each observation The delta-CC1/2 disposition keeps a weak stretch in the merge (downgraded) wherever removing it would not raise CC1/2, and the merge then carries each of its observations at the small 1/sigma^2 its scaled-up counting error gives it. R_meas counted those observations at full weight, so the frames that add next to nothing to the intensities set the number. hq-pool battery: 8xtf kept 115 weak frames (scale 0.1-0.2 of the run's) with the same CC1/2, and CC_model per shell as rc173 with 84 frames rejected, and R_meas was 1.3-1.5x higher in every shell; 9w3y (33 deg rejected -> 0, every model metric better) and lyso_x10sa_strong read the same way. The disposition itself is not segmentation-dependent: conviction is on the batch grid, not the ledger ranges, and those sets' dispositions changed because the corrected data changed. R_meas now weights each observation by its merge weight v = 1/sigma^2 under the error model (corrected_sigma on the host, ModelSigma on the GPU, with the error model of the last MergeAccum), normalised per reflection to Kish's effective count, so equal sigmas give the ordinary formula (WeightedRmeasTerms). Where the proportional term dominates - strong reflections - frames are weighted alike, as in the merge: 8xtf's lowest shell reads 15.0%, the rc173 run with 84 frames rejected 15.2%. The per-hand table is weighted the same way. R_MEAS_UNWEIGHTED / REFRES_R_MEAS_UNWEIGHTED keep the XDS/AIMLESS convention and are what to set beside XDS; the battery scorer records them. MULTIPLICITY stays a count. Effect (weighted / unweighted): lyso_x06da_ref 0.0454 / 0.0479, 8xtf 0.225 / 0.907, insu_I_x06da_ref REFRES 0.072 / 0.232. GPU and CPU paths agree to the last printed digit on lyso_x06da_ref. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CHANGELOG.md | 1 + docs/RUGNUX_REPORT.md | 8 +++ image_analysis/scale_merge/Merge.cpp | 13 ++++ image_analysis/scale_merge/Merge.h | 15 +++- .../scale_merge/RotationScaleMerge.cpp | 68 ++++++++++++++----- .../scale_merge/RotationScaleMergeGPU.cu | 43 +++++++++--- .../scale_merge/RotationScaleMergeGPU.h | 11 +-- rugnux/ResultReport.cpp | 4 ++ tests/MergeScaleTest.cpp | 21 ++++++ tools/battery/score.py | 5 +- 10 files changed, 158 insertions(+), 31 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 9504b1a74..0fd229f20 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,6 +7,7 @@ * Rugnux rejects single observations that Wilson statistics make implausible (typically a hot pixel, zinger or ice spot), including pairs and single measurements that the equivalents test cannot judge; the count is reported as `OBSERVATIONS_REJECTED_WILSON`. * Rugnux reports the X-ray bandwidth it measures from spot shapes (for multilayer and pink beams); the value is reported only and does not change processing. * Rugnux accepts a detector-modulation or absorption correction surface when held-out data support it on Fisher's z, and fits them in the order modulation, time, goniometer frame; this recovers corrections that were wrongly refused. +* Rugnux weights each observation in `R_MEAS` as the merge weights it, so weak frames kept in the merge no longer inflate it; the unweighted value is reported as `R_MEAS_UNWEIGHTED`. * Rugnux no longer stops per-frame scaling as unsettled because of a single frame that is about to be dropped as blank. * `rugnux --polarization` is documented as the polarization degree (XDS `FRACTION_OF_POLARIZATION` = (1 + p)/2); the default 0.99 suits undulators. diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index f1b68b8c9..6f0fbd676 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -51,6 +51,14 @@ not `-A` was given, but where no Bijvoet pair could be split in both hands — a `-A`, or too few pairs — the quantity does not exist and the key is absent; `COMPLETENESS=`, `MULTIPLICITY=`, `I_OVER_SIGMA=`, `R_MEAS=`, `CC_HALF=` and `WILSON_B=` follow the same rule. +**`R_MEAS=` weights each observation as the merge weights it** - by 1/sigma^2 under the error model - +with each reflection's weights normalised to their effective number, so where a reflection's +observations share one sigma it is the ordinary R_meas. A stretch the crystal barely diffracted over is +in the merge at the small weight its scaled-up counting error gives it, and is in `R_MEAS=` at that +weight too, rather than setting the number with its noise. **`R_MEAS_UNWEIGHTED=`** +(and `REFRES_R_MEAS_UNWEIGHTED=`) is the same statistic with every observation counted once, as XDS +and AIMLESS count it - the one to set beside theirs. `MULTIPLICITY=` stays a count of observations. + The last blocks before `END OF REPORT` are the authorship and the acknowledgement: who wrote rugnux, its licence (GPLv3 — free to use for academic institutions and commercial companies alike) and where releases are published, then the credit to the X-ray research community whose methods diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index dea9f923f..4c63aa15c 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -756,6 +756,19 @@ const char *SweepQualityReasonText(SweepQualityReason reason) { return "unknown"; } +bool WeightedRmeasTerms(double sum_v_absdev, double sum_v_I, double sum_v, double sum_v2, + double &num, double &den) { + if (!(sum_v > 0.0) || !(sum_v2 > 0.0)) + return false; + const double n_eff = sum_v * sum_v / sum_v2; + if (!(n_eff > 1.0)) + return false; + const double scale = n_eff / sum_v; + num = std::sqrt(n_eff / (n_eff - 1.0)) * scale * sum_v_absdev; + den = scale * sum_v_I; + return true; +} + const char *FrameDispositionCode(FrameDisposition disposition) { switch (disposition) { case FrameDisposition::Merged: return "merged"; diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index adb81c7ce..da17c9ce3 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -33,8 +33,13 @@ struct MergeStatisticsShell { double cc_ref = NAN; // Redundancy-independent merging R-factor (Diederichs & Karplus 1997), computed over the - // observations that enter the merge: R_meas = sum_hkl sqrt(n/(n-1)) sum_i|I_i-| / sum I_i. + // observations that enter the merge: R_meas = sum_hkl sqrt(n/(n-1)) sum_i|I_i-| / sum I_i. The + // rotation merge weights each observation as the merge weights it, 1/sigma^2 under the error model + // (WeightedRmeasTerms); with equal sigmas that is the formula as written. double r_meas = NAN; + // The same with every observation at weight one, as XDS and AIMLESS count it - for comparing with + // them. NaN where the two are the same by construction (the stills merge). + double r_meas_unweighted = NAN; // Anomalous signal-to-noise (XDS "SigAno" / mmCIF pdbx_absDiff_over_sigma_anomalous): // <|I(+)-I(-)|> / over acentric reflections measured in both hands. NaN when @@ -77,6 +82,14 @@ enum class FrameDisposition { const char *FrameDispositionCode(FrameDisposition disposition); // "merged", "downgraded", "rejected" +// One reflection's share of R_meas when its observations carry weights v_i: given sum v|I_i - |, +// sum v I_i, sum v and sum v^2, the numerator sqrt(n_eff/(n_eff-1)) sum v'|I_i - | and the +// denominator sum v' I_i, with the weights rescaled to v' = v n_eff / sum v so they add up to Kish's +// effective count n_eff = (sum v)^2 / sum v^2. Equal weights give the ordinary R_meas terms. False when +// the reflection has no second effective observation (n_eff <= 1), which then counts in neither sum. +bool WeightedRmeasTerms(double sum_v_absdev, double sum_v_I, double sum_v, double sum_v2, + double &num, double &den); + struct SweepQualityRange { // Inclusive, in processed-image ordinals - the numbering of _plot.txt and of every other // per-image array rugnux writes. With -s/--stride the source image is start + ordinal * stride. diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index e480d2aee..10a3580d3 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -4768,21 +4768,37 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool std::vector rmeas_num(n_shells, 0.0), rmeas_den(n_shells, 0.0); double rmeas_num_all = 0.0, rmeas_den_all = 0.0; + std::vector urmeas_num(n_shells, 0.0), urmeas_den(n_shells, 0.0); + double urmeas_num_all = 0.0, urmeas_den_all = 0.0; // R_meas and the observation counts behind the reported multiplicity. Report-only, and the last // full pass over the observations there is - so, like the correction surfaces, it is not made on a // merge whose numbers nobody reads. if (full_stats) { - // R_meas: re-walk the fulls (Mask = cell only; no ice / error-model), accumulate - // |I_i - | per reflection. - struct RmeasObs { double sum_abs_dev = 0, sum_I = 0; int n = 0, shell = -1; }; + // R_meas: re-walk the fulls (Mask = cell only; no ice), accumulate |I_i - | per + // reflection - each observation weighted as the merge weights it, v = 1/sigma^2 under the + // error model (corrected_sigma). The merge carries an observation from a frame the crystal + // barely diffracted on at the small weight its scaled-up counting error gives it; counted at + // full weight in R_meas, that frame's noise set the number while adding next to nothing to + // the intensities it describes. Where the error model's proportional term dominates - the + // strong reflections - the merge weights every frame alike, and so does this. Measured: a + // sweep keeping 115 weak frames (scale 0.1-0.2 of the run's) read the same CC1/2, + // and CC to the model as the same sweep with 84 frames rejected, and R_meas 1.3-1.5x higher + // in every shell. Per reflection the weights are normalised to Kish's effective count + // n_eff = (sum v)^2 / sum v^2, which is n where the sigmas are equal, and there this is the + // ordinary R_meas = sum sqrt(n/(n-1)) sum|I_i - | / sum I_i. + // The unweighted sums are kept beside them for r_meas_unweighted. + struct RmeasObs { double sum_abs_dev = 0, sum_I = 0, w_abs_dev = 0, w_I = 0, sum_v = 0, sum_v2 = 0; + int n = 0, shell = -1; }; std::vector rmeas(n_groups); bool did_gpu_rmeas = false; #ifdef JFJOCH_USE_CUDA if (use_gpu_merge) { // Per-group R_meas + usable count on the GPU; the shell is assigned per group (its fulls share d). - std::vector rabsdev(n_groups), rsumI(n_groups); + std::vector rabsdev(n_groups), rsumI(n_groups), rwabsdev(n_groups), rwsumI(n_groups), + rsumv(n_groups), rsumv2(n_groups); std::vector rn(n_groups), rnusable(n_groups); - gpu_->MergeRmeas(merged_I.data(), rabsdev.data(), rsumI.data(), rn.data(), rnusable.data()); + gpu_->MergeRmeas(merged_I.data(), rabsdev.data(), rsumI.data(), rwabsdev.data(), rwsumI.data(), + rsumv.data(), rsumv2.data(), rn.data(), rnusable.data()); for (int g = 0; g < n_groups; ++g) { if (rnusable[g] == 0 || acc[g].d < d_floor) continue; const auto shell = shells.GetShell(acc[g].d); @@ -4796,7 +4812,9 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool sa[*shell].total_obs += static_cast(acc[g].nh[0] + acc[g].nh[1]); if (std::isfinite(merged_I[g]) && rn[g] > 0) { auto &r = rmeas[g]; - r.sum_abs_dev = rabsdev[g]; r.sum_I = rsumI[g]; r.n = rn[g]; r.shell = *shell; + r.sum_abs_dev = rabsdev[g]; r.sum_I = rsumI[g]; r.w_abs_dev = rwabsdev[g]; r.w_I = rwsumI[g]; + r.sum_v = rsumv[g]; r.sum_v2 = rsumv2[g]; + r.n = rn[g]; r.shell = *shell; } } did_gpu_rmeas = true; @@ -4819,15 +4837,24 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool sa[*shell].total_obs++; if (std::isfinite(merged_I[o.group])) { auto &r = rmeas[o.group]; - r.sum_abs_dev += std::fabs(static_cast(I_corr) - merged_I[o.group]); - r.sum_I += I_corr; r.n++; r.shell = *shell; + const double sc = corrected_sigma(o, I_corr, sigma_corr); + const double v = 1.0 / (sc * sc); + const double dev = std::fabs(static_cast(I_corr) - merged_I[o.group]); + r.sum_abs_dev += dev; r.sum_I += I_corr; + r.w_abs_dev += v * dev; r.w_I += v * I_corr; r.sum_v += v; r.sum_v2 += v * v; + r.n++; r.shell = *shell; } } for (const auto &r : rmeas) { if (r.n < 2 || r.shell < 0 || r.shell >= n_shells) continue; + double num, den; + if (WeightedRmeasTerms(r.w_abs_dev, r.w_I, r.sum_v, r.sum_v2, num, den)) { + rmeas_num[r.shell] += num; rmeas_den[r.shell] += den; + rmeas_num_all += num; rmeas_den_all += den; + } const double factor = std::sqrt(static_cast(r.n) / (r.n - 1)); - rmeas_num[r.shell] += factor * r.sum_abs_dev; rmeas_den[r.shell] += r.sum_I; - rmeas_num_all += factor * r.sum_abs_dev; rmeas_den_all += r.sum_I; + urmeas_num[r.shell] += factor * r.sum_abs_dev; urmeas_den[r.shell] += r.sum_I; + urmeas_num_all += factor * r.sum_abs_dev; urmeas_den_all += r.sum_I; } } @@ -5000,7 +5027,9 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool // observations the merge kept (anom_cell carries that filter), so an over-rejecting run // looks better here for the same reason it does in the table above. if (ha_measured) { - std::vector row_absdev(row_mean.size(), 0.0), row_sumI(row_mean.size(), 0.0); + // Weighted as the merge weights each observation, as the table above is. + std::vector row_absdev(row_mean.size(), 0.0), row_sumI(row_mean.size(), 0.0), + row_sumv(row_mean.size(), 0.0), row_sumv2(row_mean.size(), 0.0); std::vector row_n(row_mean.size(), 0); for (size_t i = 0; i < fulls.size(); ++i) { const int cell = anom_cell[i]; @@ -5008,14 +5037,18 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool const int row = row_of_cell[cell]; if (!std::isfinite(row_mean[row])) continue; const float I_corr = fulls[i].I * fulls[i].corr; - row_absdev[row] += std::fabs(static_cast(I_corr) - row_mean[row]); - row_sumI[row] += I_corr; row_n[row]++; + const double sc = corrected_sigma(fulls[i], I_corr, fulls[i].sigma * fulls[i].corr); + const double v = 1.0 / (sc * sc); + row_absdev[row] += v * std::fabs(static_cast(I_corr) - row_mean[row]); + row_sumI[row] += v * I_corr; row_sumv[row] += v; row_sumv2[row] += v * v; row_n[row]++; } for (size_t row = 0; row < row_mean.size(); ++row) { if (row_n[row] < 2 || row_shell[row] < 0) continue; - const double factor = std::sqrt(static_cast(row_n[row]) / (row_n[row] - 1)); - ha[row_shell[row]].rmeas_num += factor * row_absdev[row]; - ha[row_shell[row]].rmeas_den += row_sumI[row]; + double num, den; + if (WeightedRmeasTerms(row_absdev[row], row_sumI[row], row_sumv[row], row_sumv2[row], num, den)) { + ha[row_shell[row]].rmeas_num += num; + ha[row_shell[row]].rmeas_den += den; + } } // The completeness denominator counted the same way: two per acentric, one per centric. if (reference_cell) { @@ -5041,6 +5074,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool ss.cc_half = sa[s].cc_half.GetCC(); ss.cc_ref = NAN; ss.r_meas = rmeas_den[s] > 0.0 ? rmeas_num[s] / rmeas_den[s] : NAN; + ss.r_meas_unweighted = urmeas_den[s] > 0.0 ? urmeas_num[s] / urmeas_den[s] : NAN; ss.abs_diff_over_sigma_anomalous = sig_den[s] > 0.0 ? sig_num[s] / sig_den[s] : NAN; ss.cc_anom = cc_anom_shell[s].GetCC(); } @@ -5056,6 +5090,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool overall.cc_half = cc_half_overall.GetCC(); overall.cc_ref = NAN; overall.r_meas = rmeas_den_all > 0.0 ? rmeas_num_all / rmeas_den_all : NAN; + overall.r_meas_unweighted = urmeas_den_all > 0.0 ? urmeas_num_all / urmeas_den_all : NAN; overall.abs_diff_over_sigma_anomalous = sig_den_all > 0.0 ? sig_num_all / sig_den_all : NAN; overall.cc_anom = cc_anom_overall.GetCC(); @@ -5075,6 +5110,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool ? ha[sh].sum_i_over_sigma / ha[sh].n_i_over_sigma : 0.0; hs.cc_half = ha[sh].cc_half.GetCC(); hs.r_meas = ha[sh].rmeas_den > 0.0 ? ha[sh].rmeas_num / ha[sh].rmeas_den : NAN; + hs.r_meas_unweighted = NAN; out.per_hand.overall.total_observations += ha[sh].total_obs; out.per_hand.overall.unique_reflections += ha[sh].unique; out.per_hand.overall.possible_unique_reflections += ha[sh].possible; diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.cu b/image_analysis/scale_merge/RotationScaleMergeGPU.cu index ccce57c9d..b4fcc4352 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.cu +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.cu @@ -450,7 +450,7 @@ namespace { double *a_swI, *a_sw, *a_swIh0, *a_swIh1, *a_swh0, *a_swh1, *a_swht0, *a_swht1, *a_d; int32_t *a_nh0, *a_nh1, *a_rejected; uint8_t *a_on_ice; - double *r_absdev, *r_sumI; + double *r_absdev, *r_sumI, *r_wabsdev, *r_wsumI, *r_sumv, *r_sumv2; int32_t *r_n, *r_nusable; uint8_t *rejected_obs; // per-full: set by MergeAccum (outlier-rejected), read by MergeRmeas }; @@ -588,8 +588,10 @@ namespace { } } - // One thread per group: R_meas accumulators (sum|I_corr - merged_I|, sum_I, n) + the count of - // observations this looser walk accepted. Mirrors MergeAndStats' R_meas re-walk (cell-only filter). + // One thread per group: R_meas accumulators (sum|I_corr - merged_I|, sum I_corr, the same weighted by + // v = the observation's merge weight 1/sigma^2 under the error model (as MergeAccumKernel weights it), + // sum v, sum v^2, n) + the count of observations this looser walk accepted. + // Mirrors MergeAndStats' R_meas re-walk (cell-only filter). // That count is NOT the per-shell total_observations - it is wider than the merge, so it would // over-report multiplicity; the host uses it only to skip empty groups. __global__ void MergeRmeasKernel(MergeParams p) { @@ -597,7 +599,7 @@ namespace { const int lo = p.gstart[g], hi = lo + p.gcount[g]; const double mI = p.merged_I[g]; const bool have = isfinite(mI); - double absdev = 0, sumI = 0; + double absdev = 0, sumI = 0, wabsdev = 0, wsumI = 0, sumv = 0, sumv2 = 0; int n = 0, nusable = 0; for (int q = lo; q < hi; ++q) { const int i = p.gperm[q]; @@ -607,10 +609,19 @@ namespace { ++nusable; if (have) { const double I_corr = double(p.I[i]) * p.corr[i]; - absdev += fabs(I_corr - mI); sumI += I_corr; ++n; + const float sigma_raw = p.sigma[i] * p.corr[i]; + const double sc = p.error_model_active + ? ModelSigma(i, p, isfinite(p.em_mean[g]) ? p.em_mean[g] : I_corr, sigma_raw) + : sigma_raw; + const double v = 1.0 / (sc * sc); + const double dev = fabs(I_corr - mI); + absdev += dev; sumI += I_corr; + wabsdev += v * dev; wsumI += v * I_corr; sumv += v; sumv2 += v * v; ++n; } } - p.r_absdev[g] = absdev; p.r_sumI[g] = sumI; p.r_n[g] = n; p.r_nusable[g] = nusable; + p.r_absdev[g] = absdev; p.r_sumI[g] = sumI; p.r_wabsdev[g] = wabsdev; p.r_wsumI[g] = wsumI; + p.r_sumv[g] = sumv; p.r_sumv2[g] = sumv2; + p.r_n[g] = n; p.r_nusable[g] = nusable; } } @@ -670,10 +681,12 @@ struct RotationScaleMergeGPU::Impl { CudaDevicePtr a_on_ice; // per group: any contributing full on an ice ring CudaDevicePtr reject_median; CudaDevicePtr reject_var_add; // the pooled test's widening - CudaDevicePtr merged_I, r_absdev, r_sumI; // R_meas per group (merged_I uploaded) + CudaDevicePtr merged_I, r_absdev, r_sumI, r_wabsdev, r_wsumI, r_sumv, r_sumv2; // R_meas per group (merged_I uploaded) CudaDevicePtr r_n, r_nusable; int merge_for_search = 0; // filter context for one MergeAndStats call double merge_min_part = 0.0; + double merge_em_a = 1.0, merge_em_b = 0.0; // the error model of the last MergeAccum, for MergeRmeas + int merge_em_active = 0; // combine: extra per-obs inputs + the one-time raw-hkl run layout CudaDevicePtr bkg, var_bkg, image_number, d_obs, px_obs, py_obs; @@ -954,6 +967,7 @@ void RotationScaleMergeGPU::MergeAccum(double error_model_a, double error_model_ Upload(d.m_half, half, d.n_fulls); Upload(d.cc_factor, frame_cc_factor, d.n_frames); + d.merge_em_a = error_model_a; d.merge_em_b = error_model_b; d.merge_em_active = error_model_active ? 1 : 0; MergeParams p{}; p.n_groups = ng; p.min_partiality = d.merge_min_part; p.for_search = d.merge_for_search; @@ -998,13 +1012,16 @@ void RotationScaleMergeGPU::MergeAccumRange(int g0, int n, double *swI, double * dl(nh0, d.a_nh0); dl(nh1, d.a_nh1); dl(rejected, d.a_rejected); dl(on_ice_out, d.a_on_ice); } -void RotationScaleMergeGPU::MergeRmeas(const double *merged_I, double *absdev, double *sumI, - int32_t *n, int32_t *nusable) { +void RotationScaleMergeGPU::MergeRmeas(const double *merged_I, double *absdev, double *sumI, double *wabsdev, + double *wsumI, double *sumv, double *sumv2, int32_t *n, + int32_t *nusable) { DeviceGuard guard(impl_->device, impl_->available); auto &d = *impl_; const int ng = d.n_groups; Upload(d.merged_I, merged_I, ng); d.r_absdev = CudaDevicePtr(std::max(1, ng)); d.r_sumI = CudaDevicePtr(std::max(1, ng)); + d.r_wabsdev = CudaDevicePtr(std::max(1, ng)); d.r_wsumI = CudaDevicePtr(std::max(1, ng)); + d.r_sumv = CudaDevicePtr(std::max(1, ng)); d.r_sumv2 = CudaDevicePtr(std::max(1, ng)); d.r_n = CudaDevicePtr(std::max(1, ng)); d.r_nusable = CudaDevicePtr(std::max(1, ng)); MergeParams p{}; @@ -1014,6 +1031,10 @@ void RotationScaleMergeGPU::MergeRmeas(const double *merged_I, double *absdev, d p.gperm = d.f_gperm.get(); p.gstart = d.f_gstart.get(); p.gcount = d.f_gcount.get(); p.merged_I = d.merged_I.get(); p.r_absdev = d.r_absdev.get(); p.r_sumI = d.r_sumI.get(); p.r_n = d.r_n.get(); p.r_nusable = d.r_nusable.get(); + p.r_wabsdev = d.r_wabsdev.get(); p.r_wsumI = d.r_wsumI.get(); + p.r_sumv = d.r_sumv.get(); p.r_sumv2 = d.r_sumv2.get(); + p.error_model_a = d.merge_em_a; p.error_model_b = d.merge_em_b; p.error_model_active = d.merge_em_active; + p.em_mean = d.m_em_mean.get(); p.var_bkg = d.f_var_bkg.get(); p.var_per_I = d.f_var_per_I.get(); p.rejected_obs = d.m_rejected.get(); const int grp_blocks = std::min(65535, (ng + BLK - 1) / BLK); @@ -1022,6 +1043,10 @@ void RotationScaleMergeGPU::MergeRmeas(const double *merged_I, double *absdev, d CudaCheck(cudaDeviceSynchronize(), "merge rmeas sync"); CudaCheck(cudaMemcpy(absdev, d.r_absdev.get(), size_t(ng) * sizeof(double), cudaMemcpyDeviceToHost), "dl absdev"); CudaCheck(cudaMemcpy(sumI, d.r_sumI.get(), size_t(ng) * sizeof(double), cudaMemcpyDeviceToHost), "dl sumI"); + CudaCheck(cudaMemcpy(wabsdev, d.r_wabsdev.get(), size_t(ng) * sizeof(double), cudaMemcpyDeviceToHost), "dl wabsdev"); + CudaCheck(cudaMemcpy(wsumI, d.r_wsumI.get(), size_t(ng) * sizeof(double), cudaMemcpyDeviceToHost), "dl wsumI"); + CudaCheck(cudaMemcpy(sumv, d.r_sumv.get(), size_t(ng) * sizeof(double), cudaMemcpyDeviceToHost), "dl sumv"); + CudaCheck(cudaMemcpy(sumv2, d.r_sumv2.get(), size_t(ng) * sizeof(double), cudaMemcpyDeviceToHost), "dl sumv2"); CudaCheck(cudaMemcpy(n, d.r_n.get(), size_t(ng) * sizeof(int32_t), cudaMemcpyDeviceToHost), "dl rn"); CudaCheck(cudaMemcpy(nusable, d.r_nusable.get(), size_t(ng) * sizeof(int32_t), cudaMemcpyDeviceToHost), "dl rnusable"); } diff --git a/image_analysis/scale_merge/RotationScaleMergeGPU.h b/image_analysis/scale_merge/RotationScaleMergeGPU.h index 4aad3fce8..730a79add 100644 --- a/image_analysis/scale_merge/RotationScaleMergeGPU.h +++ b/image_analysis/scale_merge/RotationScaleMergeGPU.h @@ -114,10 +114,13 @@ public: int32_t *nh0, int32_t *nh1, double *d_out, int32_t *rejected, uint8_t *on_ice_out); - // Per-group R_meas accumulators (sum|I_corr-merged_I|, sum_I, n, and the count this looser walk - // accepted - which the host uses only to skip empty groups, NOT as the per-shell - // total_observations); merged_I is uploaded. All arrays length n_groups. - void MergeRmeas(const double *merged_I, double *absdev, double *sumI, int32_t *n, int32_t *nusable); + // Per-group R_meas accumulators: sum|I_corr-merged_I| and sum I, the same with each observation + // weighted by its merge weight v = 1/sigma^2 under the error model of the last MergeAccum, + // sum v, sum v^2, n, and the count this looser walk accepted - which the host uses only to skip + // empty groups, NOT as the per-shell total_observations; merged_I is uploaded. All arrays length + // n_groups. + void MergeRmeas(const double *merged_I, double *absdev, double *sumI, double *wabsdev, double *wsumI, + double *sumv, double *sumv2, int32_t *n, int32_t *nusable); // Post-smooth per-frame diagnostic CC: recompute the group means from the resident (smoothed) corr // and the Pearson CC of each frame's I*corr vs its group mean, downloading only the per-frame cc / diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 5c2b47175..246f889e8 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -803,6 +803,8 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, Add(s, KeyReal("I_OVER_SIGMA", o.mean_i_over_sigma, "{:.2f}")); if (std::isfinite(o.r_meas)) Add(s, KeyReal("R_MEAS", o.r_meas, "{:.4f}")); + if (std::isfinite(o.r_meas_unweighted)) + Add(s, KeyReal("R_MEAS_UNWEIGHTED", o.r_meas_unweighted, "{:.4f}")); if (std::isfinite(o.cc_half)) Add(s, KeyReal("CC_HALF", o.cc_half, "{:.4f}")); if (std::isfinite(o.abs_diff_over_sigma_anomalous)) @@ -937,6 +939,8 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, } if (std::isfinite(ro.r_meas)) Add(s, KeyReal("REFRES_R_MEAS", ro.r_meas, "{:.4f}")); + if (std::isfinite(ro.r_meas_unweighted)) + Add(s, KeyReal("REFRES_R_MEAS_UNWEIGHTED", ro.r_meas_unweighted, "{:.4f}")); if (std::isfinite(ro.cc_half)) Add(s, KeyReal("REFRES_CC_HALF", ro.cc_half, "{:.4f}")); if (std::isfinite(ro.abs_diff_over_sigma_anomalous)) diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index 7be51a64e..408bce534 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -388,6 +388,27 @@ TEST_CASE("CCHalfFrameFactors_DeadStretchDoesNotSetTheTypicalScale") { CHECK(factor[100] == Catch::Approx(1.0).epsilon(1e-3)); } +// Equal weights give the ordinary R_meas terms, whatever their common value. +TEST_CASE("WeightedRmeas_EqualWeightsAreTheOrdinaryRmeas") { + // Three observations 9, 10, 11 of a reflection with = 10: sum|dev| = 2, sum I = 30. + double num, den; + REQUIRE(WeightedRmeasTerms(0.5 * 2.0, 0.5 * 30.0, 0.5 * 3, 0.25 * 3, num, den)); + CHECK(num == Catch::Approx(std::sqrt(3.0 / 2.0) * 2.0)); + CHECK(den == Catch::Approx(30.0)); +} + +// An observation that carries almost no information counts for almost nothing: the terms tend to those +// of the reflection without it, and a reflection left with one effective observation has none. +TEST_CASE("WeightedRmeas_NegligibleWeightDropsOut") { + // 9 and 11 at weight 1, = 10; a third at 40 with weight 1e-6. + const double v = 1e-6; + double num, den; + REQUIRE(WeightedRmeasTerms(2.0 + v * 30.0, 20.0 + v * 40.0, 2.0 + v, 2.0 + v * v, num, den)); + CHECK(num == Catch::Approx(std::sqrt(2.0) * 2.0).epsilon(1e-4)); + CHECK(den == Catch::Approx(20.0).epsilon(1e-4)); + CHECK_FALSE(WeightedRmeasTerms(0.0, 10.0, 1.0, 1.0, num, den)); +} + // A fall-off region a logistic cannot follow: CC1/2 drops through the target and comes straight back // up. The fitted crossing is then an extrapolation far past the bins it was made over, and reading // the cut off it writes the data deep into the noise; the crossing the bins themselves show is where diff --git a/tools/battery/score.py b/tools/battery/score.py index c44b3bb40..bc9d4d1fc 100644 --- a/tools/battery/score.py +++ b/tools/battery/score.py @@ -52,7 +52,7 @@ def read_report(path): # REFRES_* keys of the report read as numbers, stored lower-case under the same name REFRES_KEYS = ("refres_shells_past_limit", "refres_unique_reflections", "refres_completeness", "refres_multiplicity", "refres_i_over_sigma", "refres_r_meas", "refres_cc_half", - "refres_isa") + "refres_isa", "refres_r_meas_unweighted") def fnum(s): @@ -186,6 +186,9 @@ def judge(entry, rep, run_note): "d_min": None, "d_min_ref": ref.get("dmin"), "d_min_ref_rule": ref.get("dmin_rule"), "d_min_xds": ref.get("dmin_xds"), "res_gain_pct": None, "r_meas": fnum(rep.get("R_MEAS")), "cc_half": fnum(rep.get("CC_HALF")), + # R_MEAS weights each observation by its frame's information, as the merge does; the + # unweighted one is how XDS counts it (absent from reports before the weighting) + "r_meas_unweighted": fnum(rep.get("R_MEAS_UNWEIGHTED")), "isa": fnum(rep.get("ISA")), "completeness": fnum(rep.get("COMPLETENESS")), "multiplicity": fnum(rep.get("MULTIPLICITY")), "i_over_sigma": fnum(rep.get("I_OVER_SIGMA")), "indexing_rate": fnum(rep.get("INDEXING_RATE")), "images": fnum(rep.get("IMAGES_PROCESSED")), From 040810aa3d0d8ec91ef790ba37c2568f272ab3a7 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 25 Sep 2026 05:05:37 +0200 Subject: [PATCH 18/21] Beam centre check: decide a harmonic DISAGREE in both directions Where the file's and the measured beam centre index lattices whose primitive volumes differ by an integer factor 2-4, the pooled validation evidence decided only when the measured centre held the LARGER cell. The other direction was left to the second pass on the assumption that it drops to the smaller cell by itself. It does so only when the pass-2 indexer's free beam refinement walks the whole centre error, and that rescue is chaotic: on 7ris a 0.1 % change in the pass-1 distance switched off a 6.5 px walk, the doubled c axis (380 A) won and the run failed. Both directions are now decided on the same measurement: pooled validation spots on each lattice against its own wrong-spindle null (ValidationEvidencePrefers). No new threshold. 7ris: 44.3 % at the file's centre vs 83.5 % at the measured one; the run adopts the measured centre and the 190 A cell (R_free 0.81 -> 0.20, 1.52 A). 8pqd (3x, measured centre holds the true cell): 16.6 % vs 21.0 %, adopted in pass 1 instead of being recovered by pass 2. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- rugnux/Rugnux.cpp | 54 +++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 6289bfda4..da5945c80 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -4357,44 +4357,44 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b // Deliberately no decision on the frame counts. They are the one statistic // that must not choose: on the measured case the harmonic indexed MORE. // - // One direction is decided, and only one: where the MEASURED centre holds - // the LARGER of an integer-related pair. That is the case nothing else in - // the run repairs - the frame count leans to the sub-cell, and the second - // pass refuses a lattice larger than pass 1's (the supercell guard below), - // so a run that starts on the sub-cell stays on it. The opposite direction - // needs no help here: the second pass is free to drop to the smaller cell - // and does, on every corpus crystal that reaches this branch that way. + // Decided in either direction on the pooled excess over chance, the same + // measurement the failed-pass branch above adopts a centre on. Each lattice is + // scored against its own wrong-spindle null, so the larger cell is not credited + // for the spots a denser lattice catches by accident, and the smaller one is + // not credited for the spots it misses. // - // Decided on the pooled excess over chance, the same measurement the - // failed-pass branch above adopts a centre on. Each lattice is scored - // against its own wrong-spindle null, so the larger cell is not credited - // for the spots a denser lattice catches by accident. + // Both directions need it. Where the measured centre holds the LARGER cell, + // nothing else in the run repairs it - the second pass refuses a lattice larger + // than pass 1's (the supercell guard below). Where it holds the SMALLER one, + // the second pass can drop to it only if its own beam refinement happens to + // walk the whole centre error, and that rescue is chaotic: measured on one + // crystal, a 0.1 % change in the pass-1 distance switched off a 6.5 px walk and + // the run kept the doubled axis. const double nearest_n = std::round(ratio); const bool integer_harmonic = nearest_n >= 2.0 && nearest_n <= 4.0 && std::abs(ratio - nearest_n) < 0.15; bool adopted_measured = false; - if (integer_harmonic && alt.vol > best.vol) { + if (integer_harmonic) { const PooledEvidence at_header = pooled_evidence(*indexer, *best.result); try_beam_center(measured_x, measured_y); const PooledEvidence at_measured = pooled_evidence(*indexer, *alt.result); adopted_measured = beats_chance(at_measured) && ValidationEvidencePrefers(at_header, at_measured); + logger.Info("Beam centre check: pooled validation spots on the lattice " + "{:.1f}% at the file's centre and {:.1f}% at the measured " + "(chance {:.1f}% and {:.1f}%)", + 100.0 * at_header.on_lattice / std::max(1, at_header.spots), + 100.0 * at_measured.on_lattice + / std::max(1, at_measured.spots), + 100.0 * at_header.by_chance / std::max(1, at_header.spots), + 100.0 * at_measured.by_chance + / std::max(1, at_measured.spots)); if (adopted_measured) { - logger.Warning("Beam centre check: the larger cell is the measured " - "centre's, and it is the one the spots are on - " - "{:.1f}% of the pooled validation spots lie on it " - "against {:.1f}% on the file's cell (chance {:.1f}% " - "and {:.1f}%) - so the {:.2f}x is a real axis and not " - "a harmonic, and the run adopts the measured centre", - 100.0 * at_measured.on_lattice - / std::max(1, at_measured.spots), - 100.0 * at_header.on_lattice - / std::max(1, at_header.spots), - 100.0 * at_measured.by_chance - / std::max(1, at_measured.spots), - 100.0 * at_header.by_chance - / std::max(1, at_header.spots), - ratio); + logger.Warning("Beam centre check: the spots are on the {} cell of the " + "measured centre, not on the file's - so the {:.2f}x is " + "the file's centre error and not the crystal, and the " + "run adopts the measured centre", + alt.vol > best.vol ? "larger" : "smaller", ratio); best = alt; } else { restore_beam_center(header_x, header_y); From 96b681e50d5cf2d14a16fc49847532cb1d3f6a60 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 25 Sep 2026 06:02:50 +0200 Subject: [PATCH 19/21] PostRefine: commit the geometry when the held-out residual falls by more than its noise The joint crystal + detector fit was committed only when the held-out residual fell by at least a fixed 2 %. On a crystal whose geometry is already right that bound cuts through the noise: two builds read 1.9 % and 2.05 % for the same 0.03 px / 0.01 % move (lyso_x06da_half_image), one committed and the other did not, and everything downstream (smoothing window, stretch segmentation) followed the coin. The bar is now the residual's own noise - the standard errors of the two held-out means combined - which is the bar a round of the geometry walk already has to clear (HeldOutResidualFell). The log line carries the joint residual and that noise. Noise on the battery is 2-7 % of the residual, so the gate is about as strict as before but no longer at a fixed edge. Checked on 57 open/in-house sets and 11 private ones against the pool battery. The verdict changed on 8 open/in-house sets besides the passes that follow the 7ris/8pqd lattice change: half_image, 6qaj, lalanine, 8xtf, 9ig7, 8sqt (commit -> reject of a 1-step move) and myob_x06da_split, 6cdl (reject -> commit). Merge statistics are equal to the last digit everywhere except half_image (REFRES R_meas 1.16 -> 0.95, CC1/2 0.79 -> 0.86), 9ig7 (5 fewer rejected observations) and 8xtf, whose stretch disposition followed the coin: multiplicity 20.8 -> 15.6, R_meas 0.91 -> 0.60, R_free 0.193 -> 0.189, REFMAC R_free 0.179 -> 0.176. Private arm unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- image_analysis/geom_refinement/PostRefine.cpp | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/image_analysis/geom_refinement/PostRefine.cpp b/image_analysis/geom_refinement/PostRefine.cpp index c713826ea..a4cd73d87 100644 --- a/image_analysis/geom_refinement/PostRefine.cpp +++ b/image_analysis/geom_refinement/PostRefine.cpp @@ -799,6 +799,20 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, return nullptr; }; + // How much the held-out residual has to fall for the fit to be worth committing: more + // than its own noise, the standard errors of the two means combined - the same bar a round + // of the geometry walk has to clear (HeldOutResidualFell). A fixed fraction stood here + // (2 %), and on a crystal whose geometry was already right it cut through the noise: two + // builds read 1.9 % and 2.05 % for the same 0.03 px move, and one committed and the other + // did not. + double cv_ref_se = NAN; + { + double pos_tmp = 0.0, exc_tmp = 0.0; + if (convJ) + joint_cost(VAL, bm_f, ds_f, rv_f, q0_f, q1_f, q2_f, pos_tmp, exc_tmp, &cv_ref_se); + } + const double cv_noise = std::hypot(cv_nom_se, cv_ref_se); + // The two residual families are asked separately as well as together. Pooled, the // positional values outnumber the excitation ones about three to one where both caps // saturate, and the excitation residual is the only one that identifies the cell SCALE - @@ -813,7 +827,8 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, // a weak 7 A crystal whose fit asked for another 1 % at every one of five steps // and 4.3 % in all, against the 1.2 % its cell is actually out by. : steps >= MAX_STEPS ? "the fit used every step it was given and never settled" - : !(cv_ref < 0.98 * cv_nom) ? "the held-out residual did not improve enough" + : !(std::isfinite(cv_noise) && cv_ref < cv_nom - cv_noise) + ? "the held-out residual did not fall by more than its noise" : !(pos_ref < pos_nom) ? "the held-out positional residual did not improve" : !(exc_ref < exc_nom) ? "the held-out excitation residual did not improve" : out_of_bounds(bm_f, q2_f); @@ -876,13 +891,14 @@ PostRefineResult PostRefineRotationGeometry(PostRefineObservations observations, logger.Info("Post-refine GEOM (joint crystal + detector): dist {:.3f} -> {:.3f} mm, beam " "({:.2f},{:.2f}) -> ({:.2f},{:.2f}), cell {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} " "{:.2f} -> {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} {:.2f}, held-out positional " - "{:.3e} -> {:.3e}, excitation {:.3e} -> {:.3e} => {}", + "{:.3e} -> {:.3e}, excitation {:.3e} -> {:.3e}, joint {:.3e} -> {:.3e} +- {:.1e} " + "=> {}", dist0, ds_log[0], beam_x0, beam_y0, bm_log[0], bm_log[1], len_nom[0], len_nom[1], len_nom[2], ang_nom[0] * 180.0 / PI, ang_nom[1] * 180.0 / PI, ang_nom[2] * 180.0 / PI, len_fit[0], len_fit[1], len_fit[2], ang_fit[0] * 180.0 / PI, ang_fit[1] * 180.0 / PI, ang_fit[2] * 180.0 / PI, - pos_nom, pos_ref, exc_nom, exc_ref, verdict); + pos_nom, pos_ref, exc_nom, exc_ref, cv_nom, cv_ref, cv_noise, verdict); if (!commit) result.refused_reason = fmt::format( "{} - it wanted {:.3f} mm and cell {:.3f} {:.3f} {:.3f}", refused, From 2b476336aec914b6a3f6f5a8739bc04f2623c36a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 25 Sep 2026 06:02:50 +0200 Subject: [PATCH 20/21] rugnux: report-only supercell probe - the eight parity classes of the doubled cell Every lattice decision counts spots or frames, and the sub-lattice's strong reflections win every count, so a crystal whose cell is doubled by a weak superstructure class (9min, 6z9g) is adopted at the half cell. This asks the question in intensities instead. On 60 frames spread over the sweep, after each frame's own integration, the 2a x 2b x 2c supercell of its primitive lattice is predicted to 3 A with the frame's own refined orientation and geometry and integrated on the same engine; the reflections are summed per parity class in two shells (20-5, 5-3 A), with a fit of intensity against partiality (I = a + b p) that separates what rocks like a Bragg reflection from what sits at the node whatever the rocking. Only the tested frames are predicted and nothing is retained, so memory is bounded (the previous prototype predicted the whole run through the merge and ran out of GPU memory). The probe's integrations are kept out of the engine's own counts, which the two-pass stencil guard reads. Results are bit-identical with and without it. REPORT ONLY - it decides nothing, because on the battery it does not yet separate a weak real class from what sits at the half-integer nodes of crystals whose cell is right. Real classes: 6z9g class 101 at 24 % of the lattice's intensity (29 % rocking), 9min 100 at 19 % (4 % rocking - its real class does not rock like the lattice either). On correct cells the largest classes reach 9-12 % raw (7n2s, 9i0a, 7os3) and 3-4 % rocking (7dkp, 7os3), and 7mzt reads 40 % / 19 % on a class the deposition does not have. The log line is the population a decision has to be calibrated on. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- image_analysis/IndexAndRefine.cpp | 80 +++++++++++++++++++++++- image_analysis/IndexAndRefine.h | 53 +++++++++++++++- image_analysis/MXAnalysisWithoutFPGA.cpp | 26 +++++++- image_analysis/MXAnalysisWithoutFPGA.h | 2 + rugnux/Rugnux.cpp | 46 ++++++++++++++ tests/IndexingUnitTest.cpp | 28 +++++++++ 6 files changed, 227 insertions(+), 8 deletions(-) diff --git a/image_analysis/IndexAndRefine.cpp b/image_analysis/IndexAndRefine.cpp index b21475948..f4fca96c5 100644 --- a/image_analysis/IndexAndRefine.cpp +++ b/image_analysis/IndexAndRefine.cpp @@ -484,7 +484,8 @@ void IndexAndRefine::QuickPredictAndIntegrate(DataMessage &msg, const SpotFindingSettings &spot_finding_settings, BraggPrediction &prediction, const BraggIntegrateFn &integrate, - const IndexAndRefine::IndexingOutcome &outcome) { + const IndexAndRefine::IndexingOutcome &outcome, + const BraggIntegrateFn &probe_integrate) { if (!outcome.lattice_candidate) return; @@ -611,6 +612,78 @@ void IndexAndRefine::QuickPredictAndIntegrate(DataMessage &msg, const std::unique_lock ul(reflections_mutex); integration_outcome[msg.number] = std::move(i_outcome); } + + if (probe_integrate && std::binary_search(supercell_probe_frames_.begin(), supercell_probe_frames_.end(), + msg.number)) + ProbeSupercellFrame(msg, prediction, probe_integrate, outcome, settings_prediction); +} + +SupercellProbeFit FitSupercellProbe(const SupercellProbeClass &c) { + SupercellProbeFit f; + if (c.n < 3) + return f; + const double n = static_cast(c.n); + const double mp = c.sum_p / n, mi = c.sum_i / n; + const double spp = c.sum_pp - n * mp * mp, spi = c.sum_pi - n * mp * mi, sii = c.sum_ii - n * mi * mi; + f.mean_i = mi; + f.mean_p = mp; + f.mean_i_over_sigma = c.sum_i_over_sigma / n; + if (spp > 0.0) { + f.b = spi / spp; + f.b_se = std::sqrt(std::max(0.0, sii - f.b * spi) / (n - 2.0) / spp); + } + f.a = mi - f.b * mp; + return f; +} + +void IndexAndRefine::ProbeSupercell(std::vector image_numbers) { + std::sort(image_numbers.begin(), image_numbers.end()); + supercell_probe_frames_ = std::move(image_numbers); + supercell_probe_ = {}; +} + +SupercellProbe IndexAndRefine::GetSupercellProbe() { + const std::unique_lock ul(supercell_probe_mutex_); + return supercell_probe_; +} + +void IndexAndRefine::ProbeSupercellFrame(const DataMessage &msg, BraggPrediction &prediction, + const BraggIntegrateFn &integrate, const IndexingOutcome &outcome, + const BraggPredictionSettings &settings) { + // The band the classes are read over. A superstructure class is weakest at low resolution (the two + // halves of the cell differ in detail, not in position), so the band reaches to 3 A; 20 A keeps the + // strongest, most overlap-prone low-order reflections of the lattice out of the denominator. The + // prediction stops at 3 A as well, which is what keeps eight times the nodes of a frame affordable. + constexpr float BAND_LOW_RES_A = 20.0f; + constexpr float BAND_HIGH_RES_A = 3.0f; + + const CrystalLattice prim = outcome.lattice_candidate->ToPrimitive(outcome.symmetry.centering); + const CrystalLattice super(prim.Vec0() * 2.0f, prim.Vec1() * 2.0f, prim.Vec2() * 2.0f); + BraggPredictionSettings s = settings; + s.centering = 'P'; + s.high_res_A = std::max(settings.high_res_A, BAND_HIGH_RES_A); + ApplyPredictionRange(s, experiment, super); + const auto nrefl = prediction.Calc(outcome.experiment, super, s); + const std::vector integrated = integrate(prediction.GetReflections(), nrefl, msg.number); + + SupercellProbe frame{}; + for (const auto &r : integrated) { + if (!(r.d > BAND_HIGH_RES_A && r.d <= BAND_LOW_RES_A) || !(r.sigma > 0.0f) || !std::isfinite(r.I) + || !std::isfinite(r.partiality)) + continue; + auto &c = frame[4 * (r.h & 1) + 2 * (r.k & 1) + (r.l & 1)][r.d > 5.0f ? 0 : 1]; + c.n++; + c.sum_i += r.I; + c.sum_i_over_sigma += r.I / r.sigma; + c.sum_p += r.partiality; + c.sum_pp += r.partiality * r.partiality; + c.sum_pi += r.partiality * r.I; + c.sum_ii += static_cast(r.I) * r.I; + } + const std::unique_lock ul(supercell_probe_mutex_); + for (int i = 0; i < 8; i++) + for (int j = 0; j < 2; j++) + supercell_probe_[i][j] += frame[i][j]; } std::optional @@ -652,10 +725,11 @@ IndexAndRefine::DetermineRefineAnalyze(DataMessage &msg, const SpotFindingSettin void IndexAndRefine::ProcessImage(DataMessage &msg, const SpotFindingSettings &spot_finding_settings, BraggPrediction &prediction, - const BraggIntegrateFn &integrate) { + const BraggIntegrateFn &integrate, + const BraggIntegrateFn &probe_integrate) { auto outcome = DetermineRefineAnalyze(msg, spot_finding_settings); if (outcome && spot_finding_settings.quick_integration) - QuickPredictAndIntegrate(msg, spot_finding_settings, prediction, integrate, *outcome); + QuickPredictAndIntegrate(msg, spot_finding_settings, prediction, integrate, *outcome, probe_integrate); } bool IndexAndRefine::IndexFrameOnly(DataMessage &msg, const SpotFindingSettings &spot_finding_settings) { diff --git a/image_analysis/IndexAndRefine.h b/image_analysis/IndexAndRefine.h index 56904f78e..7d5537fe0 100644 --- a/image_analysis/IndexAndRefine.h +++ b/image_analysis/IndexAndRefine.h @@ -3,6 +3,7 @@ #pragma once +#include #include #include #include @@ -29,6 +30,33 @@ using BraggIntegrateFn = std::function( const std::vector &predicted, size_t npredicted, int64_t image_number)>; +// What the supercell probe (IndexAndRefine::ProbeSupercell) reads for one of the eight parity classes +// of the 2a x 2b x 2c primitive supercell in one resolution shell: class 4*(h&1) + 2*(k&1) + (l&1) in +// supercell indices, so class 0 is the adopted lattice and each of the other seven is one index-2 +// superstructure. The partiality sums are for the fit I = a + b p over the class: b is the part of +// it that rocks like a Bragg reflection, a what sits at its position whatever the rocking. +struct SupercellProbeClass { + int64_t n = 0; + double sum_i = 0.0, sum_i_over_sigma = 0.0; + double sum_p = 0.0, sum_pp = 0.0, sum_pi = 0.0, sum_ii = 0.0; + SupercellProbeClass &operator+=(const SupercellProbeClass &o) { + n += o.n; + sum_i += o.sum_i; sum_i_over_sigma += o.sum_i_over_sigma; + sum_p += o.sum_p; sum_pp += o.sum_pp; sum_pi += o.sum_pi; sum_ii += o.sum_ii; + return *this; + } +}; +// [class][shell], the shells 20-5 A and 5-3 A. +using SupercellProbe = std::array, 8>; + +// One parity class of the supercell probe, fitted I = a + b p over partiality p: b (with its standard +// error) is what rocks like a Bragg reflection, a what does not. +struct SupercellProbeFit { + double mean_i = 0.0, mean_i_over_sigma = 0.0, mean_p = 0.0; + double a = 0.0, b = 0.0, b_se = 0.0; +}; +SupercellProbeFit FitSupercellProbe(const SupercellProbeClass &c); + class IndexAndRefine { // When false, the current image's result is still returned via the outgoing message, but the // whole-run integration_outcome vector is not retained (viewer live/interactive use, which never @@ -92,6 +120,14 @@ class IndexAndRefine { std::vector scale_cc; std::vector > unit_cells; + // Supercell probe: the image numbers it reads, and what it has read (see ProbeSupercell). + std::vector supercell_probe_frames_; + SupercellProbe supercell_probe_{}; + std::mutex supercell_probe_mutex_; + void ProbeSupercellFrame(const DataMessage &msg, BraggPrediction &prediction, + const BraggIntegrateFn &integrate, const IndexingOutcome &outcome, + const BraggPredictionSettings &settings); + IndexingOutcome DetermineLatticeAndSymmetryRotation(DataMessage &msg); IndexingOutcome DetermineLatticeAndSymmetry(DataMessage &msg); // Shared indexing path: determine the lattice/symmetry, refine geometry, and run AnalyzeIndexing. @@ -106,7 +142,8 @@ class IndexAndRefine { const SpotFindingSettings &spot_finding_settings, BraggPrediction &prediction, const BraggIntegrateFn &integrate, - const IndexingOutcome &outcome); + const IndexingOutcome &outcome, + const BraggIntegrateFn &probe_integrate); std::unique_ptr reindex_resolver; void ScaleImage(DataMessage &msg, IntegrationOutcome& outcome); @@ -138,8 +175,20 @@ public: // Returns whether the frame indexed (a lattice was found and refined). Integration, when it runs, // is a further step gated on quick_integration. + // probe_integrate integrates the supercell probe's reflections (ProbeSupercell); a caller that + // gives none takes no part in the probe. void ProcessImage(DataMessage &msg, const SpotFindingSettings &settings, - BraggPrediction &prediction, const BraggIntegrateFn &integrate); + BraggPrediction &prediction, const BraggIntegrateFn &integrate, + const BraggIntegrateFn &probe_integrate = {}); + + // Does the lattice each frame is refined on have an index-2 superstructure? On the given frames, + // after the frame's own integration, predict the 2a x 2b x 2c supercell of its PRIMITIVE lattice + // (the per-frame refinement and geometry unchanged), integrate it out to 3 A, and sum the eight + // parity classes in two shells over 20-3 A (SupercellProbe). Class 0 is the lattice itself; the + // other seven are the seven index-2 superlattices, at most one of which a single doubling fills. + // The probe's integrations are the caller's to keep out of anything else it counts. + void ProbeSupercell(std::vector image_numbers); + [[nodiscard]] SupercellProbe GetSupercellProbe(); // Index a single frame (no integration) with the current forced rotation lattice; used to score // first-pass sampling schemes on the real per-image path. Returns whether the frame indexed. bool IndexFrameOnly(DataMessage &msg, const SpotFindingSettings &settings); diff --git a/image_analysis/MXAnalysisWithoutFPGA.cpp b/image_analysis/MXAnalysisWithoutFPGA.cpp index 93a3d736d..538201c15 100644 --- a/image_analysis/MXAnalysisWithoutFPGA.cpp +++ b/image_analysis/MXAnalysisWithoutFPGA.cpp @@ -165,6 +165,19 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output, int64_t image_number) { return bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number); }; + // The supercell probe integrates on the same engine, but what the engine counts is the run's + // measurement of its own stencil, and the probe's reflections are not the run's: keep them out. + const auto probe_integrate_fn = [this](const std::vector &predicted, size_t npredicted, + int64_t image_number) { + const BraggIntegrationCounts before = bragg_engine->Counts(); + auto ret = bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number); + const BraggIntegrationCounts after = bragg_engine->Counts(); + probe_counts.predicted += after.predicted - before.predicted; + probe_counts.bkg_starved += after.bkg_starved - before.bkg_starved; + probe_counts.bkg_starved_by_neighbour += after.bkg_starved_by_neighbour - before.bkg_starved_by_neighbour; + probe_counts.profile_fallback += after.profile_fallback - before.profile_fallback; + return ret; + }; // The radial background correction has to be decided BEFORE this image is integrated, so the // ice score is taken here rather than with the other per-image quantities at the end of the @@ -240,7 +253,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output, s.min_pix_per_spot = best_mp; SpotAnalyze(experiment, s, best_spots, output); decide_radial_background(); - indexer.ProcessImage(output, s, *prediction, integrate_fn); + indexer.ProcessImage(output, s, *prediction, integrate_fn, probe_integrate_fn); indexing_time_s += output.indexing_time_s.value_or(0.0f); } // Each indexer call reports only its own time, so the escalation's total is summed here. @@ -253,7 +266,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output, output.spot_finding_time_s = std::chrono::duration(std::chrono::steady_clock::now() - spot_finding_start_time).count(); decide_radial_background(); if (spot_finding_settings.indexing) - indexer.ProcessImage(output, spot_finding_settings, *prediction, integrate_fn); + indexer.ProcessImage(output, spot_finding_settings, *prediction, integrate_fn, probe_integrate_fn); } // Recorded whichever way the frame went. A frame holding StrongPixelLimit of them is given up @@ -372,7 +385,14 @@ void MXAnalysisWithoutFPGA::RunROIOnly(DataMessage &output) { } BraggIntegrationCounts MXAnalysisWithoutFPGA::BraggCounts() const { - return bragg_engine ? bragg_engine->Counts() : BraggIntegrationCounts{}; + if (!bragg_engine) + return {}; + BraggIntegrationCounts c = bragg_engine->Counts(); + c.predicted -= probe_counts.predicted; + c.bkg_starved -= probe_counts.bkg_starved; + c.bkg_starved_by_neighbour -= probe_counts.bkg_starved_by_neighbour; + c.profile_fallback -= probe_counts.profile_fallback; + return c; } void MXAnalysisWithoutFPGA::UpdateMaskResolution(const SpotFindingSettings &settings) { diff --git a/image_analysis/MXAnalysisWithoutFPGA.h b/image_analysis/MXAnalysisWithoutFPGA.h index 54ec41a44..573714609 100644 --- a/image_analysis/MXAnalysisWithoutFPGA.h +++ b/image_analysis/MXAnalysisWithoutFPGA.h @@ -57,6 +57,8 @@ class MXAnalysisWithoutFPGA { IndexAndRefine &indexer; std::unique_ptr prediction; std::unique_ptr bragg_engine; + // What the supercell probe's integrations added to bragg_engine's counts (see BraggCounts). + BraggIntegrationCounts probe_counts; std::unique_ptr preprocessor_buffer; const PixelMask &mask; diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index da5945c80..fd6207325 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -5042,6 +5042,25 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b } } + // Whether the lattice this pass integrates on has an index-2 superstructure. Nothing before + // integration can ask it: every count the lattice decisions use is dominated by the sub-lattice's + // strong reflections, so a crystal whose cell is doubled by a weak superstructure class is adopted + // at the half cell by every arbiter. The probe reads the question in intensities instead, on the + // same frames the first pass validates on (see IndexAndRefine::ProbeSupercell). + // + // REPORT ONLY - it decides nothing. Measured on the battery, a weak real class is not yet told apart + // from what sits at the half-integer nodes of crystals whose cell is right: a class at a fifth of + // the lattice's intensity on a crystal whose axis is truly doubled, against up to a tenth on + // correct cells (diffuse scattering and the tails of long-axis neighbours, which do not rock like + // Bragg reflections - but neither, on that crystal, does its real class). The line it logs is the + // population the decision has to be calibrated on. + if (full && indexer && experiment_.IsRotationIndexing()) { + std::vector frames; + for (const int o : select_equally_spaced_image_ordinals(images_to_process, 60)) + frames.push_back(o); + indexer->ProbeSupercell(frames); + } + // Main per-image loop, spread over N worker threads pulling from a shared counter. HDF5 reads // are serialized by the global hdf5_mutex; the analysis runs in parallel. std::atomic next_ordinal = 0; @@ -5269,6 +5288,33 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b result.cancelled = cancelled_; result.images_processed = finished_count.load(); + if (full && indexer && experiment_.IsRotationIndexing() && !cancelled_) { + const SupercellProbe probe = indexer->GetSupercellProbe(); + if (probe[0][0].n > 0 && probe[0][1].n > 0) { + std::string table; + for (int sh = 0; sh < 3; sh++) { + table += sh == 0 ? "\n 20-5 A:" : sh == 1 ? "\n 5-3 A:" : "\n 20-3 A:"; + const auto sums = [&](int c) { + SupercellProbeClass x = sh < 2 ? probe[c][sh] : probe[c][0]; + if (sh == 2) x += probe[c][1]; + return x; + }; + const SupercellProbeFit f0 = FitSupercellProbe(sums(0)); + table += fmt::format(" 000 {:.2f} a {:.1f} b {:.1f}

{:.3f} n {};", f0.mean_i_over_sigma, f0.a, + f0.b, f0.mean_p, sums(0).n); + for (int c = 1; c < 8; c++) { + const SupercellProbeFit f = FitSupercellProbe(sums(c)); + table += fmt::format(" {}{}{}: {:.2f} occ {:.2f}% rock {:.2f}+-{:.2f}% flat {:.2f}%;", + (c >> 2) & 1, (c >> 1) & 1, c & 1, f.mean_i_over_sigma, + 100.0 * f.mean_i / f0.mean_i, 100.0 * f.b / f0.b, 100.0 * f.b_se / f0.b, + 100.0 * f.a / f0.a); + } + } + logger.Info("Supercell probe (the eight parity classes of the doubled primitive cell; , " + "mean intensity, and the fit I = a + b p against the lattice's own):{}", table); + } + } + result.harmonic = HarmonicFromEvidence(harmonic_evidence); result.powder = powder_; result.powder_excluded_from_indexing = !config_.spot_finding.measured_ring_q_recipA.empty(); diff --git a/tests/IndexingUnitTest.cpp b/tests/IndexingUnitTest.cpp index acdda00fe..33f43a920 100644 --- a/tests/IndexingUnitTest.cpp +++ b/tests/IndexingUnitTest.cpp @@ -7,6 +7,7 @@ #include "../writer/HDF5Objects.h" #include "../image_analysis/indexing/IndexerFactory.h" #include "../image_analysis/indexing/PostIndexingRefinement.h" +#include "../image_analysis/IndexAndRefine.h" #include "../image_analysis/bragg_prediction/BraggPrediction.h" #include "../common/Logger.h" @@ -757,3 +758,30 @@ TEST_CASE("FFTIndexer_ManyNoiseFrames", "[Indexing]") { SUCCEED(lattices << " candidate lattices from noise frames"); } } + +TEST_CASE("FitSupercellProbe", "[SupercellProbe]") { + // A class whose intensity is 3 at any rocking plus 40 times its partiality: the fit separates the + // two, and the part that rocks is known to the noise the scatter about the line allows. + SupercellProbeClass c; + std::mt19937 rng(1); + std::normal_distribution noise(0.0, 1.0); + for (int i = 0; i < 20000; i++) { + const double p = (i % 100) / 100.0; + const double I = 3.0 + 40.0 * p + noise(rng); + c.n++; + c.sum_i += I; + c.sum_i_over_sigma += I; + c.sum_p += p; + c.sum_pp += p * p; + c.sum_pi += p * I; + c.sum_ii += I * I; + } + const SupercellProbeFit f = FitSupercellProbe(c); + CHECK(f.b == Catch::Approx(40.0).margin(0.1)); + CHECK(f.a == Catch::Approx(3.0).margin(0.05)); + CHECK(f.b_se == Catch::Approx(1.0 / std::sqrt(20000.0 * (0.9999 / 12.0))).epsilon(0.05)); + CHECK(f.mean_p == Catch::Approx(0.495)); + + // Too few reflections to fit a line: nothing. + CHECK(FitSupercellProbe(SupercellProbeClass{}).b == 0.0); +} From 9cb08b0346fe20fc3e661ea341ff6aed32f0ea23 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 25 Sep 2026 06:02:50 +0200 Subject: [PATCH 21/21] Docs: harmonic beam-centre disagreement decided both ways; geometry commit bar is the residual's noise Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C --- docs/CHANGELOG.md | 2 ++ docs/CPU_DATA_ANALYSIS_IMAGE.md | 6 +++++- docs/CPU_DATA_ANALYSIS_INDEXING.md | 2 +- docs/RUGNUX_ADVANCED.md | 2 +- 4 files changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a2928f583..118e493d3 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,6 +7,8 @@ * Rugnux rejects single observations that Wilson statistics make implausible (typically a hot pixel, zinger or ice spot), including pairs and single measurements that the equivalents test cannot judge; the count is reported as `OBSERVATIONS_REJECTED_WILSON`. * Rugnux reports the X-ray bandwidth it measures from spot shapes (for multilayer and pink beams); the value is reported only and does not change processing. * Rugnux accepts a detector-modulation or absorption correction surface when held-out data support it on Fisher's z, and fits them in the order modulation, time, goniometer frame; this recovers corrections that were wrongly refused. +* Rugnux settles a disagreement between the file's and the measured beam centre over an axis harmonic in either direction, so a file centre off along the spindle no longer leaves the run on a doubled axis. +* Rugnux commits a post-refined geometry only when the held-out residual falls by more than its own noise, instead of by a fixed 2 %. * `rugnux --polarization` is documented as the polarization degree (XDS `FRACTION_OF_POLARIZATION` = (1 + p)/2); the default 0.99 suits undulators. ### 1.0.0-rc.172 diff --git a/docs/CPU_DATA_ANALYSIS_IMAGE.md b/docs/CPU_DATA_ANALYSIS_IMAGE.md index dc00bb54f..153619e6a 100644 --- a/docs/CPU_DATA_ANALYSIS_IMAGE.md +++ b/docs/CPU_DATA_ANALYSIS_IMAGE.md @@ -178,7 +178,11 @@ majority; where both work and disagree, both cells are reported and neither is chosen, because the only arbiter available at that stage is the indexed frame count and **it points the wrong way**: acceptance is a fractional-Miller test, so a cell twice as long must place every spot twice as accurately to score the same, and a -halved axis can index *more* frames than the true cell. Where the two disagree only in Bravais class +halved axis can index *more* frames than the true cell. The one disagreement that is decided is an +axis harmonic — primitive volumes related by an integer factor 2 to 4 — and it is decided on the +**pooled validation spots** instead, each cell scored against its own wrong-spindle null, in either +direction: the measured centre is adopted when the lattice it gives, larger or smaller, carries +materially more of the spots than the file's. Where the two disagree only in Bravais class at the same primitive volume, that is said separately from a volume ratio that is an axis harmonic, which is the one the centre decides. diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index d219691ea..4ac0b252d 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -322,7 +322,7 @@ The refinement above (§7.2) runs per image against that image's spots. For rota Free: the crystal orientation, the unit cell (every parameter the crystal system leaves free, not one overall scale), the goniometer-axis direction, the detector distance and the beam centre. The positional residual on its own *is* degenerate with the cell scale — that is why this used to be split into a cell-scale step and a distance step — but the excitation residual does not involve the detector at all, so it fixes the absolute size of the reciprocal lattice and breaks the degeneracy inside the same problem. Splitting it instead cost accuracy twice over: pass 1 frees the whole lattice against a frozen distance, so the distortion it absorbs is *anisotropic* and no single scale can undo it; and whatever bias is left in that scale goes straight into the distance, which is only ever determined relative to the cell. - The fit is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual — both families of it, since the excitation residual is the only evidence of the cell scale and the positional values outnumber it about three to one — and the move stays inside its bounds: every free cell angle within 1° and the beam centre within 15 px of the nearest centre anything already believes. + The fit is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual by more than that residual's own noise (the standard errors of the two held-out means combined, the bar a round of the geometry walk has to clear) and lowers both families of it, since the excitation residual is the only evidence of the cell scale and the positional values outnumber it about three to one — and the move stays inside its bounds: every free cell angle within 1° and the beam centre within 15 px of the nearest centre anything already believes. The **distance and the cell lengths are bounded one step at a time, not as a whole**. One per cent was once a cap on the entire move, and as a cap it was the opposite of its job — a header is most worth correcting when it is most wrong, and a geometry genuinely several per cent out could never be reached (measured: a refused fit of 310.000 → 305.692 mm whose cell landed within 0.06 % of the deposited one). It is a **trust region** instead. The first solve is asked in the wide box around nominal exactly as before, so a fit that settles within one step commits unchanged; a fit that wants more is re-fitted as a *walk* of one-per-cent steps, each seeded where the last arrived and each required to lower the held-out residual, stopping where a step stops paying. A walk that uses every step it is allowed has not settled — it stopped because it ran out of steps, not because it arrived — and is refused, which is the runaway the cap stood in for, tested where it can be seen. A move of more than one step is additionally **ratified by re-indexing at where it arrived**: that is what separates the failure the cap was really aimed at (a second lattice, whose spots bias every cross-validation fold identically) from a wrong header, since a second lattice does not index better at the new geometry and a real distance error does. diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 42ab6cfe3..47d8b19ec 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -240,7 +240,7 @@ Geometry: | Option | Description | | --- | --- | -| `--beam-center-check[=off]` | Measure the beam centre from the isotropy of the scattered background on **every** run, report how far the file's value is from it — against how right this particular geometry needs it to be — and index a **second first pass** at the measured centre to see whether the two centres give the same lattice. The fit reads the projection `--detect-beam-stop` already builds, so it costs no extra frames. **On by default**; `=off` disables. On a run that indexes, the measured centre is adopted in two cases only: where the file's centre indexes nothing and the measured one indexes a majority, and where the two cells are related by an integer volume factor, the measured centre's is the **larger** one, and it carries materially more of the pooled validation spots than the file's does (each cell scored against its own wrong-spindle null, so a denser lattice is not credited for accidental hits). That one direction is the axis harmonic a centre error along the spindle produces, and nothing downstream repairs it; every other disagreement is reported with both cells and left undecided | +| `--beam-center-check[=off]` | Measure the beam centre from the isotropy of the scattered background on **every** run, report how far the file's value is from it — against how right this particular geometry needs it to be — and index a **second first pass** at the measured centre to see whether the two centres give the same lattice. The fit reads the projection `--detect-beam-stop` already builds, so it costs no extra frames. **On by default**; `=off` disables. On a run that indexes, the measured centre is adopted in two cases only: where the file's centre indexes nothing and the measured one indexes a majority, and where the two cells are related by an integer volume factor (2 to 4) and the measured centre's cell, larger or smaller, carries materially more of the pooled validation spots than the file's does (each cell scored against its own wrong-spindle null, so a denser lattice is not credited for accidental hits). That is the axis harmonic a centre error along the spindle produces; every other disagreement is reported with both cells and left undecided | | `--beam-center-search[=N\|off]` | After a first pass that indexes fewer than half the validation frames, step the centre a pixel at a time out to N px along **each** detector axis and keep the first rung that indexes a majority. **On by default** (12 px); `=off` disables. It runs only after a pass that has already failed, and spot finding is not repeated, so a run that indexes never pays for it. Both detector directions are searched: a centre error *across* the spindle collapses the indexed fraction and announces itself, while one *along* it holds the frame count up and quietly returns an axis harmonic | | `--estimate-beam-center` | Measure the direct beam before indexing, from the symmetry of the spots where the sweep reaches at least half a turn and from the radial background profile where it does not; the value in the file is kept where neither can measure it. Off by default | | `--no-fit-spindle` | With the above, keep the rotation axis given in the file instead of fitting its skew about the beam |