leonarski_fandClaude Opus 5
d5220014b3
A weak crystal inside a powder of its own microcrystals plus ice puts only ~5% of each frame's spots on its lattice. The pooled first-pass test accepted that lattice (5.5% of validation spots against 0.6% at a wrong spindle angle), but then: - every frame failed the 20% per-frame floor (LATTICE_MIN_INDEXED_FRACTION), so nothing was integrated and the merge was skipped (indexing rate 0); - the background-measured beam centre, ~3 px from the file's and correct, was not adopted because the arbiter counts validation frames, which are 0/60 at every centre. Changes: - When fewer than 1/6 of the validation frames clear the per-frame floor AND the sweep's pooled on-lattice fraction is itself below that floor, integrate every frame from the sweep's lattice (as XDS/DIALS do) and leave frame selection to scaling. Sweeps sparse only in spots per frame keep the floor. - Merge when a rotation lattice was found even if no frame "indexed" on its own. - When neither centre indexes a validation frame, adopt the measured centre if its pooled excess over chance beats the file's by 3.29 sigma. On the case above: P2 at the XDS cell, merged to 1.77 A, CC1/2 0.95, ISa 3.4 (XDS with the same lattice: ISa 3.3-5.5). Three healthy/partially-indexing rotation sets are bit-identical; a two-wavelength CBF set that currently merges 42 frames on a wrong cell moves (beam centre adopted, 755 frames). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
64 lines
4.3 KiB
C++
64 lines
4.3 KiB
C++
// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include "../../common/CrystalLattice.h"
|
|
#include "../../common/DiffractionExperiment.h"
|
|
#include "../../common/JFJochMessages.h"
|
|
|
|
// Minimum fraction of a frame's in-resolution spots that must lie on a candidate lattice for the
|
|
// frame to be that crystal's. See the frame gate in AnalyzeIndexing, which is where it is applied.
|
|
constexpr float LATTICE_MIN_INDEXED_FRACTION = 0.20f;
|
|
|
|
// Tally one image's spots by their rank in its intensity-ordered spot list: how many images had a spot
|
|
// at that rank at all (`counted`) and on how many of them it lay on the lattice (`indexed`). Ice spots
|
|
// are skipped, as they are in the frame gate. Both are added to, and their length bounds the ranks
|
|
// considered. Counts rather than weights so that the tally is exact whatever order the images are
|
|
// summed in, which is what makes the budget below independent of the thread schedule.
|
|
void AddSpotBudgetEvidence(const std::vector<SpotToSave> &spots, bool index_ice_rings,
|
|
std::vector<int64_t> &indexed, std::vector<int64_t> &counted);
|
|
|
|
// How far the fall from the peak must exceed the counting noise of the spots for the peak to be one.
|
|
//
|
|
// Under the null - the spots lie on the lattice at the same rate at every depth - the running sum
|
|
// below is a driftless random walk in the counted spots: each is worth 1 - g with probability g and
|
|
// -g otherwise, so its step has mean zero and variance g(1-g). The MAXIMUM of such a walk is positive
|
|
// whatever the data, so an argmax taken on its own cuts every dataset, including one with nothing to
|
|
// cut. What is acted on is the FALL from the peak to the end of the list, which is the maximum of the
|
|
// same walk read backwards from the end, and the reflection principle gives that maximum's null law
|
|
// exactly: P(fall > z sqrt(g(1-g)T)) = 2(1 - Phi(z)) over T counted spots in all. The search over the
|
|
// ranks is therefore already paid for and no further multiple-comparison correction is due. z is set
|
|
// for one false cut in a thousand measurements, which over a corpus the size of a rotation test set
|
|
// (tens of crystals, a measurement per pass) expects none at all.
|
|
constexpr float SPOT_BUDGET_SIGNIFICANCE_Z = 3.29f; // 2(1 - Phi(z)) = 0.001
|
|
|
|
// The spot budget those tallies support: the rank at which indexed - LATTICE_MIN_INDEXED_FRACTION *
|
|
// counted, summed over the ranks down to it, peaks. Each spot that lies on the lattice is worth
|
|
// 1 - LATTICE_MIN_INDEXED_FRACTION and each one that does not costs LATTICE_MIN_INDEXED_FRACTION - the
|
|
// same weighing the frame gate applies to a spot list as a whole - so the sum rises exactly while the
|
|
// spots at that depth are on the lattice more often than the gate's floor. Deeper than the peak they
|
|
// are not: they are no longer this crystal's reflections, and they can only push a frame towards
|
|
// rejection while adding nothing the lattice recognises.
|
|
//
|
|
// Zero - keep the whole list - when the fall from that peak to the end of the list is no larger than
|
|
// the counting noise above, which is the case whenever the spots go on lying on the lattice at the
|
|
// same rate all the way down, and the case a bare argmax gets wrong.
|
|
int64_t SpotBudgetFromEvidence(const std::vector<int64_t> &indexed, const std::vector<int64_t> &counted);
|
|
|
|
// integrate_every_frame: on rotation, integrate the frame from the sweep's lattice even where fewer than
|
|
// LATTICE_MIN_INDEXED_FRACTION of its spots lie on it - for a sweep whose lattice was accepted on the pooled
|
|
// spots although its frames, one at a time, cannot clear that floor (see Rugnux).
|
|
// spots_on_lattice, when given, receives how many of this frame's non-ice spots lie on latt - the
|
|
// numerator of the frame gate below, reported whatever the gate then decides. The rotation first
|
|
// pass's acceptance test pools that count over its validation frames, so it needs it from the sparse
|
|
// frames the gate refuses as much as from the ones it accepts.
|
|
bool AnalyzeIndexing(DataMessage &message,
|
|
const DiffractionExperiment &experiment,
|
|
const CrystalLattice &latt,
|
|
const std::vector<CrystalLattice> &extra_lattices = {},
|
|
int64_t *spots_on_lattice = nullptr,
|
|
bool integrate_every_frame = false);
|
|
|
|
|