mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 85 Wiki Activity
Files
04655bd8bf12be616ea759986fd4e1bdbc63b489
Jungfraujoch/rugnux/RugnuxFirstPassAccept.cpp
T
leonarski_fandClaude Opus 5.5 5f731db81d rugnux: the library returns its outputs in memory; the callers write the files 
The rugnux library wrote its results itself: _process.h5 through a
FileWriter, the merged/twin/_mr reflection files, the unmerged MTZs, the
maps and their MTZ, the placed models, _detector.jpg and _plot.txt, and
on a doubled cell it renamed files on disk. It now writes none of them.

- MtzTable (common/MtzTable.h): an MTZ file's header, columns and batch
  headers with no file behind it. It replaces MergedReflectionsMessage
  and MapCoefficientsMessage in the END message, with one CBOR encoding.
  WriteMtz (writer/WriteRugnuxFiles.h) is the one serialiser, used by
  rugnux and jfjoch_writer alike; it refuses a table whose columns differ
  in length, name a missing dataset, carry a malformed batch header or
  name a space group gemmi does not know. MergedMtzTable and
  UnmergedMtzTable build the tables the old writers built.
- ProcessResult::outputs (RugnuxOutputs) holds, for the output pass only:
  the answer's merge (MTZ table plus what the .cif/.hkl are written from),
  the twin hypothesis' and the model's choice's merges, the unmerged and
  unmerged-partials tables, the maps (coefficients always; the grids only
  with ProcessConfig::keep_map_grids), the placed models, the detector
  JPEG and the per-image plot table, and the start message of the
  no-lattice _process.h5. Map RMS is taken from the grid in memory, as the
  CCP4 header stated it, whether or not the map is kept.
- The per-image analysis goes to a caller-supplied RugnuxImageSink;
  rugnux and the viewer give a FileWriter-based ProcessFileSink. The
  geometry pre-pass no longer makes a _01_process.h5.
- The doubled cell keeps both output sets: the answer's in outputs, the
  other's in DoubledCell::other_outputs with its suffix; the report names
  files by suffix after the caller's prefix.
- The output prefix, write_process_h5 and the observation dump path are
  gone from ProcessConfig; --dump-observations is a callback.
- What used to be gated on having files to write (model validation and
  MR, the twin-hypothesis merges, the finalist ledger, the scaled fulls,
  the unmerged tables) is gated on the output pass instead, so a caller
  with no prefix gets the same results.
- RugnuxOutputFiles (new library JFJochRugnuxOutput, linked by rugnux,
  the viewer and the tests) writes every file from the result with the
  names and formats rugnux has always used. The viewer writes its job's
  files the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:51:20 +02:00

513 lines
30 KiB
C++
Raw Blame History

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <cstdlib>
#include "Rugnux.h"
#include "../image_analysis/structure_refinement/ModelValidation.h"
#include "../image_analysis/indexing/SpindleCuspLoss.h"
#include "../image_analysis/bragg_integration/SpotWidth.h"
#include "../image_analysis/bragg_integration/SpotFootprint.h"
#include "../image_analysis/bragg_integration/BraggStencil.h" // BRAGG_FOOTPRINT_NSIGMA
#include "../image_analysis/hot_pixels/HotPixels.h"
#include "../image_analysis/SensorAbsorption.h"
#include "DiagnosticOutput.h"
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cmath>
#include <condition_variable>
#include <deque>
#include <cstring>
#include <functional>
#include <future>
#include <mutex>
#include <numeric>
#include <map>
#include <set>
#include <sstream>
#include <thread>
#include "../reader/JFJochHDF5Reader.h"
#include "../common/JFJochMath.h"
#include "../common/ParallelFor.h"
#include "../common/Logger.h"
#include "../common/AzimuthalIntegrationMapping.h"
#include "../common/AzimuthalIntegrationProfile.h"
#include "../common/CUDAWrapper.h"
#include "../common/JFJochException.h"
#include "../common/time_utc.h"
#include "../image_analysis/MXAnalysisWithoutFPGA.h"
#include "../image_analysis/beam_stop/ShadowFinder.h"
#include "../image_analysis/IndexAndRefine.h"
#include "../image_analysis/indexing/AnalyzeIndexing.h"
#include "../image_analysis/indexing/HarmonicContamination.h"
#include "../image_analysis/geom_refinement/BeamCenterFFT.h"
#include "../image_analysis/geom_refinement/BeamCenterFromBackground.h"
#include "../image_analysis/geom_refinement/GeometryRefiner.h"
#include "../image_analysis/indexing/IndexerThreadPool.h"
#include "../image_analysis/spot_finding/ImageSpotFinderCPU.h"
#include "../image_analysis/spot_finding/SpotUtils.h"
#include "gemmi/twin.hpp"
#include "../image_analysis/azint/AzIntEngineCPU.h"
#include "../image_analysis/image_preprocessing/ImagePreprocessorCPU.h"
#include "../image_analysis/image_preprocessing/ImagePreprocessorBuffer.h"
#ifdef JFJOCH_USE_CUDA
#include "../image_analysis/image_preprocessing/ImagePreprocessorGPU.h"
#include "../image_analysis/image_preprocessing/ImagePreprocessorBufferGPU.h"
#include "../image_analysis/spot_finding/AdaptiveSpotFinderGPU.h"
#endif
#include "../image_analysis/scale_merge/Merge.h"
#include "../image_analysis/scale_merge/RfreeFlags.h"
#include "../image_analysis/scale_merge/RotationScaleMerge.h"
#include "../image_analysis/scale_merge/ResolutionCutoff.h"
#include "../image_analysis/scale_merge/ReindexAmbiguity.h"
#include "../image_analysis/scale_merge/ScalingResult.h"
#include "../image_analysis/scale_merge/SearchSpaceGroup.h"
#include "../image_analysis/geom_refinement/PostRefine.h"
#include "../image_analysis/lattice_search/LatticeSearch.h"
#include "../image_analysis/lattice_search/LePageLattice.h"
#include "../image_analysis/scale_merge/AnisotropyAnalysis.h"
#include "../image_analysis/scale_merge/FrenchWilson.h"
#include "../image_analysis/scale_merge/TwinningAnalysis.h"
#include "../image_analysis/scale_merge/TranslationalNCS.h"
#include "../image_analysis/scale_merge/HKLKey.h"
#include "../image_analysis/scale_merge/ScaleOnTheFly.h"
#include "../image_analysis/scale_merge/StillsPartialityRefine.h"
#include "../image_analysis/WriteReflections.h"
#include "../image_analysis/bragg_integration/CalcISigma.h"
#include "../common/Definitions.h"
#include "../common/CorrelationCoefficient.h"
#include <array>
#include <map>
#include "RugnuxPipeline.h"
#include "RugnuxFirstPass.h"
// The end of the first pass: whether it found a lattice at all, the no-crystal test on the pooled
// validation spots, and what the run takes from the lattice it accepts. FirstPassRotationIndexing
// gives their order.
using namespace rugnux_internal;
namespace {
// The two ways the first pass can end without a usable lattice that are about the DATA.
// They are one situation from the user's side - no crystal was found in these frames - so
// both name the cause and then give the same remedies.
constexpr const char *next_steps =
" Nothing was integrated. Check the beam centre (--beam-x / --beam-y), raise "
"--max-spots if the frames are well populated with spots, and use --mode azint to "
"write the per-image analysis for inspection.";
}
void Rugnux::FirstPassRun::RequireLattice(const FirstPass &best) const {
if (!best.result.has_value())
throw NoLatticeFound(std::string("Two-pass rotation indexing found no lattice.") + next_steps);
}
// Whether the run has a lattice at all is decided here, on the pooled spots rather than on
// the frame count the rescues above are steered by. A frame count asks each frame to clear
// a bar of its own and then counts the frames that did, which measures the spots per frame
// as much as the lattice; the pooled fraction asks the whole validation set at once and is
// judged against what the same lattice scores at a wrong spindle angle, so a sparse pattern
// is refused only when its spots genuinely do not know about the lattice.
auto Rugnux::FirstPassRun::MeasureVerdict(const FirstPass &best, const std::vector<double> &first_pass_key,
const std::optional<ValidationSpotEvidence> &memo_hit) -> Verdict {
const PooledEvidence evidence = MeasurePooled(*indexer, *best.result);
const double pooled = static_cast<double>(evidence.on_lattice)
/ static_cast<double>(std::max<int64_t>(1, evidence.spots));
const double pooled_chance = static_cast<double>(evidence.by_chance)
/ static_cast<double>(std::max<int64_t>(1, evidence.spots));
result.validation_evidence = evidence;
if (memo_hit && (memo_hit->spots != evidence.spots || memo_hit->on_lattice != evidence.on_lattice
|| memo_hit->by_chance != evidence.by_chance))
throw MemoMismatch(fmt::format("First-pass memo mismatch: stored {}/{}/{}, recomputed {}/{}/{}",
memo_hit->on_lattice, memo_hit->spots, memo_hit->by_chance,
evidence.on_lattice, evidence.spots, evidence.by_chance));
// Kept only where the pass ended on the inputs it started from.
if (!geometry_prepass && rx.FirstPassInputKey() == first_pass_key)
rx.AddFirstPassMemo(FirstPassMemo{first_pass_key, evidence});
return {evidence, pooled, pooled_chance};
}
void Rugnux::FirstPassRun::RejectChanceLattice(const FirstPass &best, const Verdict &v) const {
const PooledEvidence &evidence = v.evidence;
const double pooled = v.pooled, pooled_chance = v.pooled_chance;
// Near tie (log only): the exact binomial tail against the significance, both as log10. The
// noise is how far the tail moves for one binomial standard deviation of the count.
{
const double n = static_cast<double>(evidence.reflections + evidence.reflections_by_chance);
const double p = 1.0 / (1.0 + VALIDATION_NULL_DISPLACEMENTS);
const double log10_tail = ValidationEvidenceLog10Tail(evidence);
PooledEvidence one_more = evidence;
one_more.reflections++;
const double per_reflection = log10_tail - ValidationEvidenceLog10Tail(one_more);
NearTie("no-crystal test",
fmt::format("{} distinct reflections vs {} by chance, log10 tail {:.2f}",
evidence.reflections, evidence.reflections_by_chance, log10_tail),
fmt::format("log10 significance {:.2f}", std::log10(NO_CRYSTAL_SIGNIFICANCE)),
std::log10(NO_CRYSTAL_SIGNIFICANCE) - log10_tail,
per_reflection * std::sqrt(n * p * (1.0 - p)), "exact binomial on distinct reflections, decades");
}
if (!ValidationEvidenceBeatsChance(evidence, NO_CRYSTAL_SIGNIFICANCE)) {
{
// Name the cell and Bravais class that was rejected. The commonest cause is a metric
// symmetry promoted one class too far - the constrained cell then misses every
// reflection by the small angle the constraint snapped away - and without the cell in
// the message there is nothing to see that from.
const auto &c = best.result->search_result.conventional.GetUnitCell();
throw NoLatticeFound(fmt::format("Two-pass rotation indexing found only a lattice that "
"takes {:.1f}% of the validation spots against the "
"{:.1f}% a wrong spindle angle takes ({} distinct "
"reflections against {} over {} wrong angles) - it is "
"not this crystal's lattice. It was {}-centred {}, "
"{:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f}.{}",
100.0 * pooled, 100.0 * pooled_chance,
evidence.reflections, evidence.reflections_by_chance,
VALIDATION_NULL_DISPLACEMENTS,
best.result->search_result.centering,
gemmi::crystal_system_str(best.result->search_result.system),
c.a, c.b, c.c, c.alpha, c.beta, c.gamma, next_steps));
}
}
}
void Rugnux::FirstPassRun::AdoptLattice(const FirstPass &best, const Verdict &v) {
const double pooled = v.pooled, pooled_chance = v.pooled_chance;
const PooledEvidence &evidence = v.evidence;
indexer->ForceRotationIndexerResult(*best.result);
// A lattice the pooled spots accept while most validation frames fail the per-frame floor
// (LATTICE_MIN_INDEXED_FRACTION) on their own, because the sweep as a whole puts fewer of its
// spots on the lattice than that floor asks of each frame: the crystal is real but its
// spots are a minority of every frame - a weak crystal inside a powder of its own
// microcrystals and ice. The floor cannot tell its frames from anything else, so integrate
// every frame from the sweep's lattice, as XDS and DIALS do, and leave the frames to the
// scaling. The frames that do clear it are the upper tail of the same population: measured
// on a crystal in a powder of itself (13% of the pooled spots on the lattice), 12/60
// cleared it, and keeping the floor integrated 15% of the sweep. A sweep that is sparse
// only in spots per frame (a small molecule) keeps the floor.
const bool sparse_lattice = best.score < static_cast<int>(validation.size()) / 2
&& pooled < LATTICE_MIN_INDEXED_FRACTION;
indexer->IntegrateEveryFrame(sparse_lattice);
if (sparse_lattice)
logger.Warning("Sparse lattice: {}/{} validation frames clear the per-frame floor, but the "
"pooled spots put {:.1f}% on the lattice against {:.1f}% by chance - every "
"frame is integrated from the sweep's lattice",
best.score, static_cast<int>(validation.size()), 100.0 * pooled,
100.0 * pooled_chance);
logger.Info("First-pass spot finding: {} frames in {} batches, {:.2f} s",
spot_cache.size(), prefetch_batches, prefetch_time_s);
logger.Info("Two-pass rotation indexing found lattice (scheme '{}': {}/{} validation frames, "
"{}/{} validation spots = {:.1f}% against {:.1f}% at a wrong spindle angle; {} "
"distinct reflections against {} over {} wrong angles)",
best.name, best.score, static_cast<int>(validation.size()),
evidence.on_lattice, evidence.spots, 100.0 * pooled, 100.0 * pooled_chance,
evidence.reflections, evidence.reflections_by_chance, VALIDATION_NULL_DISPLACEMENTS);
{
const auto &c = best.result->search_result.conventional.GetUnitCell();
rx.Note(fmt::format("lattice {} {}: {:.2f} {:.2f} {:.2f} {:.1f} {:.1f} {:.1f}",
gemmi::crystal_system_str(best.result->search_result.system),
best.result->search_result.centering, c.a, c.b, c.c, c.alpha, c.beta, c.gamma));
rx.Note(fmt::format("{}/{} validation frames, {:.0f}% of their spots on it",
best.score, validation.size(), 100.0 * pooled));
}
}
// Leftover-lattice census, report only: index what the crystal's lattice leaves over, to name
// a split crystal, a twin domain or a second crystal in the report. Every spot of the spread
// and validation frames the lattice takes is marked; the rest (not on ice) goes to a fresh
// rotation indexer; a lattice it finds is judged on the validation leftovers against the same
// displaced-spindle-angle null as the first pass, its spots are marked in turn, and the search
// goes on for up to three lattices. Its own IndexAndRefine instances and indexer, so nothing
// the run goes on with is touched. Run on the pass that writes the result only.
//
// Its spots are found here; the rest runs in the background, beside the image loop, on
// copies of everything it reads, and its answer is collected before the pass returns. Not on
// a pass that only post-refines: nothing reads its result there.
void Rugnux::FirstPassRun::StartLeftoverCensus(const FirstPass &best) {
std::vector<int> census_frames = spread_ordinals;
census_frames.insert(census_frames.end(), validation.begin(), validation.end());
std::ranges::sort(census_frames);
census_frames.erase(std::unique(census_frames.begin(), census_frames.end()), census_frames.end());
PrefetchSpots(census_frames);
std::map<int, std::vector<SpotToSave>> census_spots;
for (const int f : census_frames)
census_spots.emplace(f, spot_cache.at(f));
census_future = std::async(std::launch::async,
[&logger = logger, &indexer_pool = indexer_pool, &config_ = rx.config_,
x = rx.experiment_, census_frames, census_spots, lattice = *best.result,
validation = validation, spread_ordinals = spread_ordinals,
validation_settings = validation_settings]() -> LeftoverLatticeCensus {
const auto census_start = std::chrono::steady_clock::now();
const bool ice_ok = x.GetIndexingSettings().GetIndexIceRings();
const auto &spot_cache = census_spots;
const auto rot_angle = [&](int ordinal) -> std::optional<float> {
if (const auto g = x.GetGoniometer())
return g->GetAngle_deg(static_cast<float>(ordinal)) + g->GetWedge_deg() / 2.0f;
return std::nullopt;
};
// assign[frame][spot]: -2 ice, -1 no lattice, 0 the crystal's lattice, 1.. a leftover lattice
std::map<int, std::vector<int>> assign;
for (const int f : census_frames) {
auto &a = assign[f];
a.assign(spot_cache.at(f).size(), -1);
for (size_t k = 0; k < a.size(); k++)
if (!ice_ok && spot_cache.at(f)[k].ice_ring)
a[k] = -2;
}
// Mark the still unassigned spots lattice `r` takes as `id`.
auto mark = [&](const RotationIndexerResult &r, int id) {
IndexAndRefine idx(x, indexer_pool.get(), /*retain_outcomes=*/false);
idx.ForceRotationIndexerResult(r);
idx.IntegrateEveryFrame(true);
std::mutex m_assign;
ParallelFor(static_cast<int>(census_frames.size()),
std::min<size_t>(census_frames.size(), config_.nthreads), [&](int i) {
const int f = census_frames[i];
DataMessage m{};
m.number = f;
m.spots = spot_cache.at(f);
idx.CountSpotsOnLattice(m, validation_settings);
std::unique_lock ul(m_assign);
auto &a = assign[f];
if (m.spots.size() != a.size())
return;
for (size_t k = 0; k < a.size(); k++)
if (a[k] == -1 && m.spots[k].indexed && m.spots[k].lattice == 0)
a[k] = id;
});
};
// Leftover validation spots on `r` at the true spindle angles, and the median of five
// displaced ones - the first pass's null (MeasurePooled) on the leftovers.
auto leftover_evidence = [&](const RotationIndexerResult &r) -> std::pair<int64_t, int64_t> {
IndexAndRefine idx(x, indexer_pool.get(), /*retain_outcomes=*/false);
idx.ForceRotationIndexerResult(r);
const int nv = static_cast<int>(validation.size());
auto at_shift = [&](int shift) {
std::atomic<int64_t> n = 0;
ParallelFor(nv, std::min<size_t>(validation.size(), config_.nthreads), [&](int i) {
DataMessage m{};
m.number = validation[(i + shift) % nv];
const auto &sp = spot_cache.at(validation[i]);
const auto &a = assign.at(validation[i]);
for (size_t k = 0; k < sp.size(); k++)
if (a[k] == -1)
m.spots.push_back(sp[k]);
n += idx.CountSpotsOnLattice(m, validation_settings);
});
return n.load();
};
std::vector<int64_t> chance;
for (int k = 1; k <= 5; k++)
chance.push_back(at_shift(std::max(1, k * nv / 7)));
std::ranges::sort(chance);
return {at_shift(0), chance[2]};
};
// Spots and summed intensity assigned `id`, and the share of them in the 2 densest of 8
// equal blocks of the census frames (0.25 is uniform over the sweep).
struct Tally { int64_t n = 0; double intensity = 0.0; double concentration = 0.0; };
auto tally = [&](int id) {
Tally t;
std::array<int64_t, 8> block{};
const size_t nf = census_frames.size();
for (size_t i = 0; i < nf; i++) {
const int f = census_frames[i];
const auto &a = assign.at(f);
for (size_t k = 0; k < a.size(); k++)
if (a[k] == id) {
t.n++;
t.intensity += spot_cache.at(f)[k].intensity;
// numpy.array_split: the first nf % 8 blocks hold one frame more
const size_t q = nf / 8, r = nf % 8;
block[i < r * (q + 1) ? i / (q + 1) : r + (i - r * (q + 1)) / std::max<size_t>(1, q)]++;
}
}
std::ranges::sort(block);
t.concentration = static_cast<double>(block[6] + block[7]) / std::max<int64_t>(1, t.n);
return t;
};
auto reduced = [](const RotationIndexerResult &r) {
return LatticeSearch(r.lattice.ToPrimitive(r.search_result.centering)).primitive_reduced;
};
mark(lattice, 0);
int64_t n_all = 0;
double i_all = 0.0;
for (const auto &[f, a] : assign)
for (size_t k = 0; k < a.size(); k++)
if (a[k] != -2) {
n_all++;
i_all += spot_cache.at(f)[k].intensity;
}
const Tally t0 = tally(0);
LeftoverLatticeCensus census;
census.frames = static_cast<int>(census_frames.size());
census.primary_spots_pct = 100.0 * t0.n / std::max<int64_t>(1, n_all);
census.primary_intensity_pct = 100.0 * t0.intensity / std::max(1.0, i_all);
const CrystalLattice primary_reduced = reduced(lattice);
for (int id = 1; id <= 3; id++) {
RotationIndexer ri(x, *indexer_pool);
ri.VerifyMemo(config_.verify_first_pass_memo);
for (const int f : spread_ordinals) {
std::vector<SpotToSave> left;
const auto &sp = spot_cache.at(f);
const auto &a = assign.at(f);
for (size_t k = 0; k < sp.size(); k++)
if (a[k] == -1)
left.push_back(sp[k]);
ri.ProcessImage(f, left, rot_angle(f));
}
ri.RunIndexing();
const auto r = ri.GetLattice();
if (!r)
break;
const auto [real, chance] = leftover_evidence(*r);
mark(*r, id);
const Tally t = tally(id);
const CrystalLattice r_reduced = reduced(*r);
LeftoverLattice l;
l.cell = r->search_result.conventional.GetUnitCell();
l.centering = r->search_result.centering;
l.system = gemmi::crystal_system_str(r->search_result.system);
l.volume_ratio = std::abs(r_reduced.CalcVolume()) / std::max(1.0f, std::abs(primary_reduced.CalcVolume()));
l.misorientation_deg = LatticeMisorientationDeg(primary_reduced, r_reduced);
l.z = (real - chance) / std::sqrt(std::max<double>(1.0, chance));
l.spots_pct = 100.0 * t.n / std::max<int64_t>(1, n_all);
l.intensity_pct = 100.0 * t.intensity / std::max(1.0, i_all);
l.sweep_concentration = t.concentration;
l.kind = ClassifyLeftoverLattice(l.misorientation_deg, l.volume_ratio, l.sweep_concentration);
logger.Info("Leftover lattice {}: {}-centred {} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f} {:.2f}, "
"volume ratio {:.3f}, misorientation {:.2f} deg, z {:.1f} ({} vs {} at a wrong "
"angle), {:.1f}% of non-ice spots, {:.1f}% of their intensity, sweep "
"concentration {:.2f} -> {}",
id, l.centering, l.system, l.cell.a, l.cell.b, l.cell.c, l.cell.alpha, l.cell.beta,
l.cell.gamma, l.volume_ratio, l.misorientation_deg, l.z, real, chance, l.spots_pct,
l.intensity_pct, l.sweep_concentration, LeftoverLatticeKindName(l.kind));
if (l.z >= 5.0 && l.kind != LeftoverLatticeKind::Self)
census.lattices.push_back(l);
if (real - chance < 3.0 * std::sqrt(std::max<double>(1.0, chance)) + 5)
break;
}
logger.Info("Leftover-lattice census: the crystal's lattice takes {:.1f}% of {} non-ice spots "
"({:.1f}% of their intensity) on {} frames; {} further lattice(s); {:.1f} s",
census.primary_spots_pct, n_all, census.primary_intensity_pct, census.frames,
census.lattices.size(),
std::chrono::duration<double>(std::chrono::steady_clock::now() - census_start).count());
// The last of the pass's indexing (see RunPipeline).
indexer_pool->ReleaseIndexers();
return census;
});
}
void Rugnux::FirstPassRun::ReportTiltWalk(const FirstPass &best) {
// What the walked tilt does to the prediction: the keystone at the far corner of the
// detector, against the disc the integration sums over, and the whole-pattern shift
// it is equivalent to, which the beam centre carries instead where it is refused.
const auto &tw = *best.result->tilt_walk;
const auto &g = best.result->geom;
const double start_rot1 = tw.refused ? g.GetPoniRot1_rad() : rx.experiment_.GetPoniRot1_rad();
const double start_rot2 = tw.refused ? g.GetPoniRot2_rad() : rx.experiment_.GetPoniRot2_rad();
const double walk_rad = std::hypot(tw.tilt_rad[0] - start_rot1, tw.tilt_rad[1] - start_rot2);
const double corner_px = std::hypot(
std::max<double>(g.GetBeamX_pxl(), rx.experiment_.GetXPixelsNumConv() - g.GetBeamX_pxl()),
std::max<double>(g.GetBeamY_pxl(), rx.experiment_.GetYPixelsNumConv() - g.GetBeamY_pxl()));
const double lever_px = g.GetDetectorDistance_mm() / g.GetPixelSize_mm();
const std::string what = fmt::format(
"Detector tilt: the free fit walked to {:.4f},{:.4f} deg, {:.2f} deg from the "
"{:.4f},{:.4f} deg this pass started at - further than a mounted detector is off "
"square by - which moves the far corner of the detector by {:.0f} px against an "
"integration disc of {:.1f} px, so the lattice was refined again at the starting "
"tilt with the beam centre free to take the {:.0f} px shift the tilt is equivalent "
"to, and the two were judged on the spots they index: {:.0f} at the walked tilt "
"against {:.0f} at the starting one",
tw.tilt_rad[0] * 180.0 / PI, tw.tilt_rad[1] * 180.0 / PI, walk_rad * 180.0 / PI,
start_rot1 * 180.0 / PI, start_rot2 * 180.0 / PI,
walk_rad * corner_px * corner_px / lever_px,
rx.experiment_.GetBraggIntegrationSettings().GetR1(), walk_rad * lever_px,
tw.spots_walked, tw.spots_held);
if (tw.refused)
logger.Warning("{} - the spots cannot tell the two apart, the walked tilt measured "
"nothing, and the pass runs at the tilt it started at", what);
else
logger.Warning("{} - the spots confirm the walked tilt, and the pass runs at it; the "
"file misdescribes this detector's tilt", what);
}
void Rugnux::FirstPassRun::MeasureSpotBudget(const FirstPass &best) {
// The spots in hand were already cut to the budget in force, so the measurement can only
// shorten a budget, never lengthen one - which is why RunAllPasses hands the second pass the
// list the first one started from rather than the list the first one settled on.
const int64_t budget = MeasureBudget(*indexer, *best.result);
// The measurement can honestly land under MIN_SPOT_COUNT, a floor written for a
// stills FIT, where fewer spots cannot pin a lattice down. Here the lattice is
// already in hand and merely tested - but the validation gate still needs
// ViableCellMinSpots indexed spots per frame (AnalyzeIndexing), so a shorter list
// could never index anything: keep the floor's worth of spots rather than die on
// the setter. A budget that shallow is a finding in its own right - the lattice and
// the file's geometry explain only the brightest few detections (measured on a
// header detector distance 20% off, compensated by a uniformly scaled cell that
// only matches near the beam centre) - so it is warned about, not just adopted.
const int64_t adopted = std::max<int64_t>(budget, MIN_SPOT_COUNT);
if (budget > 0 && adopted < rx.experiment_.GetMaxSpotCount()) {
if (adopted != budget)
logger.Warning("Spot budget: only the strongest {} spots per image lie on "
"the lattice (measured on {} frames) - fewer than the {} the "
"frame gate can act on, so {} are kept. A budget this shallow "
"usually means the geometry in the file (detector distance, "
"beam centre) or the lattice is wrong, not that the crystal "
"is weak", budget, static_cast<int>(validation.size()),
MIN_SPOT_COUNT, adopted);
else
logger.Info("Spot budget: keeping the strongest {} spots per image "
"(measured on {} frames; was {})", budget,
static_cast<int>(validation.size()), rx.experiment_.GetMaxSpotCount());
rx.experiment_.MaxSpotCount(adopted);
}
}
// Second pass: compare the de-novo lattice with pass 1's HERE, before integrating every image
// with it. A markedly larger cell is the bistable supercell collapse, which was already
// rejected in Run() - but only after the whole pass had run, so it cost an entire extra pass
// on a dataset that ended up on pass-1's lattice anyway. Take pass-1's result now instead:
// the same remedy, one pass earlier.
void Rugnux::FirstPassRun::Pass1SupercellGuard(const FirstPass &best) {
// Primitive volumes, like the scheme comparison above: a centred conventional cell is an
// exact integer multiple of its primitive one, so two settings of the SAME lattice differ
// by that factor and comparing the conventional cells reads a mere change of setting as a
// supercell - which then forces pass 1's setting and, with it, its lower symmetry.
const double v1 = std::abs(rx.prepass_result_->lattice
.ToPrimitive(rx.prepass_result_->search_result.centering).CalcVolume());
const double v2 = std::abs(best.result->lattice
.ToPrimitive(best.result->search_result.centering).CalcVolume());
const bool supercell = v1 > 1.0 && v2 > 1.5 * v1;
// Dropping to the primitive sub-cell of pass-1's centred lattice is the same
// disagreement seen from the other side, and the volumes above cannot see it: the two
// have the SAME primitive metric. Integrating there never predicts the reflections the
// centring implies, and the pass then merges in P1 (measured on a C-centred monoclinic
// crystal). Only this direction - a centring pass 2 finds where pass 1 had none is the
// refined geometry doing its job, and the intensity test in the search confirms it.
// Same primitive metric only: a pass 2 that lands on a primitive cell several times
// smaller is a different lattice - pass 1's supercell corrected - not a dropped centring.
const bool lost_centring = rx.prepass_result_->search_result.centering != 'P'
&& best.result->search_result.centering == 'P'
&& v1 > 1.0 && std::abs(v2 / v1 - 1.0) < 0.1;
// ... unless this pass's own axis-harmonic arbiter settled the larger cell on the spots.
// Pass 1 then never asked that question - its schemes did not tie on frames, so the
// arbiter did not run - and its smaller cell is the axis harmonic, not the reference
// (measured: a doubled axis the deposition confirms, thrown away here).
rx.supercell_settled_ = supercell && best.larger_settled;
if (rx.supercell_settled_)
logger.Info("Two-pass: that lattice ({:.0f} A^3) is {:.2f}x pass 1's ({:.0f} A^3), and "
"this pass's axis-harmonic arbiter established the larger cell on the "
"spots - keeping it", v2, v2 / v1, v1);
else if (supercell || lost_centring) {
logger.Info("Two-pass: that lattice ({}-centred, volume {:.0f} A^3) disagrees with "
"pass 1 ({}-centred, {:.0f} A^3) - integrating with pass-1's lattice instead",
best.result->search_result.centering, v2,
rx.prepass_result_->search_result.centering, v1);
RotationIndexerResult forced = *rx.prepass_result_;
if (rx.prepass_rotation_scale_ && forced.axis)
forced.axis = ScaleRotation(*forced.axis, *rx.prepass_rotation_scale_);
indexer->ForceRotationIndexerResult(forced);
}
}
Reference in new issue View Git Blame Copy Permalink