A run can make the same first pass several times - the canonical pass and the probes at the
geometry it stands at - and each of them that carried a near tie left its own re-check. The first
pass is deterministic in its inputs (FirstPassInputKey, the key the first-pass memo already trusts),
deciding late as deciding early, so a re-check whose pass read the same inputs (and the same
pre-pass flag) as an earlier one takes that one's outcome instead of running again.
Measured on 4p8r (8 re-checks, 5 on inputs of their own): re-checks 18.7 -> 13.5 s, p.hkl
identical. A diagnostic build that runs the skipped re-checks anyway and compares is used to
check the outcomes agree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Several pipeline copies of one run build FFT indexer pools on the card at once - the
rotation-scale walk's indexing probes start as the output pass begins its image loop,
and the speculative geometry probe runs beside the merge - and each indexer is a cuFFT
plan plus 0.6-0.9 GB of buffers. Construction had no admission control: a cufftPlanMany
or device allocation that found the card full failed straight through, and since
IndexerResult carries the exception that ends the run. With 7.2 GB of the card held by
another process, a small-molecule rotation set failed 4 of 4 runs this way although
each pass fits on its own.
Construction now does what the merge's allocations do: hand back the pool's unused
memory, wait for the work beside it to end, and build again. Who waits for whom is
fixed so that nothing waits in a circle:
- the main line waits for the probes (GPUWorkBeside, as the merge already does);
- a probe waits for the main line's image pass (new GPUImagePass), never for a probe,
itself included. The image pass waits for no probe, so it ends.
The walk's indexing probes now hold a GPUWorkBeside too (they are exactly that work),
and a run carries gpu_work_beside_ so its pools know which side they are on. Only when
an indexer is built changes, never what it computes; with nothing to wait for it still
fails at once.
Same run with the 7.2 GB hog: 5/5 complete (also at 7.6 and 8.0 GB), p.hkl identical
to the unhogged run; a card that cannot hold the run still fails in 0.5 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The demoted arm of the metric-symmetry check integrated and post-refined the
whole sweep even where, released from the equality, its first pass ended on
exactly the lattice the promoted class indexes. It then integrates the same
reflections the canonical pass does, realises the same held-out residual, and
the promoted class stands on the tie - 4nwv spent 14 s to report "I-centred
orthorhombic does not beat I-centred orthorhombic".
The promoted class's first pass now runs first as an indexing probe (memo
cleared, so it always records its outcome), and the demoted arm stops right
after its own first pass when that ends on the same first-pass outcome string
(lattice vectors to the last bit, path, class, geometry, spot-finding state).
No comparison is made and the canonical pass runs as before. The decision is
unchanged by construction; the geometry the arm used to hand on is the
promoted pass's own fit in that case either way.
Output-identical: p.hkl md5 and MTZ arrays equal on all 34 metric-check and
scale-walk sets of the battery (31 open/in-house + 3 private) against
56bad5fdb. The skip fired on 5 sets; metric-check step time 4nwv 14.0 -> 2.9 s,
7kcn 6.9 -> 1.6 s, in-house lysozyme sets -1.3..-1.7 s; the promoted probe
costs ~0.1-0.8 s on the sets where the lattices differ. Battery metric-check
step total 129 -> 107 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A first pass now acts on its validation frames at once (as before part B). Where
it decided a near tie that later frames can re-check, the copy of the run taken
as that pass began is kept; after the run - when every frame is in, live too, and
nothing else holds the card - the pass is run again on it as a probe deciding
late, and its outcome (lattice, path, centre, spot settings, carried centres) is
compared with the early one. Only where a re-check ends elsewhere is the run made
again, every near tie decided late: the same answer as deciding late throughout.
A run that found no lattice after carrying a near tie is made again likewise.
Live, no near tie holds the run up. In a simulated collection (frames readable
at 10 ms each, 3 repeats) the pass-1 first pass ends when the base's does (9.8
vs 10.1 s, 13.9 vs 14.0 s, 15.1 vs 15.6 s) where deciding late waited for the
end of the collection (17.9, 21.7, 23.5 s); the result follows the last frame by
+1.0/+4.6/+14.8 s against the base's +0.6/+4.1/+13.3 s (deciding late: +2.3/
+10.3/+23.1 s) - the difference is the re-checks. A set whose near tie overturns
pays a second run (+25 s on one).
Validation (78 open+inhouse sets incl. the ci tier, 14 private): p.hkl identical
to deciding late throughout on all 92 (one private set compared at this base,
the rest at e39e40f95, from which this base is output-identical on them);
identical to the base except the 3 sets where deciding late changes it (one R32
set, R_meas 86.6 -> 82.2%; two unscored small molecules). Re-checks ran on 50
sets (135 first passes; replayed deciding early, none reports a difference); the
run was made again on 6 (4 overturns, 2 no-crystal controls). Re-check time on
the other 46: median 0.9 s (7.6% of the run), at most 18 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a first-pass decision is a near tie (the margin within 2 sigma of its own
noise, the existing "near tie" lines), the same comparison can be made again on
the 60 validation frames plus 60 frames equally spaced over the rest of the
sweep (late_validation). A gate set at a share of the validation frames is
applied at the same share of all 120; a lattice that indexes none of the later
frames (a crystal that left the beam) is judged on the validation frames alone.
Decisions that are not near ties never read those frames; sweeps the pre-pass
already covers have none.
Carried: scheme choice (also in the ladder's and rescues' scheme comparisons),
the axis-harmonic arbiter, the metric-symmetry demotion gate, the sub-lattice
majority gate (only where it decides: on 74 firings the sub-lattice already
indexed fewer frames than the standing cell) and its pooled comparison, the
beam centre's pooled-spot comparison, the pass-1 fallback, the long-axis
rescue (different lattices only) and the ladder's absolute bar. Each
"near tie decided late" line says whether the later frames overturn the early
decision.
Validated deciding every near tie late, against rc175 a897b08a7 (all 60
open+inhouse sets that log a near tie plus the 28 ci sets; the 14 private
sets that log one): no verdict changes; p.hkl identical on 76/78 and 13/14.
Changed: one R32 set (R_meas 86.6 -> 82.2%, same space group, cell, d_min), one
unscored small molecule (merging statistics unchanged), one unscored private
small molecule (ISa 2.2 -> 17.8, R_meas 37 -> 4.8%, through the metric-symmetry
demotion; with the demotion not carried it is identical to the base).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A pass that finds no lattice carries score -1. That sentinel was printed as "-1/60 validation
frames" and compared as a count: the axis-sign rescue adopted the opposite sign when it scored 0
frames against the file's "no lattice" (0 > -1), and the long-axis rescue could return a re-index
that indexed nothing over a standing pass that found nothing.
Both rescues now count a missing lattice as 0 frames, so an alternative needs at least one indexed
frame to win. The log says "no lattice" (Frames()) in the axis-sign, beam-centre, long-axis and
rung lines and in the near-tie lines, whose margins also count it as 0 (log only). On the base
binary's full open+inhouse battery no set adopted a sign or a long-axis re-index against a
lattice-less pass, so no result moves.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A CUDA out-of-memory in an indexer worker was turned into an IndexerResult carrying only the
message string. The rotation indexer then returned no lattice, the first pass treated that as a
genuine negative and the rescues acted on it: in a stress run (two rugnux on one card) a run had
both first-pass schemes fail with out-of-memory, the axis-sign rescue compared 0 frames at the
flipped sign with the -1 "no lattice" sentinel, adopted the flipped sign, and the beam-centre rescue
then moved the centre - a failure of the machine used as evidence about the crystal. The CUDA
category was lost on the way, so the fail-fast check (IsFatalResourceError) never saw it.
- IndexerResult::error is now the std::exception_ptr the worker caught. RotationIndexer::RunIndexing
rethrows it before touching any state, so nothing is memoised for a failed run and the exception
keeps its category: rugnux stops with the CUDA message at once (verified with a forced indexer
failure: exit 1 after 1.9 s, "CUDA (GPU) error (...)"). FirstPass::indexer_error,
RotationIndexer::GetIndexerError and RequireLattice's machine-failure branch are gone.
- The beam-centre arm in RunAllPasses rethrows fatal resource errors like its sibling catches; it
had caught the out-of-memory and carried on at the file's centre.
- IndexAndRefine keeps per-frame indexer failures apart from frames that did not index (count +
first exception). The frame stays "not attempted" as before; the receiver logs the count and the
first error at the end of a collection (no cancel), rugnux's image pass and stills geometry
refinement rethrow it and stop. A rotation-indexer failure inside IndexAndRefine (online rotation
indexing, FinalizeRotationIndexing) is recorded the same way, so a live collection is not
cancelled by it.
- A failed device allocation or cuFFT plan now says how much was asked, how much of the card is
free, and - when the card is merely full - that another program is using the GPU or the data set
needs more than the card has, with the CPU remedy; previously "Failed to allocate device memory
(out of memory)" / "Failed to create CUFFT plan with cufftPlanMany".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first-pass spot engines (FirstPassRun::SpotEngine, SpotFindingWorkers::Engine)
were given back by move-assigning a fresh struct over them. A move-assignment goes
in declaration order, so it freed raw_image - whose bytes the analysis engine had
page-locked with cudaHostRegister (PinInputRegion, every uncompressed/CBF frame) -
and only then destroyed the engine, which unregistered the already freed range.
In between, the address may be handed to another thread's buffer. Measured in a
standalone test on this card and driver: a copy into a new buffer at a stale
registered address goes to the old pages (the data never arrives, no error), and
cudaHostRegister / cudaMallocHost on it fail with "already mapped"; eight threads
freeing in this order made unrelated threads' ordinary pageable copies return
wrong data, the correct order none. The engines now release themselves engine
first, buffer last.
The image pass had the same hazard where the writer still holds an image: the
worker read the next frame into a new one and dropped the old one, which the
writer then freed while the worker's engine still had it registered. The old
image is now held until the engine has pinned the new one.
This is the most likely cause of a sticky copy-engine fault (Xid 31, CE write to
an unmapped address) that ended a rotation run's scaling with "unspecified
launch failure": the speculative geometry probe released its spot engines on
many threads at once while the merges beside it were copying. That failure was
not reproduced in 9 reruns before the fix, so the link is inferred, not shown.
Merged data unchanged (p.hkl md5 and MTZ data identical to rc175 on 10 sets;
p.hkl identical with --write-process-h5 on a CBF set).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Where the first pass recognises two twin domains on their coincidence lattice (reticular
merohedry, index n = 2 or 3) and adopts the stronger domain's cell, the 1/n of that domain's
reflections that lie on the other domain's lattice too carry both domains' intensity. The step
now records the overlap as a lattice vector t of the standing cell (h . t divisible by n), the
rotation result carries it, and the per-image integration drops those reflections. t is a
real-space vector, so it holds through every later change of basis; it is read in sixths so a
centred conventional cell (F, I, C, R) is exact. Each unique reflection keeps its equivalents
that the other domain does not overlap; only those whose every equivalent is overlapped are lost.
The two-pass quality guard compared signal counts between a header pass that integrated
everything and a refined pass that left the overlap out, and sent the run back to the
composite lattice. A pass that left an overlap of index n out now has its counts taken up by
n/(n-1), and its axial rows are not compared with a pass that did not.
Nothing changes where the step does not fire: the field stays empty, the factor is 1.
Measured (merging the existing outputs with and without the class H.(111) = 0 mod 3, against
removing a random third): spinel-twinned F-cubic private set R_meas 25.5 -> 14.3% (random
26.7%), insu_I_x06da_low_isa 24.1 -> 16.6% (random 24.2%). With this change, end to end:
- insu_I_x06da_low_isa: I23, R_meas 24.0 -> 12.7%, ISa 5.2 -> 28.8, CC1/2 0.997 -> 1.000,
completeness 99.8 -> 95.7%, d 1.45 -> 1.47 A, model R-free 0.260 -> 0.246.
- private F-cubic twin: F432, R_meas 25.5 -> 6.3%, ISa 3.9 -> 20.4, completeness 100 -> 93%,
d 3.23 -> 3.77 A (XDS reference 4.0). 432 now passes the added-operator gate on the
all-observation merge (contrast 0.75, bound 0.72) instead of being refused there (0.62) and
rescued by the twin-immune zone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact: reflection data identical to the base on 51/51 open+inhouse and 27/27 private sets. Diagnostics only;
the full battery collects their firing counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
# Conflicts:
# rugnux/RugnuxFirstPass.cpp
# rugnux/RugnuxFirstPass.h
A crystal whose spots avoid one coset of an index-2 sub-lattice thins the overlap and the class on
neither alike, so comparing only the weaker domain's own class with the class on neither nearly
fired on a single long-axis hexagonal crystal (stopped by the congruence check alone). The class on
neither must now be four times emptier per node than the emptiest of the weaker domain's own class
and the overlap. On the evaluated private, inhouse and open sets the non-twins score at most 1.44;
the two known index-3 twins score 61 and more.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A crystal twinned by reticular merohedry of index n - obverse/reverse
rhombohedral domains, or a cubic crystal twinned by a 60 deg rotation about
[111] (the spinel law, index 3) - has its two domains' reflections on a
common lattice of n times the primitive volume of either domain's own cell.
That coincidence cell indexes more frames and more spots than the domain's
cell, so the first pass adopted it (a hexagonal/orthohexagonal or R:R cell
for an F- or I-cubic crystal), and nothing downstream could recover the
true lattice or Laue class.
A new step after the sub-lattice hypothesis (TwinCompositeHypothesis), for
n = 2 and 3, counts the validation spots on the standing cell in the four
classes of the pair of index-n sub-lattices whose union holds the most
spots: on both, on each alone, and on neither. Two domains leave the
(1 - 1/n)^2 of the nodes on neither empty but for chance coincidences; one
crystal, whatever sub-lattice a pseudo-translation favours, populates the
nodes off the stronger sub-lattice alike whether they lie on the second
sub-lattice or on neither. The step adopts the stronger domain's cell when
the two sub-lattice cells are congruent (2 % / 2 deg), the weaker domain's
own class holds at least 20 spots at more than four times the per-node density
of the class on neither, and the domain's cell indexes a majority of the
validation frames. The other domain is left to the leftover-lattice census.
Evaluated on the private, inhouse and a twin/tNCS/rhombohedral/cubic/
long-axis selection of the open arm: it fires on the two known index-3
twins (a spinel-twinned F-cubic crystal, now indexed F-cubic; the in-house
I-cubic insulin set with a 60 deg [111] second domain, R3:R -> I23) with a
weak/neither per-node density ratio of 60 and more (the class on neither
holds 0-8 spots against 250-2850 on the weaker domain), and nowhere else.
On every other set the ratio stays between 1.0 and 1.6, the top of that
range only on runs with a hundred or so validation spots, where the pair
picked as the emptiest of 7 or 13 forms carries the scatter; hence the
factor 4. Both twins now merge in their cubic group, and the sets
re-run against the release baseline otherwise did not change. Cost when it
does not fire: tens of milliseconds per first pass.
Merges two prototypes: the class census and n = 2 coverage of one, the
sub-cell congruence requirement of the other; no environment switch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Two candidates that both index fewer than a sixth of the validation frames (the bar the run keeps
a lattice on) are not two hypotheses about the crystal; on the first 34 battery sets most scheme
"ties" were 0/60 against 0/60. Log only, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Part B takes the decisions before pass 2 from the first part of a rotation sweep. Two diagnostics
now say, in the log only, where that may matter; nothing the run writes changes.
Far-end check (RugnuxFarEnd.cpp), after the image pass of the canonical pass: the sweep in 10 deg
blocks, six per-frame quantities pass 2 already holds - frames indexing on their own, share of
spots on the lattice, the cell scale the strong reflections ask for (d at the observed centroid
over d predicted), the residual rotation of each frame's refined orientation against the sweep's
lattice turned to that frame (about the spindle and off it), and the background - each late block
against the early blocks as a Student-t prediction from their own spread, family-wise at the run's
0.1 % significance over every late-block comparison. One line: "far-end check: early decisions
hold" or "far-end check: <decision> would be revisited because <evidence>".
Near ties (FirstPassRun::NearTie*): where a first-pass decision compares two candidates or a count
against a gate - scheme choice, metric-symmetry demotion, axis-harmonic arbiters, axis sign,
long-axis rescue, beam-centre gates and choices, end-wedge gate, short-axis and sub-lattice
hypotheses, ladder floor/adoption/rung choice, pass-1 fallback, no-crystal test - a margin within
two standard deviations of the comparison's own noise (binomial on validation frames, Poisson on
spot counts, the binomial standard error ValidationEvidencePrefers uses on pooled spots) is logged
as "near tie: <decision> <A> vs <B> margin x (noise y) - would carry both".
ValidationEvidencePrefers now reads its difference and standard error from
ValidationEvidenceDifference, which the near-tie lines share; same arithmetic.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The two-pass supercell guards (in the first-pass indexing and after the second
pass) revert to pass 1's lattice whenever the second pass's primitive cell is
markedly larger. On a crystal with a doubled axis pass 1's schemes did not tie
on frames, so pass 1 never asked the axis-harmonic arbiter and kept the halved
cell; pass 2's schemes tied, its arbiter established the larger cell on the
spots (17048 against 16099 validation spots), and the guard then threw it away.
The scheme comparison now records when the lattice it keeps is the larger cell
of an axis-harmonic pair the arbiter settled, and both guards leave such a
lattice in place. Where the arbiter did not settle the larger cell - every other
set the guard fires on - nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Pure refactor, no behaviour change. The 2460-line function with ~25 lambdas
becomes Rugnux::FirstPassRun (RugnuxFirstPass.h): the state the steps share
(spot cache, validation frames, schemes, committed spot-list state, the
speculative passes) as members, and one member function per step taking
and returning the standing lattice (FirstPass). FirstPassRotationIndexing is
now the ordered list of decisions:
PrefetchCertainFrames, IndexSchemes (+ centre/short-axis speculation),
AxisSignRescue, BeamCentreCheck (IndexAtMeasuredCentre,
CentreWhenFileFails, CentreByPooledSpots, CentreWhenBothIndex,
CentresDisagree), LongAxisRescue, AddEndWedge, ReleaseSpotEngines,
RequireLattice, ShortAxisHypothesis, SubLatticeHypothesis, Pass1Fallback,
last-resort FirstPassLadder, MeasureVerdict, RejectChanceLattice,
AdoptLattice, StartLeftoverCensus, ReportTiltWalk, MeasureSpotBudget,
Pass1SupercellGuard.
Files: RugnuxFirstPass.cpp (order + shared machinery: spots, scoring,
schemes, ladder), RugnuxFirstPassRescues.cpp, RugnuxFirstPassAccept.cpp.
Bodies are moved verbatim (Rugnux members reached through rx.); every log
line is unchanged. Literal duplicates folded: the two MAX_SPOT_KEYS
constants, and the feed-every-scheme / run-every-indexer loops of the
long-axis rescue and the short-axis fallback (now FeedSchemes/RunIndexing,
the same calls in the same order). The leftover census, which outlives the
pass, captures what it needs explicitly instead of [&].
Verified against the rc175 0b2dde214 binary on 16 sets (7kcn, 8v4o, 5nw5,
9rci, 6z9g, 9min, nothing_1/2, 7atg, myob/cytc_x10sa, 8a1a, kdp, insu low
ISa, 6p8j, 9qw8) and -N 8 on 7kcn/8v4o/9rci: p.mtz md5-identical, logs
identical as multisets apart from timings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With two cards or more, the speculation's spot-finding workers took card t % cards, so some of them
shared card 0 with the first pass they run beside. They now take the other cards, worker t on card
1 + t % (cards - 1): on two cards all of them on card 1, as before the previous commit, and on more
cards spread over cards 1..n-1. Spots are a pure function of the frame, so the card cannot change
them; on one card the speculation does not start. Not measured on more than one card.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The speculation that spot-finds and indexes at the measured beam centre beside the first pass
(only with two GPUs or more) put all its spot-finding workers on card 1. The file centre's spot
finding ends long before the speculation's, so the speculation is what the first pass waits on
(T4 server, kdp: main pass spot finding done at 1.3 s, speculation at 5.6 s). Its workers now
take card t % cards, as the first pass's own workers do; worker t builds and frees its engine on
the same card, since the destructor's OnWorkers maps t the same way.
Spots are a pure function of the frame, so which card finds them cannot change them. The
prototype measured kdp 25.9 -> 23.6 s on the 4-card T4 with md5 unchanged on 5 sets. On a
one-card machine the speculation does not start (get_gpu_count() < 2), so nothing changes there:
p.mtz md5 identical to rc175-next 41fab6156 on myob_x10sa, cytc_x10sa, thau_x10sa_16keV, 8a1a,
11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N, -N 8 and CUDA_VISIBLE_DEVICES=0 (5080
workstation). The multi-card gain and the 2-card risk (speculation workers now share card 0 with
the main pass) are not measured here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
79ec61c14 asked the 'end wedge' scheme inside pick_best, so every pass the
axis-sign, beam-centre and long-axis rescues run carried it, and its count went
into the pass those rescues are gated on. A consecutive wedge indexes part of
the validation frames at the WRONG rotation-axis sign as well: on 7kcn (CBF,
P4_122, the file's axis sign wrong for pass 2) the spread and the wedge gave
0/60, the end wedge 33/60, which lifted the pass over the sign rescue's
majority gate - so the flip, where the spread indexes 60/60, was never asked.
Pass 2 then integrated on the end wedge's lattice (33-49/60 frames): ISa 11.8
-> 4.7, R_meas 7.9 -> 15.3 %, wall 27 -> 43 s. The same mechanism let it
defend a file beam centre the measured one beats on 6p8j.
The end wedge is now asked once, after the rescues, at the geometry they
settled on, and only where nothing has indexed a majority; it is judged against
the standing pass as before. It can no longer displace a pass that indexes a
majority, and the passes the rescues run no longer pay for it.
Battery subset vs e99dd4949 (GPU): 7kcn back to 7c9a6c (p.mtz md5-identical
to 7c9a6c, ISa 11.8, R_meas 7.9 %); 5nw5 still pass (P212121, d 7.10, ISa
8.9); 6p8j first pass back to 7c9a6c's (measured centre, spread 60/60).
p.mtz md5-identical to e99dd4949 on 3mc4, 5ky6, 5reo, 7atg, 8v4o, 9qw8, 9rci,
cuhf2, cytidine, myob_x06da_powder_1/2, myob_x06da_split/sparse,
insu_I_x06da_low_isa, kdp/lcystine/yag/thau inhouse sets, nothing_1 (no
lattice) and six private sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This reverts commit c5a13d212.
The second-domain test fixed its target (a low-ISa insulin set) but was
evaluated 1132 times on 212 battery sets and adopted a "second domain"
on 50 of them; three sets that passed went to fail (5j23, 9qw2 and one
private set: F432 -> P432). It never asks whether the other domain is a
different orientation, so near-ties at 54:46 decide it. A narrowed
version (regr2-domain) still costs about 120-200 s battery-wide for one
difficult set. It is parked for rc176, where near-tie domains are to be
carried as two hypotheses and decided late.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The arbiter between two first-pass schemes whose cells differ by an integer volume factor
adopted the larger cell when it accounted for more validation spots than the smaller one. Each
count is made on the cloud derotated through that scheme's OWN refined geometry, so the
comparison is also one of two geometries, and the scheme that refined on frames spread over the
sweep wins it whichever cell it found. With the decisions before pass 2 taken on the first part
of a rotation sweep (5b9144410), 'spread' samples 120 deg instead of 360 and on a crystal with a
very long axis it returned a 3x harmonic of the I-centred orthorhombic cell while 'wedge' found the
true one. The 3x cell accounted for more spots than the true cell - and none of its
spots lay off its (0,1,0) sub-lattice, so the class it adds was empty. It was adopted, and the
run merged in C2 at 3x the primitive volume (I/sigma and R_meas both far worse). On the full sweep
(before part B) the roles were reversed: 'spread' found the true cell and 'wedge' a 2x harmonic,
and the count, again won by 'spread', happened to point the right way.
Now the larger cell must also account for more spots than its own sub-lattice - the lattice with
the emptiest class's translation added - counted on the same cloud. On one geometry the two
cells differ only in the added class, which is the comparison the arbiter's design comment
describes. The added class is now measured on the primitive basis (a centred basis's own
absences read as an empty class), and both cells are counted on Niggli-reduced bases, since a
fractional-Miller tolerance on a skewed basis makes the sub-lattice miss spots it contains.
The log reports both counts. The ladder's harmonic check keeps its spot-count rule.
Measured (GPU, against rc175 7c9a6c1 + the OOM fix):
a large-cell I-centred orthorhombic sweep: the 3x C2 harmonic is no longer adopted; it merges
in I222 again, with I/sigma and R_meas back to the night before part B
6h2p_native first pass no longer takes a 3x cell with an empty class (24436 vs
18613 spots); final unchanged: C2221, d 1.32, ISa 19.75, R_meas 13.6 %
p.mtz md5-identical: 5nw5, 8v4o, 9rci, 5ky6, insu_I_x06da_low_isa, nothing_1, nothing_2,
myob_x06da_powder_2, 7atg, 9qw8, 5reo, 6ze4, 9w3y, 6z9g, cuhf2, lcystine_x10sa_20keV,
myob_x06da_sparse, lyso_x06da_half_image, 8a1a.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Since the decisions before pass 2 read only the first part of a rotation sweep
(5b9144410), the first pass samples [start, PrepassEnd) with its two schemes:
'spread' (100 frames over it) and 'wedge' (consecutive frames from the start up to
the accumulation cap). Which orientations a sweep passes through first is chance,
and a sparse pattern can open on frames that do not show the long axis: on 5nw5
(P212121, 92 x 170 x 390 A, 10-25 spots a frame, 0.1 deg frames) the spread over
the first 120 deg returned a C-centred 955 A cell at every density tried (60-200
frames: 0-10/60 validation frames) and the wedge a wrong one (0/60); the FFT never
offered the true cell. The run kept the 5/60 cell as a sparse lattice and merged
P2_1 at 2.43x the deposited primitive volume, ISa 2.7. The spread over 0-90, 30-150,
60-180 or 0-180 deg indexes 57-59/60, and consecutive frames from 60 deg on 60/60:
the second half of the first part carries the lattice.
Where the two schemes index no majority of the validation frames, pick_best now
also indexes 'end wedge' - the first part read back from its end up to the
accumulation cap - and judges it against them as any later scheme is (clearly more
frames, or the axis-harmonic arbiter). It stays inside [start, PrepassEnd), so
nothing before pass 2 waits for the rest of the sweep. It is not asked on a run the
schemes index: offered there unconditionally it only added a lattice for the
axis-harmonic arbiter, which took a 3x hexagonal supercell over an I-cubic crystal's
cell on a spot count. The measured-centre arm of the beam-centre check takes it the
same way, whether its schemes were indexed speculatively or not.
Battery subset vs rc175 7c9a6c (GPU): 5nw5 fail -> pass (P212121, vol 0.996, d 7.04,
ISa 8.42, R-free(model) 0.462; baseline 7.06 / 7.76 / 0.464); myob_x06da_powder_2
d 2.26 -> 1.47 A (baseline 1.47), ISa 2.11 -> 2.34; p.mtz md5-identical on 8v4o,
9rci, 5ky6, insu_I_x06da_low_isa, nothing_1, nothing_2, 7atg, 9qw8, 5reo.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The first pass can only choose among the cells the FFT builds, and two
lattices seen together - a twin domain, or a second crystal whose reciprocal
lattice shares an index-2 sub-lattice with this one's - build the cell that
contains both, twice the volume of either. That cell indexes both domains'
spots and wins every comparison of FFT candidates; the true cell is never
offered, so nothing downstream can pick it.
Each first pass now scores the standing cell's seven index-2 sub-lattices
(same orientation and geometry) on the validation frames. The frame count
only nominates (a sub-lattice must index a majority of the frames and no
fewer than the standing cell); the decision is the pooled-spot comparison
already used between a cell and its axis harmonic (ValidationEvidencePrefers,
each lattice against its own wrong-spindle-angle null). A real superstructure
keeps its cell there because its added class holds reflections; a two-domain
union cell loses because it must place each domain's spots twice as finely
with an orientation fitted to both.
9rci (P1, deposited ~36/39/101 A): from rc173 on rugnux had merged a c-doubled
cell (b+2c, ~199 A; R-free vs model 0.57). The deposited cell was reached only
between 0f9f3a and the part-B commit (5b9144410), because pass 1's orthorhombic-
constrained post-refinement walked the tilt to -0.46 deg and the whole-sweep
re-index happened to land on the short cell there. Part B did not cause the
wrong choice; it removed that accident (rc175 with --prepass-fraction 1 still
gives the doubled cell). Now, on the early range at the file's geometry, the
(0,1/2,1/2) sub-lattice indexes 57/60 frames against 41/60 and puts 1915/5659
pooled spots on itself against 1191 (30 and 35 by chance): adopted. Result:
cell 35.90 39.17 100.78 98.6 90.3 89.9, R-free (model) 0.270, d 1.69 A,
CC1/2 0.988 (was 199 A cell, 0.569, 1.81 A, 0.952).
Battery against rc175 7c9a6c (GPU): p.mtz md5-identical on 5nw5, 8v4o, 5ky6,
insu_I_x06da_low_isa, nothing_1, nothing_2, myob_x06da_powder_2, 7atg, 9qw8,
5reo, 6ze4, 9min, 8v2t, lyso_x06da_ref, insu_H_x06da_twin, myob_x06da_split;
only 9rci changes (now the deposited cell; the verdict read pass before only because P1 matched P1). Cost: tens of ms per first pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
When neither the file's nor the background-measured beam centre indexes a majority of the
validation frames, the first pass chose between them on the pooled spots each centre's lattice
puts on itself over chance, and only then asked the long-axis rescue - at the chosen centre alone.
The lattices compared there are, by construction, not the crystal's, and the comparison is close
to a coin toss. With the decisions before pass 2 taken from the first part of the sweep (part B,
5b9144410), 8v4o (P6122, c = 545 A) tipped from the file's centre to the measured one, 3.6 px out
across the spindle, on 16.7% against 15.5%; the rescue there could not see the 545 A axis, and the
run shipped a P1 sub-cell (primitive volume 0.35x). At the file's centre the same rescue on the
same early frames indexes 60/60 (checked with --beam-center-check=off on the rc175 binary).
The rescue is now a lambda, asked at both centres in that branch so each centre is judged on the
best lattice it gives, and asked as before after the beam-centre check where that branch did not
already ask it. The early-frame principle of part B is unchanged.
8v4o: P1 2.62 A, CC1/2 0.920, R_model 0.553 -> P6122 2.12 A (pass), the file's centre kept.
p.mtz md5-identical to rc175 7c9a6c7d1 (GPU) on 5nw5, 5ky6, 5reo, 7atg, 9qw8, 9rci, cuhf2,
cytidine, insu_I_x06da_low_isa, kdp_x10sa_20keV, lcystine_x10sa_20keV/25keV, lyso_x06da_half_image,
myob_x06da_powder_2, myob_x06da_sparse, nothing_1, nothing_2 and the two private sets that reach
the same branch. 9rci (doubled-c alternative under tNCS) is a different mechanism: rc175 with
--prepass-fraction 1 also returns the doubled cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Two twin domains, or two crystals of one cell, each index every validation frame, so the frame
count the first pass picks its lattice by cannot tell them apart, and the FFT returns whichever
its sample favours. Since the decisions before pass 2 read only the first part of the sweep
(5b9144410), that sample is smaller and it can be the domain there is less of. On an I-centred
cubic crystal with a second domain at 60 deg about [111] pass 1 took the smaller domain; pass 2's
re-index then found the hexagonal P cell both domains share (3x the primitive volume), the
supercell guard handed it back to pass 1's lattice, and the run integrated the smaller domain
over the whole sweep. Every reflection the two share then carried the larger domain's intensity,
which broke the cubic 2-folds (operator CC 0.48 against 0.89 for the 3-fold about the twin axis),
the added-operator R gate refused 23 and the run merged in R3 on the cubic axes - the I centring
undeclared, primitive volume 2x.
Now the spots the chosen lattice leaves on the spread frames are indexed again, as the
leftover-lattice census does, and where they give the same lattice (Niggli-reduced primitive
edges within 2 %) in another orientation, the two are judged on the validation spots each puts
on itself over a wrong spindle angle (ValidationEvidencePrefers); the larger domain is kept. It
is decided on the same early frames, so part B's principle stands.
Measured (GPU, battery, against rc175 7c9a6c1):
insu_I_x06da_low_isa R 3:R, d 1.59, R_model 0.315 -> I 2 3, d 1.43, R_model 0.226 (= pre-B)
5reo adopts the larger domain (86.5 % against 68.4 % of validation spots):
ISa 23.0 -> 34.3, R_meas 23.6 -> 23.3 %, R_model 0.205 -> 0.202,
d 1.64 -> 1.69 A, still C2
5nw5, 8v4o, 9rci, 5ky6, 7atg, 9qw8, myob_x06da_powder_2, nothing_1, nothing_2: p.mtz
md5-identical (the check ran on most of them and kept the lattice).
Result-changing only where a second domain is adopted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The first pass's last gate - is the lattice this crystal's at all - compared the validation spots
on the lattice with the median of five displaced-spindle-angle nulls, against the null's binomial
noise floored at one spot. On the two in-house no-crystal controls it is fed a handful of genuine
stray reflections from something rotating with the sample (the same detector positions, a pair
symmetric about the beam 198 frames apart, in both sweeps, offset by the 15-frame start difference),
plus one reflection near the spindle axis that stays in diffraction for ~46 frames. With the
decisions taken from the first part of the sweep the 60 validation frames are 3 deg apart instead
of 6, so that one reflection is counted on three validation frames, and the gate passed on
4 spots of 151 (null median 0) and 7 of 72, where the whole-sweep flow had seen 3 and 3.
Now the evidence unit is the distinct Miller index - a reflection on several validation frames
is one piece of evidence - and the test uses every displaced angle's count: with no crystal the
recorded angles are one of six exchangeable arrangements, so the gate is the binomial tail of the
recorded arrangement's share at p = 1/6, at the same 0.001 significance (SPOT_BUDGET_SIGNIFICANCE_Z).
No threshold is added; the median and the one-spot floor go from the gate. The comparisons between
lattices (ValidationEvidencePrefers) are unchanged.
no-crystal control 1: 4 distinct vs 2 over 5 wrong angles, tail 0.009 -> no lattice (was P 1)
no-crystal control 2: 5 distinct vs 2, tail 0.002 -> no lattice (was P 1)
p.mtz md5 identical to rc175 7c9a6c7d1 (GPU) on 5nw5, 5reo, 7atg, 8v4o, 9qw8, 9rci, 5ky6,
insu_I_x06da_low_isa, myob_x06da_powder_2; cytidine and myob_x06da_split pass. The weakest real
lattice in the open/in-house/private battery logs has 117 spots against a null of 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The beam-centre check runs a second first pass at the centre measured from
the background, and reads the first pass's answer only once that pass is
over. With two GPUs or more, its spot finding (on the second card), scheme
feeding and FFT indexing now start beside the file centre's pass, on a
copy of the experiment at the measured centre with workers, a mapping and a
spot list of their own (SpotFindingWorkers). The check takes them only
where the experiment (rotation angles included), SpotFindingKey at the
measured centre and the spot budget are still what the run has there, and
scores the schemes on the run's own indexer as before; otherwise it runs
that pass itself as before. The spots at the two centres are not shared:
the spot list depends on the centre (ring background, resolution, budget).
With one GPU, and in the CPU build, nothing changes: on one card the two
passes share the card and the reader, and side by side they saved nothing.
FindFirstPassSpots is the first pass's per-frame spot finding, now shared;
ExperimentKey and SpotFindingKey take an experiment other than the run's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The ladder's rung says which spots let the FFT find the lattice; until now it
also fixed how deep the whole run reads each frame (the spot-finding floor),
so which floor's FFT happened to succeed decided the run's depth. On a hard
pattern that is close to a coin toss: with the decisions taken from the first
part of the sweep, a sub-angstrom crystal (7atg) tied at 60/60 on a 1.13 A
and a 1.60 A floor, kept 1.60 A for the run and lost half its validation
spots. Now the winning lattice is itself scored at each deeper floor, deepest
first, and the run reads to the deepest at which it indexes as many
validation frames as at its own. A contaminant past a floor costs the lattice
frames there and keeps the floor.
Measured (GPU unless noted), against c240690a1:
7atg ISa 21.0 -> 23.1, R_meas 5.4 -> 5.1 %, R-free(model) 0.220 -> 0.160
(whole-sweep decisions, af3ea9: 22.9, 5.2 %, 0.178)
myob_x06da_split reads to 1.92 A instead of 3.00 A: ISa 2.2 -> 2.7, R_meas 87 -> 78 %,
CC1/2 81 -> 70 %, R-free 0.382 -> 0.402 (a contaminated split crystal)
9qw8, 9zmu, 8v4o, myob_x06da_powder_1/2 (CPU): unchanged; 8v2t: p.mtz bit-identical.
Result-changing only where the ladder adopts a rung.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The pre-scan sample (beam-stop shadow, defective pixels, spot width, the background beam centre)
and the first pass's frames (the spread and wedge schemes, the validation frames, the prefetch) now
come from [start, PrepassEnd) - the part of the sweep the geometry pre-pass integrates - instead of
the whole sweep. Rugnux::PrepassEnd is the one place that range is defined, for the pre-pass and for
these decisions alike, so any change to the pre-pass range (such as a minimum rotation) moves both.
Stills keep the whole range.
With rugnux following a collection while it is written, nothing before pass 2 then waits for the
second part of the sweep: the pre-scan, both first passes and the pre-pass run while it is
collected, and pass 2 starts reading as soon as they are done. An offline run takes the same
decisions from the same frames, so the streamed result still equals the offline one by md5 (in-house
JUNGFRAU 4M thaumatin replayed at one data file per second, also at -N 8 and with the replay paused
and resumed): 2.4 s from the last data file to the result, against 8.0 s for the offline run.
Result-changing. p.mtz md5 at af3ea9cca -> here (with the 120 deg minimum of the previous commit),
GPU: myob de2ab820 -> 9bdd6916, cytc 3e5d2f91 -> 62e744a1, thau bc5e3de3 -> 3ec88c34,
kdp cf83ee5d -> cdc2c068, 8a1a 50ab659e -> 309ea9a8, 8qaw d3287922 -> c2849d52; CPU: myob 3f9d12d8 ->
18c7ac2f, cytc d30c9969 -> 40e17d97, thau 4cdbdd94 -> f14e52da, kdp f1d801f3 -> eb930714. Sweeps of
120 deg or less are unchanged by this commit (their early range is the whole sweep). Needs the
battery before it is merged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The goniometer rotation-scale walk ran between the geometry pre-pass and the
refined (output) pass, on the critical path: two indexing probes, at the stored
angles and at the pass-1 fit, each with its own first-pass spot finding.
Adopting a scale other than the stored one is rare, so the refined passes now
start at once at the stored angles, on a copy of the run, while the walk runs on
the run itself. The walk starts once the output pass has indexed, so its probe at
the stored angles finds that first pass already done (spot store, rotation-indexer
memo), and its probe at the fit takes the frames' spots from the store as well:
the spot store's key no longer carries the goniometer's angles, and the spindle
angle each stored spot carries is stamped again for the angles in hand, exactly
as DiffractionSpot::Export stamps it (verify_first_pass_memo still recomputes and
compares). The output pass waits for the walk's answer only before it writes
(the superseded flag); where the walk adopts a scale, or leaves the spot-finding
settings changed behind it, the copy's passes are thrown away and the refined
passes run again on the run itself, which is then exactly where the sequential
walk left it. A refit integrates and only follows a move off the stored angles,
so it first calls the copy off (a cancel flag of its own) and waits for it to
end: two integrations never share the card.
WHEN, not WHAT: the adopted scale and the surviving output pass's inputs are
those of the sequential flow. The console shows the check as a step only where a
scale is adopted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The body of RunPipeline's two-pass branch becomes the private member
FirstPassRotationIndexing(PipelineLocals &), unchanged (de-indented one level) behind the lines that
bind the locals it reads. Its one early exit, the indexing probe's `return result;`, becomes
`return true;`, and the call site returns the result there. The helpers only it reads (SameSpots,
IndexedMillerIndices, AddedClassOccupancy) move into the new file's anonymous namespace.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
