A centred half cell (I-centred orthorhombic, its centring class the added one) was compared to the
doubled cell's sub-lattices as if primitive, so its class was never read.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
On strong data a class at one or two percent of the lattice rises significantly (0.009 to 0.059 at 4
standard errors on a ci-tier triclinic crystal, which was then adopted doubled and failed), and a half
cell missing a few percent of its intensity still describes the crystal. The doubled deposition ends
at 0.35-0.40, those classes at 0.06.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A class that does not rise keeps the half cell whatever the doubled merge shows, so the doubled pass
is no longer run for it - on a ci-tier crystal (R-centred, class at 1-2% of the lattice) that pass ran
the card out of memory and failed the run. The doubled pass also runs without --model: placing the
model in a doubled cubic crystal's 776 A R-centred cell took twenty minutes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Doubled-cell hypothesis: the run now always merges the doubled cell under <prefix>_doubled first and
decides with both merges in hand. Adopted only where (1) the probe's class grows significantly with
resolution (three standard errors between the 20-8 A shell and the finest measured one) and (2) in
the doubled merge one pseudo-translation - half a cell plus eps, fitted over the class - modulates at
least half of the added class in every shell (FitDoubledClassTranslation: A sin^2(pi h.eps) + B per
shell, share 1 - B/mean). Where adopted, every <prefix>_doubled file takes the plain name and the
probed cell's file of that name moves to <prefix>_halfcell. Replaces the HALF_CELL_BREAKS_DOWN bound.
Measured on the forced doubled merges: modulated share per shell 0.80-1.0 out to 1.9 A on the crystal
deposited in the doubled cell; on the ones deposited in the half cell it falls to 0.01-0.34 by
2.1-2.8 A (the rising class with twinning), or stays at -0.3-0.48 at every resolution (three others);
the one class it fully explains does not rise (eps 0.1 of a cell, flat at 0.1 of the lattice).
Space-group search: the trim of a screw zone's largest absence exempted a violation "at its row's
mean" on the de-modulated intensity, so under a deep pseudo-translation a reflection at 0.015 of its
row (de-modulated by 1/0.002) could never be trimmed and alone decided the row against the screw.
The exemption now reads the measured intensity, at both sites.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Supercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own
limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the
SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice
reflection predicted on the same frame lies within r3, and each resolution shell is read against the
median of the six other parity classes - which a single doubling leaves empty - as an excess in units
of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z
beside it ("Supercell probe to the limit" in the log).
Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its
noise and at more than twice the strongest of them, in at least three shells), the run integrates and
merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along
the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's
merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both
(DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT
(the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the
class reaches a quarter of the lattice's intensity in the finest shell both are measured in).
The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a
triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed.
Measured on forced doubled-cell merges (rc175 b530c2d base): the open-arm crystal deposited in the
doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell
(odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals
deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at
8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low
resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the
neighbour correlation, the separation from the nearest lattice reflection after the r3 filter -
separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call.
Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no
longer advises -C with the doubled cell, which the indexer reduces back to the half cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact: reflection data identical to the base on 51/51 open+inhouse and 27/27 private sets. Diagnostics only;
the full battery collects their firing counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
# Conflicts:
# rugnux/RugnuxFirstPass.cpp
# rugnux/RugnuxFirstPass.h
Exact: MTZ data (gemmi) and p.hkl md5 identical to the base on 31 sets, -N 8 on 3; only the PASS= count in
p_report.txt drops where a pass is no longer run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
# Conflicts:
# rugnux/RugnuxScaleMerge.cpp
A crystal whose spots avoid one coset of an index-2 sub-lattice thins the overlap and the class on
neither alike, so comparing only the weaker domain's own class with the class on neither nearly
fired on a single long-axis hexagonal crystal (stopped by the congruence check alone). The class on
neither must now be four times emptier per node than the emptiest of the weaker domain's own class
and the overlap. On the evaluated private, inhouse and open sets the non-twins score at most 1.44;
the two known index-3 twins score 61 and more.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Where pass 1's rotation-scale fit is a step or more (STEP_BOUND, 1 %) off the
stored angles, the walk runs on its own before the refined passes instead of
beside an output pass that a refit calls off and throws away. Scheduling only:
the run ends where the sequential walk leaves it either way (the overturned
path already re-ran the refined passes on the run itself).
Exactness E: all output files identical to rc175 e9d4f2221 on 9gdj, 9qw8,
6ttn, lalanine, 9t6s, 2 runs each.
Note: "the walk will then always refit" does not hold. Of the four battery
sets with the fit >= 1 % off (9gdj 0.980, 9qw8 0.988, 9t6s 1.012, lalanine
1.012), the stored angles stand on 9t6s and lalanine, which now walk first and
lose the overlap of the walk's two probes with the output pass (~0.5-0.8 s of
probe time); 6ttn (0.998) is below the step and unchanged.
Wall (time -v, base -> new, rep1 / rep2, shared machine):
9gdj 99.6->109.7 / 107.4->101.4 (rep1 new run noisy)
9qw8 43.1->39.2 / 34.4->33.3
9t6s 18.1->18.2 / 18.9->18.5
lalanine 20.5->19.7 / 7.9->8.3
Pre-rebase run on rc175 0b2dde214: 9gdj 99.6->93.8 and 99.6->93.4.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Every round the geometry walk takes ends in a pass that writes the files
again, so the canonical pass before it spent its space-group search, merges,
analyses, model validation and files on an answer nobody reads. The walk's
first question needs only that pass's post-refinement and, for a move of less
than a step, two indexing probes. The speculative probe now runs both beside
the merge - at the fit's geometry as before, then at the geometry in hand,
which the walk otherwise indexed after the pass (a memo hit where the pass's
own first pass left one) - and GeometryWalkTakesRound reads the walk's answer
from the same first-pass memos the walk's probes then read. The canonical pass
asks it, without waiting, before its space-group search (stops there; not
with a _process.h5 open) and again at the superseded point before the
analyses. An answer not yet in leaves the pass to finish as before, so only
timing depends on when the probes finish; the walk itself is unchanged.
Exactness E (not F): merged data and all output files identical to rc175
e9d4f2221 on every walking set, 2 runs each; decisions identical. Fired:
before the search on 7ph1, 5m17, 8pqd; at the superseded point on 2xfw, 6zr0,
9t6s; on 6zqy the probes were not in by either point in these runs; 8xtf's
canonical pass is starved and never merges.
Wall (time -v, base -> new, rep1 / rep2, shared machine):
5m17 65.9->57.0 / 68.5->56.7
8pqd 49.8->43.0 / 50.0->43.1
7ph1 51.1->48.9 / 51.1->45.3
6zr0 31.1->22.7 / 24.4->21.1
2xfw 101.2->66.7 / 48.2->48.2 (rep1 base noisy)
6zqy 27.3->26.1 / 26.8->25.7
controls myob_x10sa, cytc_x10sa, 11if, kdp_x10sa_20keV, 8a1a: no change
beyond machine noise
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Where the free refinement splits a promoted equality beyond the integration
disc, RefinedPasses ran two probe passes (promoted, then demoted) and then
integrated the winner a third time as the canonical pass, although the
promoted probe and the canonical pass predict the same reflections. Now the
demoted arm runs first as a probe, and the canonical pass is the promoted arm
itself: it fits its post-refinement before the merge (lattice_arm_check_),
compares the held-out residual there with HeldOutResidualFell as before, and
stops before merging only if it loses, in which case the demoted class runs as
the canonical pass (same number of passes as before). A pass that does not
measure a post-refinement as canonical pass drops the one it fitted, and the
geometry the fits leave behind is the one the two-probe order left, so every
pass downstream starts where it did.
Exactness E: merged data and all output files identical to rc175 e9d4f2221 on
the 8 sets where the check runs (9sl0, 6yqf, 7atg, lysoI_micromax_pink,
myob_x06da_powder_2 promoted kept; 9t6s, 8v2t, lyso_micromax_pink demoted
wins), 2 runs each; the only difference is p_report.txt PASS= n-1 of n-1 where
the promoted class stands (one pass fewer is run). Decisions and residuals
identical.
Wall (time -v, base -> new, rep1 / rep2, shared machine):
9sl0 48.4->40.2 / 48.9->39.6
7atg 49.5->31.2 / 33.2->28.7
6yqf 29.4->25.1 / 30.0->39.6 (noisy)
lysoI_micromax_pink 23.2->13.0 / 14.5->13.2
myob_x06da_powder_2 13.5->12.0 / 12.6->11.5
losers 8v2t, lyso_micromax_pink, 9t6s: within +-2 s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact: P4 p.hkl identical to the base on 39 noise-subset sets; concurrent arms p.hkl md5 and MTZ data identical
to P4 on 12 sets at default -N and -N 8 (incl. 5 sets where the arms disagree). Side arm on card 2 with 3+ cards,
else on the last card when 2 x observations x 256 B fits its total memory (decided up front).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact: p.hkl md5 identical to the base on 2xfw, 6zqy, 2ygz, 8 small SMV sets, myob_x10sa, cytc_x10sa and -N 8.
SMV walkers 2xfw 48 -> 41 s, 6zqy 26 -> 23 s, 2ygz 32 -> 26 s; start-up to first log line 0.39 -> 0.18 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
set_gpu_blocking_sync() called cudaSetDevice() on every card, and since
CUDA 12 that creates the card's primary context: ~0.18 s per card on the
workstation (measured, RTX 5080), one after another, before WALL_TIME
starts. On the 4-card server that is most of the 1.73 s per run spent
outside WALL_TIME (vs 0.37 s on one card).
- The blocking-sync flag is now set with the driver's
cuDevicePrimaryCtxSetFlags, taken through cudaGetDriverEntryPointByVersion
(no link against libcuda, so a CUDA build still starts without a
driver). It creates no context; contexts created later carry the flag
(cudaGetDeviceFlags 0xc, as before; user CPU time unchanged, so no
spinning).
- Card 0's context is created by the existing cuFFT warm-up thread, beside
the file open and pre-scan; cards 1..N-1 get theirs on a new background
thread per card (create_gpu_contexts), as the pre-scan uses card 0 only.
- At exit the contexts are destroyed by one thread per card
(release_gpu_contexts, cudaDeviceReset) after the run has destroyed
everything that used them, instead of serially by the runtime's exit
handler. Microbenchmark: an explicit reset costs the same as the
implicit teardown on one card (175 ms), so no change on one card.
Which card does what is unchanged; only when contexts are made/destroyed.
Measured on the 1-card workstation (load 7-16, 2 repeats x 2 sets plus
2 earlier pairs on myob): process start -> first log line median 0.39 ->
0.18 s; last log line -> exit 0.18 -> 0.20 s (one context either way);
elapsed - WALL_TIME median 0.53 -> 0.37 s; WALL_TIME unchanged (context
creation now overlaps the input read). The multi-card gain (3 serial
context creations at start, 4 serial destroys at exit) cannot be measured
here. p.hkl md5 identical on myob_x10sa, cytc_x10sa, 2xfw, 6zqy -N 8;
all runs exit 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A crystal twinned by reticular merohedry of index n - obverse/reverse
rhombohedral domains, or a cubic crystal twinned by a 60 deg rotation about
[111] (the spinel law, index 3) - has its two domains' reflections on a
common lattice of n times the primitive volume of either domain's own cell.
That coincidence cell indexes more frames and more spots than the domain's
cell, so the first pass adopted it (a hexagonal/orthohexagonal or R:R cell
for an F- or I-cubic crystal), and nothing downstream could recover the
true lattice or Laue class.
A new step after the sub-lattice hypothesis (TwinCompositeHypothesis), for
n = 2 and 3, counts the validation spots on the standing cell in the four
classes of the pair of index-n sub-lattices whose union holds the most
spots: on both, on each alone, and on neither. Two domains leave the
(1 - 1/n)^2 of the nodes on neither empty but for chance coincidences; one
crystal, whatever sub-lattice a pseudo-translation favours, populates the
nodes off the stronger sub-lattice alike whether they lie on the second
sub-lattice or on neither. The step adopts the stronger domain's cell when
the two sub-lattice cells are congruent (2 % / 2 deg), the weaker domain's
own class holds at least 20 spots at more than four times the per-node density
of the class on neither, and the domain's cell indexes a majority of the
validation frames. The other domain is left to the leftover-lattice census.
Evaluated on the private, inhouse and a twin/tNCS/rhombohedral/cubic/
long-axis selection of the open arm: it fires on the two known index-3
twins (a spinel-twinned F-cubic crystal, now indexed F-cubic; the in-house
I-cubic insulin set with a 60 deg [111] second domain, R3:R -> I23) with a
weak/neither per-node density ratio of 60 and more (the class on neither
holds 0-8 spots against 250-2850 on the weaker domain), and nowhere else.
On every other set the ratio stays between 1.0 and 1.6, the top of that
range only on runs with a hundred or so validation spots, where the pair
picked as the emptiest of 7 or 13 forms carries the scatter; hence the
factor 4. Both twins now merge in their cubic group, and the sets
re-run against the release baseline otherwise did not change. Cost when it
does not fire: tens of milliseconds per first pass.
Merges two prototypes: the class census and n = 2 coverage of one, the
sub-cell congruence requirement of the other; no environment switch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The geometry walk re-integrates the whole sweep in every round, and on an
uncompressed SMV sweep that image loop is bound by memory traffic:
smv::ReadInto was the top function (33% of all user cycles on 2xfw), and
16 threads read 360 frames no faster than 1 thread. On Linux/macOS the
frame is now mapped and widened from the mapping, which drops the
read() copy of the whole file into a scratch buffer. Windows keeps the
ifstream read. Pixel values are unchanged; every pass benefits, not only
the walk.
Measured (RTX 5080 workstation, load 9-23 from other agents, 2 repeats,
base = rc175 0457cecaf, same flags):
2xfw WALL 48.15/48.33 -> 40.49/41.98 s, Geometry check 17.2/17.4 -> 13.6/13.8 s
6zqy WALL 26.10/.. -> 23.60/23.16 s, Geometry check 8.6 -> 7.1/7.2 s
2ygz WALL 32.05 -> 26.27 s
-N 8: 2xfw 50.09 -> 42.82 s, 6zqy 26.78 -> 22.97 s
peak RSS -0.5 GB (no scratch copy)
Microbenchmark, 360 2xfw frames, 16 threads: read 1.8-1.9 s, mmap 1.2-1.4 s.
p.hkl md5 identical to the base on every set run (2xfw, 6zqy, 2ygz, -N 8,
and the small SMV sets of the open arm).
Tried first and dropped: keeping the decoded frames in memory across the
walk's rounds. Exact, but filling 13.6 GB of fresh memory cost more than
the hits saved (2xfw round 1 +2.4 s, release +0.9 s, later rounds -0.8 s
each; CBF sets 5m17/7ph1 got slower), at +12.5 GB peak RSS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Testing every block against the other part's blocks fired on single odd blocks: a block's mean
varies with the orientation, which a handful of reference blocks cannot represent (one block of 36
on a control, a 0.04 deg residual at t = 23 on another). The rotation rate, a straight-line fit
per part, failed the same way, its standard error blind to the smooth orientation-dependent
wiggle, and it did not see the target it was added for (the residual there runs at the same rate
through both parts). Now each quantity is one comparison: the block means of the rest of the sweep
against those of the first part, Welch's t, family-wise at 0.1 % over the quantities compared.
Still log only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Two candidates that both index fewer than a sixth of the validation frames (the bar the run keeps
a lattice on) are not two hypotheses about the crystal; on the first 34 battery sets most scheme
"ties" were 0/60 against 0/60. Log only, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The projected histogram is sampled every 1/(2*max_length) of 1/d, so a row
of length L between max_length and 2*max_length folds to 2*max_length - L and
peaks there as sharply as a real row. At the default 500 A bound a 602 A axis
came back as ~400 A and a 701 A axis as ~299 A; the folded cell still indexed
59/60 and 37/60 validation frames (only the reflections whose index lands on
an integer of the folded period), so the long-axis rescue, gated on a poor
pass, never ran. The same fold also turned real >500 A lattice combinations
of shorter-axis cells into spurious shortlist rows.
Each shortlisted row is now asked once, by a direct Fourier sum over the spot
positions (no sampling, so no fold), whether the spots carry its own length or
its partner 2*max_length - L more strongly, and keeps the stronger. Partners
within 0.1 of an integer multiple are left alone (a harmonic, settled
elsewhere). Refinement accepts rows up to 2*max_length when one was unfolded.
Measured: a hexagonal P6122 c=602 A EIGER set and a P6522 c=701 A CCD set,
both indexed de novo on the folded cell before, now index the deposited cell
60/60 and merge to R-free 0.31 / 0.22 against the models (the same numbers as
-C with the deposited cell). Of 13 long-axis battery sets, 7 are bit-identical
(p.hkl md5) and 2 never unfold a row, 8v4o and 9yl4 have the same intensities (CC 1.0) without needing
the long-axis rescue, 4mtk refines a cell closer to the deposit at half the
wall time, 4nwv merges better (R_meas 0.58 -> 0.47, ISa 6.7 -> 8.4).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The first part can be the odd one (a crystal that settles or drifts over the opening degrees), so
every early block is also tested against the late ones. The rotation scale is also checked as a
rate: the slope of the residual about the spindle over the late blocks against that over the early
ones. Every far-end line now ends with each quantity's smallest p times the family size, so the
population can be read off the logs. Still log only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Stage A offered the b-unique two-fold (reference setting) on the lattice's holohedry alone, but held
its a- and c-unique siblings (pass-1 settings) to CellHostsRotations at 2e-3. On a cell refined free
~0.2 deg off orthogonal only the b-unique rung existed, so when the holohedral 222 was refused the
setting, not the data, chose the two-fold axis. Stage B already offers non-reference settings
without asking the cell, for the same reason; Stage A now does likewise where a holohedry is given.
With siblings of one order eligible together, ties were broken on the class CC alone, which a twin
law at fraction ~0.4 can win. A twin law cannot extinguish an axial row (every principal row maps to
itself or its Friedel mate), so ties now go first to the group whose pure two-fold axes carry the
more screw evidence on their own odd axial row - judged exactly as Stage B judges that zone - and
only then to the CC. Rows on 3-, 4- or 6-fold axes are not read.
Open arm: 6p8j (deposited P2_1, b = 152 A) was P 1 1 2 on the 88 A axis, now P 1 21 1 on the 152 A
axis (152 A odd row 158 nats, 46 A row 36, 88 A row refused; R-free 0.214). 9hnc and 5cc8 p.mtz
md5-identical (siblings offered but refused by added-operator R). On five passing P2_1 sets in an
orthorhombic metric the true axis carries all the evidence (offline replay), so no change there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The space-group search made its two P1 merges one after the other on one
engine: the Lorentz-filtered one, its search, then the all-observation one and
its search. Since every merge now starts from the same state, the
all-observation merge is made on a copy of the engine (CloneIngest: the host
half of the ingest copied, the device half staged on the copy's card), started
right after the ingest and read where the search needs it, so it runs beside the
P1 merge, the re-scaled search merge and the first arm. Its search then runs
beside the first arm's.
Placement: from three cards up the copy gets card 2 to itself (card 0 has the
main engine, card 1 the quality guard's). With one or two cards it shares the
last card, and only where two engines of this size fit the card's total memory
(DEVICE_BYTES_PER_OBSERVATION = 256; two engines measured at ~150 B/observation
each), so the choice never depends on what is free at the moment. On a 16 GB
card that admits sets up to ~32M observations; larger ones keep the serial
order. Nothing of the copy reaches the run before the first arm says the arm is
wanted: its log, its result fields and its per-frame write-back onto the
outcomes are taken at the point where the serial merge would have been made. A
copy that failed is merged again in series.
Results are bit-identical to the serial order (p.hkl md5 and MTZ data), at
default -N and -N 8.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The background ring's high-side clip, mean + 4*sqrt(mean), assumes counted photons. A CCD reports
ADU: on a Rigaku Saturn a background pixel scatters 4-5x its mean, so the clip sat at ~2 sigma, cut
the upper tail of every ring and added a positive offset to every reflection, compressing the
intensity statistics (a false 422 on one twinned I41 crystal).
No header carries a usable gain (SMV states none, every marCCD file the writer default of 1 ADU per
photon), so it is measured: background_gain::MeasureFrame takes the variance per unit mean of the
difference of pixels two columns apart, per level, clipped, on every fourth row of up to 16 of the
pre-scan's frames; Gain is the pair-weighted median over the levels. The clip becomes
mean + n*sqrt(max(g,1)*mean) (BraggIntegrationSettings::bkg_pixel_gain, CPU and GPU engines).
Only for SMV/marCCD input (ProcessConfig::counts_in_adu, set from the file format); HDF5 and CBF
keep g = 1, the identity. One log line reports the value. No new option.
Measured g: Saturn 944+ 4.7-5.2, Saturn 944 2.7, A200 0.68, ADSC ~0.3, marCCD ~0.3 - so the
ADSC/marCCD/A200 sets keep the photon clip exactly.
Checks (reflection data via gemmi and p.hkl, against the rc175 battery outputs / base build):
identical on 11if, 7qis, 8a1a, myob_x10sa, cytc_x10sa (photon counting), 4bwl, 5epe (ADSC,
marCCD) and 6cee (A200). Saturn sets, battery baseline -> this change:
3r6o I4122 R_meas 19.3% ISa 4.4 <|L|> .321 -> I41 (correct) 12.1% 8.1 .384
3mc4 R_meas 10.2 -> 9.8 %, ISa 12.1 -> 12.7, d_min 1.79 -> 1.75 A, <|L|> .360 -> .391
5uth 12.6 -> 11.8 %, ISa 10.4 -> 10.1, <|L|> .426 -> .483, offset vs |Fc|^2 0.21 -> 0.00
5vml 10.2 -> 9.6 %, ISa 11.0 -> 11.6, <|L|> .426 -> .475, offset 0.15 -> 0.05
5cc8 8.1 -> 7.9 %, ISa 10.8 -> 11.2, <|L|> .469 -> .494
3meb 14.5 -> 14.6 %, ISa 13.2 -> 13.2, <|L|> .442 -> .457
Space groups unchanged except 3r6o. Pre-scan cost: +0.5 s on a 9 Mpx ADSC sweep, within noise on
a 15 Mpx marCCD sweep, nothing on HDF5/CBF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ReadAttrStr and HDF5DataSet::ReadString read every string as fixed-length,
sizing the buffer by H5Tget_size. For a variable-length string (what h5py
writes for a str, UTF-8) that size is a pointer, so the read put a char*
into the buffer and returned garbage. transformation_type then never equalled
"rotation", and an EIGER master whose goniometer attributes were written that
way (depends_on, transformation_type, units on /entry/sample/transformations)
was processed as stills although it holds a valid rotation sweep.
All string reads now go through one helper, ReadHDF5String, which handles both
fixed- and variable-length strings in either character set; ReadElement<string>
already handled the variable-length case and now uses the same helper.
Test: JFJochReader_Goniometer_VariableLengthStrings reads the rotation axis
from a master with fixed-length ASCII and with variable-length UTF-8 string
attributes and depends_on dataset.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Part B takes the decisions before pass 2 from the first part of a rotation sweep. Two diagnostics
now say, in the log only, where that may matter; nothing the run writes changes.
Far-end check (RugnuxFarEnd.cpp), after the image pass of the canonical pass: the sweep in 10 deg
blocks, six per-frame quantities pass 2 already holds - frames indexing on their own, share of
spots on the lattice, the cell scale the strong reflections ask for (d at the observed centroid
over d predicted), the residual rotation of each frame's refined orientation against the sweep's
lattice turned to that frame (about the spindle and off it), and the background - each late block
against the early blocks as a Student-t prediction from their own spread, family-wise at the run's
0.1 % significance over every late-block comparison. One line: "far-end check: early decisions
hold" or "far-end check: <decision> would be revisited because <evidence>".
Near ties (FirstPassRun::NearTie*): where a first-pass decision compares two candidates or a count
against a gate - scheme choice, metric-symmetry demotion, axis-harmonic arbiters, axis sign,
long-axis rescue, beam-centre gates and choices, end-wedge gate, short-axis and sub-lattice
hypotheses, ladder floor/adoption/rung choice, pass-1 fallback, no-crystal test - a margin within
two standard deviations of the comparison's own noise (binomial on validation frames, Poisson on
spot counts, the binomial standard error ValidationEvidencePrefers uses on pooled spots) is logged
as "near tie: <decision> <A> vs <B> margin x (noise y) - would carry both".
ValidationEvidencePrefers now reads its difference and standard error from
ValidationEvidenceDifference, which the near-tie lines share; same arithmetic.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Each merge restored corr to the ingest state but kept g_partial from the merge
before it. A frame the new merge did not fit kept the G an earlier merge had
fitted it with, and the fulls' CC1/2 weight (frame_scale_fulls) read it, so a
merge depended on which merges ran before it on the same engine. RunArm now
resets g_partial to 1 at its start, as the first merge on an ingest already had
it. That is what lets two merges run side by side on two engines and give what
they give in series.
The fits themselves never warm-started (FitPerFrameG and the device kernel both
solve each frame afresh), so the change only reaches frames that a merge leaves
unfitted after an earlier one fitted them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
4MTK (P61, c 653 A, PILATUS 6M miniCBF, SBGrid 235), 4NWV (I222 virus-like
particle, 370-411 A axes, Rayonix MX-300 marCCD, SBGrid 9), 7D98 (P6122,
c 602 A, EIGER 4M miniCBF at 1.9 A, XRDa 13), 9V16 (P41212, c 451 A, EIGER1
16M HDF5, XRDa 338) and 4P8R (P6522, c 701 A, Rayonix MX-225 marCCD, IRRMC),
tagged large-cell (plus long-axis / virus). References are the deposited
space group, cell and resolution. 7D98 is pinned to its 250-frame miniCBF
sweep: the HDF5 masters beside it repeat frames 61-260 in 60-frame pieces.
4P8R's archive ships per-frame .bz2 files, decompressed after extraction.
EXTERNAL_TEST_DATA.md gains the five rows (DOIs resolved), the archive notes
and the updated collection counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Where a twin hypothesis (<prefix>_twin_<SG>) was written, the model is also
validated against that merge: as merged, and as twinned by its law -
|F_twin(h)|^2 = (1-a)|F_model(h)|^2 + a|F_model(Th)|^2 from the placed, scaled
model, one overall scale refitted per a, a scanned 0..0.5 on the working set
and R-free read at it (ValidateAgainstModel's new twin_law argument; GPU
structure factors as before). The maps and the placed model are then written
from whichever merge has the lower R-free; the report gives MAPS_FROM and the
subgroup's plain and twinned R-work/R-free and fraction. The subgroup file
follows its own validation's indexing, as the answer follows its own.
The model chooses the maps only: the space group, <prefix>.mtz and R_FREE stay
the answer's. Documented in RUGNUX_REPORT.md, RUGNUX_ADVANCED.md and
CPU_DATA_ANALYSIS_DECISIONS.md (14.9); Yeates (1997) credited.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Near-perfect twins of a subgroup are undecidable from intensities, and a merge
in the higher group cannot be twin-refined. Where the adopted group may have
absorbed a twin law - a twin gate refused it and was overruled, or a group above
P1 was adopted on a merge that read twin-like before the search - the same
observations are also merged in a subgroup and written beside the answer, with
the twin law and its fraction in _pdbx_reflns_twin.
The subgroup is the one a decision named (the group a refused promotion was
adopted over, the indexed lattice's group a metric re-ask promoted from), else
the index-2 subgroup the search confirmed as a finalist whose lost two-fold
agrees worst with the intensities. It is merged from the same engine, to the
answer's resolution, without writing per-frame scales back; it carries the
answer's free set (flags read off the answer's merge by the adopted group's
asymmetric unit) and is written in the subgroup's reference and standard
setting (a C 1 1 2_1 description is reduced to P 1 2_1 1). The fraction is
Yeates' H on the pairs the law relates (1/2 - median H, a lower bound).
At most one file; other qualifying subgroups are named in the report
(TWIN_SUBGROUP_* keys). The answer's files are byte-identical.
IndexTwoTwinSubgroups and EstimateTwinLawFraction are new and tested; the
point-group ledger now carries each hypothesis's representative group.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Whether a twin law can exist was read off a hard-coded list of Laue classes
(4/mmm, 6/mmm, m-3m and rhombohedral -3m "holohedral", everything else
eligible). That is a property of the lattice, not of the class: a C-centred
orthorhombic or a monoclinic lattice with no special metric admits no twin law
for mmm or 2/m either, and a pseudo-merohedral one admits a law the class says
nothing about. On five pseudo-merohedral sets the report said "Twinning
INDICATED - refinement needs a twin law" where the lattice admits none.
TwinLawPossible asks gemmi's find_twin_laws (Le Page two-folds within 3 deg,
as RfreeFlags and ReindexAmbiguity already do), and AnalyzeTwinning takes the
cell to decide it; in P1 or without a cell a law stays possible, so the
pre-promotion analysis and the space-group search are unchanged. Twin-like
statistics (<|L|> below 0.44) above the false-operator bar where no law is
possible are now reported as NO_TWIN_LAW: not a twin in this group, so the
symmetry may be too high or something else narrowed the distribution.
Report-only: no reflection file changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The two-pass supercell guards (in the first-pass indexing and after the second
pass) revert to pass 1's lattice whenever the second pass's primitive cell is
markedly larger. On a crystal with a doubled axis pass 1's schemes did not tie
on frames, so pass 1 never asked the axis-harmonic arbiter and kept the halved
cell; pass 2's schemes tied, its arbiter established the larger cell on the
spots (17048 against 16099 validation spots), and the guard then threw it away.
The scheme comparison now records when the lattice it keeps is the larger cell
of an axis-harmonic pair the arbiter settled, and both guards leave such a
lattice in place. Where the arbiter did not settle the larger cell - every other
set the guard fires on - nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Pure refactor, no behaviour change. The 2460-line function with ~25 lambdas
becomes Rugnux::FirstPassRun (RugnuxFirstPass.h): the state the steps share
(spot cache, validation frames, schemes, committed spot-list state, the
speculative passes) as members, and one member function per step taking
and returning the standing lattice (FirstPass). FirstPassRotationIndexing is
now the ordered list of decisions:
PrefetchCertainFrames, IndexSchemes (+ centre/short-axis speculation),
AxisSignRescue, BeamCentreCheck (IndexAtMeasuredCentre,
CentreWhenFileFails, CentreByPooledSpots, CentreWhenBothIndex,
CentresDisagree), LongAxisRescue, AddEndWedge, ReleaseSpotEngines,
RequireLattice, ShortAxisHypothesis, SubLatticeHypothesis, Pass1Fallback,
last-resort FirstPassLadder, MeasureVerdict, RejectChanceLattice,
AdoptLattice, StartLeftoverCensus, ReportTiltWalk, MeasureSpotBudget,
Pass1SupercellGuard.
Files: RugnuxFirstPass.cpp (order + shared machinery: spots, scoring,
schemes, ladder), RugnuxFirstPassRescues.cpp, RugnuxFirstPassAccept.cpp.
Bodies are moved verbatim (Rugnux members reached through rx.); every log
line is unchanged. Literal duplicates folded: the two MAX_SPOT_KEYS
constants, and the feed-every-scheme / run-every-indexer loops of the
long-axis rescue and the short-axis fallback (now FeedSchemes/RunIndexing,
the same calls in the same order). The leftover census, which outlives the
pass, captures what it needs explicitly instead of [&].
Verified against the rc175 0b2dde214 binary on 16 sets (7kcn, 8v4o, 5nw5,
9rci, 6z9g, 9min, nothing_1/2, 7atg, myob/cytc_x10sa, 8a1a, kdp, insu low
ISa, 6p8j, 9qw8) and -N 8 on 7kcn/8v4o/9rci: p.mtz md5-identical, logs
identical as multisets apart from timings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The refined geometry is no longer written as a whole p_XDS.INP (and the
report's XDS_INPUT= key is gone). Instead the report carries one section,
XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the
result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the
detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with
X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated,
as verified with XDS IDXREF). No cell or space group: XDS determines those.
The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste.
The same keywords were logged by LogXDSGeometry at the start of every pass,
with the axis in rugnux's sense; that printing is removed, so the report
section is the one place they are stated. Docs updated (RUGNUX_REPORT,
RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL).
Output only; p.mtz md5-identical on 3 battery sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The commit is already re-stamped on every build (JFJochGitInfo runs
GitInfoStamp.cmake with cmake -P; configure_file rewrites GitInfo.cpp
only when the value changed). What was missing is the case with no git:
a downloaded tarball left the SHA empty, and one unpacked inside some
other repository would have reported that repository's HEAD. The stamp
now asks git only when the source directory is itself the top of a work
tree, and otherwise records commit "unknown" beside the version from
VERSION. The reflection files' git description drops its trailing date
when there is none.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery
sets - most headers are a few pixels off. It now fires beyond three times
the tolerance; between one and three times the summary's Beam centre line
states the offset and recommends correcting the beamline value, without a
warning. The background-only branch (runs that refined no centre) uses the
same bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged observations already leave a run as the unmerged MTZ (and
the scaled fulls as the .hkl), and deposition does not come straight
from rugnux, so the _diffrn_refln block made the .cif up to ten times
larger for no consumer. The .cif is one data block again; the keys,
software rows, metadata, R_merge/R_pim/CC* and provenance stay.
ScaledFull loses the scale and frame it carried only for that block.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The REFMAC check is opt-in, so a battery run without --model-check has no
R-free pairs, and the axis range min()/max() over them crashed the script
before any figure was drawn. The R-free figure is now its own gnuplot block,
left out (with a line in summary.txt saying why) when there is nothing to
draw; d_min and the time figures are drawn as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A rotation run now writes the geometry it integrated at as an XDS input
beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the
detector axes carrying the tilt, wavelength, the rotation axis the run
integrated at (ProcessResult::used_rotation_axis, after any sign rescue),
oscillation (with an adopted rotation scale), starting angle/frame, data
range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left
commented where the group is not in its number's reference setting),
FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the
master file as an x_??????.h5 template and a LIB= line to point at the
reader plugin; a CBF series gets its ???? template and its own frame
numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch
(2010) Acta Cryst. D66, 125-132.
The rotation axis goes out negated: rugnux turns the crystal by -phi about
its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with
XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis
indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the
CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged
(apart from the job list): 99.8% complete, ISa 23.3, direct beam within
0.15 px and cell within 0.07 A of rugnux's.
Output only; no result changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Overloads: the integrated reflections whose signal disk held a saturated
pixel are counted on the integration outcome before the merge, over the
whole range and over the reflections coarser than 4 A, where they
concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An
OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution
reflections, and recommends attenuation / a shorter exposure. Measured: a
strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an
ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%.
Overlaps: both integration engines count the signal disks a nearer
neighbour's prediction takes pixels from (the overlap ownership the default
--overlap exclude already builds), and those left under --overlap-minpk of
their own - BraggIntegrationCounts::overlapped / overlapped_major (log line
"Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning
fires from 1% in the second and recommends a longer distance, finer slicing
or a smaller beam; on 10 battery sets it fires on the two long-axis ones
(8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both;
documented in RUGNUX_REPORT.md.
Counters only; nothing reads them. p.mtz md5-identical to the rc175
battery on 10 sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The merged .mtz and the unmerged .mtz carry, after the existing "From
Rugnux <version>" record, HISTORY records with the git commit and date,
the command line, the input file (absolute path) and a cheap identity of
it - size and modification time, the master file's for HDF5, nothing
hashed - and the collection date. MTZ allows 30 records of 80
characters; a longer value runs on over records starting with '+', cut
so that no record ends in a blank a reader would strip.
The mmCIF names the commit in _software.description and the command
line, input file and identity under _jfjoch_provenance (the local prefix
the other private items use; no PDBx item holds them), the first two as
text fields so any character survives.
OutputProvenance is filled once in the CLI and travels in ProcessConfig
to every writer of a reflection file. Reflection data are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
