The spot-width stage of the pre-scan copied every preprocessed frame back to the host (72 MB a
frame on a 16M detector, page-locked host mirrors per worker) for measurements that read a few
pixels about each spot. Now:
- MeasureSpotsOnTiles: the flux curves and the footprint read (2*24+1)^2 squares about each spot,
gathered on the card (ImagePreprocessorBufferGPU::GatherTiles) - no copy_image_to_host, no host
mirror. The flux curves are bounded at R_BKG_OUT = 20 px. The footprint keeps its arithmetic; a
per-iteration check that its window is inside the square, and a spot whose window walks off it is
measured again from scratch on an 80 px square, then on the whole frame. Where the squares would
be more pixels than the frame, the frame is read. Same pixels in the same order, so the same bits.
Harness over 34 rotation sets (~1500 frames): 0 differences; misses at 24 on a few spots of most
frames, at 54 on up to 80% of frames of a few sets, at 80 on none.
- The azimuthal mapping and the per-worker spot engines are built while the frames are read.
- The tiers no longer wait for each other: workers run on through the visit order, each tier's
answer is drawn from exactly its own frames, and frames measured past the tier the answer stops
at are thrown away - a frame is measured the same whichever tier it is read for.
- PreScanMemo keeps the spot measurement, the defective pixels and the background gain under the
pre-scan key plus the spot-finding and azimuthal settings, so a copy of the run that pre-scans
again on the same inputs reads no frames.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
- Likelihood (Read 2001; Z-score over random positions as McCoy et al. 2005): log-likelihood gain of
Rice/Woolfson E distributions, sigma_A from the pLDDT coordinate error with its scale refined, to
2.5 A (or the 20000 lowest-resolution reflections). Further-copy TF peaks ranked by the gain on the
search's reflections (peaks that lower it dropped); a copy whose TF Z is below 8 also stands out where
its refined likelihood Z-score reaches 8 - only where the copies placed are as many as Matthews makes
most likely. The likelihood is computed only where a translation function does not stand out.
Copy structure factors exact from one FFT of the cell's grid, or read off the molecular transform
where it reaches the likelihood's resolution.
- PAE domains: the AlphaFold DB's predicted_aligned_error JSON beside the model (same name pattern) is
read automatically; average linkage on the symmetrised PAE (10 A, domains >= 40 residues). Where the
whole model is placed nowhere, each domain is searched by itself side by side; placed domains must
agree, and a domain alone may only confirm the answer's group.
- Twin comparison: where the answer's merge chose and the twin hypothesis' merge exists, the chosen
placement read as a crystal in the subgroup (its symmetry images as copies, the subgroup's setting
found from the two cells) against the model searched for afresh in the subgroup, both against the
data twinned by the law (fraction refined on the working set, Yeates 1997); the subgroup is taken by
a 0.03 R-free lead plus its own null/margin -> TWIN_SUBGROUP, written to p_mr with law and fraction.
With --af-model rugnux makes the subgroup merge wherever a metric re-ask promoted the indexed group.
- tNCS rule: a group with fewer copies beside tNCS (screws bridged by the vector, or the enantiomorph)
no longer leaves the choice undecided where the chosen copies, carried into it and refined, fit
worse by 0.03 in R-free.
- Speed: the molecular transform's FFT on the GPU (cuFFT, serialised), no rigid body for trials that
lower the likelihood, re-search only with tNCS, PAE domains searched in parallel.
Survey (193 sets, MR replay on stored merges): single GREEN 58 -> 68, multi 48 -> 58, partial 8 -> 9;
4bwl (and, in full runs, 2wnq and a withdrawn entry of the same protein) TWIN_SUBGROUP P21 instead of
a wrong AGREES; a P21212 set with tNCS now SPACE_GROUP P21212 instead of UNDECIDED; no wrong call.
CPU and GPU builds make the same decisions on 18 demo/multi-copy/twin sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The beam-stop projection (ShadowAccumulatorGPU) and the defective-pixel sums (HotPixelFinderGPU) are
now kept one per card, fed by workers pinned t % gpus like the other pre-scan workers, and folded
into the first card when they are read (cudaMemcpyPeer + an integer add / max-with-count kernel).
All of them are integer sums or a maximum with the first-value rule, so the fold is order-free and
the result is identical to one card's. With one card nothing is folded.
The pre-scan sample is read once and kept (compressed as stored; up to 2 GiB, a frame past that is
read again where it is wanted) for the spot measurement and the defective pixels, which read every
frame again each before - a CBF/marCCD/SMV frame was decoded on the host three times.
A beam-stop detection that finds no shadow leaves the mask as it was, so every probe copy of the run
repeated it on the same inputs. Its answer (shadow, ring centre, mean projection) is now kept in a
memo shared with the copies, keyed on the sampled frames, the geometry the rings are drawn in, the
polarization and the binary mask; up to 4 entries.
Debug TIMING lines for the pre-scan's parts (read and projection, beam stop, defective pixels, spot
measurement). New tests: two accumulators/finders folded match one fed every frame.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Copy number. The hard "Vm >= 4: MULTI_COPY, no decision" rule is gone. Copies
are searched one after another up to the largest copy number with a Matthews
probability of 1% (a Gumbel distribution with Kantardjieff & Rupp's 2003 mode
and mean, no resolution dependence); COPIES_LIKELY reports the most probable.
Sequential search. Copy n's translation function carries copies 1..n-1 as a
known partial structure: the cross term 2 Re(conj(F_known) F(t)) goes on the
same FFT grid (CPU FFTW and GPU cuFFT; KnownStructureFactors, AssemblyCorrelation).
Its peaks, past those that clash, are ranked by the correlation of the whole
assembly with E^2 (the TF itself, linear in |F|^2, rewards copying a placed
copy onto an origin shift), the first that stands out (TF Z >= 8), packs and
lowers R-free by 0.01 is kept; each copy is refined beside the others held
fixed (RefineRigidBody's fixed copies), and the first copy again beside the
second, the earlier ones once more at the end.
Packing: ClashFraction counts C-alpha clashes with other copies and all their
mates, binned (7mzt: 51 s of clash checks -> 1.9 s).
Search quality: rotation-function peaks hill-climbed off the 6 deg sampling
(6ze4: RF Z 10.4 -> 16.0, 3.3 deg off the deposited orientation -> 0.2); the
first copy goes down the TF list until three placements that pack are
refined, so a group is CLASH only where no offered orientation packs; groups
are searched side by side, and every group the best one leads is searched
again with the best one's copy orientations tried first, the better search kept.
tNCS: the native Patterson of E^2 (5-10 A) off the lattice; placed copies'
orientations join every further copy's search; and where the tNCS vector turns
one group's screw axes into another's (TncsRelatesGroups), a group whose search
stopped at fewer copies does not lose to the other - UNDECIDED. Measured on an
in-house tNCS crystal (vector c/3 along the 3-fold): both hands fit with three
copies to within 0.01 in R-free, and where one hand's search failed the other
was taken as ENANTIOMORPH; now UNDECIDED.
Null: the replicates are placed by the translation function in their own
orientation, as the search placed the real one (a null at the real centroid
reads high), and the gate is 4.51 = t(8) sqrt(1 + 1/9), 3 sd of a nine-draw
null, the gate --model's MODEL_FIT is now read at.
Speed: placements scored on the GPU structure-factor engines; further copies'
trials score with the first copy's bulk solvent held, the assembly refitted at
the end. Survey of 193 sets (jfjoch_mr_replay on stored merges, RTX 5080
shared with other jobs): MR median 1.15 s, max 24 s; demos 0.8-1.1 s.
Report: COPIES and COPIES_LIKELY columns, MODEL_PRIOR_COPIES.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Three defects the battery's wrong-model null found (5 sets x 5 deposited models of unrelated proteins):
CC(model, data) significance. The per-shell sigma was Fisher's transform on independent reflections,
but the data and a scaled model share the falloff across a shell, the anisotropic B fitted to these
very data, the bulk solvent and the symmetry enhancement epsilon - whatever the model is. 21 of 24
wrong models read CC_MODEL_CONFIRMED_TO_D_MIN to the data's d_min (10-15 sigma in most shells).
The correlation is now formed on I/epsilon and |F_model|^2/epsilon, and its null is empirical: the
data at h against the model at h + d for the 26 nearest lattice offsets, which keeps everything smooth
in reciprocal space and loses what the model knows of h. SIGMA = (z - mean) / max(sd, 1/sqrt(N-3)) in
Fisher z over those 26, a new CC_NULL column shows the null, and a shell confirms only beside a coarser
shell that also clears 3 sigma (one shell alone did so once in 216 wrong-model shells). Prototyped on
the p_maps.mtz of a full open-arm run and the null runs: wrong models confirm 0/24 (was 21/24), the
deposited models confirm 197/200 (was 198/200 in the prototype's reading; the one lost has CC_MODEL
0.05-0.13 in every shell). The report no longer says a model cannot agree by accident.
MODEL_FIT gate. Accepted at 3.0 sample sds of a nine-draw null, as if the sd were known; an unrelated
P1 helical model on a P3(1) helical crystal was ACCEPTED at +3.05. Against 48 draws it reads +2.09 and
the best random placement fits as well as it does, and drawing the null about the cell axes instead
changes nothing (+2.15): the cause is the nine-draw sd, not the orientation. The gate is now 3 sigma
allowing for that: t(8) * sqrt(1 + 1/9) = 4.51. The deposited models of the open arm read +7.0 and up.
The indexing lead stays at 3 sigma (INDEXING_MARGIN_SIGMA): it is read only behind the fit gate, and
a deposited model settles one at +4.15.
Rigid-body GPU reserve. The finest zone's observations were counted by the data's d, but the zones
count them by the d of the index in the model's ASU, and a model's group need not keep the data's
metric (an I2(1)3 model on I222 data: 2320 in the zone against 1712 reserved), so the validation died
with "zone over the engine's reserve". Counted the zones' way, under every indexing probed.
Tests: ModelValidation_CCModelUnrelatedModelConfirmsNothing (Fisher alone reads +4.7 on an unrelated
model with an anisotropic falloff; the new sigma stays under 1), ModelValidation_WrongModelInAGroupOffTheDataMetric
(fails with the old count), ResultReport_ModelValidationSection (CC_NULL column, lone-shell rule).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
gemmi's table lists some groups a second time on another origin ("P 42 21 2a", "P 21212(a)",
"I 2 2 2a", "C 1 21 1"). SameRotationGroups kept them as separate candidates, so the true group tied
with itself in R-free and the choice came out UNDECIDED: five single-copy sets of the --af-model
survey (and one partial-model set) ended there. A group equal to one already listed up to an origin
shift on the twelfths is now left out.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- An MR ENANTIOMORPH decision is taken into the answer's label only as --model takes a model's
enantiomorph: ValidateAgainstModel runs on the placed model (in a scratch directory, removed after)
and its adopted_model_enantiomorph - fit over its null, anomalous map at the atoms not inverted -
decides; otherwise ENANTIOMORPH_VETOED. The hand-check numbers go to the MODEL PRIOR section.
- Rotation and translation functions on the GPU (MolecularReplacementGPU.cu): one block per
orientation for the rotation function; per orientation the copies, the bin weights, a gather of the
Crowther-Blow coefficients (CSR lists, fixed order, no atomics) and a cuFFT C2R, block-reduced
statistics finished on the host in a fixed order. The CPU path (FFTW) is unchanged in what it
computes; the orientation-independent part of the translation function (TranslationSetup) is now
built once per group on either path.
- Rescoring: a placement clashing with its mates over twice the limit is not refined, one rigid-body
engine per group, and the null is built only for the candidate the choice falls on.
Measured MR step on the RTX 5080 (before -> after, seconds; hand check extra where it runs):
3r6o 1.77 -> 0.67, neg 2.14 -> 0.37, 11if 1.20 -> 0.51 (+0.6 hand check), 7ris 1.46 -> 0.56 (+0.6),
9zmu 4.71 -> 1.07 (+0.9). Rotation 0.50 -> 0.02 s and translation 1.17 -> 0.13 s on 9zmu.
Mirrored-detector runs of two anomalous lysozyme sets: the hand check vetoes (map inverted at the
model's atoms); the unmirrored runs adopt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
IndexerThreadPool_FFTIndexerWaitsForGPUWorkBeside filled only the test thread's
card, but an indexer worker takes its card round-robin (pin_gpu: a process-wide
counter modulo the device count). On the CI runner, a container given all of the
host's GPUs, the probe's worker could land on a card nobody had filled, build its
indexer and fail "CHECK(probe.Run(x, recip).error != nullptr)"; the two checks that
expect a wait passed there for the same wrong reason. On a one-card workstation the
worker is always on device 0, so it never showed.
Every card is now filled, each under its own device, and released the same way, and
the fill is topped up right before each Run so memory given back in between by
anything else does not let the indexer fit. The pools are trimmed on every card at
the end. What the test checks is unchanged.
Run 20x alone and with the [CUDAMemHelpers],[AdaptiveSpotFinderGPU] group, also with
another process holding 4 GB of the card: all pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pools' indexers return their memory to the allocation pool when destroyed; without a
trim the next GPU test in the same process (AdaptiveSpotFinderGPU_RunToRunReproducible)
found the card taken and failed with out of memory.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A predicted model (AlphaFold mmCIF, pLDDT in the B column) is trimmed at pLDDT 70, given B-factors
from the pLDDT (Oeffner et al. 2022) and placed by molecular replacement, one copy in the ASU:
- rotation function on a uniform 6 deg sampling of the point group's rotation ASU, each orientation
scored by the correlation of E^2 with the symmetry-summed |F|^2 read off the model's molecular
transform, computed once by FFT and interpolated (Navaza 1994);
- full-symmetry Crowther & Blow translation function (one FFT per orientation and group, through
MapFromFPhi/FFTW) for the 50 best orientations in every group with the merge's rotations;
- the best three placements per group through RefineRigidBody (GPU where there is one), bulk solvent
and R-free to 3 A, a C-alpha packing check against symmetry mates, Matthews, and the
random-orientation null for a placed candidate.
Decision: PLACED = TF Z >= 8, clash <= 5%, Vm in [1.5, 4), null >= 3 sd. Within the answer's merge
the placed group with the lowest R-free is chosen where it leads every other packing group by 0.03;
the twin hypothesis' merge is searched, and chosen, only where no group of the answer's merge can
hold the model at all (Matthews, packing). An enantiomorph relabels the answer as --model does;
another group goes to <prefix>_mr.mtz/.cif plus <prefix>_mr_model.pdb; the MODEL PRIOR report
section carries the table. The search runs on the CPU in both builds; without --af-model nothing
changes.
Measured (RTX 5080, MR step): 3r6o twin I41 chosen over I4122 (Matthews + clash) in 1.8 s, solution
0.09 A CA rmsd from Phaser's; 11if P41 -> P43 1.2 s; 7ris P3121 -> P3221 1.7 s; 9zmu P6122 -> P6522
4.7 s; 11if's model against 3r6o's data NOT_PLACED.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Where the measured spot width widened the integration radius, the canonical pass
integrated the whole sweep at the widened radius, and only then did the starvation
guard read how many background rings the neighbouring predictions crowd; over the
bound (1.13%) the pass was thrown away and run again at the fixed radius.
The geometry pre-pass now counts that same quantity at the widened radii without
integrating there (BraggIntegrationEngineCPU::CountNeighbourCrowding: the neighbour
mask and the ring pixels the run's mask leaves readable, no image) on 20 frames
spread over the pre-pass, with the standard error of the ratio over those frames.
Over the bound, the canonical pass integrates at the fixed radius from the outset.
A count within NEAR_TIE_SIGMA of the bound is a near tie in the part-B sense: the
canonical pass counts the widened radii again on 20 frames over the whole sweep,
and where that puts them under the bound the run is made again deciding late. A
run deciding late counts nothing early and decides as before. Under the bound the
canonical pass integrates at the widened radius and the measured guard stays as it
was, so a pre-pass count that is too low (its prediction holds fewer tails than
the canonical pass's, or it stood on another lattice class) costs only the pass
it cost before. No new threshold.
Battery against 46be3f647 (the 14 open sets that re-ran at the fixed radius, the
9 open/in-house sets closest under the bound, the ci tier, 2 private sets):
- p.hkl and MTZ data identical on all 51 open/in-house sets and on one private set.
- The 14 re-running sets: rugnux wall 534 -> 447 s (7qij -27, 4nwv -12, 7yzx -10,
9fhc -9, 5src -6 s, on a shared machine). 8sqq and 8xtf count under the bound on
the pre-pass and re-run as before. 4nwv (1.69 +- 0.28%) and one other set were
near ties, re-checked on the whole sweep (1.72%, 1.61%), ending where they did.
- One private, densely crowded set: same space group and d_min, R_meas 28.6 ->
39.3%. Its old re-run re-indexed starting from the spot budget the thrown-away
pass had measured (152 of 1000). With that budget given (--max-spots 152), the
new run gives 30.6%. The difference comes from that path, not from the radius.
- The count costs ~13 CPU-s on the heaviest set (65k predictions a frame). Over
the 22 ci sets that count, pre-pass integration took 45.2 -> 45.8 s.
- A build forcing the near tie and its overturn on 4nwv re-checks on the whole
sweep (1.80% against the engine's 1.81%) and makes the run again deciding late.
It ends with the same p.hkl.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Several pipeline copies of one run build FFT indexer pools on the card at once - the
rotation-scale walk's indexing probes start as the output pass begins its image loop,
and the speculative geometry probe runs beside the merge - and each indexer is a cuFFT
plan plus 0.6-0.9 GB of buffers. Construction had no admission control: a cufftPlanMany
or device allocation that found the card full failed straight through, and since
IndexerResult carries the exception that ends the run. With 7.2 GB of the card held by
another process, a small-molecule rotation set failed 4 of 4 runs this way although
each pass fits on its own.
Construction now does what the merge's allocations do: hand back the pool's unused
memory, wait for the work beside it to end, and build again. Who waits for whom is
fixed so that nothing waits in a circle:
- the main line waits for the probes (GPUWorkBeside, as the merge already does);
- a probe waits for the main line's image pass (new GPUImagePass), never for a probe,
itself included. The image pass waits for no probe, so it ends.
The walk's indexing probes now hold a GPUWorkBeside too (they are exactly that work),
and a run carries gpu_work_beside_ so its pools know which side they are on. Only when
an indexer is built changes, never what it computes; with nothing to wait for it still
fails at once.
Same run with the 7.2 GB hog: 5/5 complete (also at 7.6 and 8.0 GB), p.hkl identical
to the unhogged run; a card that cannot hold the run still fails in 0.5 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The GPU engines are built per worker per step, and each took its page-locked host buffers with
cudaHostAlloc / cudaHostRegister and gave them back with cudaFreeHost / cudaHostUnregister. Those
calls map and lock pages under the process's memory-map lock, so all workers queue behind each
other: on the 5080 workstation a myoglobin run made ~3400 of them (6-22 thread-s, depending on the
load), 9gdj ~10400 (124 thread-s); on a four-socket host they cost 20-30 ms each.
- CudaHostPtr takes its memory from a pool: a freed buffer is kept in a power-of-two size class for
the next engine, up to 512 MiB in all; a buffer above 64 MiB is never kept. Rugnux::Run hands the
pool back at its end.
- PinnedVector (std::vector with the pool's allocator) replaces the registered vectors of
AdaptiveSpotFinderGPU and ImagePreprocessorGPU; the Bragg predictors download into a pooled
pinned buffer and copy into their output, instead of registering it.
- BSLZ4DecoderGPU copies the compressed chunk into a pooled pinned buffer before the upload, so it
is one DMA instead of the driver's pageable staging (9gdj: 163 GB pageable H2D, 52 s of copy
time, -> 147 GB pinned, 13 s).
Measured with nsys on the workstation: myob 3398 -> 1396 calls (674 of them the release at the
end), cytc 2986 -> 1270, 9gdj 10428 -> 1748 (124 -> 3.7 thread-s). Output identical (p.hkl md5 on
myob, cytc, 9gdj; CI tier battery).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
With -N left to choose, rugnux now confines the process (main thread, the CPUs
RestoreThreadAffinity hands back, and what AvailableCpus counts) to the CPUs of
the packages its cards hang off, and sizes its threads to them. Nothing changes
where the allowed CPUs sit in one package, there is no card or a card's NUMA
node is unknown, or the cards' packages already hold every allowed CPU. A mask
set from outside (taskset, numactl, a cpuset) is kept: the decision works
within it. An explicit -N keeps the old behaviour.
On a 48-CPU server with four cards on two of its four NUMA nodes, a rotation
run took 24.5-24.7 s confined to those nodes' 24 CPUs against 27.7-27.9 s on
all 48 (scaling and merging 8.1 -> 6.2 s), and two larger sets moved
95-99 -> 84-87 s and 131-135 -> 113-115 s. On a single-socket machine this is
a no-op.
The decision itself (CpusOfGpuPackages) is a pure function, tested on
synthetic 4-package, 2-package hyper-threaded, cards-on-both, no-card and
restricted-mask topologies.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The demoted arm of the metric-symmetry check and the rotation-scale walk's
refits integrated and post-refined every frame of the sweep, only to read a
held-out residual or a scale fitted on top. They now integrate only the frames
the geometry pre-pass read (prepass_frames_only_, PrepassEnd), as the pre-pass
asked its own fit: the run decides early on the evidence it has.
Metric-symmetry check: the canonical pass, the promoted arm, fits its
comparison post-refinement on the same frames (GatherPostRefineObservations
takes a frame limit). That fit only measures and leaves nothing on the run;
the canonical pass then measures its own fit over every frame where it
measures one anyway, as any canonical pass does. A decision whose margin from
the HeldOutResidualFell line lies within NEAR_TIE_SIGMA of its noise is a near
tie in the part-B sense: the canonical pass also fits over every frame while
its reflections are in hand, and after the run the demoted arm runs again on
every frame (deciding late) and is judged against that fit. Where the check
ends elsewhere the run is made again deciding late, which asks both arms on
all the frames as before. No new threshold.
Rotation-scale walk: a refit proposes the next scale; the validation frames of
the whole sweep still decide whether it is taken, so it needs no re-check.
Battery, 31 open/in-house metric-check and scale-walk sets + 3 private,
against 56bad5fdb:
- p.hkl and MTZ arrays identical on 32 of 34. Every metric-symmetry decision
ends as before. 6yqf decided a near tie the other way on 1200 of 2000
frames, the re-check on all frames overturned it and the run made again
gives base's output (24 -> 55 s). 6cdl and 9vyb were near ties the re-check
confirmed.
- 9gdj: the scale walk takes one more refit and adopts 0.97020 instead of
0.97045; R_meas 11.41 -> 11.36 %, R_free 0.1709 -> 0.1708, d_min, SG same.
- One private scale-walk set: the pre-pass refit pointed up where the
all-frame one pointed down; the whole-sweep validation frames prefer the
higher scale, and the merge improves on every statistic.
- ci tier: 0 sets changed beyond noise (only 9gdj's data differ).
- rugnux wall over the 31 sets 1059 -> 993 s (with 6yqf's re-run), metric
check step 129 -> 85 s; ci tier 640 -> 626 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
CalcGlobalWilsonB fits ln<I> against 1/d^2 from 4 A to d_min. With d_min just under 4 A that window is a
sliver, and after the shells past the signal limit are dropped it held 0-2 shells, so the estimate was
NaN (or a fit over almost no range); and any B above 200 A^2 was discarded, which is every crystal
diffracting to 4-7 A.
Now the window is never narrower in 1/d^2 than the 4-3 A window a 3 A data set gets: a narrower one is
widened towards low resolution (d_min 3.15 A -> from 4.39 A, 3.61 A -> 5.96 A, 3.87 A -> 7.40 A; data
coarser than ~4.4 A use everything, as before). The 200 A^2 cap is dropped for the dataset-wide
estimate (kept for the per-image one). The window is logged, printed with the statistics and reported
(WILSON_B_RANGE_A, developer key); the correlation is reported as before. Data with d_min <= 3 A are
unchanged.
Report only: the value feeds the statistics text, the log, the report and the mmCIF
_reflns.B_iso_Wilson_estimate; nothing in scaling, merging or the MTZ reads it.
Measured on the open arm (replica on the a897b08 merged data, confirmed by rugnux runs): 12 sets
change, 7 previously NaN now finite - 5nw5 315, 9z44 493, 6r72 302 A^2 (were above the cap; model mean
B 328/310/114, deposited Wilson 388/322/179), 9yzk 210 (model 383, ctruncate 243, xtriage 104), 7qij 104
(model 194, ctruncate 150), 9yl4 78 (model 146), 8qq7 35 at 4 shells (model 197; its signal ends well
inside the written 3.15 A). 6yqf 132 (corr 0.51) -> 96 (corr 0.69; model 57, ctruncate 145,
xtriage 114); 9rcs 69 -> 148 (model 74, ctruncate 131). At 3-4 A the estimators themselves disagree by
1.5-3x, so these are better-defined numbers, not exact ones. p.hkl and MTZ data identical to the base
(e34a5fb88) on every set compared (ci tier, private arm, the Wilson sets).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The verdict gated only deltaB_linear, the Debye-Waller part of the tensor, against the data set's own
systematic floor. A deficit that does not follow s^2 (a FLAT signature) can stop the signal far earlier
in one direction and still read NOT_DETECTED: one open-arm set reported directional limits of
3.18/3.18/2.14 A beside NOT_DETECTED. The cone limits (<I/sigma> = 2 in 20 deg cones) were computed but
never consulted.
Now each statement is decided on its own evidence and either establishes anisotropy:
- tensor: unchanged gate;
- directional limits: at the best direction's limit the weakest cone's <I/sigma> is below the best
cone's by more than 3.5 of their joint standard error (both read linearly in s^2 between shell
centres, as the limits are), the weakest limit is measured (not censored at the data edge), and the
limits are at least 0.25 A apart - the spread the report already calls "noticeable".
Why the magnitude term: with the standard errors alone (the counting statistics of the cones) the
weakest cone is "significantly" below the best on 172 of 216 battery sets, down to limits 0.5% apart
(measured with a replica of ConeLimits on the a897b08 battery's merged data). The cones differ
systematically (multiplicity and coverage depend on direction), so significance alone made DETECTED
near-universal. A refusal (CANNOT_DETERMINE: dose ramp, short sweep, triclinic, no unmerged data) is not
overruled - its reasons bias the cones too - but now reports the measured directional limits.
Report: ANISOTROPY_DETECTED_BY (TENSOR/DIRECTIONAL/BOTH/NONE); developer keys for the two cones'
<I/sigma> and errors; the summary row names the test that carried the verdict, and CANNOT_DETERMINE
rows give the directional limits; the mmCIF verdict comment says so. Report only: no intensity, flag or
decision downstream reads the verdict (the French-Wilson prior uses the tensor regardless).
Measured (battery replica, 225 sets with limits): directional statement established on 95; 23
NOT_DETECTED -> DETECTED (directional ratio 1.14-1.59); of the 17 sets with ratio > 1.2 and an
uncensored weakest limit, 11 flip, 5 stay CANNOT_DETERMINE with limits reported, 1 (sparse, noisy
cones) stays NOT_DETECTED. ANISOTROPY warnings (spread > 0.5 A) appear on 6 more sets. Confirmed with
rugnux runs on the ci tier, those 17 and the private arm: p.hkl and MTZ data identical to the base
(e34a5fb88) on every set compared.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RugnuxFarEnd.cpp used M_PI, which MSVC does not define without
_USE_MATH_DEFINES, and broke the Windows viewer build (C2065). The
ResultReport_SupercellCheck test still looked for the supercell warning's
wording from before the doubled-cell change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
TCPStreamPusher::SendImage is called from the receiver's frame-transformation
threads. It used a 128-deep per-connection queue and waited when it was full.
With images routed per file ((n / images_per_file) % writers), every writer
receives a whole file as a burst at the full frame rate (1000 images/file at
2 kHz fills 128 in ~64 ms), so the queue filled on every burst and all
transformation threads ended up waiting on one writer. They then stopped
returning FPGA frame buffers, and the FPGA dropped packets: partial images,
images without a background estimate, data loss on every writer. The ZeroMQ
pusher has an unbounded queue and never blocks, which is why it coped.
- The per-connection queue is unbounded: every element owns a distinct
image-buffer slot, so the bound is the image buffer's slot count. When slots
run out the receiver drops only the write (GetImageSlot() == nullptr), after
analysis. Absorbing one file burst therefore needs an image buffer of about
images_per_file slots (~20 GB for a 9M at 1000 images/file); below that,
images are still dropped, now as writer-queue-full instead of packet loss.
- SendImage never blocks. A dead or wedged writer is still caught in SendAll
(15 s silence / 60 s without progress), after which WriterThread returns the
queued slots unsent.
- The 200 ms fallback in StopDataCollectionThreads is gone: at END it closed
the connection of any writer whose queue stayed full for 200 ms - a normal
state for a slow but healthy writer - discarding up to 128 queued images and
never sending END. END now follows the whole backlog.
- A per-connection enqueue mutex makes "check active, enqueue" atomic against
"clear active, enqueue the end sentinel": an image could land behind the
sentinel when a connection was torn down mid-run, leaking its slot
(CheckIfBufferReturned then threw) or being sent as stale data in the next
run. New stress test SendImageRacingEnd_NoSlotLeak (verified to fail with
the race widened and the lock removed).
- TearDownConnection marks the connection broken first, so the destructor
returns a backlog instead of sending it.
Writer side:
- jfjoch_writer -b/--rcv_buffer sets the TCP receive socket buffer (was
hardcoded to the kernel default).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The model scale searched b_sol only over 10-80 A^2. That is the range
Fokine & Urzhumtsev report for models at ordinary B, but on low-resolution
entries whose deposited B is 200-400 A^2 the fit sat on the 80 A^2 ceiling
(and k_sol on its 0.1 floor): a solvent edge far sharper than the atoms it
lies against, reaching too deep in resolution. R-free came out 0.05-0.10
above REFMAC (0 cycles) and phenix.model_vs_data on the same MTZ, model
and flags.
The ceiling is now max(80, 2 x the model's mean atomic B) (SolventBMax),
for the reported fit and for the rigid-body placement's per-zone solvent
(CPU, GPU and the host fallback alike). The grid keeps eight b_sol values
across a wider box, so it costs no more points; the default box, and so
every model with mean B <= 40 A^2, is bit-identical to before.
Measured (10 open-arm sets with --model, GPU build, vs the same base):
high-B set, 7.0 A: R-free 0.462 -> 0.364, b_sol 80 -> 379 (REFMAC 0.366,
REFMAC's own b_sol ~371; model_vs_data 0.291);
the rigid-body step now commits
high-B set, 3.9 A: 0.410 -> 0.331, b_sol 80 -> 388, k_sol 0.10 -> 0.225
(REFMAC 0.328, model_vs_data 0.314); null 10.1 -> 14.2 sigma
B~200 set, 3.2 A: 0.408 -> 0.402, b_sol 80 -> 120; enantiomorph label
decision unchanged (27.2 -> 25.8 sigma)
7 other sets (mean B 16-57): bit-identical R, k_sol, b_sol, placement.
p.hkl md5 and the MTZ reflection data are identical to the base on all 10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A CUDA out-of-memory in an indexer worker was turned into an IndexerResult carrying only the
message string. The rotation indexer then returned no lattice, the first pass treated that as a
genuine negative and the rescues acted on it: in a stress run (two rugnux on one card) a run had
both first-pass schemes fail with out-of-memory, the axis-sign rescue compared 0 frames at the
flipped sign with the -1 "no lattice" sentinel, adopted the flipped sign, and the beam-centre rescue
then moved the centre - a failure of the machine used as evidence about the crystal. The CUDA
category was lost on the way, so the fail-fast check (IsFatalResourceError) never saw it.
- IndexerResult::error is now the std::exception_ptr the worker caught. RotationIndexer::RunIndexing
rethrows it before touching any state, so nothing is memoised for a failed run and the exception
keeps its category: rugnux stops with the CUDA message at once (verified with a forced indexer
failure: exit 1 after 1.9 s, "CUDA (GPU) error (...)"). FirstPass::indexer_error,
RotationIndexer::GetIndexerError and RequireLattice's machine-failure branch are gone.
- The beam-centre arm in RunAllPasses rethrows fatal resource errors like its sibling catches; it
had caught the out-of-memory and carried on at the file's centre.
- IndexAndRefine keeps per-frame indexer failures apart from frames that did not index (count +
first exception). The frame stays "not attempted" as before; the receiver logs the count and the
first error at the end of a collection (no cancel), rugnux's image pass and stills geometry
refinement rethrow it and stop. A rotation-indexer failure inside IndexAndRefine (online rotation
indexing, FinalizeRotationIndexing) is recorded the same way, so a live collection is not
cancelled by it.
- A failed device allocation or cuFFT plan now says how much was asked, how much of the card is
free, and - when the card is merely full - that another program is using the GPU or the data set
needs more than the card has, with the CPU remedy; previously "Failed to allocate device memory
(out of memory)" / "Failed to create CUFFT plan with cufftPlanMany".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Narrow, guarded: the doubled pass runs only where the added class grows with resolution and is resolved; the doubled
cell is adopted only where one pseudo-translation explains the added class in every shell and the point group does not
rise. Fixes the one remaining lattice failure (half cell -> deposited cell, R-free 0.574 -> 0.275); p.hkl identical on
the ci tier, the supercell gate set, 78 further supercell-candidate sets and all private sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A shell is read only where at least half of the class's reflections are clear of the lattice's by r3.
On a 701 A axis 26-43 of ~36000 per shell were clear; they read at the lattice's own strength, the
doubled merge (93% of predictions overlapping a neighbour) passed the one-translation test, and the
run was adopted doubled and failed. With the rule, of ~134 sets with probe logs only 9min and one
I-centred crystal reach the doubled pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
On strong data a class at one or two percent of the lattice rises significantly (0.009 to 0.059 at 4
standard errors on a ci-tier triclinic crystal, which was then adopted doubled and failed), and a half
cell missing a few percent of its intensity still describes the crystal. The doubled deposition ends
at 0.35-0.40, those classes at 0.06.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Exact: p.hkl md5 and MTZ data identical on every real set (34 open/inhouse incl. the ci tier, all 27 private);
both no-crystal controls now ~7 decades inside the bar instead of 0.3-0.9.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Exact: p.hkl md5 and MTZ reflection data identical on 18 sets. 4nwv R-free 0.545 -> 0.297 and cell in the depositor's
frame; written cells corrected on 9rcs, 9b22, 9qw8, myob_x06da_powder_1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
- ModelValidation reads the model and, where it carries non-given NCS operators
(_struct_ncs_oper 'generate' / MTRIX without the given flag), expands them
(gemmi expand_ncs, vendored assembly.cpp from GEMMI 0.7.5, MPL-2.0 as the rest
of gemmi_gph). Same in the battery's model_check.py and depdata_check.py.
An icosahedral-virus entry deposited as 3 of the 45 chains of its ASU was
scored against a fifteenth of its atoms: R-free 0.545 -> 0.297.
- The indexing the model chooses (AdoptModelFrame) now moves the cell with the
indices (reseat_integrated in the main path; CellInBasis in the forecast,
twin-file and stills/scale paths). A near-symmetry law (b within 2% of c, beta
a degree off 90, a~b in P1) relabelled the reflections only, so the files were
written with two axes swapped or beta flipped relative to the indices. Where
the law moved the cell, the validation is remade on the moved cell.
Measured on 18 open/in-house sets with models: p.hkl and MTZ reflection data
bit-identical on all; cell and model R unchanged where the law is exact; the
cell now matches the deposited one on the 6 pseudo-merohedral cases (orthorhombic
a/c swap, three monoclinic beta flips, a P1 alpha/beta swap), R-free within
+-0.001 there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Doubled-cell hypothesis: the run now always merges the doubled cell under <prefix>_doubled first and
decides with both merges in hand. Adopted only where (1) the probe's class grows significantly with
resolution (three standard errors between the 20-8 A shell and the finest measured one) and (2) in
the doubled merge one pseudo-translation - half a cell plus eps, fitted over the class - modulates at
least half of the added class in every shell (FitDoubledClassTranslation: A sin^2(pi h.eps) + B per
shell, share 1 - B/mean). Where adopted, every <prefix>_doubled file takes the plain name and the
probed cell's file of that name moves to <prefix>_halfcell. Replaces the HALF_CELL_BREAKS_DOWN bound.
Measured on the forced doubled merges: modulated share per shell 0.80-1.0 out to 1.9 A on the crystal
deposited in the doubled cell; on the ones deposited in the half cell it falls to 0.01-0.34 by
2.1-2.8 A (the rising class with twinning), or stays at -0.3-0.48 at every resolution (three others);
the one class it fully explains does not rise (eps 0.1 of a cell, flat at 0.1 of the lattice).
Space-group search: the trim of a screw zone's largest absence exempted a violation "at its row's
mean" on the de-modulated intensity, so under a deep pseudo-translation a reflection at 0.015 of its
row (de-modulated by 1/0.002) could never be trimmed and alone decided the row against the screw.
The exemption now reads the measured intensity, at both sites.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Where the first pass recognises two twin domains on their coincidence lattice (reticular
merohedry, index n = 2 or 3) and adopts the stronger domain's cell, the 1/n of that domain's
reflections that lie on the other domain's lattice too carry both domains' intensity. The step
now records the overlap as a lattice vector t of the standing cell (h . t divisible by n), the
rotation result carries it, and the per-image integration drops those reflections. t is a
real-space vector, so it holds through every later change of basis; it is read in sixths so a
centred conventional cell (F, I, C, R) is exact. Each unique reflection keeps its equivalents
that the other domain does not overlap; only those whose every equivalent is overlapped are lost.
The two-pass quality guard compared signal counts between a header pass that integrated
everything and a refined pass that left the overlap out, and sent the run back to the
composite lattice. A pass that left an overlap of index n out now has its counts taken up by
n/(n-1), and its axial rows are not compared with a pass that did not.
Nothing changes where the step does not fire: the field stays empty, the factor is 1.
Measured (merging the existing outputs with and without the class H.(111) = 0 mod 3, against
removing a random third): spinel-twinned F-cubic private set R_meas 25.5 -> 14.3% (random
26.7%), insu_I_x06da_low_isa 24.1 -> 16.6% (random 24.2%). With this change, end to end:
- insu_I_x06da_low_isa: I23, R_meas 24.0 -> 12.7%, ISa 5.2 -> 28.8, CC1/2 0.997 -> 1.000,
completeness 99.8 -> 95.7%, d 1.45 -> 1.47 A, model R-free 0.260 -> 0.246.
- private F-cubic twin: F432, R_meas 25.5 -> 6.3%, ISa 3.9 -> 20.4, completeness 100 -> 93%,
d 3.23 -> 3.77 A (XDS reference 4.0). 432 now passes the added-operator gate on the
all-observation merge (contrast 0.75, bound 0.72) instead of being refused there (0.62) and
rescued by the twin-immune zone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Supercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own
limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the
SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice
reflection predicted on the same frame lies within r3, and each resolution shell is read against the
median of the six other parity classes - which a single doubling leaves empty - as an excess in units
of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z
beside it ("Supercell probe to the limit" in the log).
Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its
noise and at more than twice the strongest of them, in at least three shells), the run integrates and
merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along
the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's
merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both
(DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT
(the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the
class reaches a quarter of the lattice's intensity in the finest shell both are measured in).
The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a
triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed.
Measured on forced doubled-cell merges (rc175 b530c2d base): the open-arm crystal deposited in the
doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell
(odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals
deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at
8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low
resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the
neighbour correlation, the separation from the nearest lattice reflection after the r3 filter -
separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call.
Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no
longer advises -C with the doubled cell, which the indexer reduces back to the half cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The line put the distinct-reflection count against a normal-approximation bar, which at a
handful of reflections reads the wrong way: a control the exact test rejected (5 vs 2 by
chance, tail 10^-2.70 against 10^-3) was printed 0.59 above the bar. It now prints the exact
binomial tail and the significance, both as log10, with the margin in decades and the noise as
the tail's change over one binomial standard deviation of the count. The tail is summed in logs
(ValidationEvidenceLog10Tail), so it stays finite for a real lattice; the decisions compare the
same tail in log10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The final no-crystal test (RejectChanceLattice) used the 0.001 binomial level of the comparisons
between candidate lattices. The lattice it tests is the winner of some fifty indexing attempts,
each fitted to an accumulated cloud that already holds the long-lived reflections it then re-finds
on the validation frames, so that null is optimistic by roughly the number of attempts. The two
in-house no-crystal controls passed it by one or two distinct reflections (tails 10^-2.06 and
10^-2.70 against 10^-3).
The test now asks 1e-9 (Z ~ 6); the rescue comparisons keep 0.001. With two reflections by chance
the controls would now need 15 distinct reflections instead of 4-5. Real lattices are tens of
decades past either level (weakest seen: 10^-83).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The projected histogram is sampled every 1/(2*max_length) of 1/d, so a row
of length L between max_length and 2*max_length folds to 2*max_length - L and
peaks there as sharply as a real row. At the default 500 A bound a 602 A axis
came back as ~400 A and a 701 A axis as ~299 A; the folded cell still indexed
59/60 and 37/60 validation frames (only the reflections whose index lands on
an integer of the folded period), so the long-axis rescue, gated on a poor
pass, never ran. The same fold also turned real >500 A lattice combinations
of shorter-axis cells into spurious shortlist rows.
Each shortlisted row is now asked once, by a direct Fourier sum over the spot
positions (no sampling, so no fold), whether the spots carry its own length or
its partner 2*max_length - L more strongly, and keeps the stronger. Partners
within 0.1 of an integer multiple are left alone (a harmonic, settled
elsewhere). Refinement accepts rows up to 2*max_length when one was unfolded.
Measured: a hexagonal P6122 c=602 A EIGER set and a P6522 c=701 A CCD set,
both indexed de novo on the folded cell before, now index the deposited cell
60/60 and merge to R-free 0.31 / 0.22 against the models (the same numbers as
-C with the deposited cell). Of 13 long-axis battery sets, 7 are bit-identical
(p.hkl md5) and 2 never unfold a row, 8v4o and 9yl4 have the same intensities (CC 1.0) without needing
the long-axis rescue, 4mtk refines a cell closer to the deposit at half the
wall time, 4nwv merges better (R_meas 0.58 -> 0.47, ISa 6.7 -> 8.4).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Stage A offered the b-unique two-fold (reference setting) on the lattice's holohedry alone, but held
its a- and c-unique siblings (pass-1 settings) to CellHostsRotations at 2e-3. On a cell refined free
~0.2 deg off orthogonal only the b-unique rung existed, so when the holohedral 222 was refused the
setting, not the data, chose the two-fold axis. Stage B already offers non-reference settings
without asking the cell, for the same reason; Stage A now does likewise where a holohedry is given.
With siblings of one order eligible together, ties were broken on the class CC alone, which a twin
law at fraction ~0.4 can win. A twin law cannot extinguish an axial row (every principal row maps to
itself or its Friedel mate), so ties now go first to the group whose pure two-fold axes carry the
more screw evidence on their own odd axial row - judged exactly as Stage B judges that zone - and
only then to the CC. Rows on 3-, 4- or 6-fold axes are not read.
Open arm: 6p8j (deposited P2_1, b = 152 A) was P 1 1 2 on the 88 A axis, now P 1 21 1 on the 152 A
axis (152 A odd row 158 nats, 46 A row 36, 88 A row refused; R-free 0.214). 9hnc and 5cc8 p.mtz
md5-identical (siblings offered but refused by added-operator R). On five passing P2_1 sets in an
orthorhombic metric the true axis carries all the evidence (offline replay), so no change there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The background ring's high-side clip, mean + 4*sqrt(mean), assumes counted photons. A CCD reports
ADU: on a Rigaku Saturn a background pixel scatters 4-5x its mean, so the clip sat at ~2 sigma, cut
the upper tail of every ring and added a positive offset to every reflection, compressing the
intensity statistics (a false 422 on one twinned I41 crystal).
No header carries a usable gain (SMV states none, every marCCD file the writer default of 1 ADU per
photon), so it is measured: background_gain::MeasureFrame takes the variance per unit mean of the
difference of pixels two columns apart, per level, clipped, on every fourth row of up to 16 of the
pre-scan's frames; Gain is the pair-weighted median over the levels. The clip becomes
mean + n*sqrt(max(g,1)*mean) (BraggIntegrationSettings::bkg_pixel_gain, CPU and GPU engines).
Only for SMV/marCCD input (ProcessConfig::counts_in_adu, set from the file format); HDF5 and CBF
keep g = 1, the identity. One log line reports the value. No new option.
Measured g: Saturn 944+ 4.7-5.2, Saturn 944 2.7, A200 0.68, ADSC ~0.3, marCCD ~0.3 - so the
ADSC/marCCD/A200 sets keep the photon clip exactly.
Checks (reflection data via gemmi and p.hkl, against the rc175 battery outputs / base build):
identical on 11if, 7qis, 8a1a, myob_x10sa, cytc_x10sa (photon counting), 4bwl, 5epe (ADSC,
marCCD) and 6cee (A200). Saturn sets, battery baseline -> this change:
3r6o I4122 R_meas 19.3% ISa 4.4 <|L|> .321 -> I41 (correct) 12.1% 8.1 .384
3mc4 R_meas 10.2 -> 9.8 %, ISa 12.1 -> 12.7, d_min 1.79 -> 1.75 A, <|L|> .360 -> .391
5uth 12.6 -> 11.8 %, ISa 10.4 -> 10.1, <|L|> .426 -> .483, offset vs |Fc|^2 0.21 -> 0.00
5vml 10.2 -> 9.6 %, ISa 11.0 -> 11.6, <|L|> .426 -> .475, offset 0.15 -> 0.05
5cc8 8.1 -> 7.9 %, ISa 10.8 -> 11.2, <|L|> .469 -> .494
3meb 14.5 -> 14.6 %, ISa 13.2 -> 13.2, <|L|> .442 -> .457
Space groups unchanged except 3r6o. Pre-scan cost: +0.5 s on a 9 Mpx ADSC sweep, within noise on
a 15 Mpx marCCD sweep, nothing on HDF5/CBF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ReadAttrStr and HDF5DataSet::ReadString read every string as fixed-length,
sizing the buffer by H5Tget_size. For a variable-length string (what h5py
writes for a str, UTF-8) that size is a pointer, so the read put a char*
into the buffer and returned garbage. transformation_type then never equalled
"rotation", and an EIGER master whose goniometer attributes were written that
way (depends_on, transformation_type, units on /entry/sample/transformations)
was processed as stills although it holds a valid rotation sweep.
All string reads now go through one helper, ReadHDF5String, which handles both
fixed- and variable-length strings in either character set; ReadElement<string>
already handled the variable-length case and now uses the same helper.
Test: JFJochReader_Goniometer_VariableLengthStrings reads the rotation axis
from a master with fixed-length ASCII and with variable-length UTF-8 string
attributes and depends_on dataset.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Near-perfect twins of a subgroup are undecidable from intensities, and a merge
in the higher group cannot be twin-refined. Where the adopted group may have
absorbed a twin law - a twin gate refused it and was overruled, or a group above
P1 was adopted on a merge that read twin-like before the search - the same
observations are also merged in a subgroup and written beside the answer, with
the twin law and its fraction in _pdbx_reflns_twin.
The subgroup is the one a decision named (the group a refused promotion was
adopted over, the indexed lattice's group a metric re-ask promoted from), else
the index-2 subgroup the search confirmed as a finalist whose lost two-fold
agrees worst with the intensities. It is merged from the same engine, to the
answer's resolution, without writing per-frame scales back; it carries the
answer's free set (flags read off the answer's merge by the adopted group's
asymmetric unit) and is written in the subgroup's reference and standard
setting (a C 1 1 2_1 description is reduced to P 1 2_1 1). The fraction is
Yeates' H on the pairs the law relates (1/2 - median H, a lower bound).
At most one file; other qualifying subgroups are named in the report
(TWIN_SUBGROUP_* keys). The answer's files are byte-identical.
IndexTwoTwinSubgroups and EstimateTwinLawFraction are new and tested; the
point-group ledger now carries each hypothesis's representative group.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Whether a twin law can exist was read off a hard-coded list of Laue classes
(4/mmm, 6/mmm, m-3m and rhombohedral -3m "holohedral", everything else
eligible). That is a property of the lattice, not of the class: a C-centred
orthorhombic or a monoclinic lattice with no special metric admits no twin law
for mmm or 2/m either, and a pseudo-merohedral one admits a law the class says
nothing about. On five pseudo-merohedral sets the report said "Twinning
INDICATED - refinement needs a twin law" where the lattice admits none.
TwinLawPossible asks gemmi's find_twin_laws (Le Page two-folds within 3 deg,
as RfreeFlags and ReindexAmbiguity already do), and AnalyzeTwinning takes the
cell to decide it; in P1 or without a cell a law stays possible, so the
pre-promotion analysis and the space-group search are unchanged. Twin-like
statistics (<|L|> below 0.44) above the false-operator bar where no law is
possible are now reported as NO_TWIN_LAW: not a twin in this group, so the
symmetry may be too high or something else narrowed the distribution.
Report-only: no reflection file changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
