A new last column, "disposition", gives each frame's merge disposition as
a number to plot directly: 1 merged, 0.5 downgraded, 0 rejected; nan for
a frame that never got a scale (never reached the merge) and on a run
that merged nothing, as the file writes every value nothing measured.
Existing columns keep their indices.
0.5 is a marker, not a weight. "Downgraded" is a label applied after the
merge (RotationScaleMerge::CountSweepDisposition marks the merged frames
inside a flagged sweep-quality range); the merge applies no factor of its
own to them - they are carried at whatever weight their own scale and
sigmas give - so there is no fixed or per-frame reduction factor that
could honestly be written instead.
Writer-only change: on three open-arm sets with downgraded and rejected
frames p.hkl is identical to the same commit without it, the first ten
columns are unchanged, and the column's 1/0.5/0 counts equal the report's
merged/downgraded/rejected counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Bounded memory and efficient closing of old data files on long runs, a failure ending the run in
an error state the broker hears about, and the pusher's log line when a stalled writer makes it
drop images; the master-file ordering moves here from the Fixes line. Owner-requested edit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Every stage between the socket and the HDF5 write was bounded by a message count and never by
bytes: ZeroMQ receive watermark 100, cbor_fifo 200, outside_fifo 1000, repub_fifo 200 - up to
some 1500 images in flight per writer process, each a whole compressed image, so 6-12 GB for a
JUNGFRAU 4M and 30-60 GB for a 9M, and the writer's memory was a quantity nobody had chosen.
ReceiveRingBuffer is a fixed ring of bytes owned by the ImagePuller base class, so the ZeroMQ
and TCP pullers share it. A message is received straight into it (TCP reads the payload into
the ring; ZeroMQ copies the message in) and travels as a shared_ptr region in ImagePullerOutput;
the deserialised image points into that region, republish sends it as is, and the bytes come
back when the last holder lets go. Space is reclaimed in arrival order: a region released out
of turn - republish finishes with messages in its own order - is marked and waits for the ones
in front of it. Because of that, what republish may hold is capped at a quarter of the ring,
and an image over the cap is not republished, exactly as one that finds the republish queue
full; otherwise a stalled republish consumer would pin the ring and slow the writer to its
rate. A message larger than the whole ring (a START or END of a very long run) goes on the heap,
so it cannot wedge the receiver.
A full ring stalls the receive thread, and what that does depends on the stream. On TCP the
pusher waits behind the writer's BUSY heartbeats and gives up only after a minute in which no
image at all was written. On ZeroMQ the pusher's send blocks for one second, then drops the
image and sends without waiting for the rest of the run - so on ZeroMQ a ring that is smaller
than the writer's stalls costs images, where the old 1500-message queues absorbed them. That
is the trade: the budget is now a number, and it has to be chosen. jfjoch_writer sets it with
-m/--rcv_ring_mb, default 2048 MB per writer process; the ZeroMQ receive watermark default goes
from 100 to 10 messages, since it now only counts what sits beyond the ring. The ring's peak
use is in the end-of-run log line, and docs/JFJOCH_WRITER.md has a Memory section with the
sizing rule.
The puller's own default is no ring at all, which keeps every other user as it was: the
broker's DECTRIS stream2 puller (JFJochServices, JFJochReceiverService) stays zero-copy with
the ZeroMQ message as the buffer, and the broker's direct HDF5 writer (HDF5FilePusher ->
FileWriter) never goes through a puller. The puller threads now also catch std::exception, as
the writer does since the previous commit: deserialising the END message of a million-image
run allocates a vector per per-image array, and a bad_alloc leaving the thread ended the process.
Tests: ReceiveRingBufferTest (reuse, wrap-around, full ring, out-of-order release, heap
fallback, zero size, stop, a region outliving its ring); ZMQImageCommTest_SmallRingBuffer (64
one-megabyte images through a three-image ring, nothing lost, peak within the bound); one of
the two writers in TCPImageCommTest_2Writers_WithAck runs on a three-image ring.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A new page, RUGNUX_MODEL_AND_REFERENCE.md, beside the other rugnux pages in the toctree and
linked from RUGNUX.md and the --model section of RUGNUX_ADVANCED.md. It lists every option that
brings in a model or reference data (--model, --mr, the PAE file beside a prediction, -z and
--reference-column, -S, -C, --force-rotation-lattice, --report-resolution), sorts every decision
into data-only, model-informed label in <prefix>.mtz (hand past the anomalous veto, indexing
ambiguity, setting), model-driven separate files (<prefix>_mr) and report-only statistics, walks a
--model run step by step, says exactly where a model can touch the space group and what guards it
(the null, the anomalous veto, the indexing margin, the MR rules, the wrong-model battery step),
and describes the free set and model bias (a deposited model's R-free on rugnux's own test set is
not cross-validated; -z with the PDB's structure factors inherits the depositor's set). Includes the
one pre-merge use of a model: serial stills with -C/-S, where it breaks the per-image indexing
ambiguity. Documentation only; behaviour as merged in rc175 (7ed970503).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
- All 84 DOIs checked (82 on Crossref, 2 DataCite): title, journal, volume, pages and first author
match; Matthews 1968 and Yeates 1997 links percent-encode their parentheses; the MOSFLM link
points at a page that answers.
- autoPROC: an explicit acknowledgement of Clemens Vonrhein (Global Phasing) for the principle that a
pipeline should verify the metadata it is given - beam centre, geometry, goniometer - and correct
it where it is wrong; Vonrhein et al. (2011) Acta Cryst. D67, 293-302 (DOI checked).
- Zhang, Sauter et al. (2006) DISTL, already credited in SpotFindingSettings.h, gets its entry.
- Molecular replacement: tNCS looked for in the native Patterson as Read, Adams & McCoy do; the same
credit at DetectTranslationalNcs (comment only).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
One page listing the CUDA engines by pipeline stage - decode and preprocessing, per-image analysis
(azimuthal and ROI integration, spot finders, extractor), pre-scan accumulators, indexing,
prediction and integration, the scaling engine, model validation and molecular replacement - each
with what it computes, how closely it agrees with the CPU, and a link to the section describing the
method. Added to the toctree after CPU_DATA_ANALYSIS_DECISIONS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
- CPU_DATA_ANALYSIS_IMAGE §0: CBF byte-offset and marCCD widening are decoded on the card; the host
decoders are the CPU build's path.
- CPU_DATA_ANALYSIS_DECISIONS §14: the null runs for every model the rigid body is tried on; the
device choice is made once on the card's total (structure factors and rigid body a quarter, scale
fits an eighth) and a CUDA failure ends the validation, deciding nothing; a subgroup the model fits
is written as <prefix>_mr; new §14.10 on molecular replacement (model kind, rotation and
translation functions, likelihood, Matthews, tNCS and P1 rules, null, what it may change).
- CPU_DATA_ANALYSIS: scope and references gain molecular replacement.
- CPU_DATA_ANALYSIS_INDEXING: the near tie decided early and re-checked once every frame is in.
- RUGNUX: merging is the default and _process.h5 optional; live following of a collection with a
broker URL; --model's MR fallback.
- RUGNUX_INSTALL, RUGNUX_OVERVIEW, RUGNUX_TUTORIAL, RUGNUX_REPORT: GPU work list, <prefix>_mr outputs,
MR in --mode mx for rotation and stills, rigid body only in --mode scale.
- RUGNUX_ADVANCED: -N default as the usage states it.
- battery README: the ci tier has 28 sets.
- ModelValidation.cpp: a comment still said a GPU failure reran the validation on the CPU (comment only).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
RUGNUX_ADVANCED's --model page described the rigid body alone. It now says how the unified flow
places a model: an experimental model by the rigid body, checked against its null in the answer's
or the twin hypothesis' merge, and by molecular replacement where it is not placed; a prediction
straight to molecular replacement; --mr skipping the rigid body; MODEL_PLACEMENT. It says what a
placed model may change in <prefix>.mtz (the hand, and from the rigid body an indexing-ambiguity
reindex) and that another group or the twin subgroup the model chooses goes to <prefix>_mr.mtz/.cif
with <prefix>_mr_model.pdb/.cif. --mode scale places by the rigid body only (no molecular
replacement, --mr refused); a full run does molecular replacement on serial stills too (checked:
a stills run with a predicted model reports MODEL_PLACEMENT= MR). The options table gains --mr
and the --model row the fallback.
RUGNUX_INTEGRATION no longer says rugnux does no molecular replacement, and ACKNOWLEDGEMENT names
--model beside --mr for predicted models.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The --mr line now says what the unified --model flow does for a user: a predicted model is
accepted, and a model the rigid body does not place goes to molecular replacement, with --mr
forcing it. One viewer line for the Files panel refresh (F5 and the poll) and the combo boxes
that no longer reset during live collection. Owner-requested edit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
One --model now does the right thing by itself:
- A prediction (AlphaFold/ModelCIF, read from the file) goes straight to molecular replacement.
- An experimental structure is placed by the rigid body as before. The null now runs on every
--model run (no NOT_TESTED): "placed" = the fit beats its null (4.51 sd) in the answer's merge
or in the twin hypothesis' merge. Placed: behaviour unchanged (hand and reindex may go into
p.mtz); a subgroup the model fits and prefers is written as <prefix>_mr.
- Not placed (or no crystal cell): molecular replacement with the experimental-model path. It may
relabel the hand into p.mtz past the anomalous veto, never reindexes; another group or the twin
subgroup goes only to <prefix>_mr. Maps, R-free and CC come from validating the MR placement with
every decision switched off; MODEL_FIT stays the rigid body's verdict.
- The twin hypothesis' merge is made in memory with a model where a metric re-ask promoted the
group or the model's group has fewer rotations than the adopted one; written and reported only
where the model chooses it.
- --mr is kept as "--model, searched for by MR straight away".
- Report: MODEL_PLACEMENT= RIGID_BODY|MR|NOT_PLACED, MODEL_DECISIONS_TAKEN gains MR_ENANTIOMORPH,
MODEL_CHOICE_FILE_PREFIX/SPACE_GROUP; the MR keys move into 5. MODEL VALIDATION; MODEL_FIT absent
where no rigid body ran. MR NOT_PLACED is never a warning.
- Battery wrong-model step: check (c) also fails a wrong model placed by MR.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
- The option is --mr <file>. What kind of model it is is read from the file: a prediction where it
is ModelCIF (_ma_qa_metric) or has no cell and one B value per residue in 0-100 (AlphaFold PDB/mmCIF);
an experimental structure otherwise. A prediction is handled as before (pLDDT >= 70 trim, B from
pLDDT, PAE domains where the AlphaFold DB's PAE file sits beside it). An experimental structure is
taken as its polymer - no waters, ligands, hydrogens or alternative conformers, occupancy 1, isotropic
B as refined - with a coordinate error of 0.67 A (a prediction's at pLDDT 90) for sigma_A; a B column
above 100 no longer stops the search.
- The report section is MOLECULAR REPLACEMENT (EXPERIMENTAL), its keys MR_* (MR, MR_MODEL_FILE,
MR_MODEL_KIND, MR_DECISION, MR_SPACE_GROUP, MR_COPIES, MR_HAND_CHECK, MR_SECONDS, ...); the <prefix>_mr
outputs are unchanged.
- The twin hypothesis' merge that is made only for the model to test (a metric re-ask promoted the
indexed lattice's group) is written and reported only where the model chooses the subgroup.
Raw deposited coordinates of nine control sets, unconverted: every one placed and confirmed against the
deposition (all chains as one body), the groups the same as with the prediction or the prepared model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
- Likelihood (Read 2001; Z-score over random positions as McCoy et al. 2005): log-likelihood gain of
Rice/Woolfson E distributions, sigma_A from the pLDDT coordinate error with its scale refined, to
2.5 A (or the 20000 lowest-resolution reflections). Further-copy TF peaks ranked by the gain on the
search's reflections (peaks that lower it dropped); a copy whose TF Z is below 8 also stands out where
its refined likelihood Z-score reaches 8 - only where the copies placed are as many as Matthews makes
most likely. The likelihood is computed only where a translation function does not stand out.
Copy structure factors exact from one FFT of the cell's grid, or read off the molecular transform
where it reaches the likelihood's resolution.
- PAE domains: the AlphaFold DB's predicted_aligned_error JSON beside the model (same name pattern) is
read automatically; average linkage on the symmetrised PAE (10 A, domains >= 40 residues). Where the
whole model is placed nowhere, each domain is searched by itself side by side; placed domains must
agree, and a domain alone may only confirm the answer's group.
- Twin comparison: where the answer's merge chose and the twin hypothesis' merge exists, the chosen
placement read as a crystal in the subgroup (its symmetry images as copies, the subgroup's setting
found from the two cells) against the model searched for afresh in the subgroup, both against the
data twinned by the law (fraction refined on the working set, Yeates 1997); the subgroup is taken by
a 0.03 R-free lead plus its own null/margin -> TWIN_SUBGROUP, written to p_mr with law and fraction.
With --af-model rugnux makes the subgroup merge wherever a metric re-ask promoted the indexed group.
- tNCS rule: a group with fewer copies beside tNCS (screws bridged by the vector, or the enantiomorph)
no longer leaves the choice undecided where the chosen copies, carried into it and refined, fit
worse by 0.03 in R-free.
- Speed: the molecular transform's FFT on the GPU (cuFFT, serialised), no rigid body for trials that
lower the likelihood, re-search only with tNCS, PAE domains searched in parallel.
Survey (193 sets, MR replay on stored merges): single GREEN 58 -> 68, multi 48 -> 58, partial 8 -> 9;
4bwl (and, in full runs, 2wnq and a withdrawn entry of the same protein) TWIN_SUBGROUP P21 instead of
a wrong AGREES; a P21212 set with tNCS now SPACE_GROUP P21212 instead of UNDECIDED; no wrong call.
CPU and GPU builds make the same decisions on 18 demo/multi-copy/twin sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Copy number. The hard "Vm >= 4: MULTI_COPY, no decision" rule is gone. Copies
are searched one after another up to the largest copy number with a Matthews
probability of 1% (a Gumbel distribution with Kantardjieff & Rupp's 2003 mode
and mean, no resolution dependence); COPIES_LIKELY reports the most probable.
Sequential search. Copy n's translation function carries copies 1..n-1 as a
known partial structure: the cross term 2 Re(conj(F_known) F(t)) goes on the
same FFT grid (CPU FFTW and GPU cuFFT; KnownStructureFactors, AssemblyCorrelation).
Its peaks, past those that clash, are ranked by the correlation of the whole
assembly with E^2 (the TF itself, linear in |F|^2, rewards copying a placed
copy onto an origin shift), the first that stands out (TF Z >= 8), packs and
lowers R-free by 0.01 is kept; each copy is refined beside the others held
fixed (RefineRigidBody's fixed copies), and the first copy again beside the
second, the earlier ones once more at the end.
Packing: ClashFraction counts C-alpha clashes with other copies and all their
mates, binned (7mzt: 51 s of clash checks -> 1.9 s).
Search quality: rotation-function peaks hill-climbed off the 6 deg sampling
(6ze4: RF Z 10.4 -> 16.0, 3.3 deg off the deposited orientation -> 0.2); the
first copy goes down the TF list until three placements that pack are
refined, so a group is CLASH only where no offered orientation packs; groups
are searched side by side, and every group the best one leads is searched
again with the best one's copy orientations tried first, the better search kept.
tNCS: the native Patterson of E^2 (5-10 A) off the lattice; placed copies'
orientations join every further copy's search; and where the tNCS vector turns
one group's screw axes into another's (TncsRelatesGroups), a group whose search
stopped at fewer copies does not lose to the other - UNDECIDED. Measured on an
in-house tNCS crystal (vector c/3 along the 3-fold): both hands fit with three
copies to within 0.01 in R-free, and where one hand's search failed the other
was taken as ENANTIOMORPH; now UNDECIDED.
Null: the replicates are placed by the translation function in their own
orientation, as the search placed the real one (a null at the real centroid
reads high), and the gate is 4.51 = t(8) sqrt(1 + 1/9), 3 sd of a nine-draw
null, the gate --model's MODEL_FIT is now read at.
Speed: placements scored on the GPU structure-factor engines; further copies'
trials score with the first copy's bulk solvent held, the assembly refitted at
the end. Survey of 193 sets (jfjoch_mr_replay on stored merges, RTX 5080
shared with other jobs): MR median 1.15 s, max 24 s; demos 0.8-1.1 s.
Report: COPIES and COPIES_LIKELY columns, MODEL_PRIOR_COPIES.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Three defects the battery's wrong-model null found (5 sets x 5 deposited models of unrelated proteins):
CC(model, data) significance. The per-shell sigma was Fisher's transform on independent reflections,
but the data and a scaled model share the falloff across a shell, the anisotropic B fitted to these
very data, the bulk solvent and the symmetry enhancement epsilon - whatever the model is. 21 of 24
wrong models read CC_MODEL_CONFIRMED_TO_D_MIN to the data's d_min (10-15 sigma in most shells).
The correlation is now formed on I/epsilon and |F_model|^2/epsilon, and its null is empirical: the
data at h against the model at h + d for the 26 nearest lattice offsets, which keeps everything smooth
in reciprocal space and loses what the model knows of h. SIGMA = (z - mean) / max(sd, 1/sqrt(N-3)) in
Fisher z over those 26, a new CC_NULL column shows the null, and a shell confirms only beside a coarser
shell that also clears 3 sigma (one shell alone did so once in 216 wrong-model shells). Prototyped on
the p_maps.mtz of a full open-arm run and the null runs: wrong models confirm 0/24 (was 21/24), the
deposited models confirm 197/200 (was 198/200 in the prototype's reading; the one lost has CC_MODEL
0.05-0.13 in every shell). The report no longer says a model cannot agree by accident.
MODEL_FIT gate. Accepted at 3.0 sample sds of a nine-draw null, as if the sd were known; an unrelated
P1 helical model on a P3(1) helical crystal was ACCEPTED at +3.05. Against 48 draws it reads +2.09 and
the best random placement fits as well as it does, and drawing the null about the cell axes instead
changes nothing (+2.15): the cause is the nine-draw sd, not the orientation. The gate is now 3 sigma
allowing for that: t(8) * sqrt(1 + 1/9) = 4.51. The deposited models of the open arm read +7.0 and up.
The indexing lead stays at 3 sigma (INDEXING_MARGIN_SIGMA): it is read only behind the fit gate, and
a deposited model settles one at +4.15.
Rigid-body GPU reserve. The finest zone's observations were counted by the data's d, but the zones
count them by the d of the index in the model's ASU, and a model's group need not keep the data's
metric (an I2(1)3 model on I222 data: 2320 in the zone against 1712 reserved), so the validation died
with "zone over the engine's reserve". Counted the zones' way, under every indexing probed.
Tests: ModelValidation_CCModelUnrelatedModelConfirmsNothing (Fisher alone reads +4.7 on an unrelated
model with an anisotropic falloff; the new sigma stays under 1), ModelValidation_WrongModelInAGroupOffTheDataMetric
(fails with the old count), ResultReport_ModelValidationSection (CC_NULL column, lone-shell rule).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The ENANTIOMORPH hand check ran a whole ValidateAgainstModel on the placed model: model read and frame
probe, a rigid body, a 9-replicate null (~0.5 s of the ~0.6-0.9 s), sigma_A, three maps written. The
decision needs only --model's rule: the model fits its null (the MR choice has already required its
own null of the same construction) and the anomalous map at the model's atoms is not inverted.
ReadModelAnomalousMap (ModelValidation) computes F_calc and F_mask to the data's resolution (GPU engine
where there is one), carries them through the bulk solvent MR scored the placement with (the overall
scale and anisotropic B do not move phases), and reads the anomalous map with the same code as the
validation: its coefficient construction and its reading at the atoms are factored out of Validate
(anomalous_coefficients, read_anomalous_map) and used by both, so plain --model is unchanged.
Hand check 0.6/0.6/0.9 s -> 0.027/0.028/0.040 s on 11if/7ris/9zmu; same ADOPTED/VETOED on all seven
cases tried (the three demos, two anomalous lysozyme sets and their mirrored-detector runs), with the
scatterer means and deepest troughs within 0.1 sigma of the full validation's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- An MR ENANTIOMORPH decision is taken into the answer's label only as --model takes a model's
enantiomorph: ValidateAgainstModel runs on the placed model (in a scratch directory, removed after)
and its adopted_model_enantiomorph - fit over its null, anomalous map at the atoms not inverted -
decides; otherwise ENANTIOMORPH_VETOED. The hand-check numbers go to the MODEL PRIOR section.
- Rotation and translation functions on the GPU (MolecularReplacementGPU.cu): one block per
orientation for the rotation function; per orientation the copies, the bin weights, a gather of the
Crowther-Blow coefficients (CSR lists, fixed order, no atomics) and a cuFFT C2R, block-reduced
statistics finished on the host in a fixed order. The CPU path (FFTW) is unchanged in what it
computes; the orientation-independent part of the translation function (TranslationSetup) is now
built once per group on either path.
- Rescoring: a placement clashing with its mates over twice the limit is not refined, one rigid-body
engine per group, and the null is built only for the candidate the choice falls on.
Measured MR step on the RTX 5080 (before -> after, seconds; hand check extra where it runs):
3r6o 1.77 -> 0.67, neg 2.14 -> 0.37, 11if 1.20 -> 0.51 (+0.6 hand check), 7ris 1.46 -> 0.56 (+0.6),
9zmu 4.71 -> 1.07 (+0.9). Rotation 0.50 -> 0.02 s and translation 1.17 -> 0.13 s on 9zmu.
Mirrored-detector runs of two anomalous lysozyme sets: the hand check vetoes (map inverted at the
model's atoms); the unmirrored runs adopt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A predicted model (AlphaFold mmCIF, pLDDT in the B column) is trimmed at pLDDT 70, given B-factors
from the pLDDT (Oeffner et al. 2022) and placed by molecular replacement, one copy in the ASU:
- rotation function on a uniform 6 deg sampling of the point group's rotation ASU, each orientation
scored by the correlation of E^2 with the symmetry-summed |F|^2 read off the model's molecular
transform, computed once by FFT and interpolated (Navaza 1994);
- full-symmetry Crowther & Blow translation function (one FFT per orientation and group, through
MapFromFPhi/FFTW) for the 50 best orientations in every group with the merge's rotations;
- the best three placements per group through RefineRigidBody (GPU where there is one), bulk solvent
and R-free to 3 A, a C-alpha packing check against symmetry mates, Matthews, and the
random-orientation null for a placed candidate.
Decision: PLACED = TF Z >= 8, clash <= 5%, Vm in [1.5, 4), null >= 3 sd. Within the answer's merge
the placed group with the lowest R-free is chosen where it leads every other packing group by 0.03;
the twin hypothesis' merge is searched, and chosen, only where no group of the answer's merge can
hold the model at all (Matthews, packing). An enantiomorph relabels the answer as --model does;
another group goes to <prefix>_mr.mtz/.cif plus <prefix>_mr_model.pdb; the MODEL PRIOR report
section carries the table. The search runs on the CPU in both builds; without --af-model nothing
changes.
Measured (RTX 5080, MR step): 3r6o twin I41 chosen over I4122 (Matthews + clash) in 1.8 s, solution
0.09 A CA rmsd from Phaser's; 11if P41 -> P43 1.2 s; 7ris P3121 -> P3221 1.7 s; 9zmu P6122 -> P6522
4.7 s; 11if's model against 3r6o's data NOT_PLACED.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
TCPStreamPusher::SendImage is called from the receiver's frame-transformation
threads. It used a 128-deep per-connection queue and waited when it was full.
With images routed per file ((n / images_per_file) % writers), every writer
receives a whole file as a burst at the full frame rate (1000 images/file at
2 kHz fills 128 in ~64 ms), so the queue filled on every burst and all
transformation threads ended up waiting on one writer. They then stopped
returning FPGA frame buffers, and the FPGA dropped packets: partial images,
images without a background estimate, data loss on every writer. The ZeroMQ
pusher has an unbounded queue and never blocks, which is why it coped.
- The per-connection queue is unbounded: every element owns a distinct
image-buffer slot, so the bound is the image buffer's slot count. When slots
run out the receiver drops only the write (GetImageSlot() == nullptr), after
analysis. Absorbing one file burst therefore needs an image buffer of about
images_per_file slots (~20 GB for a 9M at 1000 images/file); below that,
images are still dropped, now as writer-queue-full instead of packet loss.
- SendImage never blocks. A dead or wedged writer is still caught in SendAll
(15 s silence / 60 s without progress), after which WriterThread returns the
queued slots unsent.
- The 200 ms fallback in StopDataCollectionThreads is gone: at END it closed
the connection of any writer whose queue stayed full for 200 ms - a normal
state for a slow but healthy writer - discarding up to 128 queued images and
never sending END. END now follows the whole backlog.
- A per-connection enqueue mutex makes "check active, enqueue" atomic against
"clear active, enqueue the end sentinel": an image could land behind the
sentinel when a connection was torn down mid-run, leaking its slot
(CheckIfBufferReturned then threw) or being sent as stale data in the next
run. New stress test SendImageRacingEnd_NoSlotLeak (verified to fail with
the race widened and the lock removed).
- TearDownConnection marks the connection broken first, so the destructor
returns a backlog instead of sending it.
Writer side:
- jfjoch_writer -b/--rcv_buffer sets the TCP receive socket buffer (was
hardcoded to the kernel default).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Narrow, guarded: the doubled pass runs only where the added class grows with resolution and is resolved; the doubled
cell is adopted only where one pseudo-translation explains the added class in every shell and the point group does not
rise. Fixes the one remaining lattice failure (half cell -> deposited cell, R-free 0.574 -> 0.275); p.hkl identical on
the ci tier, the supercell gate set, 78 further supercell-candidate sets and all private sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The doubled pass now runs only on resolved, rising classes (two sets of ~134 with probe logs), so the
twenty-minute model placement in a doubled cubic cell can no longer arise, and the adopted answer is
validated again (9min: R-free 0.275 against the deposited model, 0.574 in the half cell).
Final rules, measured on gate + ci tier + all sets with a de-modulated axial row + 49 sets with a
baseline supercell occupancy >= 8% + the private arm: 9min fail -> pass (P212121, doubled cell);
every other set run keeps its verdict and a bit-identical p.hkl.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
On strong data a class at one or two percent of the lattice rises significantly (0.009 to 0.059 at 4
standard errors on a ci-tier triclinic crystal, which was then adopted doubled and failed), and a half
cell missing a few percent of its intensity still describes the crystal. The doubled deposition ends
at 0.35-0.40, those classes at 0.06.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A class that does not rise keeps the half cell whatever the doubled merge shows, so the doubled pass
is no longer run for it - on a ci-tier crystal (R-centred, class at 1-2% of the lattice) that pass ran
the card out of memory and failed the run. The doubled pass also runs without --model: placing the
model in a doubled cubic crystal's 776 A R-centred cell took twenty minutes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Supercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own
limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the
SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice
reflection predicted on the same frame lies within r3, and each resolution shell is read against the
median of the six other parity classes - which a single doubling leaves empty - as an excess in units
of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z
beside it ("Supercell probe to the limit" in the log).
Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its
noise and at more than twice the strongest of them, in at least three shells), the run integrates and
merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along
the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's
merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both
(DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT
(the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the
class reaches a quarter of the lattice's intensity in the finest shell both are measured in).
The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a
triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed.
Measured on forced doubled-cell merges (rc175 b530c2d base): the open-arm crystal deposited in the
doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell
(odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals
deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at
8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low
resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the
neighbour correlation, the separation from the nearest lattice reflection after the r3 filter -
separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call.
Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no
longer advises -C with the doubled cell, which the indexer reduces back to the half cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Reframe the Global Phasing / autoPROC acknowledgement as inspiration and
name the ideas: header metadata (beam centre, geometry, axis) must not be
trusted blindly (autoPROC, Clemens Vonrhein in particular); the beam
centre is determined from the images (rugnux does it its own way); and
diffraction anisotropy must be analysed and reported (STARANISO, cited
with the reference the STARANISO server recommends; no DOI exists).
One-line pointers in data-analysis §1.4 and §13.5, and both works added
to the CPU_DATA_ANALYSIS references. The Sheriff & Hendrickson, Popov &
Bourenkov and Evans & Murshudov credits for the estimators are kept.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The de-novo FFT indexers implement the 1D-projection method of Steller,
Bolotovsky & Rossmann (1997), which was already credited; the 3D Fourier
transform indexing idea it descends from (Bricogne 1986, EEC PSD software
workshop, Phase III, LURE, p. 28 - no DOI) was not. Add it to
ACKNOWLEDGEMENT.md, the CPU_DATA_ANALYSIS references, the §5 text and the
in-source credit, and add the DOI of Powell (1999) to the MOSFLM paragraph.
3D-FFT papers (Campbell 1998, Gildea et al. 2014) are not cited: the code
does not compute a 3D transform.
Add a section acknowledging Global Phasing (Bricogne and colleagues) and
autoPROC (Vonrhein et al. 2011) as the expert-system model rugnux aims at.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A crystal twinned by reticular merohedry of index n - obverse/reverse
rhombohedral domains, or a cubic crystal twinned by a 60 deg rotation about
[111] (the spinel law, index 3) - has its two domains' reflections on a
common lattice of n times the primitive volume of either domain's own cell.
That coincidence cell indexes more frames and more spots than the domain's
cell, so the first pass adopted it (a hexagonal/orthohexagonal or R:R cell
for an F- or I-cubic crystal), and nothing downstream could recover the
true lattice or Laue class.
A new step after the sub-lattice hypothesis (TwinCompositeHypothesis), for
n = 2 and 3, counts the validation spots on the standing cell in the four
classes of the pair of index-n sub-lattices whose union holds the most
spots: on both, on each alone, and on neither. Two domains leave the
(1 - 1/n)^2 of the nodes on neither empty but for chance coincidences; one
crystal, whatever sub-lattice a pseudo-translation favours, populates the
nodes off the stronger sub-lattice alike whether they lie on the second
sub-lattice or on neither. The step adopts the stronger domain's cell when
the two sub-lattice cells are congruent (2 % / 2 deg), the weaker domain's
own class holds at least 20 spots at more than four times the per-node density
of the class on neither, and the domain's cell indexes a majority of the
validation frames. The other domain is left to the leftover-lattice census.
Evaluated on the private, inhouse and a twin/tNCS/rhombohedral/cubic/
long-axis selection of the open arm: it fires on the two known index-3
twins (a spinel-twinned F-cubic crystal, now indexed F-cubic; the in-house
I-cubic insulin set with a 60 deg [111] second domain, R3:R -> I23) with a
weak/neither per-node density ratio of 60 and more (the class on neither
holds 0-8 spots against 250-2850 on the weaker domain), and nowhere else.
On every other set the ratio stays between 1.0 and 1.6, the top of that
range only on runs with a hundred or so validation spots, where the pair
picked as the emptiest of 7 or 13 forms carries the scatter; hence the
factor 4. Both twins now merge in their cubic group, and the sets
re-run against the release baseline otherwise did not change. Cost when it
does not fire: tens of milliseconds per first pass.
Merges two prototypes: the class census and n = 2 coverage of one, the
sub-cell congruence requirement of the other; no environment switch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
4MTK (P61, c 653 A, PILATUS 6M miniCBF, SBGrid 235), 4NWV (I222 virus-like
particle, 370-411 A axes, Rayonix MX-300 marCCD, SBGrid 9), 7D98 (P6122,
c 602 A, EIGER 4M miniCBF at 1.9 A, XRDa 13), 9V16 (P41212, c 451 A, EIGER1
16M HDF5, XRDa 338) and 4P8R (P6522, c 701 A, Rayonix MX-225 marCCD, IRRMC),
tagged large-cell (plus long-axis / virus). References are the deposited
space group, cell and resolution. 7D98 is pinned to its 250-frame miniCBF
sweep: the HDF5 masters beside it repeat frames 61-260 in 60-frame pieces.
4P8R's archive ships per-frame .bz2 files, decompressed after extraction.
EXTERNAL_TEST_DATA.md gains the five rows (DOIs resolved), the archive notes
and the updated collection counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Where a twin hypothesis (<prefix>_twin_<SG>) was written, the model is also
validated against that merge: as merged, and as twinned by its law -
|F_twin(h)|^2 = (1-a)|F_model(h)|^2 + a|F_model(Th)|^2 from the placed, scaled
model, one overall scale refitted per a, a scanned 0..0.5 on the working set
and R-free read at it (ValidateAgainstModel's new twin_law argument; GPU
structure factors as before). The maps and the placed model are then written
from whichever merge has the lower R-free; the report gives MAPS_FROM and the
subgroup's plain and twinned R-work/R-free and fraction. The subgroup file
follows its own validation's indexing, as the answer follows its own.
The model chooses the maps only: the space group, <prefix>.mtz and R_FREE stay
the answer's. Documented in RUGNUX_REPORT.md, RUGNUX_ADVANCED.md and
CPU_DATA_ANALYSIS_DECISIONS.md (14.9); Yeates (1997) credited.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Near-perfect twins of a subgroup are undecidable from intensities, and a merge
in the higher group cannot be twin-refined. Where the adopted group may have
absorbed a twin law - a twin gate refused it and was overruled, or a group above
P1 was adopted on a merge that read twin-like before the search - the same
observations are also merged in a subgroup and written beside the answer, with
the twin law and its fraction in _pdbx_reflns_twin.
The subgroup is the one a decision named (the group a refused promotion was
adopted over, the indexed lattice's group a metric re-ask promoted from), else
the index-2 subgroup the search confirmed as a finalist whose lost two-fold
agrees worst with the intensities. It is merged from the same engine, to the
answer's resolution, without writing per-frame scales back; it carries the
answer's free set (flags read off the answer's merge by the adopted group's
asymmetric unit) and is written in the subgroup's reference and standard
setting (a C 1 1 2_1 description is reduced to P 1 2_1 1). The fraction is
Yeates' H on the pairs the law relates (1/2 - median H, a lower bound).
At most one file; other qualifying subgroups are named in the report
(TWIN_SUBGROUP_* keys). The answer's files are byte-identical.
IndexTwoTwinSubgroups and EstimateTwinLawFraction are new and tested; the
point-group ledger now carries each hypothesis's representative group.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The refined geometry is no longer written as a whole p_XDS.INP (and the
report's XDS_INPUT= key is gone). Instead the report carries one section,
XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the
result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the
detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with
X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated,
as verified with XDS IDXREF). No cell or space group: XDS determines those.
The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste.
The same keywords were logged by LogXDSGeometry at the start of every pass,
with the axis in rugnux's sense; that printing is removed, so the report
section is the one place they are stated. Docs updated (RUGNUX_REPORT,
RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL).
Output only; p.mtz md5-identical on 3 battery sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery
sets - most headers are a few pixels off. It now fires beyond three times
the tolerance; between one and three times the summary's Beam centre line
states the offset and recommends correcting the beamline value, without a
warning. The background-only branch (runs that refined no centre) uses the
same bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged observations already leave a run as the unmerged MTZ (and
the scaled fulls as the .hkl), and deposition does not come straight
from rugnux, so the _diffrn_refln block made the .cif up to ten times
larger for no consumer. The .cif is one data block again; the keys,
software rows, metadata, R_merge/R_pim/CC* and provenance stay.
ScaledFull loses the scale and frame it carried only for that block.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A rotation run now writes the geometry it integrated at as an XDS input
beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the
detector axes carrying the tilt, wavelength, the rotation axis the run
integrated at (ProcessResult::used_rotation_axis, after any sign rescue),
oscillation (with an adopted rotation scale), starting angle/frame, data
range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left
commented where the group is not in its number's reference setting),
FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the
master file as an x_??????.h5 template and a LIB= line to point at the
reader plugin; a CBF series gets its ???? template and its own frame
numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch
(2010) Acta Cryst. D66, 125-132.
The rotation axis goes out negated: rugnux turns the crystal by -phi about
its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with
XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis
indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the
CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged
(apart from the job list): 99.8% complete, ISa 23.3, direct beam within
0.15 px and cell within 0.07 A of rugnux's.
Output only; no result changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Overloads: the integrated reflections whose signal disk held a saturated
pixel are counted on the integration outcome before the merge, over the
whole range and over the reflections coarser than 4 A, where they
concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An
OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution
reflections, and recommends attenuation / a shorter exposure. Measured: a
strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an
ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%.
Overlaps: both integration engines count the signal disks a nearer
neighbour's prediction takes pixels from (the overlap ownership the default
--overlap exclude already builds), and those left under --overlap-minpk of
their own - BraggIntegrationCounts::overlapped / overlapped_major (log line
"Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning
fires from 1% in the second and recommends a longer distance, finer slicing
or a smaller beam; on 10 battery sets it fires on the two long-axis ones
(8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both;
documented in RUGNUX_REPORT.md.
Counters only; nothing reads them. p.mtz md5-identical to the rc175
battery on 10 sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The merged .mtz and the unmerged .mtz carry, after the existing "From
Rugnux <version>" record, HISTORY records with the git commit and date,
the command line, the input file (absolute path) and a cheap identity of
it - size and modification time, the master file's for HDF5, nothing
hashed - and the collection date. MTZ allows 30 records of 80
characters; a longer value runs on over records starting with '+', cut
so that no record ends in a blank a reader would strip.
The mmCIF names the commit in _software.description and the command
line, input file and identity under _jfjoch_provenance (the local prefix
the other private items use; no PDBx item holds them), the first two as
text fields so any character survives.
OutputProvenance is filled once in the CLI and travels in ProcessConfig
to every writer of a reflection file. Reflection data are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the
entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on
_reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/
_diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a
loop naming Rugnux for 'data reduction' and for 'data scaling'.
The experiment as the input states it: _diffrn.ambient_temp from the
sample temperature, the source's general class (from NXmx
/entry/source/type, which the HDF5 reader now reads), facility and
beamline, the detector make/model in _diffrn_detector.type (where
.detector used to carry it) and its class in .detector where the name
settles it, and the collection date from /entry/start_time.
R_merge and R_pim are formed beside R_meas from the same per-reflection
sums (stills and rotation merges) and written with CC* overall and per
shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the
_reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as
three new columns of the shell tables. R_pim following Weiss (2001),
CC* following Karplus & Diederichs (2012).
A second data block, data_sample_unmerged, carries the scaled unmerged
observations of a rotation merge as _diffrn_refln (measured index,
frame, scaled I and sigma, pdbx_scale_value) - the same records as the
.hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now
carries the scale and the frame. The unmerged MTZ stays unscaled.
Reflection data are unchanged: the merged _refln rows, the .mtz columns
and the .hkl are written as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The pre-scan's beam-centre check and the post-refinement both measure where
the direct beam lands, but the report only carried the centre the run used.
Section 1 now sets the file's direct beam (FILE_DIRECT_BEAM) beside the
background's (BACKGROUND_DIRECT_BEAM, with its sigma), the shift of the
refined centre from the file's (BEAM_CENTRE_SHIFT) and the tolerance the
geometry asks for (BEAM_CENTRE_TOLERANCE: BeamCenterNeed_pxl, the number the
log's check prints, here on the run's own cell). A shift above the tolerance
is a BEAM_CENTRE_OFF warning naming the file's, refined and background
centres, the difference in px and mm and the value to correct the beamline
to; a run that refined no centre is warned on the background alone where
its fit resolves the difference (3 sigma). The summary gains a Beam centre
line.
The refined centre, not the background, is the trigger wherever one exists:
on a battery set where XDS's refined direct beam and rugnux's agree to 1 px,
the background fit is 12 px away from both. Report-only; no result changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Full open and in-house battery on 0b2dde214 (run 20261009-1447_0b2dde_rc175-0b2dde2-full) against
the 10-07 battery on 9ced4c444, without the REFMAC check: the R-free figure and numbers are kept from
the rc.174 battery. Report regenerated from the run with the current battery tools (verdicts scored
by the current scorer, so KDP passes as an indistinguishable alternative), commentary written by
hand. 5ky6 and the low-ISa cubic insulin set go from pass to fail, accepted losses of taking the
pre-pass-2 decisions from the first part of the sweep; every other verdict is unchanged. The whole
battery took 4898 s against 5795 s, most of it model validation now on the GPU (278 s against
1024 s); without it the HDF5 sets are at the same speed. Time, zoomed time, in-house time and
resolution figures regenerated with paper_plots.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This reverts commit a825fb6cb.
The per-frame cell-scale retry restored the space group of its one
target (a P2 set whose cell grows with dose) but not its data: R_meas
went 26 -> 37 % and d_min 1.54 -> 1.62 A against the pre-part-B run. It
patches the early, low-dose cell of pass 2 frame by frame with a
+-2 % range, 0.1 % step and sqrt(n) margin chosen on that set, and runs
inside AnalyzeIndexing on every set for one difficult one. A late cell
belongs to rc176's far-end check of the early decisions instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This reverts commit f257d51ea.
The block-wise moving-shadow mask fixed its one target (a half-image
lysozyme set whose pin turns with the sample) but fired on 47 sets of
the full battery: on one small-molecule set the mask covered 51 % of
the detector and SHELXL R1 doubled (0.10 -> 0.20), on another the mask
grew from 8 to 21 %. The pre-scan cost rose by +87 s on the HDF5 sets
and +499 s on the CBF sets (the commit claimed +0.1 s / +1-5 s per set).
A narrowed version (regr2-mask) still costs +25-45 s battery-wide, about
+0.3 s per HDF5 set, for one difficult set - more than the owner's
budget allows. It is parked for rc176, where a moved shadow is to be
found by a far-end check of the early decisions instead.
The documentation paragraph the commit added goes with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pin, loop or capillary on the goniometer shadows one place for part of the rotation only. The
beam-stop mask is one for every frame and is read off the mean over the pre-scan sample, so a pixel
such hardware covers on fewer than about half the sampled frames is kept and integrated attenuated
on the frames it is covered on. Since the pre-scan samples only the first part of the sweep (part B,
5b9144410), a sweep whose moving shadow is short in the first half and long in the second loses it:
on the in-house half-image lysozyme sweep the mask fell from 32.8% to 21.6% of the detector, the
second half of the sweep was integrated through the shadow, ISa dropped 6.9 -> 4.1, R_meas 116 ->
240%, the cell refined a != b, and the attenuated intensities erased the a-axis screw evidence
(merged rc175: P4_1 2 2 instead of P4_1 2_1 2).
The sample is now also read in blocks of 10 consecutive frames (rotation only); each block's shadow
is found on its own with the rings about the whole sample's centre, and the blocks' pieces that
reach beyond the 30 px band the whole sample's edge can grow over (PENUMBRA_MAX_PX) are added whole.
A pixel shadowed on most frames of any longer stretch is shadowed on most frames of one block, so
nothing finer is needed; pieces within the band are a block redrawing a static stop's edge and are
left out, so a sweep without moving hardware keeps its mask bit for bit. Part B's principle is kept:
everything is still decided on the first part of the sweep.
Measured (GPU, against merged rc175 eac0833c8, 19 sets): half-image lysozyme P4_1 2_1 2, ISa 4.1 ->
9.7, R_meas 240 -> 113%, CC1/2 0.955 -> 0.979, R-free(model) 0.331 -> 0.282 (last night 6.9 / 116%).
p.mtz md5-identical on 14 of 19 (8v4o, 9rci, 5ky6, 6yqf, 6ze4, 5reo, 9qw8, insu_I low-ISa,
insu_H notwin, myob powder 1/2, myob split, nothing_1/2); changed: 5nw5 (a dim crescent at the stop
edge now masked: ISa 8.4 -> 8.9, CC1/2 0.937 -> 0.980, R-free 0.462 -> 0.429), 7atg (0.85% added at
the detector edge, statistics unchanged), 9min (0.03%, fail -> fail unchanged), 5jk4 (<0.01%,
unchanged). Cost: pre-scan +0.1 s on HDF5 sweeps, +1-5 s on large file-per-image (CBF, marCCD)
sweeps, where the blocks are re-read and their masks computed on the host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
