The header offset comes from TIFF tag 34710 unchecked and is handed to
ReadPrefix, which allocates that many bytes before reading. This runs
during format autodetection, on files nobody has said are marCCD, so a
foreign or corrupt TIFF could have the program allocate ~4 GB for a
probe. The probe now reads at most what the file holds; a short read was
always the normal outcome and the caller already checks the size.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
MarCCD::CanRead and SMV::CanRead read the first plausible file of a
directory outside their own try, and the CLI made all three CanRead
calls outside the try that reports a bad input. "rugnux <dir>" whose
alphabetically-first plausible file is unreadable therefore terminated
with no message. CanRead now answers false for anything it cannot read,
and the CLI asks the question where it can report the answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
SweepLayout rejected any inter-frame difference below half the header's
oscillation width as read-back jitter. A series that steps by less than
it exposes - overlapping wedges, or a writer that puts the whole sweep's
range in that field - then had every difference rejected, no step was
found, and the files fell into the "series that never turns" branch,
which lays them out end to end and, unlike the gapped path, said
nothing. The jitter bound is now absolute (1e-4 deg, below any step an
instrument makes and above 32-bit float read-back noise), and the
end-to-end fallback warns when the angles do span a rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A lattice arm that indexed nothing (or post-refined nothing) measures no
held-out residual, and PASS_DECISION then read "realised a held-out
residual of nan against 1.234e-03 ..., so the metric symmetry stands" -
a comparison the run could not make, stated as one it did. The fallback
itself is right: the metric symmetry stands. Only the claim changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
twin_zone_decision was documented as empty when no twin gate fired, but
an undecided read - no zone reflections, or evidence inside the margin -
wrote it too, and the report then printed "a twin gate refused this
promotion, and the twin-immune zone of the operators it adds decided it"
over a line reading "undecided".
An undecided read now goes to the log and leaves the field alone, so the
field holds what was decided and the prose is true of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The report decided whether to name a twin law by substring-matching
ProcessResult::twin_zone_decision, a field every zone read overwrites.
A run reads the zone up to four times, so the match saw the last
comparison made rather than the refusal that stood, and TWIN_LAW and the
_pdbx_reflns_twin mmCIF loop could appear on a crystal whose refusal was
overturned - or be missing on one whose refusal stood.
TwinZoneVerdict already returns the verdict; it now records it on the
result, and the decision point reads that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The twin-law report block dereferenced the adopted space group without the
guard every other use of it carries; a de-novo run whose search refused a
point group but left no adopted group would crash there, after the merge.
The joint post-refinement's distance-correlation array was read on every run
that holds the header distance, but only written when the solve is usable, so
an unusable first solve printed four indeterminate values as correlations.
The merge-degradation gate ran neither of its two branches when the data
confirm more than one operator and the best of them is itself close to the
random-pairing end: the contrast has no scale to read on such a merge, but the
candidate was then promoted on no merge-degradation evidence at all. It now
falls back to the noise-floor ratio, as it does when only one operator is
confirmed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Four lines for 118 commits: the broker GPU-memory leak that cancelled
collections, the rotation scaling loop now running to convergence (which
moves merged intensities and every decision read off them), the
per-frame sweep placement that makes gapped and angle-wrapped CCD/CBF
series read correctly, and the two diagnostic files a rotation run now
writes without being asked.
Left out deliberately, as developer-facing or niche: -A no longer
changing what a rotation run decides, --report-resolution, the R-free
holohedry keying, the mmCIF twin loop, and every internal refactor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A handful of open-arm rows disagree with their deposition on a knife edge that no test
available to us settles. They were handled three different ways - silently overridden to
our answer, marked unscored, or left failing - and none of the three says what is true:
either answer is acceptable as long as the program picks one of them.
A manifest row can now list `ref_alternatives`. Each entry replaces the reference fields
it names - a space group, a cell, or both - and the row passes if the answer matches any
of its references, the deposition included. `ref` keeps the deposited values verbatim in
every case. Every alternative must carry `why`: an accepted alternative with no stated
reason raises rather than passing, so the mechanism cannot be used to launder a failure.
The report keeps these rows visible rather than folding them into the passes: a summary
column counting them, their own segment in the verdict bars, and a section naming each
row, what we read, what was deposited and the reason both are accepted.
Five rows use it. Four are symmetry: a tetragonal row where the refinement test is split
and its spread exceeds the effect, and three trigonal rows where we read a higher point
group - one where the evidence favours our answer, one where our own twin-immune test
favours the deposition, one unresolved in either direction. The fifth is a cell: a real
tNCS supercell whose (0,1/2,1/2) sublattice is what was deposited, both being correct
descriptions of the same lattice. The documentation frames all of them as open questions,
not as errors in a deposition, and states the limit: a merohedral twin at exactly one half
and true higher symmetry predict identical intensities, so no test can close them even in
principle.
`test_score.py` covers the new path: both answers accepted, the other hand of an
alternative, a third answer still failing, the cell case, and the missing-justification
schema error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Written on every rotation run, next to the reflections, with no flag to
switch them on:
<prefix>_detector.jpg - the mean projection the beam-stop pre-scan built,
which is the very image the shadow detection tested, coloured with the
viewer's own default map from common/ColorScale.h: white-to-indigo for the
counts, grey for the module and chip gaps, coral for the detected shadow,
magenta for the rest of the pixel mask. A shadow claimed where the
background is unbroken, or a dark patch left uncoloured, is then visible at
a glance instead of being a pixel count in the log. The projection is the
one the pre-scan already holds, so no frame is read twice; the picture is
rendered there and held as JPEG bytes until the canonical pass writes it,
because on a two-pass run the pre-scan belongs to the superseded pass.
<prefix>_plot.txt - one row per processed image, whitespace-separated with
a single '#' legend line over its own columns, so gnuplot plots it as it
stands: image, spindle angle, background, resolution, spot count, scale,
sigma_M, CC to merge, and whether the image's observations reached the
merged data. A quantity nothing measured is written nan, so the columns
never shift and no row is ever left out.
sigma_M is emitted as the pipeline produces it - a per-image Gaussian
rocking width, smoothed in frame order and filled with the run median where
a frame could not be fitted - and the documentation says so rather than
presenting it as a crystal property.
<prefix>_image.dat stays: the in-tree comparison scripts read it and it
carries the smoothing wedge, which the new file does not. The docs name
_plot.txt as the one to script against.
Measured on a 1800-image rotation sweep: 29.6/30.4 s before against
29.8/30.5 s after, and the merged reflections are byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Five open-arm rows do not reproduce their deposited space group or cell. Four are
scoring rows that disagreed with a published entry with nothing written down about
why, which is not a state a reader should have to discover from the manifest.
A tetragonal row whose deposited asymmetric unit holds two chains related by the very
two-fold the higher group adds, to 0.16 A CA RMSD - coordinate error at its
resolution - is overridden to the higher group. Merging there costs 0.0006 in R_meas
for 1.75x the multiplicity and correlates better with the deposited model, and an
independent reference implementation reads the same point group on our own P1 merge.
The refinement comparison is recorded as SPLIT: one purpose-built program makes the
higher group better at half the parameters and calls the deposited assignment
incorrect, an independent restrained refinement on a symmetry-consistent free set
makes the lower group better, and the gap between refinement protocols exceeds the
difference being measured.
One trigonal row becomes unscored rather than overridden: the evidence points at a
higher point group but the deposited copies differ by more than coordinate error, and
a purpose-built program settles on a different supergroup whose operators these data
do not support. The set runs; it is not scored.
Two trigonal rows had already been overridden, by editing the reference itself, with
no reason recorded. The deposition is restored verbatim to ref and the change moved
into ref_override, which leaves the effective reference and every verdict unchanged.
One of them carries strong centric-zone evidence; the other is marked OPEN, because
the one test that separates real symmetry from pseudo-symmetry reads against the
promotion the row is scored on, and may belong unscored instead.
The documentation frames all of these as open questions rather than as errors in the
PDB, and states the limit that applies to every one of them: a merohedral twin at a
fraction of exactly one half and a crystal that genuinely has the higher symmetry
predict identical intensities, so no test available to us can show that an operator
holds exactly rather than very nearly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The departure from a = b that decides whether the metric-symmetry arms run at
all was read by pushing the free (triclinic) refinement through the search
result's `reindex`. That matrix is stale for a whole population: where the walk
matches a metrically hexagonal lattice on its C-centred orthorhombic character,
RotationIndexer re-expresses `conventional` in hexagonal axes but leaves
`reindex` describing the setting it replaced. The C-centred setting has
b = a*sqrt(3) BY CONSTRUCTION, so a against b there is
2(sqrt(3)-1)/(sqrt(3)+1) = 53.6 % apart on every hexagonal lattice there is,
whatever the crystal.
So the gate fired on the whole hexagonal and trigonal population - eighteen
sets in one corpus run, all reporting 53.1-53.7 % - and the arms then decided
two classes that fit equally well on differences in the third or fourth
significant figure. Three of those coin flips landed on the ortho-hexagonal
supercell and took a P 61 2 2, a P 65 2 2 and a P 31 with them.
The departure is now computed by LengthEqualityDeparture, which derives the
primitive-to-conventional map from the pair of cells the search result carries
instead of trusting `reindex`, and reads a against b in that basis. The two
cells describe one lattice, so the map is the integer relabelling of its basis
vectors. The measurement no longer depends on which character the walk happened
to match.
Re-measured, the hexagonal and trigonal sets report 0.00-0.41 %, the band the
tetragonal and cubic proteins were already in, and the gate does not fire on any
of them; the genuine pseudo-symmetric small-molecule cases stand alone at 0.98,
1.85 and 2.61 %. The three lost groups come back, and the sets that merely
tolerated the noise are bit-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The beam-centre check indexes a second first pass at the measured centre and,
where the two centres return cells related by an integer volume factor, says so
and decides nothing. That abstention is right in general - the validation frame
count leans to the sub-cell, since a cell twice as long must place every spot
twice as accurately to score the same - but it leaves one direction
unrecoverable. Where the MEASURED centre holds the LARGER cell, the run starts
on the sub-cell and stays there: the second pass re-finds the larger cell at the
refined geometry and its supercell guard refuses it for being larger than
pass 1's. The run then merges a halved axis at full indexing rate, with good
merging statistics and nothing to say it is wrong.
Decide that one direction, on the pooled spots rather than on the frames: the
excess each lattice puts on itself over its own wrong-spindle null, which is the
measurement the failed-pass branch of the same check already adopts a centre on,
and the one statistic that does not credit a denser lattice for accidental hits.
The opposite direction is deliberately left alone - a second pass is free to drop
to the smaller cell and does, on every corpus crystal that reaches this branch
that way.
Measured on a tetragonal crystal whose header centre is 3.6 px out across the
spindle, more than the geometry absorbs: the file's centre gives a halved a axis
(43/60 frames, 25.4% of the pooled spots on the lattice) and the measured one the
deposited cell (60/60, 43.3%). Before, the run merged the halved cell and the
deposited model refined to R-free 0.575; after, it merges the deposited lattice
to 1.62 A with R-free 0.254 against the deposition's 0.203.
Five other corpus crystals reach the same branch with the measured centre on the
SMALLER cell; all five are unchanged (space group, cell and R-free identical), as
are two crystals whose halved or doubled axis comes from elsewhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Smoothing the per-frame scale inside the loop runs before the guard that drops frames whose scale
collapsed, and the window is a geometric mean - so a frame the crystal did not diffract on was both
dragging its whole window down by its logarithm and being lifted toward its neighbours, which is how
it escaped the guard meant to drop it. Measured: on sweeps with a genuinely dead stretch the guard
went from eight fires at up to a million times below the median to five at six hundred times, and on
a long clean sweep with a dead block from six fires to none.
A frame already below the guard's floor is now left out of every window and keeps its own fitted
scale, so what reaches the guard is the scale the fit produced. The floor is the guard's own, read
the same way - no new criterion and no new constant.
The guard's fires come back where the fit still collapses those frames (on the two dead-stretch
sets to seven and six, at up to three and seven million times below the median; on the long clean
sweep to six, the count it had before), and with them the error-model asymptote: ISa 7.9 back to 9.2
on one dead-stretch set, 36.7 to 38.0 on the clean one. Where the restraint stops the fit from
driving a frame below the floor at all there is nothing to exclude and nothing changes. No result
moves the wrong way: the fine-sliced, small-cell and powder sets keep the merges the restraint
recovered - two of them improve, one by CC1/2 0.916 to 0.943 - and every space group is unchanged,
including the short-sweep tetragonal pair and the de-novo orthorhombic set the loop was fixed for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A screw zone's evidence drops its largest member to defend against one badly-measured
reflection whose sigma lies about it, and such a reflection is by construction weak - a
fraction of the row it sits on. A member that has passed the violation test AND stands
at or above the mean of its row's own present class is a different animal: not a
measurement that moved, but a reflection that is there. Trimming it removed the single
datum that refutes the claim, and the violation-count deferral then read the inflated
evidence to forgive the very violation that had been trimmed out of it.
Nested screw ORDERS are decided entirely on this. 6_1 extinguishes l != 6n and 6_2/6_4
extinguish l != 3n, so the two differ only on l = 3n not 6n. On a hexagonal crystal
whose 00l row holds five present reflections, the strongest of the whole row lay in
that difference: trimmed, 6_1 read the row as perfectly dead and won on the count of
absences alone - nine at 50.6 nats with one violation against seven at 43.3 with none
- and the run reported the wrong screw order with the right one ranked below it. With
the violation left in, 6_1 reads 6.8 and is refused. An independent POINTLESS run on
the same P1 merge puts the 6_1 condition at probability 0.000 and the 3n condition at
0.998.
Trim only among members that are not both flagged present and at full row strength.
Both halves of the condition are needed and the corpus separates them: the reflection
above stands at 1.94 of its row's mean, where two monoclinic crystals whose 0k0 are
genuinely dead carry one violation each at 0.31 and 0.74 of their row - the
mis-measurement the trim exists for, and one that costs a real 2_1 if it stays in.
Zones with no violations are bit-identical, and so is every candidate whose absent
class is clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The Bravais class is chosen by a walk that reads two axes as EQUAL when they
agree to a fixed relative tolerance, and the class carrying that equality is
then imposed on everything below: the cell is refined with a = b, reflections
are predicted from it, and they are integrated at those predicted positions.
On a small cell the tolerance is far wider than the spot positions resolve, so
a genuinely orthorhombic crystal whose a and b differ by 2 % is integrated as
tetragonal and every decision downstream is read off the wreckage.
Measure the equality instead of assuming it. The indexer already refines each
candidate a second time with nothing held, so the first pass now reports how
far that free refinement leaves the two axes apart. A relative split of eps
displaces a reflection at radius r by eps/2 * r pixels: where that displacement
at the far corner of the detector stays inside the integration disc, imposing
the equality moves nothing out of its own box and the higher symmetry is kept
with no extra pass. Where it does not, both hypotheses are run as probe-only
passes - the promoted class, and the class the same walk carries when it is
granted no length equality at all - and whichever realises the lower held-out
positional residual is kept, the promoted class on a tie.
Two small-molecule sets whose axes differ by 2 % now index, refine and merge in
their own orthorhombic lattice instead of a tetragonal mean: one goes from a
cell 1 % wrong and P 1 at CC1/2 0.19 to the deposited cell within 0.2 % and
P 2 2 2 at CC1/2 0.95, the other from a tetragonal mean to a cell matching its
reference to 0.4 %. Protein sets whose symmetry is real (P41212, P4222, I23,
P6422, F4132) are unchanged: their free refinements leave the axes 0.02-0.10 %
apart, a few tenths of a pixel, so the question is never asked. Where it was
asked on a weak sweep whose free refinement diverged, the arms decided for the
higher symmetry and the output was identical.
No new threshold: the comparison is the integration radius the run already
integrates at, and the arms are judged by HeldOutResidualFell, as the geometry
walk's rounds are. postrefine_probe_only_ returns for the arms' sake - a pass
run only to measure stops before the scaling engine is built, so neither arm
pays for a merge or a space-group search.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The alternating scale fit has a second degenerate direction beyond the gauge, and that one is not
harmless. A frame whose scale drifts below its neighbours' loses its vote in the references it is
fitted against - an observation's weight there goes as G^2 - which moves the references away from
it, which drives its scale further down. Where every frame is measured against hundreds of
reflections the references barely notice one frame and the loop settles; on a fine-sliced sweep of
a small cell, where a frame holds two or three reflections and a rocking curve spans a dozen or
more frames, the frames at the tails of the curves lose against the frames at their peaks and the
loop walks the bulk of the sweep down fifteen decades. The convergence test then said the loop had
settled, because its own weight (frame observations at G^2) vanishes for exactly the frames that
are collapsing.
So the scale is now smoothed over the rotation INSIDE the loop, at the window the merge already
smoothed G over afterwards (the post-loop smoothing is the same operation and is not repeated):
every frame in a window shares the window's geometric mean, no frame can lose its vote alone, and
what a frame's own observations cannot determine its neighbourhood's do. The window is floored at
six rocking curves - within one curve a change of scale and an error of the partiality model are
the same thing, and a window holding a curve or two fits the model's error as scale. The step test
is weighted by each frame's observation count alone, and a loop whose step has not fallen below
nine tenths of its smallest value for five rounds stops and says so rather than walking further.
Measured against the previous behaviour: a fine-sliced sweep that collapsed to P1 with CC1/2 0.000
merges in its true orthorhombic group at CC1/2 0.95; a monoclinic small-cell set goes from an
unusable 2.13 A merge to 0.81 A at R_meas 11.5%; another from R_meas 33% / ISa 2.2 to 6.1% / 14.2;
two powder-ring sets and a short-sweep tetragonal one improve. The space group is unchanged on
every protein control measured, and the short-sweep and de-novo orthorhombic gains of the
convergent loop are kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A 2700 frame, 540 degree sweep was refused at load as "screening images taken at
scattered angles". Nothing about it is irregular: every consecutive difference is
a clean step, and the headers simply write the start angle modulo 360, so the
second and third revolutions repeat angles the first already had.
The unwrap handled that correctly. What failed was the STEP. It was taken to be
the smallest difference between two adjacent frames - a single recorded value, and
of all the samples the one biased furthest low by read-back noise. These headers
are written from 32-bit floats, so that one sample was 1.3e-4 low, and placing a
frame by dividing its distance from the FIRST frame by that step multiplied the
error by the frame number: by frame 1924 the drift had reached a quarter of a step
and the on-grid check refused the rest of the sweep.
The fix separates the two jobs the step was doing. The smallest difference is now
only a guess, used to count how many steps apart each pair of NEIGHBOURS is; a
local difference carries the noise of one reading and nothing accumulates, however
long the sweep. The step itself is then a straight line fitted through (steps,
angle) over every frame, so the noise averages out instead of one unlucky pair
setting the scale for thousands of frames. Over the whole corpus this changes the
step for 4 of 82 numbered series and leaves the other 78 bit-identical; of the 4,
three were drifting by 0.17 to 0.35 of a step across the sweep and one by 1e-8.
Counting steps between neighbours is also what carries a sweep past a full turn:
each fold is one step forward, so the count keeps climbing through 360 and a frame
taken on the second revolution lands beyond the first rather than on top of it. A
series that genuinely goes BACK - two sweeps of one crystal concatenated - folds to
a large negative count in a single pair and still collides, so that refusal keeps
working; its message now says what it means.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The "the cell metric carries more rotations than the class in hand" ask fired only
when the metric carried MORE. A lattice class is named from a Niggli-reduced cell on
relative tolerances, so it can also name more symmetry than the lattice carries: a
cell whose gamma is 1.4 degrees off 120 is named hexagonal, and one with a = b whose
gamma is 2 degrees off 90 is named tetragonal, where Le Page - which measures each
two-fold's obliquity - finds four rotations and a C-centred orthorhombic conventional
cell at 0.02 to 0.05 degrees. Asked only on ">", the comparison was then 4 against 12
(or 8) and the ask stayed silent, so the crystal was searched under a holohedry its
own lattice does not have, every operator of that holohedry was refused, and it fell
to a primitive monoclinic subgroup - the one mechanism built to rescue that case
disarmed by the very over-call it exists to correct.
Ask whenever the two differ. The ask adopts only on a strictly higher point group
confirmed by the same search with the same gates, so asking against a lower metric
class cannot demote; it costs one search.
Measured on three rotation datasets whose true lattice is C-centred orthorhombic and
which were indexed on its primitive sub-cell: two now reach the C-centred cell and
adopt C 2 2 2 and C 2 2 2_1, both agreeing with an independent POINTLESS run on the
same P1 merge to the third decimal of every cell edge. The third reaches the same cell
and is then refused by the merge chi^2 ratio at 1.96 against a bound of 1.85 - a
separate gate, untouched here. Eight datasets that decide correctly today, five of
them exercising this ask, are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The translational-NCS vector was refined only on d >= 4 A, the band its
detection gate reads, where an error of 0.02 costs the contrast almost
nothing. The L-test's partner steps and the twin-immune zone's per-class
normalisation then assign a class, cos(2 pi h.u), to reflections at full
resolution, where the phase error is |h| times the error in the vector: past
about |h| = 12 the classes were no longer the physical ones, and on one corpus
crystal they were anti-correlated with them over a whole band.
The vector is now re-refined against all the intensities, up a ladder that
doubles the number of reflections read at each step and starts each step from
the previous one's answer, so the phase is never extrapolated further than it
is known. The objective is the correlation between E^2 and cos(2 pi h.u) -
neither the gate's max/min bin ratio nor the fitted amplitude survives a
full-resolution population, both being ratios that run away where the cosine
has little variance. The gate itself is untouched: which crystals are called
is unchanged, only where the vector points.
Measured as that correlation in bands of |h|, before against after:
0.57/0.61, 0.42/0.52, 0.19/0.30, 0.14/0.18 on one crystal and 0.14/0.52,
-0.15/0.42, 0.00/0.20, -0.02/0.09 on another. The acentric control of the
twin-immune zone moves towards its analytic 0.736 where the classes change
(0.671 -> 0.748 on one), and no space-group or twinning verdict moves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The 8XBP deposition (RCSB, verified against the cached mmCIF) gives
C 1 2 1, 148.29 50.78 60.21 90 92.33 90, collected 2023-06-23. The
sweep in the open arm was collected 2023-06-21 and gives b = 51.83,
2.0% from the deposited b, with a and c agreeing to 0.1%; the reduced
primitive volume ratio is 1.02 and the set fails the lattice test.
DIALS 3.29 indexes the same master de novo at b = 51.777(11), 0.09%
from rugnux and 90 sigma from the deposit, and the deposited model
refines against the rugnux merge at R-free 0.30 / CC 0.89. The
deposited cell therefore does not describe these images.
ref_override supplies the DIALS cell in place of the deposited one.
The deposited space group and d_min are kept, and ref still records
the deposition. No scoring code and no threshold is changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Whether a sweep determines its detector distance at all was decided by running the canonical pass
twice - once at the post-refined distance, once at the header's - and keeping whichever realised the
lower held-out residual, the header's on a tie. The first-stage test that sent a run down that arm
compared the free and held fits on the POOLED held-out residual, and pooling is what made it
uninformative: the positional family cannot see the distance along the degenerate direction, so it
dilutes the one family that can, while the unpaired standard error of a heavy-tailed mean of squares
is 2-25 % of the mean against in-fit differences of 0.1-1 %. The test therefore said "cannot tell" on
four fifths of the fitted sweeps and the arm ran on most of a corpus, buying by prior what it could
not measure.
The two hypotheses are now compared on the one residual family that can tell them apart. The
excitation residual never involves the detector, so it is blind to the distance itself; what it sees
is the cell scale, and a held fit at a wrong header distance is forced into a wrong cell scale by the
spot positions, which the rocking angles then refuse - measured on a sweep whose header was 1.4 %
long, the held fit's held-out excitation residual is seventeen times the free fit's. Where freeing
the distance lowers that residual by more than its own standard error the free fit is committed as
before; where it does not, the held fit is committed - header distance, refined beam, cell,
orientation and axis - and the walk and the commit run held. The positional residual is deliberately
not consulted: its in-fit gain along the degenerate direction is the one re-integration erases.
The question is asked only while the run is still at the file's distance. A run that has walked off
the header has already refuted that hypothesis by re-integrating, and asking it again at every round
stalls a walk short of its fixed point, because the excitation standard error at the walk's tail is
outlier-dominated (measured: a walk stopped 0.4 % early, seven passes, cell 0.65 % off against
0.26 %, ISa 10.6 against 14.0).
So there is no arm, and with it go the two probe passes that measured it and the canonical pass
the losing arm used to cost: the decision costs two Ceres solves. On thirteen
rotation sweeps covering both verdicts, every decision the arm took by evidence or by its tie rule is
reproduced at the fit, except where the excitation family sees what the pooled test could not and the
free distance - the better cell against an external reference - is taken instead; merged intensities
are unchanged where the verdict is.
POSTREFINE_DISTANCE_HELD now means "the committed fit held the header distance", and is cleared where
a later geometry walk left it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A series of one file per image was laid out end to end: the rotation start came
from the first file and the step from the difference between the first two, so a
series with frames missing came out compressed - one 179.8 degree deposited sweep
of 1108 files out of 1800 was read as 111 degrees, and every frame past the first
gap was analysed at the wrong spindle angle. Indexing then found a lattice that
took 4% of the validation spots, and two other gapped series aborted outright
with "it is not this crystal's lattice".
Every one of these formats writes each image's own start angle in its own header,
so the sweep is fully recoverable. The new reader/SweepLayout places each frame at
the slot its own angle puts it in and leaves a missing frame as a gap - a slot with
no file, which ReadRawImage reports as nothing to read, which every image loop in
the pipeline already passes over. The goniometer's start + increment * image_number
is then the true angle of every image, and the sweep range, the per-10-degree
delta-CC1/2 batches and the sweep-quality ledger all read the rotation the headers
describe. The rotation step is the smallest move between two frames that really are
adjacent, not the first pair.
The three readers shared this code by duplication; it is now written once. The same
place refuses what is not a sweep rather than averaging it into one: headers that
disagree about the detector distance, the beam centre, the wavelength or the
oscillation width, angles that do not sit on a single step (a folder of screening
shots), or two frames claiming the same angle - each naming the frames. A series
that does not turn at all is left exactly as it was.
A directory holding fewer files than its own numbering spans is also reported, with
both counts: that is the signal that a sweep was not unpacked or copied whole, which
otherwise shows up only as a resolution nobody can explain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The full-resolution merge-degradation gate divided the added operators' mean
intensity-weighted R by the smallest R anywhere in the crystal. That denominator is an
extreme order statistic, so it has a pole: one unusually clean operator condemns every
other genuine one. It is not a rare accident. A rotation about an axis near the spindle
maps a reflection onto one recorded at nearly the same detector position, so the lab-frame
systematics cancel for that operator alone; measured over the rotation corpus the spread
WITHIN a genuine group reaches 2.0-2.6x, the whole width of the old bound of 2.0. On a weak
cubic crystal whose near-spindle 3-fold read 0.074 against 0.16-0.22 for its ten other
genuine operators, every one of them was refused, the search kept only the group generated
by the reference operator itself (which reads 1.00 by construction), and the same 222
hypothesis on the same merge read 2.45 under a cubic enumeration and 1.02 under an
orthorhombic one - a statistic that moves with which OTHER operators were enumerated.
The added operators are now placed on the scale the crystal itself defines, between the R
of unrelated reflections and the R of the best-agreeing operator:
contrast = (random_pairing_r - r_added) / (random_pairing_r - global_best_operator_r)
random_pairing_r is new: the same intensity-weighted R over shell-matched pairs of
reflections no symmetry relates, measured on the merge (deterministic, no RNG). Both ends
are hypothesis-free, and the form has no pole - an unusually clean reference widens the
denominator by a few per cent instead of driving the divisor towards zero. The test
abstains where the best-agreeing operator is itself no better than half way to unrelated
reflections, i.e. where the merge holds no clean end to measure from, as it already
abstains when there is no second operator at all.
Calibrated over 149 rotation datasets with a known answer, read as the gate reads it:
genuine promotions reach down to 0.77 and the worst false one reads 0.695, so the bound is
0.72. It keeps every refusal the old ratio made on that corpus except the weak cubic one,
which now promotes to its cubic group; the twins that read above the genuine range are
refused by the H gate and the twin-immune zone, as before.
The reported value, the point-group report line, the finalist ledger and the
gate-fired-and-was-overridden note all follow the contrast.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A screw axis whose row the sweep never recorded - it lies in the spindle's
blind cone, or outside the resolution range - is not a group the data refused,
it is a question nobody asked. The search already offered the whole set in
SPACE_GROUP_ALTERNATIVES, but the per-zone screw table that would say it in
words is printed for the SELECTED candidate only, and the selected candidate
in exactly this case is the one with no screw zones, so the run's account of
the open axis was a blank.
The search now names the axes on which two SELECTED candidates disagree about
whether the row carries screw absences at all, with why the row could not be
judged (never recorded / no control class). The report writes the axes as
SPACE_GROUP_SCREW_UNDETERMINED= beside SPACE_GROUP_ALTERNATIVES and explains
them in prose; the adoption logs a warning naming the axis and the set. An
enantiomorphic or origin-ambiguous pair predicts the same absences on every
row and is not named here - that ambiguity is the hand, or the origin.
Nothing about the decision moves: the group adopted, the alternatives and the
written .mtz/.cif/.hkl are exactly as before, because a reflection file cannot
hold "maybe a screw".
The battery scorer mirrors its existing "hand only" rule: a set differing from
its reference only by a screw the run reports as undeterminable, with the
reference among the groups it offered, scores unscored/screw_undetermined
instead of a sym_screw failure. All three conditions are necessary, so a screw
called wrongly where the row WAS measured stays a failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The zone verdict that arbitrates a refused promotion read its zone absolutely, against the
centric and acentric Wilson expectations, and an absolute reading is only as good as the
normalisation under it. On two refused 622 promotions - a 6/m crystal and a 312 one, whose added
operators disagreed at 4.7x and 5.2x the parent's H and merged to an R_meas of 0.34 under the
higher group - every class read centric: a 67 A^2 anisotropy, isotropically normalised in bins
of 100, spread one shell's expected intensity over a factor of 15 between its directions, and
the acentric control read 1.00 against its 0.74. The zone read the same as the control, +0.10
nats per reflection, and +124 nats over 947 reflections rescued a twin law.
Two things change. The anisotropy is fitted on the acentric reflections of the shells read
(ln I = c + s^T Q s, by least squares) and its deviatoric part taken out of every intensity
before anything is normalised; that alone brings the controls of the crystals measured to
0.73-0.77. What no normalisation removes, the control then certifies: an acentric population
reads -0.130 nats per reflection when the normalisation is right, a twinned one reads below
that, so whatever the control reads above it is the normalisation's - anisotropy, a
pseudo-translation, a pseudo-centring, noise all inflate every class towards centric alike -
and the zone, normalised the same way, carries the same per reflection; the calibrated evidence
has it taken off, and that is what the verdict reads. The two rescued twin laws now read -130
and -78 nats and stay refused, with the law named; the genuine promotions measured read +137 to
+580 (a pseudo-centred orthorhombic crystal whose control reads 0.99 still +334); the trigonal
and hexagonal partial twins -129 to -420 as before. The report prints the control's excess and
the calibrated evidence beside the raw one.
The metric-lattice re-ask carried a second copy of the two-arm rule without the zone: a P31
partial twin whose 32 the main rule had refused on the zone was promoted by that ask's
Lorentz-filtered arm. It now puts the same verdict to a refusal the other arm would outvote.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A screw's predicted-absent class is one axial row - half a dozen to a few dozen
reflections - and its evidence is a SUM over them, so it is decided by its
largest member. The file's own LIMIT comment said so; 7n2s is that limit firing
on real data. Between two scaling passes that differed only in which weak frames
were rejected, one of eight dead 0k0 moved from 14 +- 9 to 99 +- 10 while the
other seven did not move at all, and the zone fell from 30.1 nats to 17.1 and
lost the 2(1) under a bound of 20. That reflection was never measured to the
precision its sigma claimed: its two half-set merges read 198 and 2.5.
The zone's sum is now taken with its single largest member dropped and rescaled
for the trim - divided by n - H_n, the expected sum of the other n-1 under the
null, and multiplied back by n. ScrewZoneEvidence reads the result exactly as
before: same statistic, same floor, same bound, same calibration, with a robust
estimate of the zone's deadness in place of a fragile one. One member only,
whatever the zone holds: a zone with two strong absences is a zone that is not
extinct. On a uniformly dead zone the rescale under-states by 2.2 nats at eight
absences and 3.9 at sixty-four - it only ever refuses, never claims. Glide zones
keep the untrimmed sum: a plane holds hundreds to thousands of reflections and
no single one can carry the verdict.
7n2s -> P 1 21 1 (zone 27.6 nats, set by the seven reflections that did not
move), matching its deposit; a second monoclinic crystal decided six nats under
the bound (7 absent, 1 violation, 13.9 nats) reaches 21.9 and its 2(1) as well.
Unchanged on 7mzt, 7k1l, 11if, 9hs7, 9zlo and four in-house reference sets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Both arrays are one record per integrated observation - tens of millions on a fine-sliced long axis,
gigabytes each - and both carry the raw hkl only to sort and group on. A Miller index needs sixteen
bits (|h| <= a / d_min, in the hundreds even on the longest axis at atomic resolution), which takes
the ingest sort key from 24 to 20 bytes and the post-refine partial from 32 to 28.
Same comparisons, same order; merged output byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Where the pre-pass cannot tell whether its distance move paid, the two hypotheses were judged by
running each one as a full canonical pass - scaling, space-group search, correction surfaces,
every merge, the model step and the output files - although the comparison reads exactly one
number from each: the held-out residual the post-refinement measures right after integration,
before the scaling engine is even built. Where the header arm won, a third full pass then
reproduced the second's numbers digit for digit to write the files.
Both arms are now measured before the canonical pass by passes that stop as soon as that
post-refinement has measured, and only the arm that wins is run as a canonical pass. So a run
that takes the arm pays two image loops plus one canonical pass instead of two or three
canonical passes.
Result-neutral by construction - the residuals compared, the comparison, the geometry adopted
and the pass that writes are unchanged. Verified byte-identical p.mtz, p.hkl, p.cif and p_P1.mtz,
and identical reports bar the command line and the clock, on eight rotation datasets covering
both arms (seven where the header wins, one where the post-refined distance does).
Measured on the datasets whose logs show the arm: it takes an open-arm corpus pass from 261 to
218 minutes, the in-house arm from 25 to 22 and the private arm from 24 to 19, with the worst
single set going 594 -> 316 s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Three follow-ups to the converging scale loop, all on sweeps with a stretch the crystal barely
diffracted on:
- The sweep ledger's scale channel (and with it delta-CC1/2's "normal frame" test) and the
space-group search's scale floor measure a frame against the precision-weighted typical frame
G_ref (TypicalFrameScale) instead of the run median: on a sweep that spent most of its turn out
of the beam the median frame is itself a dead one, and every stretch then reads as typical. The
two collapse guards stay on the median on purpose - a frame that collapsed toward zero and was
not quite dropped has its fulls re-fitted with a scale of 1/G, a cubic mean is then theirs, and
the floor read against it dropped three quarters of every live frame's fulls (measured).
- delta-CC1/2's sigma-tau statistic enters each reflection with the information it carries (the
same per-frame factor as the CC1/2 weight), so a dead stretch, whose scaled-up noise flooded the
mean error variance of every mixed reflection and read as harm up to the 25% cap, now costs
about nothing and is left to the ledger.
- The scale loop's step test weighs each frame by its merge weight (its observations at its scale
squared): a dead frame's scale is fitted on noise and wanders by orders of magnitude every
iteration, carries nothing into the merge, is dropped after the loop, and must not hold the loop
open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
leonarski_fandClaude Opus 5