Merged dq-sm-absorb, whose ingest fix (corr_ingested carries the recomputed partiality, not the
predictor's) repairs the same defect as this branch's unit partial scale on the path without
partial scaling. The two did not double-apply - the unit scale rebuilt corr from scratch - but
they differ by the incident flux, and measured head to head the flux costs the sparse sweeps:
SHELXL R1 on four in-house organic sweeps 0.0459/0.0396/0.0709/0.0413 with it, 0.0421/0.0393/
0.0697/0.0405 without; on five open small-molecule sets equal or better without (0.0707 ->
0.0660 on one). Only the cubic absorbing sweep prefers it (0.119 vs 0.139): there the
background really does fall with the absorbed beam. So the unit partial scale and its device
kernel are removed and the ingest fix stands alone; the flux meter's smoothing, which only
mattered on that path, is reverted, so the partial-scaling path sees the flux exactly as
before. The penalised per-frame scale of the fulls stays.
Also restores two characters of CPU_DATA_ANALYSIS_INTEGRATION.md that the previous commit's
rewrite had turned from a stray carriage return into a line break.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Stage A's gates all compare the operators' disagreement with a reference - a parent group, the best
operator, the noise floor - and a pseudo-symmetric structure or a twin defeats every one of them: the
added operators correlate nearly as well as real ones. On the open arm that over-called 2wnz and 2xfw
(deposited P2_1, pseudo-merohedral C-orthorhombic metric) as C222_1, and 7bgu (deposited P1 with
b = c to 2.7%) as C2.
New gate, for a candidate that holds every rotation of its lattice (gemmi find_lattice_symmetry at
3 deg obliquity, with the lattice centring now passed in SearchSpaceGroupOptions::lattice_centring):
no twin law exists for such a group, so intensities merged under it that read like a twin's are not a
twin. AnalyzeLTestUnderMerge reads the Padilla-Yeates L-test twice on the same pairs of the P1 merge -
as measured, and with every intensity replaced by its plain orbit mean under the candidate - so the
narrowing is the merge's own doing and not a property of the data. Refused when the merged <|L|>
reads twin-like (0.375 <= <|L|> < 0.44) and the averaging closed a third or more of the gap from the
unmerged <|L|> to 0.375.
Calibration on the C++ search merges (targeted battery sympg-pg1): genuine lattice-holohedral groups
close 0.01-0.25 of the gap (highest 6z9g, indexed on half its deposited cell, 0.23-0.25; 5vml 0.16-
0.17; 7kcn 0.06-0.09); the over-calls 2wnz 0.47, 2xfw 0.46-0.47, 7bgu 0.79. A merge reading below
0.375 is not read - nothing a twin or a false operator does reaches it, so something else compresses
the intensities: 8c3e (0.353) and the genuine 9zmu (0.351, 494 A axis) look alike there. Near-
perfect twins (4bwl 0.14-0.16, 2wnq 0.13-0.20) are out of reach by construction, and 6p8j (0.19-0.30,
deposited as a twin of P2_1) is left alone. Cost: 170 ms per test on a 730k-reflection merge, only for
candidates that pass every other gate and hold the lattice's full symmetry.
Targeted battery sympg-final (44 open/in-house sets: the over-calls and a control panel of genuine
lattice-holohedral, twinned and twin-like sets) and the private arm: 2wnz C2221 -> P1211 (R_free with
the deposited model 0.227), 2xfw C2221 -> P1211 (0.215), 7bgu C121 -> P1 (R_meas 0.28 -> 0.14), all
the deposited groups. Every other set keeps its space group; against the rc174-cand runs of the same
sets, open and private, nothing else moves beyond run time. Still over-called: 8c3e, 4bwl, 2wnq,
6p8j, 3r6o (reasons above; 3r6o and 8c3e read below 0.375).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Every partial's corr is formed at integration as image_scale_corr = LP*QE*flight / partiality with
the predictor's per-frame partiality. Ingest then recomputes every partiality from the
frame-order-smoothed mosaicity (and the one exact-Bragg angle per rocking event), but corr_ingested
kept the predictor's 1/p. Where partials are scaled every fitted frame's corr is rebuilt from the
recomputed partiality, so that path only saw it in the first reference. Where they are not - fewer
than 50 rocking events per frame: small molecules, and long-wavelength protein sweeps - corr is what
the 3D combine divides each partial by, and the full became a weighted mean of I/p_pred while the
event's recomputed fractions summed to 1.00: the combined fulls tracked 1/sum(p_pred) (r = -0.96)
and scattered 13% rms about the plain sum of the same partials.
corr_ingested is now rescaled by p_pred/p_recomputed once at ingest.
Targeted battery against rc174-cand: every set on the partial-scaling path is bit-identical
(5reo, 9qw8, lyso_x06da_ref, thau_x10sa_0p1deg, lyso_x06da_atten_wedge, myob_x06da_split,
cytc_x06da_1, insu_I_x06da_5keV/6keV). Small molecules, SHELXL R1 against the COD model:
aspirin 20 keV 0.0515 -> 0.0432 (ISa 10.3 -> 30.9), aspirin 25 keV 0.0451 -> 0.0396, HEPES
0.0452 -> 0.0420, citric acid 0.0747 -> 0.0726, dnba 0.0472 -> 0.0273, metformin 0.0405 -> 0.0326,
nidppe 0.0498 -> 0.0426, lalanine 0.081 -> 0.057, cytidine 0.077 -> 0.071. Long-wavelength
proteins on the fulls-alone path: lyso_x06da_5keV ISa 17.9 -> 27.5, thau_bl1a_3p8keV 21.7 -> 28.4,
thau_bl1a_4p6keV 26.4 -> 33.5. One regression: lcystine_x10sa_25keV (a polycrystalline aggregate)
now refuses 622 and reports P31 instead of P6122.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The French-Wilson prior of each reflection is now epsilon * K_shell * a(h), with
a(h) = exp(-1/2 s^T B s) from the deviatoric tensor AnalyzeAnisotropy already fits
(the form it is fitted in) and K_shell = sum(I/eps) / sum(a), so a shell's priors still
average to its measured mean. The amplitudes are made isotropically at the merge as
before and made again once the tensor exists (full pipeline and --mode scale). Only
F/SIGF and F(+)/F(-) change; IMEAN/I(+)/I(-) are bit-identical. Applied whenever a
tensor was fitted, with no detection gate: a near-isotropic tensor gives a(h) ~ 1 and
the isotropic prior back, and the prior wants the best estimate of <I> along h whatever
its cause. Follows ctruncate's anisotropic prior (Ballard & Stein, CCP4); credit in
ACKNOWLEDGEMENT.md, CPU_DATA_ANALYSIS.md and at the algorithm.
--model scaling (ModelScaling.cpp) was checked: k_overall + symmetry-constrained
anisotropic B + flat bulk solvent fitted on the working set, against the same FW F
written to the MTZ - as REFMAC/phenix.refine do. No change needed.
Evidence (REFMAC 10-cycle restrained refinement of the deposited model, R-free on
the depositor's free reflections shared by both data sets; base = rc174 processing,
same IMEAN):
set base new d set base new d
9rcs 0.3475 0.3494 +0.0019 8qq7 0.4452 0.4503 +0.0051
9yzk 0.3192 0.3171 -0.0021 9hs7 0.2898 0.2465 -0.0433
6yqf 0.4642 0.4543 -0.0099 5nw5 0.3256 0.3206 -0.0050
7n2s 0.3126 0.2998 -0.0128 6z8o 0.2927 0.2892 -0.0035
6qaj 0.3390 0.3038 -0.0352 6moj 0.2756 0.2651 -0.0105
6r72 0.3826 0.3797 -0.0029 7qij 0.3239 0.3140 -0.0099
anisotropic sets: median -0.0075, mean -0.0107, 10/12 better
isotropic controls: 5reo -0.0014, 7kcn +0.0003, 6fid +0.0002, 11if 0.0000
rugnux's own --model R-free moves the same way (median about -0.019; 5nw5 +0.006),
R_model shell-scaled too; dep_cc_delta unchanged (intensity based). The adoption rule
(median gain >= 0.005 on the anisotropic sets, no control worse than +0.002) is met.
The two sets that lose are the one with a FLAT resolution signature (8qq7) and 9rcs,
where the exp form drives the dead direction's prior to ~0 beyond 3.7 A.
For scale: ctruncate's own anisotropic prior on the same merges moved the same
REFMAC R-free by a median of only -0.0008 (9hs7 +0.026).
Inhouse lyso_x06da_ref, thau_x10sa_0p1deg: every battery metric unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Three defects in the per-frame scaling of sparse (small-molecule, weak) rotation sweeps:
1. The fulls' per-frame scale pooled sparse frames by their RAW full count, but in a
high-symmetry group with many systematic absences most usable counts stayed under the
minimum, so most frames were never fitted and kept corr = 1 beside pinned, fitted frames
(two gauges in one reference). The release step then read the gauge offset (a constant
134x on a cubic Ia-3d small-molecule sweep) as signal and gave each frame exp(kept_f * 4.9) - the e^-5
errors on a quarter of the frames, R1 0.62. A fixed box window also cannot follow a
100x absorption ramp over a few degrees, and frames under the credible floor, exempt
from the window, ran away to 1e-7.
Now each round fits every frame on its own fulls (no pooling, no minimum), and the scale
is a penalised second-difference smoother of log G (Whittaker/Eilers), each frame at
the information of its fit, lambda by cross-validation over blocks one rocking curve
wide (interleaved single frames leak through shared rocking curves and chose to follow
every frame). After convergence the existing ShrinkToRestrained hands back the per-frame
deviation its neighbour shares. Pooling and the box window are gone from the fulls loop;
the partials loop is unchanged.
2. With partial scaling off (< 50 rocking events per frame) the partials kept the
integration-time corr: no incident-flux correction and not the partiality of the
ingest-smoothed geometry, because only the partial scaling loop rewrote corr. They now
get corr = prescaling_corr / partiality at G = 1 (host, and a device kernel).
3. The flux meter (per-frame mean background) jumped 30x between neighbouring frames of a
sparse sweep - on a few reflections it measures which reflections the frame holds. It
is read through the same smoother at the precision of each frame's mean.
SHELXL R1(>4sig) against the published structures, rc174-cand -> this, on the in-house
small-molecule sweeps: cubic Ia-3d 0.615 -> 0.119 (XDS 0.088); four organic sweeps
(monoclinic / orthorhombic) 0.0515 -> 0.0459, 0.0451 -> 0.0396, 0.0747 -> 0.0709,
0.0452 -> 0.0413; ISa up to 11.6 -> 31. Raw flux instead of smoothed costs 0.002-0.003 R1 on
the first two. A weak, decaying protein sweep on the fulls-only path: ISa 6.7 -> 27.1.
CPU and GPU paths agree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The SHELX .hkl test removed its file while its own ifstream still held it
open, which Windows refuses. The GPU-vs-gemmi model-scale check held b_star
to 1e-4 of its largest element alone, which on an F4132 case fitted at
B 0.6 A^2 failed on a 1e-4 A^2 difference; it now also allows 0.001 A^2
of B.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Where the file's centre and the background-measured one index the same lattice in the same class,
the check kept the file's centre and carried nothing forward. But a centre off by about one
reflection spacing indexes the same cell just as well, with reflections given an index one out over
part of the sweep, and only the merge shows it. On 9hnc (pseudo-tetragonal P2, c* spacing 11.3 px on
the detector) the post-refinement walked the file's centre to 1219.4 px while the background measured
1229.95 +- 0.34 px: the run merged at R_meas 0.58, ISa 1.3, CC1/2 0.84 and adopted P121.
Now the measured centre is kept as a hypothesis when it is further from the file's than this
geometry absorbs (and than 3 sigma of its own fit), and RunAllPasses runs the first pass there only
when the file's centre merged inconsistently: CC1/2 below 0.9 in the lowest-resolution shell of the
search merge, where every reflection is strong. Over the corpus runs read that shell at 0.967-0.998;
9hnc's file centre reads 0.610. Running it unconditionally would cost a second first pass on 107 of
227 battery sets.
MeasuredCentreWins on the same lattice compared only the strong-reflection count, which the
misindexed arm won by 2% (335800 vs 329330). It now also lets the measured centre win when its
inner-shell CC1/2 beats the file's by MAX_CC_HALF_LOSS - 0.997 against 0.610 on 9hnc.
9hnc: P121 R_meas 0.578 ISa 1.33 CC1/2 0.844 -> P1211 (= deposited) R_meas 0.129 ISa 13.5 CC1/2 0.998.
Its screw now reads 2_1 (32 absent, 0 violations) beside a detected (1/2,1/2,0) pseudo-translation,
against the deposited P121, so the battery verdict moves pass -> fail on the screw alone; R_free with
the deposited model 0.443 -> 0.255.
Battery sympg-bc1 (22 targeted sets, 13 of them in the same-lattice branch with a centre further
off than the geometry absorbs): only 9hnc tries the measured centre; the others read 0.984-0.999 in
the inner shell and are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The re-selection among tied higher-group candidates left
sohncke_space_group at the old lowest-numbered pick, so the report read
SPACE_GROUP_NAME P 42 21 2 beside SOHNCKE_SPACE_GROUP P 4 2 2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
refined_shared() recomputed R-free as sum|F - FC_ALL_LS| / sum F from REFMAC's output MTZ. That
MTZ's F is not the input F: REFMAC writes it with its overall anisotropic scale applied. On
strongly anisotropic data the scale multiplies the noise of the weak direction, and the recomputed
R comes out far above REFMAC's own R-free over the very same reflections.
9RCS (deltaB ~250 A^2, deposit STARANISO-corrected): the recomputed shared-set R-free was 0.503
against REFMAC's own 0.368 on the same 378 free reflections (deposit 0.366/0.365), which is the
whole of the +0.137 the open arm reported for this set. Across the 152 untwinned rc173 sets the
two agree to a median 0.0001 and to within 0.03 everywhere else.
Both refinements' free sets are the shared set by construction, so REFMAC's own R-free is the
shared-set number already, as the twin branch used it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Two first-pass decisions taken on frame counts at their noise floor, both found on a hexagonal
small-molecule crystal embedded in a powder of its own microcrystals (the non-ice powder rings
index on the crystal's own cell; 13-16% of the pooled validation spots lie on the lattice):
- Sparse lattice. A lattice whose pooled spots fall under the per-frame floor (20%) is integrated
on every frame only if fewer than 1/6 of the validation frames clear the floor. At 12/60 the floor
was kept, so only the frames in the upper tail of the same population were integrated: 267 of
1800 frames merged, 22.7k rocking events -> 4.2k fulls, completeness 74%. The bar is now the
majority (score < N/2). 25 keV sweep: completeness 74% -> 99% to 0.56 A, frames rejected
1533 -> 137, SHELXL R1 (COD model) 0.150 -> 0.156 / 0.150 -> 0.139 depending on the COD entry.
No other set of the overnight battery (open, in-house, private) is in the changed band
(pooled < 20% with 10-29 of 60 frames).
- Axis sign. The opposite spindle sign was adopted on 1/60 frames against 0/60; every rescue after
that ran on the wrong sign and the 20 keV sweep ended in "No crystal lattice found" (with some
compiler flags the file's sign scored 9/60 and the run indexed). The flipped lattice must now
also beat chance on the pooled spots, the test the run puts its final lattice to; a sparse
pattern at the right sign still passes (2/60 frames, 89% of the validation spots). 20 keV
sweep: fails -> P 61 2 2, 5.41/56.0 A, SHELXL R1 0.137 (was 0.76 in the runs that indexed).
tools/battery/shelx_check.py: the fixed-model R1(F) now fits SHELXL's extinction coefficient
together with the scale (scan over x), since extinction belongs to the crystal and otherwise
dominates the strongest reflections (HEPES 0.099 -> 0.059).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Mapped every single-axis screw candidate of every primitive-lattice set in
battery 3 (open + in-house) and the private arm against the deposited group.
False zones with no violation read at most +4 nats (plus one pseudo-
translation row at +30.5 that no bound separates); true zones go down to
+10. The true zones under 20 are all short monoclinic rows (b ~ 25-30 A:
four to six 0k0-odd reflections inside the search's resolution range, on
weak data at a few percent of their row), refused at 20 on five crystals
that are P2_1: four myoglobin sweeps (10.0-16.3 nats) and 6cs9 (10.8).
The rc174-cand myob_x06da_split call sat at 21.1, one refit away from P2.
Re-scoring the stored candidate tables at the new bound changes only those
five plus 3r6o (I4_1 2 2 newly eligible, toward the deposited I4_1 screw);
no adopted group elsewhere. New test section pins both sides: four 0k0-odd
at 2% of their row are claimed (11.3 nats), at 20% they are not.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
When a refused point group is re-asked on the adopted group's merge (twin-immune
zone / merge comparison), that merge no longer holds the adopted group's
screw-absent axial reflections. The higher search then had nothing to judge on
those rows, every candidate of the point group tied, and the lowest-numbered -
the one claiming no screw - was written. Measured on an open-arm P4_2 2_1 2
crystal (8v2t): adopted P2_1 2_1 2_1, promoted by the twin-immune zone, all
three axial rows "no control class", P 4 2 2 written.
Among the candidates the higher search cannot separate, take the first that
keeps every axial absence the adopted group claims (as the centring already
is taken), drop from the alternatives the ones that do not, and stop reporting
those rows as undetermined. Rows the adopted group left present are still in
the merge and still decided by the higher search.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Its deposited sweep lacks 40 deg (frames 561-720 are not in the
archive; the depositors processed it as two sweeps), so it tests
gap handling rather than processing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
SRS Daresbury PX10.1 (6RYM), EMBL Hamburg DORIS X13 (7BGU), NSLS X29A and X25
(8V4J, 8V2T), ESRF ID14-2 (5JK4), Australian Synchrotron MX2 (6CS9), CLSI 08B1-1
(7UDI), SSRF BL17UM (9LXL), MAX IV BioMAX (9S02), ALBA XALOC (6GVK) - marCCD,
ADSC (Quantum 4/210r/315), PILATUS and EIGER2 files. Each read and processed with
the current rugnux; 7BGU (C2 imposed on a P1 lattice) and 7UDI (P41 under-called
to P2) disagree with the deposition. Multi-sweep archives pinned to one sweep.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Pseudo-merohedral and merohedral twins, pseudo-symmetric cells and their
negative controls, chosen to test the space-group search's promotions:
- 8c3e (P3121, merohedral twin ~0.23, home-source HyPix CBF)
- 4bwl, 2wnq, 2xfw, 2wnn (P21 NAL crystals with the pseudo-merohedral
twin law -h,-k,h+l at fractions 0.49/0.46/0.10/0.33; ADSC SMV) and 2wnz,
the same crystal form deposited untwinned
- 6oww, 6p8j, 9qw2 (P21 with near-90 beta or a~c), 9qvv (I222, b=c),
5ojv (P21212, a~c), 7q6j (P212121, b~c)
Data staged in /data/scout_twin, linked from the open data root.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The XDS references of aspirin (20, 25 keV), citric acid, HEPES and YAG were made with
OVERLOAD=65534, half of the EIGER2's saturation value (133202). CORRECT ignored 13-313
reflections per set whose peak pixel lay between the two - the strongest, most
extinguished reflections - so the references described a data set without them.
Re-ran only the CORRECT step on the unchanged INTEGRATE.HKL with every input parameter
reconstructed from the old CORRECT.LP and OVERLOAD=133201 (saturation - 1, as the other
in-house sets already use); a rerun at 65534 reproduced the old CORRECT.LP. The old outputs
are kept in xds_overload_half/ beside each data set. `refs --write` picked up the new values.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
BraggIntegrationEngineGPU_MatchesCPU gains three footprint sections (spaced, crowded under overlap
exclude, with the radial background correction): both engines classify the summation ellipse, the
grown ring and the footprint Gaussian alike. Integration chapter and changelog describe the measured
footprint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The integrator's r1 disk and r2..r3 background ring are fixed in pixels and chosen from spots near
the beam. On small-molecule data at 20-25 keV a spot's standard deviation grows from ~1 px near the
beam to ~5 px at the edge (radially from parallax/obliquity, tangentially from the crystal's
azimuthal spread), so the r1 = 4 disk holds a quarter of the flux there, the background ring a third
of it, and the in-disk second moments the Gaussian is built from saturate near r1^2/4. On top of
that, the profile/summation runaway guard sent 20-30% of these reflections - the strong, wide ones -
back to the truncated r1 box sum.
- SpotFootprint: every pre-scan spot (width frames) is measured with a window that follows it
(3 sigma, iterated, re-centred), radially and tangentially; the medians per distance-from-beam bin
become BraggIntegrationSettings::Footprint. Installed only where some bin outgrows r1, and on the
adaptive side like the radius (pre-pass without; the starvation guard falls back to the settings
without it).
- BraggStencil: where 3 sigma > r1 the background ring starts at 3 sigma along and across the radius,
the summation region is the r1 disk plus the 3-sigma footprint ellipse (so the guard's fallback is a
complete intensity), and the per-reflection Gaussian takes the footprint widths. Compact spots keep
the stencil bit for bit. Both engines build it from the same header.
SHELXL against COD (R1 / fixed-XDS-model R1(F)): citric acid .101/.230 -> .077/.055, HEPES
.070/.179 -> .048/.050, aspirin 20 keV .059/.070 -> .052/.061, aspirin 25 keV unchanged, L-cystine
25 keV unchanged (.145 -> .144).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The SHELX HKLF 4 file now has one record per full reflection - its partials summed,
the per-frame scale and every correction applied, sigma(I) as the merge weighted it -
at the index it was measured at, not averaged with its equivalents: the chemical
crystallographer's convention, so SHELXL computes Rint and Rsigma itself. Outliers
the merge rejected and fulls beyond its resolution cut are left out; no batch column
(it would select a BASF scale in SHELXL). The engine hands the fulls back only for a
merge that may be written (RotationScaleMerge::SetExportScaledFulls), so the search
merges, the pre-pass and the P1 cross-check carry no copy; the fulls follow the same
relabelling as the merged reflections (merge_to_written). Stills keep the merged
file. --mode scale writes the unmerged form too.
Validation: p.mtz md5 unchanged on myob/cytc/thau (GPU). SHELXL on the same runs,
merged-old vs unmerged-new (COD models, harness /data/tmp/sm_shared):
aspirin 20 keV Rint 0 -> 0.071, Rsigma 0.036 -> 0.043, R1 0.0964 -> 0.0969,
wR2 0.312 -> 0.310, GooF 1.53 -> 1.47; HEPES 20 keV Rint 0 -> 0.163, Rsigma 0.057
-> 0.068, R1 0.0903 -> 0.0897, wR2 0.318 -> 0.263, GooF 1.57 -> 1.10; the "input
data appear to be merged" warning is gone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
shelx_check.py refines the row's published structure (manifest key "cod", cached from the
Crystallography Open Database into the site's cod_cache) against rugnux's p.hkl with one fixed
recipe: data reindexed into the COD setting (lowest-R1 integer matrix), non-H anisotropic, H fixed,
EXTI, MERG 2, three rounds of SHELXL's suggested weights. It records R1/wR2/GooF/EXTI/WGHT/residual
density/R(int)/R(sigma)/K of the strongest bin and a fixed-model R1(F) (|Fc| of the COD model as
published, gemmi). Reported, never scored. The report gets a small-molecule table; compare lists
SHELXL R1/wR2/GooF/EXTI deltas; report/compare fill the check in for older runs.
COD entries matched by Niggli-reduced cell and space group: aspirin 7050897, citric acid 5000063,
HEPES 2224210, YAG 2003066, L-cystine 1513328 (2005, replaces the 1959 model for refinement),
cytidine 2001311, 3,5-dinitrobenzoic acid 4510615, L-alanine 2104782, metformin HCl 2108029,
NiCl2(dppe) 2012031. cuhf2 has no reference cell and no match: left without one.
SHELXL is called, not shipped (site key "shelxl" or PATH; it comes with CCP4).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The error model was fitted once, on deviations from a mean that weighs every full by its COUNTING
variance, and the outlier test's median took the same weights. Where equivalents disagree beyond
counting statistics, the low-count observations dominate that centre: on a strongly absorbing crystal
(YAG, Ia-3d, equivalents spread over a factor 100 after scaling) the mean of (0 4 0) sat at 21k among
observations from 11k to 1.9M, a ran into its bound (100), b came out at 800% internally, and the
six-sigma test about the biased median removed 63% of the observations - the strong ones.
Now, after the first fit, the model is refitted about the model-weighted mean (each full weighted by
the variance the fitted model gives it at the reflection's mean) until a and b settle, and the
rejection median takes the same weights. Where counting statistics are right nothing moves. Following
Blessing (1997) J. Appl. Cryst. 30, 421-426. GPU path: the refitted means are uploaded (SetEmMean).
Measured (SHELXL R1(>4sigma) against the COD model, sm-a's harness; rc174 scaling):
YAG 0.556 -> 0.127 (XDS 0.083 merged), rejected 8055 -> 53, normalised deviations calibrated
(median |z| 0.62-0.69 in every intensity decile); aspirin 20 keV 0.0964 -> 0.0958; citric acid
0.161 -> 0.159; HEPES 0.0903 -> 0.0899; L-cystine 25 keV 0.1425 -> 0.1456;
aspirin 25 keV 0.094 -> 0.106 (fixed-model R1 0.107 -> 0.173): its strong equivalents split into two
frame-dependent populations from the per-frame partial scaling (sm-a's dq-smallmol), which the old
under-sized sigmas happened to cut; with that scaling fixed (f69339ce6 + pooling, --no-scale-partials)
this change is neutral to better on every small molecule (aspirin 20 .0618 -> .0586, aspirin 25
.0456 -> .0454, citric .1093 -> .1006, HEPES .0703 -> .0697, YAG .649 -> .222; SHELXL GooF ~1.1).
=> ship together with the scaling fix.
Proteins (GPU full runs): CC1/2 and R_meas unchanged to 0.002; ISa myob 9.06 -> 8.20, thau 52.5 -> 47.5,
cytc 25.8 -> 25.4, lyso 29.4 -> 29.4 (still above XDS's 5.2 / 44.5 / 31.8 / 28.3 except cytc).
CPU build gives the same statistics as the GPU build on aspirin 20 keV and myob.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Replaces the prototype --no-scale-partials switch with a rule read off the data: a merge whose
sweep holds fewer than 50 rocking events per frame takes its per-frame scale from the fulls alone
(scale-fulls, a sparse frame fitted over its neighbours - PoolHalfWidth), and logs that it did.
The rocking-event walk already run for the smoothing window now also returns its event count.
Within one rocking curve a partial's scale and an error of the partiality model are the same
thing; on a sparse fine-sliced sweep the partial fit takes the one for the other and imprints an
hkl-dependent bias common to all equivalents. Measured populations: small-molecule sweeps 2.6-23
events per frame, protein sets 84-900; on the proteins the fulls-only scale leaves model R-free
unchanged (+-0.002 over the smoke tier's open-arm sets) but lowers ISa, so they keep the partial
scale. p.mtz md5 unchanged on the three profiling sets, GPU and CPU builds.
SHELXL against the COD structures, R1(>4sig) rc174 -> this: aspirin 20 keV 0.096 -> 0.062,
aspirin 25 keV 0.094 -> 0.046, citric acid 0.161 -> 0.109, HEPES 0.090 -> 0.070 (XDS 0.030-0.038);
SHELXL's weight a comes off its 0.2 cap on all four. CPU and GPU paths agree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
The same source built with and without -march=x86-64-v3 gave different results on battery sets
(9zmu axis-harmonic supercell arbiter fired in one build only; 8rud resolution cut 1.69 vs 1.70 A;
myob_x06da_powder_1 cut 1.07 vs 1.42 A; 7mzt short-axis first pass; lcystine_x10sa_20keV indexed
vs no lattice), against the rule that no decision may depend on compiler flags.
Two mechanisms, found by building the merged tree four ways (baseline, x86-64-v2, x86-64-v3,
x86-64-v3 -mno-fma) with and without -ffp-contract=off and comparing p.mtz:
- FMA contraction. GCC contracts a*b+c whenever the target has FMA. With -ffp-contract=off the
x86-64-v3 build gives a p.mtz byte-identical to the baseline build on 8 of 9 sets (7mzt, 8rud,
9zmu, insu_I_x06da_5keV_2, myob_x06da_powder_1, myob_x10sa, cytc_x10sa, thau_x10sa_16keV), on
both the GPU and the CPU build. Cost: none measurable (user core-s, CPU build, x86-64-v3 vs the
same with -ffp-contract=off: 1877/1874, 3285/3253, 2212/2195 on myob/cytc/thau; GPU likewise
within noise). Set project-wide for C, C++ and CUDA host code; MSVC does not contract under
/fp:precise.
- SIMD width. lcystine still differed: baseline and x86-64-v2 (128-bit) agreed, x86-64-v3 with or
without FMA (256-bit) disagreed - Eigen's HouseholderQR in the FFT indexer's candidate refinement
(PostIndexingRefinement.cpp) reduces column norms over all spots in packets of the target width.
Replaced by the 3x3 normal equations summed in spot order in double. All four builds now agree on
all nine sets.
Changes results of the default x86-64-v3 build (contraction off); lcystine_x10sa_20keV now gives no
lattice in every build (its first pass is a knife-edge: 0/60 vs 9/60 validation frames before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
Prototype switch --no-scale-partials: skip the per-frame scaling of the partials and let the
fulls (scale-fulls) carry the per-frame scale, as XDS does. Without partial scaling a frame whose
fulls number fewer than MIN_REFLECTIONS is fitted over the nearest frames on either side that
together hold enough (PoolHalfWidth), on the host and in the GPU kernel. Default path unchanged
(p.mtz md5 identical on the three profiling sets).
Why: on fine-sliced small-molecule sweeps the per-frame partial scale and the partiality model are
degenerate within a rocking curve, and the fit swings G 0.23..1.2 with a 180 deg period (XDS's own
frame scale: 0.79..0.99). That imprints an hkl-dependent bias common to all equivalents, which
R_meas/CC1/2/ISa cannot see but a refinement against the known structure does. And scale-fulls
never fitted a frame on such data: a full is filed under one frame, about 8 per frame, below
MIN_REFLECTIONS, so every frame kept G = 1.
Measured with SHELXL refining the COD structures (R1 >4sig), default -> switch:
aspirin 20 keV 0.096 -> 0.062, aspirin 25 keV 0.094 -> 0.046, citric acid 0.161 -> 0.109,
HEPES 0.090 -> 0.070 (XDS 0.030-0.038). Proteins lose ISa with the switch (myob 9.1 -> 7.6,
cytc 25.8 -> 13.7), so it is not a default; the choice is to be made from the data.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K5K8jvPPbmCrbqnWkddTuB
leonarski_fandClaude Opus 5.5