The background ring's high-side clip, mean + 4*sqrt(mean), assumes counted photons. A CCD reports
ADU: on a Rigaku Saturn a background pixel scatters 4-5x its mean, so the clip sat at ~2 sigma, cut
the upper tail of every ring and added a positive offset to every reflection, compressing the
intensity statistics (a false 422 on one twinned I41 crystal).
No header carries a usable gain (SMV states none, every marCCD file the writer default of 1 ADU per
photon), so it is measured: background_gain::MeasureFrame takes the variance per unit mean of the
difference of pixels two columns apart, per level, clipped, on every fourth row of up to 16 of the
pre-scan's frames; Gain is the pair-weighted median over the levels. The clip becomes
mean + n*sqrt(max(g,1)*mean) (BraggIntegrationSettings::bkg_pixel_gain, CPU and GPU engines).
Only for SMV/marCCD input (ProcessConfig::counts_in_adu, set from the file format); HDF5 and CBF
keep g = 1, the identity. One log line reports the value. No new option.
Measured g: Saturn 944+ 4.7-5.2, Saturn 944 2.7, A200 0.68, ADSC ~0.3, marCCD ~0.3 - so the
ADSC/marCCD/A200 sets keep the photon clip exactly.
Checks (reflection data via gemmi and p.hkl, against the rc175 battery outputs / base build):
identical on 11if, 7qis, 8a1a, myob_x10sa, cytc_x10sa (photon counting), 4bwl, 5epe (ADSC,
marCCD) and 6cee (A200). Saturn sets, battery baseline -> this change:
3r6o I4122 R_meas 19.3% ISa 4.4 <|L|> .321 -> I41 (correct) 12.1% 8.1 .384
3mc4 R_meas 10.2 -> 9.8 %, ISa 12.1 -> 12.7, d_min 1.79 -> 1.75 A, <|L|> .360 -> .391
5uth 12.6 -> 11.8 %, ISa 10.4 -> 10.1, <|L|> .426 -> .483, offset vs |Fc|^2 0.21 -> 0.00
5vml 10.2 -> 9.6 %, ISa 11.0 -> 11.6, <|L|> .426 -> .475, offset 0.15 -> 0.05
5cc8 8.1 -> 7.9 %, ISa 10.8 -> 11.2, <|L|> .469 -> .494
3meb 14.5 -> 14.6 %, ISa 13.2 -> 13.2, <|L|> .442 -> .457
Space groups unchanged except 3r6o. Pre-scan cost: +0.5 s on a 9 Mpx ADSC sweep, within noise on
a 15 Mpx marCCD sweep, nothing on HDF5/CBF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The commit is already re-stamped on every build (JFJochGitInfo runs
GitInfoStamp.cmake with cmake -P; configure_file rewrites GitInfo.cpp
only when the value changed). What was missing is the case with no git:
a downloaded tarball left the SHA empty, and one unpacked inside some
other repository would have reported that repository's HEAD. The stamp
now asks git only when the source directory is itself the top of a work
tree, and otherwise records commit "unknown" beside the version from
VERSION. The reflection files' git description drops its trailing date
when there is none.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged observations already leave a run as the unmerged MTZ (and
the scaled fulls as the .hkl), and deposition does not come straight
from rugnux, so the _diffrn_refln block made the .cif up to ten times
larger for no consumer. The .cif is one data block again; the keys,
software rows, metadata, R_merge/R_pim/CC* and provenance stay.
ScaledFull loses the scale and frame it carried only for that block.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Overloads: the integrated reflections whose signal disk held a saturated
pixel are counted on the integration outcome before the merge, over the
whole range and over the reflections coarser than 4 A, where they
concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An
OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution
reflections, and recommends attenuation / a shorter exposure. Measured: a
strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an
ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%.
Overlaps: both integration engines count the signal disks a nearer
neighbour's prediction takes pixels from (the overlap ownership the default
--overlap exclude already builds), and those left under --overlap-minpk of
their own - BraggIntegrationCounts::overlapped / overlapped_major (log line
"Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning
fires from 1% in the second and recommends a longer distance, finer slicing
or a smaller beam; on 10 battery sets it fires on the two long-axis ones
(8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both;
documented in RUGNUX_REPORT.md.
Counters only; nothing reads them. p.mtz md5-identical to the rc175
battery on 10 sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The merged .mtz and the unmerged .mtz carry, after the existing "From
Rugnux <version>" record, HISTORY records with the git commit and date,
the command line, the input file (absolute path) and a cheap identity of
it - size and modification time, the master file's for HDF5, nothing
hashed - and the collection date. MTZ allows 30 records of 80
characters; a longer value runs on over records starting with '+', cut
so that no record ends in a blank a reader would strip.
The mmCIF names the commit in _software.description and the command
line, input file and identity under _jfjoch_provenance (the local prefix
the other private items use; no PDBx item holds them), the first two as
text fields so any character survives.
OutputProvenance is filled once in the CLI and travels in ProcessConfig
to every writer of a reflection file. Reflection data are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the
entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on
_reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/
_diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a
loop naming Rugnux for 'data reduction' and for 'data scaling'.
The experiment as the input states it: _diffrn.ambient_temp from the
sample temperature, the source's general class (from NXmx
/entry/source/type, which the HDF5 reader now reads), facility and
beamline, the detector make/model in _diffrn_detector.type (where
.detector used to carry it) and its class in .detector where the name
settles it, and the collection date from /entry/start_time.
R_merge and R_pim are formed beside R_meas from the same per-reflection
sums (stills and rotation merges) and written with CC* overall and per
shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the
_reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as
three new columns of the shell tables. R_pim following Weiss (2001),
CC* following Karplus & Diederichs (2012).
A second data block, data_sample_unmerged, carries the scaled unmerged
observations of a rotation merge as _diffrn_refln (measured index,
frame, scaled I and sigma, pdbx_scale_value) - the same records as the
.hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now
carries the scale and the frame. The unmerged MTZ stays unscaled.
Reflection data are unchanged: the merged _refln rows, the .mtz columns
and the .hkl are written as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Structure factors, maps and the null on the GPU (with gemmi's mask shrink step), the second
validation started on a forecast beside the first, change-of-basis candidates scored at the same
time, validation work spread over the visible cards, the merge's GPU engine freed before the
validation, a CUDA error failing the validation instead of restarting it on the CPU, and no
first-validation maps where the forecast one is kept. Output files md5-identical to the CPU path's
reflection files and placed model on 8sa8, 8xtg, 9ea5, 8tyy, 7qis and myob; maps within float
tolerance of the CPU path and bit-identical run to run and at -N 8. Validation time 8sa8 24.0 ->
7.4 s, 8xtg 18.4 -> 5.0 s, 9ea5 19.1 -> 3.0 s, 8tyy 47.7 -> 3.0 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The multi-GPU placement (replicate threads pinned with pin_gpu) and the
second-validation memory rule read RigidBodyGPUPool and get_gpu_count(),
which a build without CUDA does not have; both are now inside
JFJOCH_USE_CUDA, where the pool can exist at all. Nothing changes in a
CUDA build. Non-CUDA rugnux and jfjoch_test build; [ModelValidation] 22
cases pass there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Where the validation in the model's setting, started beside the first on
the forecast (b5980c9ba), is going to be kept, it writes the maps, the map
coefficients and the placed model on the written axes, and the first
validation's p_2fofc/p_fofc/p_anom.ccp4 and p_maps.mtz were overwritten
unread. ModelValidationSchedule::maps_superseded is asked once the
verdict, the setting and the indexing are decided, before any map is
made; the first validation then makes none (the map coefficients and the
MTZ are not built, the three map transforms not run, nothing written).
The answer is the same rule that keeps the forecast validation
afterwards (forecast_kept, now one lambda for both). The kept validation
also brings back the reflections it relabelled, free set included, so
relabel_output does not reindex and redraw them a second time. Where the
forecast is discarded or was never started, everything runs as before,
and modelpar's delete-before-rewrite stays for those cases. If the kept
validation then fails, a warning says no maps were written.
Log lines that disappear (first validation only, kept-forecast runs):
"mean 2mFo-DFc density at atom centres = ...", "anomalous difference map
from ... Bijvoet pairs; strongest density ...", "wrote p_2fofc.ccp4, ...";
in their place: "no maps made here - the validation in the model's
setting writes them". (The "anomalous density ... inverted" warning would
also go; it cannot be raised where a forecast is kept.)
Verified on 8sa8, 8xtg, 9ea5 (forecast kept), 8tyy, 7qis, myob_x10sa (no
forecast): p.mtz, p.hkl, p.cif, the three maps, p_maps.mtz and
p_model.cif/pdb md5-identical to the merge commit, repeated and at -N 8;
the discard path (8sa8 with a rotated model) identical files and log.
Time saved is small with the maps already on the GPU: 8sa8 7.6 -> 7.4 s,
8xtg 5.1 -> 5.0 s, 9ea5 within noise.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Both sides kept: modelpar's parallel basis scoring, the second validation
started on a forecast beside the first (write gate, schedule parameter),
the multi-GPU placement and delete-before-rewrite; gpusf's GPU structure
factors, maps and null engine, failure-instead-of-restart, and the merge
engine released before the validation (now just before modelpar's
ValidateAgainstModel call, after the forecast lambda is set up).
Placement: each validation's structure-factor engines (d_min and the
null's) are made on the card of the thread that runs it - the main
thread's for the first validation, card 1 % count for the speculative
second, which pins itself there - so two validations on two cards use
both, as the rigid-body pools do.
One memory rule for both, per card, from total memory, up front: a
validation plans at most half of its card - its structure-factor engines
a quarter together (was half for the d_min engine alone), its rigid-body
engines a quarter (RigidBodyGPUPool) - and the second validation runs
beside the first only where twice the first's plan (rigid-body planned
bytes + the d_min engine, twice it where a null is coming, the null's
engine being no larger) fits half of all cards' memory together (was:
twice the rigid-body plan within a quarter). The card's total is read
once when the engine is made; a CUDA error there fails the validation
like any other.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The null - nine random placements of the model, and the real model's side
of the comparison - is scored to a coarser limit (3.5 A, or where the free
set allows), and those structure factors still ran on the CPU, one model
at a time on each replicate thread. Where the d_min structure factors are
on the GPU, a second ModelStructureFactorsGPU engine at the null's limit
is now made, on the same card, before any replicate starts; every
replicate and the real side go through it, so all of them are computed
the same way. It is decided like the first engine, from the card's total
memory: the two together must stay within half of it, else the null stays
on the CPU (logged). The replicates' threads may sit on other cards
(modelpar's 8d6e9d3a4); each call works on the engine's own device. Where
the null is scored to d_min it shares the d_min engine, as before.
Measured (validation time, before -> after; CPU path for reference):
P3_2 1.55 A: 11.2 -> 9.0 s (CPU 17.7)
P2_1 oblique: 5.1 -> 4.5 s (CPU 7.9)
C2 1.11 A: 12.9 -> 13.2 s (CPU 23.3) - its null time is in the
scale fits, which stay on the CPU
p.mtz md5-identical in all four variants per set (before, after, CPU,
-N 8); maps and map MTZ bit-identical after vs -N 8; the null's sigma
moves only in the second decimal (+113.39 -> +113.38, +86.13 -> +86.14).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two
threads that drive the merging tail were not: the quality guard took its card with set_gpu(1),
and the main engine runs on the pipeline's own thread, which goes on to everything else. On a
multi-socket server their pageable copies to the card went through a remote socket.
RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost,
IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous
device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding
covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads
the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use
std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not
squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses
pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask
themselves; the new test checks both that and that a pinned caller keeps its own mask.
No flag, no environment variable. On one NUMA node (this workstation) only the device setting
happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base =
rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa,
cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs),
-N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise,
e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for
multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A CUDA error on the validation's GPU path (rigid body, structure factors,
maps) used to start the whole validation again on the CPU - a migration
part way through, which makes how long a run takes, and in principle what
its rigid bodies converge to, depend on a device fault. It now ends the
validation with ok = false and a logged reason ("the GPU failed during the
validation (...)"), which the run reports as MODEL_NOT_VALIDATED. A
validation that did not finish decides nothing, so the reflection files
are those of a run without a model. Where there is no GPU at all, or the
up-front rule sends the work to the CPU, nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ModelStructureFactorsGPU computes what compute_model_factors() and
map_from_coefficients() compute on the CPU - F_calc from the model's
density (IT92, Refmac-compatible blur, unblurred as prepare_asu_data()
does) and F_mask from the Refmac bulk-solvent mask, both on the
reflections prepare_asu_data(d_min) lists, in its order; and a map from
ASU coefficients on the grid get_size_for_hkl(coef, 0, 3.0) sizes - on a
device. Made once per cell, group, resolution and model, then evaluated
as often as the coordinates change, so refinement or MR can call it in a
loop. The device is an explicit parameter; every call leaves the calling
thread's current device as it found it.
Pieces:
- ModelDensityGPU: the rigid body's deterministic brick gather, moved
out of RigidBodyGPU.cu into a component of its own (ModelMaskGPU's
pattern); the rigid body uses it unchanged. MAX_BRICKS_PER_AXIS 8 ->
16, so fine grids with high-B atoms (lysozyme at 1.2 A, a 0.9 A P1
cell) are no longer refused; existing zones are gridded identically.
- One copy of the content is gridded and the symmetry composed in
reciprocal space (SymmetryComposition), operators applied on the fly;
the mask is ModelMaskGPU (every image of every atom, islands, shrink).
- Maps: gemmi's get_f_phi_on_grid() in ZYX order on the host (the
coefficients written are the same), in-place cuFFT c2r, transposed back
to XYZ on the device. One map at a time, in the engine's buffers.
Decided once, up front, per card, from its TOTAL memory: the engine's
bytes (16 N + cuFFT work + reflections, N the larger of the structure-
factor and map grids) must be at most half the card - the rigid body's
engines take at most a quarter beside it. Otherwise, or where the gather
cannot grid the cell, the CPU path runs, logged with needed vs total.
Anything to a resolution other than d_min (the null's 3.5 A fits) stays
on the CPU, so all replicates and the real model's side of the null are
computed the same way. A CUDA failure takes the existing path: the
validation restarts on the CPU.
Measured, model validation total per run (CPU path -> GPU), 16 GB card:
F432 215 A cubic, 1.30 A, 500^3 grid: 47.7 -> 15.6 s (two validations;
14.3 -> 2.4 and 33.4 -> 13.2, the rest of the second is writing the
three 0.5 GB maps); F_calc + F_mask 5.7 s -> 0.05 s
C2 1.11 A: 23.9 -> 13.6 s; P3_2 1.55 A: 18.2 -> 10.2 s;
P2_1 1.25 A: 13.4 -> 6.5 s; F4_132 328 A: 13.0 -> 5.5 s;
P6_5: 8.8 -> 4.2 s; P4_3 0.97 A: 4.6 -> 2.5 s; P1 0.92 A: 4.2 -> 2.2 s;
small P1: 3.2 -> 1.3 s; P6_1: 8.8 -> 6.2 s; lysozyme: 1.8 -> 1.4 s.
p.mtz md5-identical on all 13 sets. Against the CPU path: FC within
1e-4 of mean |F|, phases of the strong half within 0.003 deg, maps within
1e-4 (2mFo-DFc) and 7e-4 (mFo-DFc) of their rms; every logged R, CC,
FOM, k_sol and anomalous site list identical at the printed precision,
except where a rigid-body commit sat on an exact R-free tie (0.2155 ->
0.2155) and fell the other way (R-work 0.2127 vs 0.2129). The GPU result
is bit-identical run to run and with -N 8 (maps, map MTZ, placed model).
Peak device memory of the engine: 2.5 GB at 500^3 (process total peaked
at 14.4 GB with what the merge still holds).
Tests: ModelStructureFactorsGPU_MatchesCPU (five groups, 3.5 and 1.5 A:
same reflections, F_calc <= 1e-5 of mean |F|, F_mask 2e-7 rms, repeat
bit-identical), ModelStructureFactorsGPU_MapMatchesCPU (<= 5e-6 of rms).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The GPU bulk-solvent mask had no shrink step (SolventMasker::shrink(),
set_margin_around() with Refmac's r_shrink = 0.8 A), so every rigid-body
zone whose grid has an offset within 0.8 A was sent to the CPU. That is
not only fine grids: in an oblique setting (P2_1, beta ~141 deg) the
3.5 A zone's lattice-plane spacing is 0.69 A, and on such a set the real
fit and all nine null replicates ran their rigid bodies on the CPU.
The shrink is now two passes over the grid: mark the solvent points with
a macromolecule point among gemmi's near offsets, then turn into solvent
every macromolecule point with such an edge point at any stencil offset
(near or far, gemmi's split at the coarsest grid step). That is gemmi's
rule in both of its branches, and the stencil is built on the host in
double exactly as gemmi builds it. ModelMaskGPU::ShrinkIsNoOp() and
RigidBodyGPUEngine::MaskSupports() are gone; the rigid body no longer
refuses such zones.
Verified (ModelMaskGPU_ShrinkMatchesGemmi): the shrink alone on gemmi's
post-island mask is bit-identical, and the whole GPU mask equals gemmi's
put_mask_on_grid() bit for bit (0 differing points) on the five test
groups at 1.5 A and an oblique P2_1 cell at 3.5 A; repeats are identical.
Measured on the oblique-setting set of the open arm (P2_1, 14k atoms,
1.66 A): model validation 17.2 s -> 5.2 s with the GPU structure factors
of the next commit in place (the first validation's ten rigid bodies,
~12 s on the CPU, now take well under a second). p.mtz md5-identical;
the null moves from +73.1 to +86.1 sigma (the replicates are refined on
the GPU, which agrees with the CPU to rounding), verdict unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The validation in the model's setting writes its three maps and the map
MTZ over the ones the first validation wrote seconds before, and a
two-pass run writes over the first pass's. XFS (like ext4) flushes a file
that was truncated and rewritten when it is closed, so each overwrite
forced ~155 MB out to the disk on the spot. Measured on /data (XFS on a
hard disk): three 155 MB files rewritten 3.16 s, written fresh 0.62 s,
removed and written again 0.66 s. The contents are the same bytes either
way.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The rigid-body pool put all of its engines on the calling thread's current
card, so a validation used one GPU whatever the machine had. Now, by fixed
rules decided up front and never by momentary free memory:
- RigidBodyGPUPool::Create puts engine i on card (d + i) % count, d being
the calling thread's card; the pool restores that card afterwards
(an engine's constructor sets its own) and an engine is released on
its own card.
- The threads that run the null's replicates are pinned with pin_gpu
(which also binds them to the card's NUMA node where that is enabled),
replicate thread t to card (d + 1 + t) % count, and Acquire() hands a
thread an idle engine on its own card where there is one, any other
otherwise. With one card this is exactly the previous back-of-the-list
choice.
- The validation started on the forecast runs on card 1 % count, so with
two cards or more it is on the other card from the first; the up-front
memory rule becomes: twice the planned engine bytes within a quarter of
the cards' total memory taken together.
The engines' kernels are deterministic (no floating-point atomics) and an
engine's result does not depend on which engine it is, so on cards of one
model the numbers are those of one card; what changes is only where the
work runs. On this one-card workstation the multi-card path cannot be
exercised: md5 of p.mtz and the validation outputs are identical to the
base, and the card-count arithmetic was checked by reading only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Where a model that fits is written on other axes than the data, the files
take its setting and the validation is made a second time on the
relabelled data. That second validation depends on the first only through
the setting and the indexing the first settles, and both are known long
before the first has finished: the change of basis right after the frame
scoring, the indexing the probe prefers before the null. So ValidateAgainstModel
now reports them (ModelFrameForecast, through ModelValidationSchedule::
on_forecast), and the second validation is started there, on the
relabelling AdoptModelFrame and relabel_output would make, applied to a
copy of the merge - beside the first one's null, real fit and maps.
It is kept only where the first decides exactly what was forecast (the
model fits, same change of basis, same indexing; a model asserting the
other enantiomorph is not forecast, as the label is decided last, on the
anomalous map). Until then its log is held (Logger::Buffered, replayed
where the serial run logged it) and its files wait on a gate
(ModelValidationSchedule::write_gate) placed before the first map is
written; otherwise it is released with false and returns unwritten, and
the serial validation runs as before.
GPU memory: two validations at once take twice the rigid-body engines.
The parallel start is decided up front from sizes, never from what is
free: allowed where twice what the first pool asked for (bytes per engine
times the engines wanted) fits a quarter of the card's TOTAL memory, the
share one validation may take. Threads: both validations submit to the
one ParallelFor pool from threads outside it (the second runs on a
std::async thread, as the null's replicates do), so no pass runs inline
on a pool worker and the pool's size bounds the workers.
Measured on the loaded 16-core workstation, TIMING model validation:
8sa8 30.6 -> 19.7 s, 8xtg 19.9 -> 13.9 s, 9ea5 22.1 -> 16.4 s (this and
the previous commit together). p.mtz, p.hkl, p.cif, the three maps,
p_maps.mtz and p_model.cif/pdb md5-identical to the base on 8sa8, 8xtg,
9ea5, 7qis and myob_x10sa; the validation's log lines identical as a set
(the null's replicate lines were already in completion order).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Each candidate setting of the model was scored one after another on the
coarse shell (a density, an FFT and an isotropic scale fit, 0.2-0.5 s
each), and a run with a model in another setting scores them in both of
its validations. They share nothing but what they read, so they now run
on ParallelFor, each on its own copy of the model, and are read back in
their own order: the log lines, the ranking and the tie-break (first of
equal R wins) are those of the serial loop.
Measured (16-core workstation, loaded): 8sa8 3 candidates 0.64 s ->
0.22 s; 9ea5 3 candidates 1.47 s -> 0.50 s; 8xtg 1.45 s -> 0.40 s.
p.mtz, the maps, the map MTZ and the placed model md5-identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This reverts commit a825fb6cb.
The per-frame cell-scale retry restored the space group of its one
target (a P2 set whose cell grows with dose) but not its data: R_meas
went 26 -> 37 % and d_min 1.54 -> 1.62 A against the pre-part-B run. It
patches the early, low-dose cell of pass 2 frame by frame with a
+-2 % range, 0.1 % step and sqrt(n) margin chosen on that set, and runs
inside AnalyzeIndexing on every set for one difficult one. A late cell
belongs to rc176's far-end check of the early decisions instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
--model validation (battery-only for users) was 18% of the battery's time. Every
number it produces is unchanged to the bit (p.mtz, maps, placed model and every
model-validation line of the report md5/diff-identical on 11 open sets); only
when and where the work runs changes:
- The bulk-solvent grid fit (FitModelScale, most of the CPU time) fits each
solvent pair on a copy of gemmi::Scaling's target that takes
|Fcalc + k_sol exp(-b_sol s^2) Fmask| once per pair instead of at every
solver evaluation; same expressions, same types (new test checks a grid
point against gemmi's own Scaling fit with ==).
- Fcalc density and the solvent mask are made on two threads; the model's
structure factors beside the GPU engine reservation.
- The indexing probe fits the relabellings concurrently.
- The null's replicates run beside the real model's placement (they start
from a snapshot of the model as read); one GPU engine per replicate plus
one for the real fit instead of a cap of 4 (engines are interchangeable
and deterministic).
- The 2mFo-DFc, mFo-DFc and anomalous maps are made and written
concurrently; the placed model is written beside the reflection files.
- A rigid-body zone whose solvent-mask grid needs gemmi's shrink is sent to
the CPU when the engines are reserved (ModelMaskGPU::ShrinkIsNoOp), instead
of failing on the GPU and validating everything again on the CPU - the
same CPU result, without the wasted first attempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A rotation frame is refined on the sweep's one cell (orientation and beam per frame, never the
cell), and a crystal whose cell grows with the dose leaves that cell behind: the per-frame gate
(20 % of the in-resolution, non-ice spots on the lattice) then refuses the frames it misses as if
the crystal had gone. A frame that fails the gate is now tried once more on the sweep's cell scaled
isotropically - steps of 0.1 %, out to +-2 %, nearest first - to the scale that puts most of its
non-ice spots on the lattice, re-refined there, and integrated on it where it then clears the gate.
The scale must gain more spots than sqrt(count), so forty trial scales cannot turn a chance spot on a
sparse frame into a frame. Ice-ring spots are not counted, as the gate does not count them. A frame
the sweep's cell already admits is untouched, and only the per-image integration path does this
(not the first-pass validation scoring).
Found on 5ky6 (P 1 21 1, 564 x 0.5 deg, B rising by 46 A^2 over the sweep), which part B
(5b9144410, decisions before pass 2 from the first part of the sweep) turned from P 1 21 1 into
P 1 2 1. The mechanism:
- The first pass now indexes on [0, 141 deg). A per-frame probe of the isotropic scale that puts
most spots on the canonical lattice reads -0.1 % on frame 0, +0.4 % at 50 deg, >= +0.6 % from
115 deg on (at frame 230: 62 spots on the lattice at the sweep's cell, 112 at +0.6 %). Sampling
the whole sweep (before part B) handed pass 2 a cell averaged over the dose, so frames to ~147 deg
and part of the last wedge still cleared the gate; the early-range cell is the low-dose cell, and
every frame from 113 deg on was refused (221 of 564 frames merged, completeness 83 %).
- The axial 0k0 row is recorded in those frames: the merge held 1 screw-absent axial reflection
instead of 15, so the 2_1 screw could not be decided (P2 by default), and R-free 0.243 -> 0.263.
- Separately, on the early range the FFT candidate matched a false C-orthorhombic setting
(gamma 92 deg) whose refutation fell straight to triclinic instead of P-monoclinic, so the cell
was refined free (cell dev 0.64 -> 1.03 %). Not changed here (see the report).
Integrating every frame from the sweep's cell instead (the sparse-lattice switch forced on) also
recovers P 1 21 1, but at 1.63 A with ISa 5.0 and R_meas 45 %: the late frames are then integrated
on a cell they have left.
Result (GPU, battery verdicts against the rc175 full run 7c9a6c7d1):
5ky6 fail P 1 2 1, d 1.50, R-free 0.263, compl 83.0 % -> pass P 1 21 1, d 1.62, R-free 0.234,
compl 99.6 % (pre-part-B: P 1 21 1, 1.54, 0.244, 96.9 %)
9rci pass -> pass, compl 86.5 -> 94.7 %, d 1.81 -> 1.76
9qw8 pass -> pass, R-free 0.329 -> 0.326, ISa 5.2 -> 5.5
8v4o fail -> fail (its part-B lattice loss is untouched), compl 22.3 -> 23.1 %
p.mtz md5 unchanged: 5nw5, insu_I_x06da_low_isa, nothing_1, nothing_2, myob_x06da_powder_2,
7atg, 5reo, 6fvz, 6hwj, 6pxc, 8tha, 9crw, 9jzo, 6zqr, 6rym, thau_x10sa_0p1deg, cytc_x10sa.
Catch2: Rugnux_Rotation, RotationIndexer* pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The device correction-surface fit (gpu-tail2) keeps its per-term arrays - the
terms, their parity, the group CSR, the three subsets' orders and the per-term
weights, about 60 bytes a term - as engine members, and nothing freed them
after the last surface. They stayed on the card through the merge, where the
GPU Wilson outlier test (gpu-tail) then asks for ~96 bytes per full. On a
large rotation set (139.8 M partials, 13.35 M fulls, 16 GB card) the engine's
resident arrays left 1242 MiB free for the first Wilson test and 998 MiB for
the one after the automatic resolution cut, against 1222 MiB needed: the run
stopped with "Wilson outlier test ... run the CPU build" although nothing ran
beside it. The surfaces' buffers are released once the last surface is
fitted (the next pass's surfaces build their own), which gives back 905 MiB
there.
The up-front checks of the three dataset-sized tail steps (Wilson test, pair
sort, error-model fit) also read free memory as cudaMemGetInfo reports it,
which counts what the allocation pool holds with no buffer in it as taken.
They now hand that back and, only if it is still short, wait for GPU work
running beside the merge (GPUWorkBeside, the speculative probe) to end before
giving up - as the engine's own allocations already do. That changes when a
step runs, never what it computes; a set that does not fit still fails at
once with the same "run the CPU build" message.
p.mtz: the large set completes (and is identical with the Wilson test forced
onto the host); myob 9bdd6916 and 8a1a 309ea9a8 unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The R_meas accumulators per group and the device's per-group sums were eight
group-length locals plus a 56-byte record per group, value-initialised on one
thread for every merge that reports statistics. They are now members, reset in
parallel, and the device branch's pass that assigns each group its shell and
copies its sums runs on all threads; the per-shell observation counts are
integers, gathered per block. The shells' R_meas sums are still added in group
order, as before.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Which groups are written is decided first (KeptIndices, in group order), then
each written group's MergedReflection is formed on its own, on all threads,
into a result sized once - where the serial loop pushed back up to millions of
records, growing a vector that each merge allocated afresh. The same records in
the same order.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The rejected flags and the observation keys and halves (per full), the merged
intensities (per group) and the Wilson observation records were locals: on a
large cell every one of the run's merges allocated, page-faulted and zeroed
them again on one thread - hundreds of megabytes per merge on 8qaw. They are
now members, only resized, and reset with ParallelChunks where a pass needs
them reset. The error-model samples are no longer cleared before the device
download overwrites them (the host path still clears before it appends).
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The fused adaptive spot finder's azimuthal profile (and the standalone AzIntEngineGPU) summed the
corrected pixel values - pixel times a float correction - with float atomicAdd, in shared memory
and then across blocks. Float addition is not associative, so the per-ring sums depended on the
order the threads and blocks happened to arrive in, and on the grid size chosen for the GPU. The
profile, and the per-image background estimate read from it, moved in their last bits between two
runs of the same command: on a thaumatin JUNGFRAU 4M sweep one bkg value in _plot.txt differed in
the 4th digit between runs. The raw ring sums that drive detection were already exact integers;
only the reported profile was affected.
Each corrected value (and its square) is now quantised to 2^-12 on its own and summed as a signed
64-bit integer, the convention the raw ring sums and the Bragg-integration profile already use.
Integer addition is associative, so the sums are the same whatever the order or grid.
The repeat test of the fused finder now also requires the profile and its std to repeat bit for
bit (it failed on the old kernel at the first repeat); the AzIntEngineGPU test re-runs 20 times.
Fused finder benchmark: 0.313 -> 0.324 ms/frame.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With the fulls resident (the GPU combine ran), ApplyCellSurface no longer builds
its terms on the host and uploads them: SurfaceBuildTerms makes them from the
resident fulls, this surface's cell per full and the current corr (the only two
uploads) - which fulls give a term, the terms in fulls order, the group CSR over
them and each subset's per-(block, cell) segments - and SurfaceShellWeights the
resolution gauge's inputs: the shell edges (order statistics of 1/d^2 read off a
sort), each group's shell and each (cell, shell) bin's weight. Every order the
host made is a stable counting sort, which a stable radix sort of the indices
reproduces; the per-bin weights are summed one bin per thread in fulls order
with the host's rounding. The existing exact surface kernels read them as
before. Without resident fulls (the observation dump) the host path runs.
p.mtz md5 and the surfaces' log lines unchanged on myob, cytc, thau, kdp, 8a1a
and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With two GPUs or more the two-pass quality guard's merge of the pre-pass
frames gets a scaling engine of its own on device 1 (RotationScaleMerge::
SetGpuDevice), and the main engine's device ingest and first merge no
longer wait for it to hand the card back. The first merge waits only for
the guard's ingest, the last time the guard reads the outcomes: that merge
writes per-frame fields (mosaicity among them) back into them.
Neither merge reads what the other writes, and a merge gives the same bits
on any card of one model, so only when the guard runs changes. With one
GPU, and in the CPU build, the order is the one it was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
std::thread::hardware_concurrency() counts every CPU of the node, so a job
given a CPU set (taskset, a cpuset, a Slurm allocation) or a container CPU
quota started one thread per node CPU: the default -N, the shared worker
pool and every "0 = all threads" default. AvailableCpus() (ThreadAffinity)
counts the start affinity mask, capped by the cgroup CPU quota (v2 cpu.max,
v1 cfs_quota/period, the process's own cgroup and then the mounted root),
read once. Outside Linux it is hardware_concurrency(), so the viewer tree
stays portable. It replaces every hardware_concurrency() call outside the
tests and the vendored pocketfft.
Only how many threads run changes; the passes split their work by n alone,
so the results do not depend on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
FitErrorModelGPU puts the samples in the host's rank order with four stable
radix passes, so the equal-count bins hold the same samples; computes every
normalised deviation with the host's rounding, so every bin median is the
host's to the bit; and hands the per-bin terms to the host's own update step,
now ErrorModelUpdate (a pure extraction from FitErrorModel). The one thing
that differs is the order each bin's two sums are added in: a fixed tree on the
device, the order nth_element happened to leave the bin in on the host. a and
b^2 can therefore differ from the host fit in their last bits - measured on two
synthetic pools: relative difference 2e-16 and 3e-13 in a, 5e-16 in b^2. The
device fit is deterministic (same pool, same bits).
p.mtz md5 nevertheless unchanged on myob, cytc, 8a1a and 8qaw (GPU build): the
difference does not survive the float rounding of the merged sigmas there. A
knife-edge decision elsewhere could still see it, which is why this is kept as
its own commit.
The host fit took 0.4-0.6 s per fit on the large merges, about half of it the
initial equal-count split, with only 16-way parallelism in the iterations.
Device peak about 72 bytes per sample (the rank sort; each stage's buffers are
freed when it is done); the free memory is checked first and a device without
it stops with a message naming the CPU build.
Kept last on the branch so that it can be dropped on its own.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Every full's observation key and its half - the parity of the key's rank among
its group's usable fulls - are integer work, so the device gives the host's
halves exactly: one thread per full for the keys, one per group for the ranks.
The host receives both (the anomalous statistics rank the same keys again within
each Bijvoet cell) instead of walking every full's record for them.
p.mtz md5 and p.cif unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The strong-reflection asymptote is read from per-group sums of I*corr, its
square and the counting variance at the group mean - a walk over every full's
record on the host. In the GPU build a kernel takes them, one thread per group
in index order with the host's rounding (CountingVarianceExact); the host keeps
the strong-group selection and the median.
p.mtz md5, p.cif and the error-model lines (which print the asymptote)
unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The merges of the largest sets run with 12-13.6 GB of a 16 GB device already
taken by the resident observations, so the transient footprint of this pass is
what decides whether it fits. Release the sort keys as soon as the d order is
known and the noise ratios once their per-shell medians are read - on the
device, rather than downloading every sorted value. Peak about 96 bytes per
observation; the free memory is checked first and a device without it stops
with a message naming the CPU build.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The multiplicative spread of the outlier band is the weighted median of
(|ln(I/median)|, weight) pairs, one per usable full, and its sort was the bulk of
that step. SortPairsGPU sorts them on the GPU (by the second key, then stably
by the first) and leaves the sequence std::sort leaves: two pairs that compare
equal are the same two numbers. The values themselves (a logarithm) and the
running sum stay on the host.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build); a Catch2 case
checks the device order against std::sort.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The per-group sums the widening is measured from - each hand's inverse-variance
mean under the error model's sigmas, the group's d and the d range - are a walk
over every full's record on the host. In the GPU build a kernel takes them, one
thread per group in index order, with the host's model sigma to the bit
(ModelSigmaExact: every product rounded on its own); the host keeps the shells
and their medians. The host path is the same sums, now kept per group so both
paths hand the shell step one array.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
MergeEmSamples now forms each sample whole on the device - the counting
variance at the reflection's expected intensity included, every product rounded
on its own as the host build rounds it - and compacts them in fulls order with a
scan, so the host receives exactly the vector it used to build. That replaces
the download of three full-length double arrays and a flag array, and the host
pass that compacted them while reading every full's eighty-byte record.
p.mtz md5 and the error-model lines unchanged on myob, cytc, 8a1a and 8qaw
(GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
WilsonOutliersGPU runs the test of WilsonOutliers.cpp step for step on the
device and returns the same result to the bit: the two orders the host sorts
into are total orders, so a stable radix sort of the indices reaches the same
sequence; every sum is taken by one thread over its shell or reflection in the
host's order; each product the host rounds on its own (-ffp-contract=off) is
rounded with __dmul_rn; medians are read off sorts; and the logarithms (t, z,
u, the tail scale) are taken on the host from downloaded counts and medians.
MergeAndStats calls it where the merge itself is on the device.
8a1a final P61 merge, "rsm merge and statistics" 4.35 -> 2.82 s (the test runs
twice there, once per error-model fit). p.mtz md5 unchanged on myob, cytc,
8a1a and 8qaw (GPU build). A Catch2 case checks the device result against the
host one on a population that reaches every branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The automatic cut fitted its logistic to one Pearson CC1/2 per bin, which
has no bound on what one pair can do to it. In a weak shell a single
reflection with a huge intensity in both halves - an artefact the halves
reproduce, or one observation with a sigma fifty times the shell's - carries
nearly all of the covariance. Each bin's reflections are now dealt into up
to seven groups by their indices (groups of at least ten reflections), and
the bin reads the median of the groups' CC1/2. On a clean shell every group
estimates the same CC1/2, so the median is the Pearson value; an outlier
takes one group with it and the median does not follow. Only the cut
decision changes; the reported shell table is unchanged.
Found on a weak split myoglobin crystal: one reflection carried 50-99 % of
the covariance in most fit bins past 2.6 A (e.g. 43647 in a bin with median
I 1.3), reading them at CC1/2 0.93-0.98 where the median of groups reads
0.2. The cut swung between 1.21 and 2.03 A with --prepass-fraction; it is
now 1.85 A at 0.5 and 1.89 A at 1 (XDS 1.51 A, rc.174 1.96 A).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pixel that reads at most half of its own sector's level on average, by more
than the bound a hot pixel's excess has to clear, is now masked as dead - the
mirror of the hot rule. Nothing but a defect keeps a pixel dark frame after
frame, and every reflection whose disk covers it is integrated short.
The finder now also runs on CCD sweeps (no sensor depth): a dead CCD pixel
reads its offset, far below any ring with background, and the hot rule's
ten-fold ratio does not depend on Poisson counts either.
Found on 5jk4 (ADSC Q4, 2304x2304): the last row and the first two columns
read the offset (~20 ADU against ~60-100 around them) and were unmasked.
Reflections touching them merged to large negative intensities, and one
observation at y=2298 with sigma 10334 swamped the outer shell's half-set
CC1/2 (3.9 %) and the overall CC1/2 (0.786; 0.998 at rc.174). With the
6630 dead pixels masked the fall-off is clean: CC1/2 0.998, cut 0.96 A
(fit 1.01 A), at --prepass-fraction 0.5 and 1 alike. Counting-detector
sets with low background are untouched (myoglobin X06DA: 0 dead).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Rotation prediction enumerated the whole (2h+1)(2k+1)(2l+1) box every frame and kept the ~1% whose
rotation solution lands near the frame. A solution within phi_limit of the frame needs
|f| = ||p0|^2 + 2 S0.p0| <= 2 |S0_perp| |p0| phi_limit, and along an (h, k) column f is a quadratic
in l, so the l that can pass are at most two intervals. Those are walked - widened by 1% in f and by
a whole index at each end - in the same order and with the same arithmetic; the rest the phi test
rejected anyway. Without a phi limit (min_zeta 0) the whole column is walked as before. The GPU
kernel is unchanged.
A new test predicts 1000 random lattices, geometries, axes and rocking widths both ways and
requires the same reflections in the same order, every value bit for bit (30000 cases passed once
locally). CPU build, two interleaved warm runs: 8a1a output loop 241.5 -> 216.3 s, pre-pass loop
90.7 -> 73.8 s, whole run 427.0 -> 384.2 s; 8qaw (with the gzip change before this) loops
116.6 / 251.3 -> 94.4 / 206.8 s, user CPU 14700 -> 12718 s. GPU build within noise.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ApplyCellSurface copied and sorted every term's 1/d^2 to read 19 order statistics off it, for each
of the four surfaces of the final merge. The edges are now selected: one pass counts the values
under each top-16-bit pattern (a non-negative float orders as its bits), a second collects the few
bins the ranks fall in, and nth_element picks the value inside each. An order statistic is a value,
so the edges are the ones the sort gave.
8a1a, GPU build: the shell set-up of the four surfaces 0.55 -> 0.26 s (instrumented run), the
correction-surface window 6.32 -> 6.12 s and the final merge 11.03 -> 10.82 s (two interleaved warm
runs each); whole run 55.78 -> 55.45 s. 8qaw surfaces 2.95 -> 2.83 s.
Tried in the same window and not kept, as they measured nothing: the serial stable counting sorts
(surface group order, decay terms, fulls CSRs, delta-CC1/2 frame order) on all threads; delta-CC1/2
workers uncapped from 8; measuring two rounds of the edge search at once. The delta-CC1/2 window
(2.4 s) is bound by its longest chain of ranges, one serial accumulation, not by the worker count.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged scaled fulls the final merge hands back for the .hkl were gathered by one thread
pushing back over every full. Which fulls is now KeptIndices (the same list, in the same order),
and each record is made side by side.
The same records in the same order: p.mtz and the .hkl byte-identical (myob, thau, cytc, kdp,
cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a, 8tyy, CPU build myob). 8a1a, no other
load, interleaved: the P61 merge's statistics stage 3.94 -> 3.71 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
