Both arrays are one record per integrated observation - tens of millions on a fine-sliced long axis,
gigabytes each - and both carry the raw hkl only to sort and group on. A Miller index needs sixteen
bits (|h| <= a / d_min, in the hundreds even on the longest axis at atomic resolution), which takes
the ingest sort key from 24 to 20 bytes and the post-refine partial from 32 to 28.
Same comparisons, same order; merged output byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The device merge's per-group sums were downloaded into a full-length host array per field and then
unpacked into the per-group accumulators, so every merge held two full copies of them - on a large
cell searched in P1 that is most of a gigabyte beside the accumulators themselves. MergeAccum now
leaves the sums on the device and MergeAccumRange downloads a million groups at a time, each slice
unpacked straight into the accumulators.
Same values, same integer reject count; merged output byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The whole-run passes retain every frame's integrated reflections until scaling is done - thousands
of vectors of a few megabytes each, allocated by the image workers in the allocator's per-thread
arenas. When a pass hands them back, most of that memory stays in those arenas as holes, and the
next pass's workers (new threads) do not reuse it, so on a fine-sliced long axis gigabytes of freed
reflections were carried to the end of the run.
IndexAndRefine now copies each retained frame's reflections into a ReflectionArena: 64 MiB blocks,
each its own mapping, carved by a bump pointer and returned to the system in one piece when the
last vector in them is gone. IntegrationOutcome::reflections becomes a std::vector with an allocator
that uses the arena when given one and plain new/delete otherwise (copies go to the heap), so the
read sites are unchanged; the few functions that took the vector by type now take a span.
No arithmetic changes; merged output byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The ingest-time geometry smoothing (delta_phi from the smoothed lattice, one exact-Bragg angle per
rocking event, partiality from the smoothed mosaicity) used to run over a 32-byte host record per
observation, copied from the source reflections and held beside them. It now reads hkl, frame, d
and zeta straight from the source reflections, frame by frame, and keeps arrays only for what it
rewrites (delta_phi, partiality) and what the rocking-event walks read in raw-hkl order
(image_number, and on the resident path the usability flag): 13 bytes an observation instead of 32.
The full-Obs path runs the same code and copies the two rewritten fields back.
Same arithmetic in the same order; merged output byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Pooled run B: twin report-only evidence and free-R on the lattice holohedry,
information-weighted resolution cutoff with weighted outlier median and ISa
rework, sparse-lattice integration and beam-centre arbiter, GPU/CPU collapsed
scale upload, geometry walk on the realised residual, --model speed-ups with
FFTW structure factors, memory reductions for large-cell sweeps, -A decisions
Friedel-merged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Data quality - resolution, CC1/2, R_meas, completeness, ISa - is no longer a
pass/fail criterion anywhere; it is reported beside the reference in the tables
and plots for a human to judge. The CC1/2 noise ratio against XDS stays as a
reported guide.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The x86/ARM split asked for on macOS already exists: BitShuffleBlock.h (fbc507839) sends
the per-block transform to bitshuffle_hperf everywhere except aarch64 with NEON, where it
uses the classic bitshuffle's NEON path, and both call sites go through it. Nothing new is
needed for Apple Silicon - clang --target=arm64-apple-macos defines __aarch64__ and
__ARM_NEON exactly as aarch64 Linux does, both vendored files compile to arm64 Mach-O
objects, the seam resolves to bshuf_(un)trans_bit_elem and bshuf_using_NEON() is 1. The
header now says why the switch is a preprocessor one: a universal build compiles it once
per architecture, which a CMake-time answer cannot follow.
What had never happened is the NEON code actually running. It has now, under
qemu-aarch64 in the project's cross image: 504 blocks (elem 1/2/4/8, 8 elements up to the
128 kB block, odd multiples of 8, random and detector-like data) plus 54 whole-buffer
bitshuffle/LZ4 streams with element counts that are no multiple of 8 or of the block. The
output is byte-identical (same md5) across aarch64 NEON, aarch64 without NEON (seam falls
back to hperf's portable code), x86 hperf AVX2 via ifunc, hperf SSE2, hperf portable
fallback, and classic SSE2 / AVX2 / scalar; every implementation decodes every other.
Throughput, one thread on a loaded Zen 3, 16/32-bit, GB/s encode/decode: hperf AVX2
~9-12/~10-11, classic AVX2 ~8/~5.5-8, classic SSE2 ~4-5/~4.5-5.5, hperf portable fallback
auto-vectorised ~2.8/~5, not vectorised ~1/~1.8. Classic 128-bit SIMD beating hperf's
fallback is the x86 stand-in for the choice the seam makes on ARM; no ARM hardware was
available, so the NEON-vs-fallback ranking on a real core is still unmeasured.
The one thing that would have failed on aarch64 is the Bshuf_SSE test, which required
SSE2 outright. It now accepts NEON as well, which also makes it the check that a Mac or
DGX Spark build did not end up on the scalar path.
Not built: no CMake configure or jfjoch_test build was run on this machine (busy); the
test expression was compiled and run standalone on x86 only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 5957b15493dfd0cc8a18fb780a18c2573a6eccde)
The owner's scoring decision: no R_meas criterion, and merge quality
judged only by CC1/2 relative to XDS.
- The global rule (R_meas > 60% or CC1/2 < 0.5 fails) is gone.
- XDS arms: a set fails `merge` when (1/CC1/2 - 1) of the reference-range
table is more than twice XDS's, XDS's CC1/2 taken at the bottom of its
0.1% rounding. CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S at any CC1/2, and
E goes as 1/observations: 2x is XDS's merge with half its observations.
Stored as cc_half_noise_ratio. No REFRES table or no XDS CC1/2: no
criterion.
- Open arm: no merge criterion; CC1/2 and R_meas stay reported numbers.
- Low-resolution R_meas is reported, not scored: the lowest shell of the
own table, the reference-range table and XDS's (lowres_*), with
lowres_r_meas_ratio in the like-for-like table and a ratio plot.
- Every schema-3 run is re-scored when it is read (report, compare,
baseline delta) with today's scorer and today's manifest rows, so both
sides of a comparison are scored alike. results.json keeps the verdicts
as scored at run time; rows without a lattice keep them.
Re-scoring the rc172 reference run: inhouse 29 -> 27 pass (three new
merge fails, one R_meas fail lifted), open 137 -> 138 (one R_meas fail
lifted).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
None of this has been built on a Mac - there is none yet. It is the list a read-only audit of the
viewer/rugnux subtree produced, plus a serial -fsyntax-only pass of every reachable .cpp with
clang 16 + libc++ on Linux, which found exactly one error (the first item).
- JFJochDatasetInfoChartView: std::vector<fftwf_complex> does not compile with libc++, whose
construct_at refuses an array element type (float[2]). Use std::vector<std::complex<float>> and
the reinterpret_cast every other FFTW call site already uses.
- libcurl: GSSAPI off on Linux, and neither TLS nor GSSAPI on macOS. The viewer never sets
CURLOPT_HTTPAUTH, so Negotiate was dead weight that cost a krb5-devel build dependency; on macOS
curl's FindGSS refuses the system Heimdal outright, and with Secure Transport gone from curl
(8.15) TLS would mean a Homebrew OpenSSL - the only host library a Mac build would need.
Linux keeps OpenSSL. CURL_USE_GSSAPI is forced OFF rather than left unset so an existing build
tree drops its cached ON.
- libjpeg-turbo ExternalProject: CMAKE_SYSTEM_NAME/PROCESSOR were forwarded unconditionally, which
puts even a native sub-build into cross-compiling mode, and CMAKE_OSX_ARCHITECTURES / SYSROOT /
DEPLOYMENT_TARGET were not forwarded at all. Now the same rule the zlib-ng sub-build follows.
- ShadowAccumulatorGPU.cu was added on the JFJOCH_USE_CUDA option (default ON) instead of
JFJOCH_CUDA_AVAILABLE like every other .cu, so a machine without nvcc got a CUDA source in a
target with no CUDA language.
- CMAKE_OSX_DEPLOYMENT_TARGET defaults to 12.0 (overridable). Left unset, CMake takes the build
machine's OS version and the .dmg starts nowhere older.
- Standard headers that were only arriving transitively (<chrono>, <cmath>, <limits>, <cstring>,
<atomic>, <thread>, <string>); newer libc++ releases keep removing such transitive includes.
Checked: the seven changed sources pass clang 16 + libc++ -fsyntax-only. The CMake changes are
not configured or built.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015eAE2K7i5JGDwgwifiCfuA
(cherry picked from commit 5a7282759a)
parse_correct_lp also reads the first row of CORRECT.LP's last resolution
table: its high-resolution limit (dmin_low) and R_meas (r_meas_low).
inhouse.json regenerated with `refs --arm inhouse --write`; the only
change is the two new keys per set (the private manifest was regenerated
the same way, outside the repository).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
H5Pget_filter2 takes cd_nelmts as the number of elements the cd_values buffer can hold and
writes up to that many. It was given sizeof(cd_values) = 32 for an 8-element array, so a
filter carrying more than 8 parameters would have been written past the end of the buffer.
Bitshuffle carries 5, so nothing overflowed in practice.
The "Weird value" message printed cd_values[1] while the test above it is on cd_values[4].
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015eAE2K7i5JGDwgwifiCfuA
(cherry picked from commit dd1af9576c)
-A set MergeFriedel(false), and RotationScaleMerge builds one grouping for the
whole pass from it, so every decision on an anomalous rotation run was taken on
hand-separated groups: per-frame scaling, the correction surfaces, the outlier
median, the error model, the CC1/2 half-sets and the resolution cut, the
delta-CC1/2 frame ledger and the space-group search. That halves the
multiplicity each of those gates was calibrated at, and flipped the bare run's
symmetry, resolution and frame-rejection calls on several sets - none of them
through the anomalous signal (a 422 refused on an added-operator R, a 23
refused on a chi^2 gate, a screw row read on hand rows, thin hands exempt from
the outlier median letting one hot pair own a CC1/2 bin).
A rotation run now always merges with Friedel's law: the Rugnux constructor
turns MergeFriedel back on when the run is rotation (IsRotationIndexing, which
--force-still clears), and --mode scale does the same for its rotation merge.
Nothing is lost for anomalous use: the merge always keeps the Bijvoet split, and
I(+)/I(-), F(+)/F(-), SigAno and CCanom are written and reported from it
whether or not -A was given. -A on a rotation run is therefore a no-op on the
processing, which is the point; stills keep -A as their only anomalous route.
A bare run is untouched.
This is step 1 of the -A design (decide with pairs merged, present with them
split). The per-hand statistics table that -A should select is not built yet,
so an -A rotation run now reports the Friedel-merged table (FRIEDELS_LAW=
TRUE, Laue multiplicity and completeness). The thin-hand pair median of the
previous commit is unreachable from rugnux on rotation data but kept for any
direct MergeFriedel(false) caller of RotationScaleMerge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Data quality - resolution, CC1/2, R_meas, completeness, ISa - is no longer a
pass/fail criterion anywhere; it is reported beside the reference in the tables
and plots for a human to judge. The CC1/2 noise ratio against XDS stays as a
reported guide.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The owner's scoring decision: no R_meas criterion, and merge quality
judged only by CC1/2 relative to XDS.
- The global rule (R_meas > 60% or CC1/2 < 0.5 fails) is gone.
- XDS arms: a set fails `merge` when (1/CC1/2 - 1) of the reference-range
table is more than twice XDS's, XDS's CC1/2 taken at the bottom of its
0.1% rounding. CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S at any CC1/2, and
E goes as 1/observations: 2x is XDS's merge with half its observations.
Stored as cc_half_noise_ratio. No REFRES table or no XDS CC1/2: no
criterion.
- Open arm: no merge criterion; CC1/2 and R_meas stay reported numbers.
- Low-resolution R_meas is reported, not scored: the lowest shell of the
own table, the reference-range table and XDS's (lowres_*), with
lowres_r_meas_ratio in the like-for-like table and a ratio plot.
- Every schema-3 run is re-scored when it is read (report, compare,
baseline delta) with today's scorer and today's manifest rows, so both
sides of a comparison are scored alike. results.json keeps the verdicts
as scored at run time; rows without a lattice keep them.
Re-scoring the rc172 reference run: inhouse 29 -> 27 pass (three new
merge fails, one R_meas fail lifted), open 137 -> 138 (one R_meas fail
lifted).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
parse_correct_lp also reads the first row of CORRECT.LP's last resolution
table: its high-resolution limit (dmin_low) and R_meas (r_meas_low).
inhouse.json regenerated with `refs --arm inhouse --write`; the only
change is the two new keys per set (the private manifest was regenerated
the same way, outside the repository).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
On the resident (GPU) path the ingest's narrow host record sits on top of the source reflections
for the whole smoothing and partiality recompute, and on a long axis that moment is the run's
memory high-water mark. Store its Miller indices in 16 bits and move the rocking-event flag into
their padding: 40 -> 32 bytes a partial. Stage the device upload in slices of two million
observations rather than eight, which takes the thirteen staging arrays from ~400 MB to ~100 MB.
Neither changes a value.
Measured on a 3600-frame long-axis rotation set (-N 6): VmHWM 13.79 -> 12.80 GiB on top of the
previous commits (15.0-15.1 GiB before any of them); merged MTZ/HKL/CIF, P1 MTZ, per-image table and
the report are byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
At 16 keV a native crystal has no usable anomalous signal, so XDS's CORRECT
was rerun with FRIEDEL'S_LAW=TRUE and the set no longer runs with -A.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The post-refinement probe on a canonical pass only measures, and it reads nothing the scaling
engine produces - the reflections' hkl, I, sigma and positions are final once the images are
integrated, and the engine writes back per-frame fields only. It ran after the first merge,
beside the engine's arrays, where its gathered observations and rocking events set the pass's
memory high-water mark. Run it before the engine is built instead; the pre-pass keeps its
original place (it consumes the smoothed mosaicity the merge writes back).
Measured on a 3600-frame long-axis rotation set: the canonical pass's high-water drops from
14.8 GB to under 12.8 GB (sampled RSS, -N 6); merged MTZ/HKL/CIF, P1 MTZ, per-image table and the
report are byte-identical, the probe's own result included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
--report-resolution now gets XDS's own d_min, the range its CORRECT.LP totals
cover, so the comparison is exact; shells past rugnux's cut print as not
merged. The derived reference d_min stays what rugnux's own cut is scored
against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ingest wrote every observation's 24-byte SortKey into an index-order array and then scattered that
array into its h buckets - two key arrays alive together at the top of the ingest. Count the
buckets per chunk of frames straight from the source reflections and scatter from them instead:
a frame's reflections are consecutive in the key numbering, so the chunks are consecutive index
ranges and each bucket lands in the same index order as before. The key array is byte-identical.
BuildInRangeObservations also hands back the remap temporaries (old runs, per-run maps, new_idx)
before the observation array is built rather than at the end of the function.
Measured on a 3600-frame long-axis rotation set (95.7 M partials in the geometry pre-pass): the
pre-pass ingest high-water drops 15.45 -> 14.1 GB (sampled RSS, -N 6); merged MTZ/HKL/CIF, the P1
cross-check MTZ and the per-image table are byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
--model and --report-resolution are both processing-inert (merged reflections
bit-identical with and without them), so the bare/model and bare/xds variant
pairs collapse into one command per arm:
open: rugnux -o p --no-export-unmerged --model <deposited coords> <input>
(no --model where there is no deposited model)
inhouse/private: rugnux -o p --no-export-unmerged [-A] --report-resolution
<reference d_min>,<XDS d_max> <input>
Every row is rugnux's own result (own cut vs the reference, as `bare` was), and
on the XDS arms also carries the REFRES_* table, scored against CORRECT.LP's
completeness, multiplicity, R_meas, CC1/2 and ISa, with ISa and R_meas ratio
plots and a like-for-like table; REFRES_SHELLS_PAST_LIMIT > 0 marks the row as
coverage (rugnux's cut coarser than the reference). Nothing is forced on the
processing any more. On the open arm the space group is scored on the data's
own determination (SOHNCKE_SPACE_GROUP where SPACE_GROUP_ENANTIOMORPH is
ASSUMED_FROM_MODEL), the model's label kept as sg_label; R_FREE/R_WORK/CC_MODEL
/MODEL_FIT are trend fields (placement-only, own free set); REFMAC stays opt-in.
results.json schema 3: one row per set, no variant/first_read; older runs are
read through their bare rows (open-arm model R-factors folded in), so compare
and report keep working against them. Every time is now a first read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Finalize reserved npredicted reflections and kept only the ones whose fit succeeded. On a dense
long-axis pattern more than half the predicted reflections lose their background ring, so each
retained per-image vector carried more dead capacity than data for the rest of the pass. Count the
kept ones first and reserve exactly that. No result changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The own statistics table's grid bottoms 0.1% below the finest kept reflection,
and the observation-level counts (N_obs, R_meas, the Bijvoet split behind SigAno
and CCanom) re-walk the fulls, which still hold every ingested group. Groups in
that band just below the auto cut were erased from the merge, so they were not
in N_uniq, but their observations still counted in N_obs and R_meas of the
finest shell. The own table now floors those counts at the cut by group d, the
rule the erase applies - the same floor the reference-range table already used
(now passed as a double, so the comparison is exactly the erase's).
Report-only: on an auto-cut cubic in-house set the written MTZ, HKL, P1 MTZ, unmerged
MTZ and image table are byte-identical; TOTAL_OBSERVATIONS 378754 -> 377933,
MULTIPLICITY 40.31 -> 40.22 (now equal to the reference-range table's), finest
shell N_obs 60873 -> 60052, R_meas 1136.9% -> 1129.2%, CCanom -0.6% -> -0.3%;
overall R_MEAS, CC1/2, SigAno unchanged at printed precision; the mmCIF's
pdbx_number_measured_all / pdbx_redundancy / last shell row follow. A
detector-limited lysozyme run (no auto cut) is byte-identical throughout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The INDEXING_AMBIGUITY warning was suppressed whenever --model was given on
rotation data, decided before the model was read. A model that then decided
nothing - not tested, rejected, or a probe winner that did not beat its own null
- still silenced it, so WARNING_COUNT, PATHOLOGY_FLAGS and possibly VERDICT
differed from a run without the model although the written reflections were
identical.
The warning is now issued exactly as without a model, and withdrawn (from the
warnings and from the statistics text) after model validation only where the
indexing probe decided the indexing: the model fits and the winner's R-free
margin beats the random-placement null (ModelValidationResult::indexing_decided,
set where the decision is taken). A reference MTZ, or the model reference on
serial stills with -C and -S, suppresses it up front as before.
Verified bare vs --model on three open-arm sets: merged MTZ data identical in
all three; warnings identical where the model decided nothing (identity probe
without null; no twin law); withdrawn where the probe decided (+33 sigma).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rot3d merge tests each observation against the weighted median of its
reflection, and only from three observations up: at two the median is one of
them. With -A each hand of a Friedel pair is its own reflection, so a P1
anomalous merge sits at about two observations a hand and almost nothing is
tested - on one P1 sweep the merge rejected 0 observations with -A against
tens of thousands without it. A single observation a hundred times its
partner's intensity (300 sigma apart, a 3.9 A reflection) then survived into
the merge, carried 94% of the weighted variance of its CC1/2 bin, took that
bin from 0.80 to 0.18, and the automatic cut with it: 3.46 A and UNUSABLE,
where the same data without -A cut at 1.40 A.
A hand with fewer than three observations now takes the weighted median of
both hands together, where the pair has three. I(+) and I(-) differ by the
anomalous signal, a few percent of I and far inside the six-sigma test, so
the mate supplies the observations the hand is missing. A hand with three of
its own keeps its own median exactly as before, and a Friedel-merged run is
unchanged. The medians are formed on the host, so the device merge reads the
same ones.
Measured with -A: the P1 sweep 3.46 A / UNUSABLE -> 1.49 A (1.63 A XDS
reference; 1.40 A without -A, unchanged), 0 -> 3,699 rejections. Two lysozyme
and one insulin set: same cut, space group, CC1/2 and ISa. On one of the
lysozyme sets 16 more observations are rejected, every shell's CCanom is
unchanged, and the overall CCanom goes 0.31 -> -0.02: the whole-range figure
was being carried by a handful of wild pairs, while the shells ranged -18% to
+12% and XDS's overall anomalous correlation is 1%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
Comparing a run with another program's table has meant running rugnux AT that
program's resolution range (--scaling-high-resolution), which is a different
run: the range moves the cut, the space-group decision and everything after
them, so the comparison buys itself a different answer. --report-resolution
<dmin>[,<dmax>] instead leaves the run alone and adds a second table to
section 3 of the report - the REFRES_* keys and a shell table - binned from the
same merged reflections over the range given, with the completeness
denominator enumerated over that range and the shells in equal steps of 1/d^2
so they read row for row against a CORRECT.LP at the same range. Report-only:
the merged files and every decision are byte-identical with and without it.
The table holds only what the run kept. Where the reference range is finer
than the run's own limit, the shells past it are printed as not merged (with
their possible count) rather than as zeros, REFRES_SHELLS_PAST_LIMIT counts
them so a consumer can tell "not merged" from a measured zero, REFRES_
COMPLETENESS counts their reflections as missing, and the other overall numbers
are over the shells the run reached; nothing is read from the observations the
run judged to carry no signal. REFRES_ISA is the error model refitted on the
reflections of the table alone, in XDS's convention (rotation only; the stills
model is fitted over the whole range already).
On the rotation path the statistics block of MergeAndStats becomes a lambda
over a shell grid, called once for the run's own grid and once for the
reference one; the reference call floors every observation-level count at the
cut by group d, the rule the erase applied. The stills MergeStats takes a
declared range, whose bounds are the grid's whether or not any reflection
reaches them. Both --mode mx and --mode scale report it, the viewer's command
line echoes it, and the docs describe the keys.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gemmi::transform_map_to_f_phi conjugates the forward FFT at the end (a
structure factor is the sum over exp(+2 pi i h.x), a forward FFT the sum over
exp(-2 pi i h.x)); MapToFPhi did not, so every model-path Fcalc and Fmask came
out with its phase negated. Amplitudes, and so the R-factors and the scale,
were unaffected; the phases of the maps and everything read off them (map
coefficients, density at atom centres) were not. Caught by
ModelValidation_MapToFPhiMatchesGemmi, which now passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The per-shell SigAno and CCanom of the rotation merge were accumulated on the
report grid, but the overall pair of numbers - the SIGANO / CC_ANOM keys and
the mmCIF's pdbx_absDiff_over_sigma_anomalous - were summed over every Bijvoet
pair the fulls hold, including the ones past the automatic resolution cut that
the written reflections do not contain. On a run the cut trims, the overall
line therefore described more data than the shells above it add up to. Both
overall numbers now count only the pairs that land in a shell, as the rest of
the table does; a run with a manual limit, whose ingest already ends at the
limit, is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
FFTW's planner (every fftwf_plan_* and fftwf_destroy_plan) shares global state
and is not thread-safe; executing a plan is. FFTIndexerCPU and BeamCenterFFTCPU
each guarded their planning with a lock of their own, TranslationalNCS and the
viewer's spectrum with none, and ModelFFT with a third - which does not stop
two of them planning at once. common/FFTWPlannerLock.h holds the one mutex they
all now take. No numerical change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
A long-running broker began cancelling every data collection with
Device decoding failed (CUDA (GPU) error (out of memory)), falling back to host decompression
CUDA (GPU) error (out of memory)
while nvidia-smi showed the cards less than a fifth full. Two defects, both in CUDAMemHelpers.h and
both from the pooled allocator (cudaMallocAsync) that came with rc.162.
The leak. cuda_allocation_stream() kept one stream per (thread, device) in a thread_local map of raw
cudaStream_t and never destroyed them. That was written against rugnux, where the worker threads live
as long as the process. The broker starts fresh std::async threads for every data collection - 16 or
64 of them - so every collection left that many streams behind. Measured: 0.56 MB of device memory
per leaked stream, linear to 4928 streams, never returned, with nothing on the host side growing.
The RAII wrapper (CudaStream) was there but not used at this site, and using it as-is - a stream
destroyed when its thread exits - would not have been safe: ShadowFinder builds its GPU accumulator
on a throw-away std::async thread and frees it from another thread long after, and that free is
ordered on the allocating thread's stream. So the streams are still never destroyed, but a thread
now only borrows one: CudaStream objects live in a process-wide per-device idle list, a thread takes
one on first use and hands it back when it exits. Their number is bounded by the threads that were
ever alive at once instead of by the threads ever started. The list itself is deliberately leaked, so
that nothing calls into CUDA during static destruction.
Replaying the broker's pattern against the real header, 60 collections of 64 threads:
before 3840 streams, 260 -> 2424 MB of device memory
after 64 streams, 260 -> 358 MB
The stale error. Every helper here throws a named message, yet the log carried the raw CUDA string,
so the failure came through a cuda_err() and not from an allocation. CudaDevicePtr falls back to
cudaMalloc when cudaMallocAsync fails, silently - but the failed call stays behind as the thread's
last error, and the cudaGetLastError() that follows the next kernel launch reports it. The buffers
were all allocated; the frame was lost anyway, once on the device-decode route (caught, hence the
warning) and once more on the host fallback (fatal). The pooled attempt failing, and a stream that
cannot be created, are both handled by falling back, so both now clear the error they leave.
What finite resource the production cards ran out of at under 4 GB used was not established - no
cap on the number of streams was found up to 4928 on the card this was measured on. The leak is the
only thing on this path that grows with uptime.
tests/CUDAMemHelpersTest.cpp: later threads end up on the same stream, concurrent threads on
different ones, a buffer is freed cleanly after its allocating thread has exited (and another has
borrowed its stream), and a pool that cannot serve a request leaves no error behind - the last by
capping a memory pool at 4 MB so that the pooled attempt fails and the fallback succeeds.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The rigid-body target is an Fcalc + solvent mask + scale re-fit per
evaluation, and its forward-difference Jacobian made six of them one after the
other after the central one - 7 of the ~8 evaluations per LM iteration.
The six shifted evaluations now run in parallel, column j on its own Evaluator
over its own copy of the model (an evaluation moves every atom, so two cannot
share one), set to the same zone and given the central evaluation's bulk-solvent
pair - the pair the serial loop's shifted evaluations used, since the zone's
solvent is fitted on the zone's first evaluation and every Evaluate starts with
the central one. Each column's arithmetic is the serial one and the Jacobian is
assembled in column order, so the refinement is bit-identical; the evaluation
count is kept as the serial loop kept it (up to and including a failing column).
The per-zone solvent fit (FitModelScale) gets the thread count as well.
Inside a null replicate (a pool worker) the columns run inline, as before.
To check: RIGID_BODY_* and MODEL_* keys and md5 of maps/.mtz/_model.cif identical
with and without this commit on the audit set; rigid-body seconds on a large
model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The crystal does have a single P2 lattice, holding ~6% of each frame's spots
on top of ice and protein powder; XDS indexes it with the ice rings excluded.
The reference is now that P2 run to 2.0 A instead of a failed P1 run, and the
set is renamed from aggregate to sparse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A rejected rigid-body placement put the atoms back and then recomputed the full
Fcalc and solvent mask at the restored positions. Nothing reads them: the real
model's ModelState is only fitted again after its own compute_model_factors
(the re-placement on the merged indexing, the null-failure fallback), and a
null replicate's state is discarded. The atoms are still put back; the
structure factors are left as they are.
Where the lattice has twin laws, the indexing probe fits the identity labelling
first, and place_and_fit then made the identical fit again - same
ModelState (nothing moves the model between the two), same reflections. The
probe now hands that Fit over, for the real model when the identity wins and
always for a null replicate (placed against the merged indexing). fit_model is
deterministic, so the reused Fit is the one the second call produced.
Bit-identical by construction: no value that is read changes. To check: MODEL_*
keys and md5 of maps/.mtz/_model.cif identical with and without this commit on
the audit set, including sets with twin laws and a null.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
XDS was rerun with the pin shadow masked and the frames where the crystal
leaves the beam excluded; the reference is now P43212 to 1.65 A instead of a
failed P1 run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FitModelScale fits ~113 (k_sol, b_sol) grid points, each a full
Levenberg-Marquardt over every working reflection, and it is the largest single
cost of --model (every fit_model call: the first fit, the twin-law probe, the
fit after the rigid body, each null replicate). It ran on one thread.
Each point starts from fit_isotropic_b_approximately(), which sets k_overall
and b_star from the data and the point's solvent pair alone, so no point
depends on the one before it. The points of each pass (coarse, then the
refinement around the coarse winner) now run in contiguous chunks, each on its
own copy of the Scaling, and the winner is read off afterwards in grid order
with the serial rule (lowest finite R, first on a tie). Each point's arithmetic
is unchanged, so the result is the serial loop's bit for bit. Where
fit_isotropic_b_approximately() has five or fewer reflections to fit on it
returns without setting anything and points would chain, so there the grid is
still walked serially on the caller's Scaling.
Inside a null replicate (a pool worker) the chunks run inline, as before.
To check: MODEL_* keys and md5 of the maps/.mtz/_model.cif identical with and
without this commit on the audit set; model-phase time on a fit-dominated set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
The battery is bound by reading images from disk, so instead of one battery run per setting each
set now runs in all its variants in a row while its images are still in the page cache:
open arm bare, then model (rugnux --model <deposited coordinates>)
XDS arms bare, then xds (XDS's resolution range forced, -A where XDS was anomalous)
bare (plain rugnux <input>) runs first on every arm, so its time always carries the set's disk
read whichever variants are selected; each row records first_read and the report's timing table
says which variant's times are cold. --variants runs a subset; --unforced is gone (--variants
bare). Rows, work dirs (work/<arm>/<set>/<variant>/), compare and the report are per variant;
compare pairs (arm, set, variant) on the variants both runs have on an arm, and a schema-1 run is
read as one variant. The model variant takes the coordinates and published R-free from
model_check's RCSB cache (site key pdb_cache) and records rugnux's R-free/R-work and the ratio;
the REFMAC check is now opt-in (--model-check) and its keys moved to refmac_*. results_schema 2.
model_sweep.py is retired: the model variant replaces it (its --spot/--scaling-low-resolution 50
were rugnux's defaults, so the command is the same).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every map -> structure-factor transform on the model path - Fcalc from the
model density and Fmask from the bulk-solvent mask, in the fit, the rigid-body
evaluations (two per evaluation, ~120 evaluations per placement), the frame
probe and the stills model reference - went through gemmi's vendored pocketfft,
single-threaded. They now go through MapToFPhi (rugnux/ModelFFT.{h,cpp}): an
FFTW r2c (fftw3f, already fetched and linked) planned with the guru interface on
the same layout gemmi uses (u fastest, w halved), scaled by V/N, into the same
FPhiGrid, so gemmi's prepare_asu_data() extracts the reflections exactly as
before. gemmi's vendored code is not modified. The F -> map transforms of the
output maps are left on gemmi.
Plans are FFTW_ESTIMATE only (MEASURE times candidates and could choose
differently between runs), made under a mutex - FFTW's planner is not
thread-safe, and the null replicates and the parallel Jacobian call this
concurrently - cached per grid size for the life of the process, and executed
with fftwf_execute_dft_r2c on fftwf_malloc buffers (new-array execution needs
the planning arrays' alignment, which a std::vector does not promise).
NOT bit-identical with pocketfft: the two libraries order their arithmetic
differently, so structure factors move in the last float bits and everything
downstream (scale fit, rigid body, R-factors, maps) can move in the last
printed digit. Verify by tolerance, separately from the parallelisation commits:
MODEL_* keys equal to printed precision and the same MODEL_FIT / hand /
indexing decisions on the audit set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
On the XDS-referenced arms rugnux merges Friedel mates separately whenever
XDS ran with FRIEDEL'S_LAW=FALSE, in forced and unforced mode alike, so the
merging statistics compare like with like.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rotation geometry walk kept re-integrating while the post-refinement at
the adopted geometry still committed a move, up to four rounds. It never ended
on its own: every walking run hit the cap, and with the cap lifted to twelve
every one hit twelve. Each round's fit predicted a held-out gain the
re-integration at its geometry then did not realise - the realised residual
stayed flat from the first round, or passed a minimum and rose again while the
distance crept on along the distance/cell-scale degeneracy - so the cap decided
where on the drift the run stopped.
A round is now kept only when the held-out residual measured by the pass
re-integrated at its geometry falls below the best round's by more than the
standard error of the difference. The first round that does not ends the walk,
and the run goes back to the best round (one more pass, since the canonical
files were written by the last). The walk also stops when re-indexing pushes
the fitted cell back on the same length in the same direction on two rounds
running: the fit keeps asking for a cell nothing ratifies. The cap is 8; a
walk still paying at the cap keeps its last (= best) round, sets
geometry_not_converged, and the report flags GEOMETRY_NOT_CONVERGED. Either
way the kept round is judged against the header-geometry pass by
RefinedPassIsWorse, as before. PASS_DECISION lists distance and realised
residual per round and why the walk ended.
Runs whose post-refinement does not commit a move of more than one step do not
walk and are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each post-refinement now records, beside what it commits, what the integration
it was handed realises at the geometry it started from: the held-out residual
at nominal (the same value the log prints on the left of "held-out ... ->"),
its standard error over the held-out residual values, and the cell the pass's
own indexing refined. Nothing here changes a decision.
Two helpers read them: HeldOutResidualFell (a pass at a new geometry realises a
lower residual than another by more than the standard error of the difference)
and ReindexPushesCellBack (re-indexing at a committed geometry returns, on the
length the fit moved most, a cell on the side the fit moved away from). The
geometry walk in RunAllPasses uses both in the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DropCollapsedFullScales zeroes the corr of fulls whose frame scale collapsed on
the host copy of the fulls; on the resident-fulls (GPU) path that zero reached
the device only with the corrected corr pushed after the surfaces. The merge
that measures CC1/2 before the correction surfaces runs in between, so on the
GPU it still merged the dropped frames' fulls while the host path excluded
them - the two builds measured different observations.
The corr is now pushed to the device right after the drop, and the later push
covers only the surfaces and the frame rejection.
This changes the input of the two-pass quality guard (cc_half_before_corrections)
on GPU runs where a full scale collapsed; the final merge is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A weak crystal inside a powder of its own microcrystals plus ice puts only
~5% of each frame's spots on its lattice. The pooled first-pass test accepted
that lattice (5.5% of validation spots against 0.6% at a wrong spindle angle),
but then:
- every frame failed the 20% per-frame floor (LATTICE_MIN_INDEXED_FRACTION),
so nothing was integrated and the merge was skipped (indexing rate 0);
- the background-measured beam centre, ~3 px from the file's and correct,
was not adopted because the arbiter counts validation frames, which are 0/60
at every centre.
Changes:
- When fewer than 1/6 of the validation frames clear the per-frame floor AND
the sweep's pooled on-lattice fraction is itself below that floor, integrate
every frame from the sweep's lattice (as XDS/DIALS do) and leave frame
selection to scaling. Sweeps sparse only in spots per frame keep the floor.
- Merge when a rotation lattice was found even if no frame "indexed" on its own.
- When neither centre indexes a validation frame, adopt the measured centre if
its pooled excess over chance beats the file's by 3.29 sigma.
On the case above: P2 at the XDS cell, merged to 1.77 A, CC1/2 0.95, ISa 3.4
(XDS with the same lattice: ISa 3.3-5.5). Three healthy/partially-indexing
rotation sets are bit-identical; a two-wavelength CBF set that currently
merges 42 frames on a wrong cell moves (beam centre adopted, 755 frames).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The (a, b) error-model fit already refuses to report ISa when its top intensity
bin has no leverage on b. It still printed 1/b when b had leverage but came out
at zero within its own error - the strongest bins scatter no more than
counting statistics say (the fitted b^2 is negative and clamps to 0), or there
are too few samples for the bin medians to mean anything. ISa then reports the
noise in b as an I/sigma: on one low-resolution sweep consecutive merges of the
same data gave ISa 0, 28, 46, 113 and 130.
The standard error of b^2 is taken from the 16 bins' own scatter about the
fitted line, and when b^2 is less than two standard errors above zero the
merge result carries isa_resolved = false. Only what is printed follows it:
the report's ISA key reads "undetermined", the summary line says so, the
asymptote is not printed and the mmCIF carries "?". The fitted ISa itself is
unchanged and is still what the space-group search's present-reflection cut
and the refused-point-group arbitration read, so no decision moves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The resolution cutoff, the shell table and the overall CC1/2 counted every
unique reflection equally. The merge itself is inverse-variance weighted, so
an observation from a frame the crystal barely diffracted on enters it at
1/G^2 of a good one - honestly, with its sigma - but a reflection measured
only on such frames is scaled-up noise that then counts as much as a
well-measured pair in every Pearson CC1/2 read off the merge. On a sweep
where half the frames are weak the curve collapses at every resolution: the
cut lands at 3.7 A on a P1 crystal whose good frames reach 1.5 A, and the
UNUSABLE verdict fires (CC1/2 0.40 beside I/sigma 10).
Each merged reflection now carries cc_weight: the precision its half-sets
would have had with every observation at the run's typical frame scale, over
the precision they have. G_ref = sum G^3 / sum G^2 over the usable
observations is the precision-weighted typical scale, which the dead frames
cannot drag down however many there are; the factor per observation is
max(1, (G_ref/G)^2), with G the frame's total scale (partial scale, flux and
the fulls' own G) taken before the correction surfaces and before collapsed
frames are dropped, so nothing intensity- or resolution-dependent enters it.
On a sweep without a weak stretch every weight is 1 and the CC1/2 is the
plain Pearson it was. The cutoff fit, the shell table and the overall CC1/2
(and so the UNUSABLE verdict and the report's shell checks) all read the same
weighted statistic.
The two extra per-group sums are accumulated on both merge paths, the host
loop and MergeAccumKernel, from one per-frame factor array; a host recompute
of the device sums agrees to 1e-15 relative on every merge after the
corrected corr is uploaded, and a host-merge run gives the same cut, space
group, CC1/2 and ISa on five sets.
Weighting by the half-set error variance alone (1/(v0+v1)) is not this: v
grows with the intensity, so it weights the weak end of the intensity
distribution and biases homogeneous data coarser.
Measured (written resolution, together with the weighted outlier median):
a P1 sweep with a long weak stretch 3.73 -> 1.40 A against a 1.63 A XDS
reference, UNUSABLE withdrawn (overall CC1/2 0.40 -> 0.97); a second 3.61 ->
3.00 A; one with most of the sweep out of beam 7.35 -> 5.20 A. Lysozyme,
thaumatin and two insulin sets unchanged to 0.01 A (one lysozyme sweep with a
weak wedge 1.13 -> 1.16 A, from the median), same space groups throughout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-reflection median the merge rejects outliers against was an
unweighted median of the scaled intensities. On a sweep where the crystal
barely diffracts over a long stretch, observations from those frames are
scaled up by 1/G together with their sigmas, and in P1 at multiplicity ~3
two of them outvote one well-measured observation: the median becomes
noise and the well-measured observation is rejected against its own small
sigma. On one P1 sweep with ~44% such frames this rejected 52,574
observations (2,131 once those frames are dropped) and took the
correlation of the merged intensities with an external reference from
0.92 to 0.68 at low resolution. The median is now weighted by
1/(sigma*corr)^2, which leaves it unchanged where the observations are
comparably precise: rejections 3,932, correlation 0.92.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FITTED_RESOLUTION was withheld whenever the fit was finer than the d_min of
the last shell whose CC1/2 still reached the target. A shell's CC1/2 is an
average over the shell, so on every ordinary fall-off the crossing lies
inside the first shell whose average is below the target - 0.01-0.1 A finer
than the number the check allowed - and the report said "no resolution could
be fitted" on data whose fall-off is textbook: 68 of 100 battery datasets,
with the fit 0.01-0.02 A inside the first sub-target shell. The comparison is
now against that shell's d_min, and the RESOLUTION_FIT flag fires only where
the fit is finer than the shell the crossing can lie in.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Under a twin law T a reflection whose twin mate is itself (up to the true
group and Friedel) is untouched at every twin fraction. For each index-2
subgroup H of the adopted point group, those are the reflections centric in
the group and acentric in H: centric if the operators the group adds over H
are real, acentric if they are a twin law or a pseudo-symmetry - the one
intensity statistic that still separates the two at fraction 0.5, where
every operator statistic reads "real".
Read on the P1 cross-check merge: epsilon-1 reflections in shells with
<I/sigma> >= 5 (noise inflates every class towards centric), each class
normalised against its own mean in bins of ~100 reflections and within the
two phase classes of a detected pseudo-translation, Wilson outliers above
E^2 = 20 dropped. Reported per subgroup: the added operators, n,
<|E^2-1|> +- SE read absolutely against 0.968 / 0.736, and the centric-over-
acentric Wilson log-likelihood in nats with both densities convolved with
each reflection's error (flooring E^2 at its sigma instead read a genuine
1.2 A lysozyme zone as acentric), with the acentric control beside it.
Log, report prose and TWIN_ZONE_n keys. Nothing reads it back; no decision
changes.
On the reference sets: 6toc P4222, all three subgroups centric (1.04-1.07,
+253 to +534 nats); 6iu9 P3121 over P31 acentric (0.789, -294 nats); 5j23
R32 over R3 acentric (0.805, -304 nats, tNCS-class normalised); lysozyme
P41212 centric (0.90, +476 to +1297); a P21 myoglobin centric (0.930, +135).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
