When neither the file's nor the background-measured beam centre indexes a majority of the
validation frames, the first pass chose between them on the pooled spots each centre's lattice
puts on itself over chance, and only then asked the long-axis rescue - at the chosen centre alone.
The lattices compared there are, by construction, not the crystal's, and the comparison is close
to a coin toss. With the decisions before pass 2 taken from the first part of the sweep (part B,
5b9144410), 8v4o (P6122, c = 545 A) tipped from the file's centre to the measured one, 3.6 px out
across the spindle, on 16.7% against 15.5%; the rescue there could not see the 545 A axis, and the
run shipped a P1 sub-cell (primitive volume 0.35x). At the file's centre the same rescue on the
same early frames indexes 60/60 (checked with --beam-center-check=off on the rc175 binary).
The rescue is now a lambda, asked at both centres in that branch so each centre is judged on the
best lattice it gives, and asked as before after the beam-centre check where that branch did not
already ask it. The early-frame principle of part B is unchanged.
8v4o: P1 2.62 A, CC1/2 0.920, R_model 0.553 -> P6122 2.12 A (pass), the file's centre kept.
p.mtz md5-identical to rc175 7c9a6c7d1 (GPU) on 5nw5, 5ky6, 5reo, 7atg, 9qw8, 9rci, cuhf2,
cytidine, insu_I_x06da_low_isa, kdp_x10sa_20keV, lcystine_x10sa_20keV/25keV, lyso_x06da_half_image,
myob_x06da_powder_2, myob_x06da_sparse, nothing_1, nothing_2 and the two private sets that reach
the same branch. 9rci (doubled-c alternative under tNCS) is a different mechanism: rc175 with
--prepass-fraction 1 also returns the doubled cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
IDXREF writes the h,k,l it assigns back into SPOT.XDS, and its choice of basis is not deterministic
(sign flips between runs), so an md5 of the whole file could differ between layouts holding the same
pixels and finding the same spots. Compare x, y, frame and intensity only - what a read plugin
actually determines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With a DECTRIS detector the broker starts the writers - and so learned the collection's start
message - only when the detector's own start message arrived, after /start had returned. A reader
started right after /start found no collection (404) and had to wait for, and race with, the
detector.
- /start registers the collection in LiveCollection (Expect) as it is accepted, so GET /live/events
is accepted from then on. The image pusher fills in the start message when it starts the writers
(Begin) - at once on the FPGA path, when the detector's start message arrives on the DECTRIS path.
- The event stream's first event is now `start`, carrying that start message (CBOR, base64) with
run number, file prefix, image count and images per file; it replaces `collection`. Until it is
known the stream only keeps alive. file and end follow as before.
- However the measurement thread ends, it ends a collection the pusher did not end (EndIfOpen), so a
cancel or a failure before the detector streamed reaches the reader as `end` with an error. The
wait is event-driven: it ends with the start message or with the measurement.
- GET /live/start.cbor is removed: the stream carries the start message, one route less.
- rugnux takes the start message from the stream (BrokerFeed::WaitForStart), so it can be started
right after /start; files reported before it attached the reader are replayed to it.
Tests: LiveCollection_ExpectedAtStart; BrokerFeed_StartFromEventStream (an event stream whose start
message comes late, then a file and the end).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The beam-centre check runs a second first pass at the centre measured from
the background, and reads the first pass's answer only once that pass is
over. With two GPUs or more, its spot finding (on the second card), scheme
feeding and FFT indexing now start beside the file centre's pass, on a
copy of the experiment at the measured centre with workers, a mapping and a
spot list of their own (SpotFindingWorkers). The check takes them only
where the experiment (rotation angles included), SpotFindingKey at the
measured centre and the spot budget are still what the run has there, and
scores the schemes on the run's own indexer as before; otherwise it runs
that pass itself as before. The spots at the two centres are not shared:
the spot list depends on the centre (ring background, resolution, budget).
With one GPU, and in the CPU build, nothing changes: on one card the two
passes share the card and the reader, and side by side they saved nothing.
FindFirstPassSpots is the first pass's per-frame spot finding, now shared;
ExperimentKey and SpotFindingKey take an experiment other than the run's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The ladder's rung says which spots let the FFT find the lattice; until now it
also fixed how deep the whole run reads each frame (the spot-finding floor),
so which floor's FFT happened to succeed decided the run's depth. On a hard
pattern that is close to a coin toss: with the decisions taken from the first
part of the sweep, a sub-angstrom crystal (7atg) tied at 60/60 on a 1.13 A
and a 1.60 A floor, kept 1.60 A for the run and lost half its validation
spots. Now the winning lattice is itself scored at each deeper floor, deepest
first, and the run reads to the deepest at which it indexes as many
validation frames as at its own. A contaminant past a floor costs the lattice
frames there and keeps the floor.
Measured (GPU unless noted), against c240690a1:
7atg ISa 21.0 -> 23.1, R_meas 5.4 -> 5.1 %, R-free(model) 0.220 -> 0.160
(whole-sweep decisions, af3ea9: 22.9, 5.2 %, 0.178)
myob_x06da_split reads to 1.92 A instead of 3.00 A: ISa 2.2 -> 2.7, R_meas 87 -> 78 %,
CC1/2 81 -> 70 %, R-free 0.382 -> 0.402 (a contaminated split crystal)
9qw8, 9zmu, 8v4o, myob_x06da_powder_1/2 (CPU): unchanged; 8v2t: p.mtz bit-identical.
Result-changing only where the ladder adopts a rung.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
For demonstrating and testing a collection followed while it is written:
- jfjoch_stream2_replay replays a stored dataset as a DECTRIS detector's stream2 (ZeroMQ PUSH on the
stream2 port): start message, every image at a given rate, end message.
- fake_simplon.py stands in for the detector's SIMPLON control interface: answers the configuration
and status requests the broker makes, accepts every command, reads back idle. It needs port 80.
Not installed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
rugnux <path>/<prefix>_master.h5 http://<broker>:<port>/ - when the master file is not on disk yet -
processes the broker's current collection as it is written:
- the start message comes from GET /live/start.cbor; the given path has to end in the collection's
master file name, and the data files are read next to it;
- JFJochStartMessageReader builds the dataset from the start message (detector, geometry,
goniometer, mask) and reads the images straight from the data files; a read of an image whose file
the broker has not reported waits for it;
- BrokerFeed reads GET /live/events in a thread of its own and hands the reader each closed file. On
end, the run finishes with the images the collection delivered (a read past them returns no image).
A broken stream, another collection, or a refused follower makes every waiting read throw
LiveCollectionLost, which ends the run like a fatal resource error;
- the token is taken from JUNGFRAUJOCH_HTTP_TOKEN (the variable the viewer uses) and sent only in
the Authorization header; without it, and with no master file on disk, rugnux stops at once;
- rotation data and --mode mx only; plain http only (TLS is not linked into rugnux).
With the master file on disk, the URL is ignored and the run is offline as before. The streamed run
does not read the master file and is not compared with it: its result can differ from an offline run
of the same files in the last bits of the metadata.
Tests ([Live]): LiveCollection, the server-sent-event parser, the FileWriter callback, the TCP
pusher's closed-file reports with two real writers, and JFJochStartMessageReader against the
master the writer produced.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
So that rugnux can process a collection while it is written, the broker reports each data file once
a writer has closed it and renamed it into place:
- Writer: FileWriter takes a callback, called for every closed data file. StreamWriter uses it on TCP
to send an acknowledgement with ack_for = FILE_CLOSED (new TCPFrameType 10): file number and the
images the file holds. TCP protocol version 5. Files closed at END are reported before the END
acknowledgement.
- LiveCollection (image_pusher): the start message the writer of the master file received, the
closed files, the end. Fed by the TCP pusher (FILE_CLOSED acknowledgements; end once every writer
acknowledged END) and by the in-process HDF5 pusher (the FileWriter callback). The ZeroMQ pusher
has no back channel and offers none.
- Two routes, protected by the dataset's bearer tokens like the other dataset routes:
- GET /live/start.cbor: that start message;
- GET /live/events: server-sent events (collection, file, end, superseded) with the history first
and a keepalive every 10 s. One follower at a time; a second gets 409.
Documented in jfjoch_api.yaml; they add no schema, so the generated C++ model is unchanged.
- The HTTP server's thread pool is set explicitly (16, up to 64): an event stream holds a thread for
the whole collection.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The final master marks a finished collection for anyone watching the directory (rsync, autoprocessing,
a reader following the collection), so it must not appear before a data file it links to.
- FileWriter: at END every data file is closed (renamed into place) before the master is finalized
and renamed. A data file that fails to close does not cost the master; the failure is reported
after it. For NXmxIntegrated, file 0 is the master's own image dataset, so there a failure still
stops the master.
- With several writers, the master was written by the writer of the first socket as soon as it got
END, while the others could still be closing their data files. END now goes to every other writer
first:
- TCP: the connections are taken in reverse order, and each END waits for its acknowledgement,
which a writer sends only after closing its files;
- ZeroMQ, with a writer notification socket: END to every other socket, then their notifications
(the same ones Finalize collected before, now collected here), then END to the first socket.
Without a notification socket there is nothing to wait on and the order is all it changes.
Tests: TCPImageCommTest_MasterWriterGetsEndLast (three mock writers, the master's END after the
other two acknowledged), StreamWriterTest_ZMQ_MasterAfterAllDataFiles (two real writers, the second
started only after the collection was sent; the master's ctime is after every data file's). Both
fail without this change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The pre-scan sample (beam-stop shadow, defective pixels, spot width, the background beam centre)
and the first pass's frames (the spread and wedge schemes, the validation frames, the prefetch) now
come from [start, PrepassEnd) - the part of the sweep the geometry pre-pass integrates - instead of
the whole sweep. Rugnux::PrepassEnd is the one place that range is defined, for the pre-pass and for
these decisions alike, so any change to the pre-pass range (such as a minimum rotation) moves both.
Stills keep the whole range.
With rugnux following a collection while it is written, nothing before pass 2 then waits for the
second part of the sweep: the pre-scan, both first passes and the pre-pass run while it is
collected, and pass 2 starts reading as soon as they are done. An offline run takes the same
decisions from the same frames, so the streamed result still equals the offline one by md5 (in-house
JUNGFRAU 4M thaumatin replayed at one data file per second, also at -N 8 and with the replay paused
and resumed): 2.4 s from the last data file to the result, against 8.0 s for the offline run.
Result-changing. p.mtz md5 at af3ea9cca -> here (with the 120 deg minimum of the previous commit),
GPU: myob de2ab820 -> 9bdd6916, cytc 3e5d2f91 -> 62e744a1, thau bc5e3de3 -> 3ec88c34,
kdp cf83ee5d -> cdc2c068, 8a1a 50ab659e -> 309ea9a8, 8qaw d3287922 -> c2849d52; CPU: myob 3f9d12d8 ->
18c7ac2f, cytc d30c9969 -> 40e17d97, thau 4cdbdd94 -> f14e52da, kdp f1d801f3 -> eb930714. Sweeps of
120 deg or less are unchanged by this commit (their early range is the whole sweep). Needs the
battery before it is merged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The two-pass geometry pre-pass integrates the first --prepass-fraction
(0.5) of the sweep, but never less than 120 deg of rotation (header
oscillation), so a sweep up to 120 deg is pre-passed whole and one up to
240 deg gets 120 deg. Wider sweeps are unchanged. The range is computed
in one place, Rugnux::PrepassEnd.
Evidence (fraction sweep with the rc175-p1-baseline binary, 10 sweeps
of 360 deg and 10 of 180 deg; pre-pass distance vs the full-sweep fit):
median error 0.081 % at 90 deg, 0.042 % at 120, 0.042 % at 150 and
0.033 % at 180 on the 180-deg sweeps; 0.032 / 0.021 / 0.013 % at
120 / 150 / 180 deg on the 360-deg sweeps. 120 deg on a wide sweep
matches what the whole-sweep pre-pass gave a 180-deg sweep. Below
~60 deg the pre-pass misses the distance outright (6rym: held at the
header, 3.4 % off; 6vww and 6fwc: distance not freed). Merged quality
moves by under 1 % at any angle >= 90 deg.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The goniometer rotation-scale walk ran between the geometry pre-pass and the
refined (output) pass, on the critical path: two indexing probes, at the stored
angles and at the pass-1 fit, each with its own first-pass spot finding.
Adopting a scale other than the stored one is rare, so the refined passes now
start at once at the stored angles, on a copy of the run, while the walk runs on
the run itself. The walk starts once the output pass has indexed, so its probe at
the stored angles finds that first pass already done (spot store, rotation-indexer
memo), and its probe at the fit takes the frames' spots from the store as well:
the spot store's key no longer carries the goniometer's angles, and the spindle
angle each stored spot carries is stamped again for the angles in hand, exactly
as DiffractionSpot::Export stamps it (verify_first_pass_memo still recomputes and
compares). The output pass waits for the walk's answer only before it writes
(the superseded flag); where the walk adopts a scale, or leaves the spot-finding
settings changed behind it, the copy's passes are thrown away and the refined
passes run again on the run itself, which is then exactly where the sequential
walk left it. A refit integrates and only follows a move off the stored angles,
so it first calls the copy off (a cancel flag of its own) and waits for it to
end: two integrations never share the card.
WHEN, not WHAT: the adopted scale and the surviving output pass's inputs are
those of the sequential flow. The console shows the check as a step only where a
scale is adopted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The R_meas accumulators per group and the device's per-group sums were eight
group-length locals plus a 56-byte record per group, value-initialised on one
thread for every merge that reports statistics. They are now members, reset in
parallel, and the device branch's pass that assigns each group its shell and
copies its sums runs on all threads; the per-shell observation counts are
integers, gathered per block. The shells' R_meas sums are still added in group
order, as before.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Which groups are written is decided first (KeptIndices, in group order), then
each written group's MergedReflection is formed on its own, on all threads,
into a result sized once - where the serial loop pushed back up to millions of
records, growing a vector that each merge allocated afresh. The same records in
the same order.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The rejected flags and the observation keys and halves (per full), the merged
intensities (per group) and the Wilson observation records were locals: on a
large cell every one of the run's merges allocated, page-faulted and zeroed
them again on one thread - hundreds of megabytes per merge on 8qaw. They are
now members, only resized, and reset with ParallelChunks where a pass needs
them reset. The error-model samples are no longer cleared before the device
download overwrites them (the host path still clears before it appends).
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Plugin 1.1.0 (2629f3821) passes a saturated pixel of a signed image (INTx_MAX) through with its
value so XDS flags it against OVERLOAD; only the error marker (INTx_MIN signed, UINTx_MAX unsigned)
becomes -1. The legacy test still expected INT16_MAX -> -1. It now checks the saturated value
explicitly and also that module-edge (bit 30) and chip-edge (bit 31) mask pixels read as -2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The fused adaptive spot finder's azimuthal profile (and the standalone AzIntEngineGPU) summed the
corrected pixel values - pixel times a float correction - with float atomicAdd, in shared memory
and then across blocks. Float addition is not associative, so the per-ring sums depended on the
order the threads and blocks happened to arrive in, and on the grid size chosen for the GPU. The
profile, and the per-image background estimate read from it, moved in their last bits between two
runs of the same command: on a thaumatin JUNGFRAU 4M sweep one bkg value in _plot.txt differed in
the 4th digit between runs. The raw ring sums that drive detection were already exact integers;
only the reported profile was affected.
Each corrected value (and its square) is now quantised to 2^-12 on its own and summed as a signed
64-bit integer, the convention the raw ring sums and the Bragg-integration profile already use.
Integer addition is associative, so the sums are the same whatever the order or grid.
The repeat test of the fused finder now also requires the profile and its std to repeat bit for
bit (it failed on the old kernel at the first repeat); the AzIntEngineGPU test re-runs 20 times.
Fused finder benchmark: 0.313 -> 0.324 ms/frame.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With the fulls resident (the GPU combine ran), ApplyCellSurface no longer builds
its terms on the host and uploads them: SurfaceBuildTerms makes them from the
resident fulls, this surface's cell per full and the current corr (the only two
uploads) - which fulls give a term, the terms in fulls order, the group CSR over
them and each subset's per-(block, cell) segments - and SurfaceShellWeights the
resolution gauge's inputs: the shell edges (order statistics of 1/d^2 read off a
sort), each group's shell and each (cell, shell) bin's weight. Every order the
host made is a stable counting sort, which a stable radix sort of the indices
reproduces; the per-bin weights are summed one bin per thread in fulls order
with the host's rounding. The existing exact surface kernels read them as
before. Without resident fulls (the observation dump) the host path runs.
p.mtz md5 and the surfaces' log lines unchanged on myob, cytc, thau, kdp, 8a1a
and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With two GPUs or more the two-pass quality guard's merge of the pre-pass
frames gets a scaling engine of its own on device 1 (RotationScaleMerge::
SetGpuDevice), and the main engine's device ingest and first merge no
longer wait for it to hand the card back. The first merge waits only for
the guard's ingest, the last time the guard reads the outcomes: that merge
writes per-frame fields (mosaicity among them) back into them.
Neither merge reads what the other writes, and a merge gives the same bits
on any card of one model, so only when the guard runs changes. With one
GPU, and in the CPU build, the order is the one it was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
std::thread::hardware_concurrency() counts every CPU of the node, so a job
given a CPU set (taskset, a cpuset, a Slurm allocation) or a container CPU
quota started one thread per node CPU: the default -N, the shared worker
pool and every "0 = all threads" default. AvailableCpus() (ThreadAffinity)
counts the start affinity mask, capped by the cgroup CPU quota (v2 cpu.max,
v1 cfs_quota/period, the process's own cgroup and then the mounted root),
read once. Outside Linux it is hardware_concurrency(), so the viewer tree
stays portable. It replaces every hardware_concurrency() call outside the
tests and the vendored pocketfft.
Only how many threads run changes; the passes split their work by n alone,
so the results do not depend on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Plugin:
- Module-edge (bit 30) and chip-edge (bit 31) pixels are marked -2 (untrusted). Chip-edge pixels
are split from a larger pixel with integer division and read low at low counts; feeding them to
XDS as real pixels was what made the three plugins disagree in CI.
- A saturated pixel in a signed image (INTx_MAX) is passed through with its value, so XDS counts
it as an overload; only the error marker (INTx_MIN / UINTx_MAX) becomes -1.
- Release version 1.1.0 set in CMake (library name and start message). VERSION_* in plugin.h stay
0: XDS refuses a plugin whose info-array major version is not 0.
- Built and packaged on Linux only; the untested macOS archive is dropped.
- The XDS linking permission is removed from LICENSE; the plugin is GPL-3.0.
CI:
- XDS.INP beam centre corrected to 1091/1137 (ORGY was 7.7 px off, which put IDXREF on a knife
edge where one plugin's run fell into a false minimum).
- Each XDS case must reach CORRECT, IDXREF spot SD must be < 2 px, and all file layouts read by
the same plugin must give identical SPOT.XDS.
Docs: recommend Durin (PSI or Global Phasing build); mask-bit table rewritten from measurement.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
FitErrorModelGPU puts the samples in the host's rank order with four stable
radix passes, so the equal-count bins hold the same samples; computes every
normalised deviation with the host's rounding, so every bin median is the
host's to the bit; and hands the per-bin terms to the host's own update step,
now ErrorModelUpdate (a pure extraction from FitErrorModel). The one thing
that differs is the order each bin's two sums are added in: a fixed tree on the
device, the order nth_element happened to leave the bin in on the host. a and
b^2 can therefore differ from the host fit in their last bits - measured on two
synthetic pools: relative difference 2e-16 and 3e-13 in a, 5e-16 in b^2. The
device fit is deterministic (same pool, same bits).
p.mtz md5 nevertheless unchanged on myob, cytc, 8a1a and 8qaw (GPU build): the
difference does not survive the float rounding of the merged sigmas there. A
knife-edge decision elsewhere could still see it, which is why this is kept as
its own commit.
The host fit took 0.4-0.6 s per fit on the large merges, about half of it the
initial equal-count split, with only 16-way parallelism in the iterations.
Device peak about 72 bytes per sample (the rank sort; each stage's buffers are
freed when it is done); the free memory is checked first and a device without
it stops with a message naming the CPU build.
Kept last on the branch so that it can be dropped on its own.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Every full's observation key and its half - the parity of the key's rank among
its group's usable fulls - are integer work, so the device gives the host's
halves exactly: one thread per full for the keys, one per group for the ranks.
The host receives both (the anomalous statistics rank the same keys again within
each Bijvoet cell) instead of walking every full's record for them.
p.mtz md5 and p.cif unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The strong-reflection asymptote is read from per-group sums of I*corr, its
square and the counting variance at the group mean - a walk over every full's
record on the host. In the GPU build a kernel takes them, one thread per group
in index order with the host's rounding (CountingVarianceExact); the host keeps
the strong-group selection and the median.
p.mtz md5, p.cif and the error-model lines (which print the asymptote)
unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The merges of the largest sets run with 12-13.6 GB of a 16 GB device already
taken by the resident observations, so the transient footprint of this pass is
what decides whether it fits. Release the sort keys as soon as the d order is
known and the noise ratios once their per-shell medians are read - on the
device, rather than downloading every sorted value. Peak about 96 bytes per
observation; the free memory is checked first and a device without it stops
with a message naming the CPU build.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The multiplicative spread of the outlier band is the weighted median of
(|ln(I/median)|, weight) pairs, one per usable full, and its sort was the bulk of
that step. SortPairsGPU sorts them on the GPU (by the second key, then stably
by the first) and leaves the sequence std::sort leaves: two pairs that compare
equal are the same two numbers. The values themselves (a logarithm) and the
running sum stay on the host.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build); a Catch2 case
checks the device order against std::sort.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The per-group sums the widening is measured from - each hand's inverse-variance
mean under the error model's sigmas, the group's d and the d range - are a walk
over every full's record on the host. In the GPU build a kernel takes them, one
thread per group in index order, with the host's model sigma to the bit
(ModelSigmaExact: every product rounded on its own); the host keeps the shells
and their medians. The host path is the same sums, now kept per group so both
paths hand the shell step one array.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
MergeEmSamples now forms each sample whole on the device - the counting
variance at the reflection's expected intensity included, every product rounded
on its own as the host build rounds it - and compacts them in fulls order with a
scan, so the host receives exactly the vector it used to build. That replaces
the download of three full-length double arrays and a flag array, and the host
pass that compacted them while reading every full's eighty-byte record.
p.mtz md5 and the error-model lines unchanged on myob, cytc, 8a1a and 8qaw
(GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
WilsonOutliersGPU runs the test of WilsonOutliers.cpp step for step on the
device and returns the same result to the bit: the two orders the host sorts
into are total orders, so a stable radix sort of the indices reaches the same
sequence; every sum is taken by one thread over its shell or reflection in the
host's order; each product the host rounds on its own (-ffp-contract=off) is
rounded with __dmul_rn; medians are read off sorts; and the logarithms (t, z,
u, the tail scale) are taken on the host from downloaded counts and medians.
MergeAndStats calls it where the merge itself is on the device.
8a1a final P61 merge, "rsm merge and statistics" 4.35 -> 2.82 s (the test runs
twice there, once per error-model fit). p.mtz md5 unchanged on myob, cytc,
8a1a and 8qaw (GPU build). A Catch2 case checks the device result against the
host one on a population that reaches every branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The automatic cut fitted its logistic to one Pearson CC1/2 per bin, which
has no bound on what one pair can do to it. In a weak shell a single
reflection with a huge intensity in both halves - an artefact the halves
reproduce, or one observation with a sigma fifty times the shell's - carries
nearly all of the covariance. Each bin's reflections are now dealt into up
to seven groups by their indices (groups of at least ten reflections), and
the bin reads the median of the groups' CC1/2. On a clean shell every group
estimates the same CC1/2, so the median is the Pearson value; an outlier
takes one group with it and the median does not follow. Only the cut
decision changes; the reported shell table is unchanged.
Found on a weak split myoglobin crystal: one reflection carried 50-99 % of
the covariance in most fit bins past 2.6 A (e.g. 43647 in a bin with median
I 1.3), reading them at CC1/2 0.93-0.98 where the median of groups reads
0.2. The cut swung between 1.21 and 2.03 A with --prepass-fraction; it is
now 1.85 A at 0.5 and 1.89 A at 1 (XDS 1.51 A, rc.174 1.96 A).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pixel that reads at most half of its own sector's level on average, by more
than the bound a hot pixel's excess has to clear, is now masked as dead - the
mirror of the hot rule. Nothing but a defect keeps a pixel dark frame after
frame, and every reflection whose disk covers it is integrated short.
The finder now also runs on CCD sweeps (no sensor depth): a dead CCD pixel
reads its offset, far below any ring with background, and the hot rule's
ten-fold ratio does not depend on Poisson counts either.
Found on 5jk4 (ADSC Q4, 2304x2304): the last row and the first two columns
read the offset (~20 ADU against ~60-100 around them) and were unmasked.
Reflections touching them merged to large negative intensities, and one
observation at y=2298 with sigma 10334 swamped the outer shell's half-set
CC1/2 (3.9 %) and the overall CC1/2 (0.786; 0.998 at rc.174). With the
6630 dead pixels masked the fall-off is clean: CC1/2 0.998, cut 0.96 A
(fit 1.01 A), at --prepass-fraction 0.5 and 1 alike. Counting-detector
sets with low background are untouched (myoglobin X06DA: 0 dead).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Rotation prediction enumerated the whole (2h+1)(2k+1)(2l+1) box every frame and kept the ~1% whose
rotation solution lands near the frame. A solution within phi_limit of the frame needs
|f| = ||p0|^2 + 2 S0.p0| <= 2 |S0_perp| |p0| phi_limit, and along an (h, k) column f is a quadratic
in l, so the l that can pass are at most two intervals. Those are walked - widened by 1% in f and by
a whole index at each end - in the same order and with the same arithmetic; the rest the phi test
rejected anyway. Without a phi limit (min_zeta 0) the whole column is walked as before. The GPU
kernel is unchanged.
A new test predicts 1000 random lattices, geometries, axes and rocking widths both ways and
requires the same reflections in the same order, every value bit for bit (30000 cases passed once
locally). CPU build, two interleaved warm runs: 8a1a output loop 241.5 -> 216.3 s, pre-pass loop
90.7 -> 73.8 s, whole run 427.0 -> 384.2 s; 8qaw (with the gzip change before this) loops
116.6 / 251.3 -> 94.4 / 206.8 s, user CPU 14700 -> 12718 s. GPU build within noise.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
SlurpGz cleared the reused read buffer and grew it back in 1 MB resizes, so every frame
zero-filled its whole decompressed size before gzread overwrote it. It now grows the buffer only
past what it already holds and trims it to the bytes read; the bytes are the same.
8qaw (Eiger 16M, .cbf.gz): memset 5.7% -> 0.5% of the run's user cycles (perf, GPU build). The GPU
build's image loops do not move (pre-pass / output loop 15.28 / 28.63 -> 15.16 / 28.62 s): they are
not bound by the decode. The CPU build's loops are; with the prediction change after this one, its
8qaw loops go 116.6 / 251.3 -> 94.4 / 206.8 s.
Tried and not kept: reading the next frame on a thread of its own while the worker analyses the
current one (two raw images per worker, two page-locked regions in ImagePreprocessorGPU). Exact,
but the 8qaw loops got slower (15.3 / 28.6 -> 16.1 / 29.8 s) and myob's too (2.10 -> 2.40 s): with
the decode overlapped the loop is still waiting on the uploads.
p.mtz byte-identical on 8qaw, GPU and CPU builds; the HDF5 sets do not reach this code (and are
identical with the commit after this one).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ApplyCellSurface copied and sorted every term's 1/d^2 to read 19 order statistics off it, for each
of the four surfaces of the final merge. The edges are now selected: one pass counts the values
under each top-16-bit pattern (a non-negative float orders as its bits), a second collects the few
bins the ranks fall in, and nth_element picks the value inside each. An order statistic is a value,
so the edges are the ones the sort gave.
8a1a, GPU build: the shell set-up of the four surfaces 0.55 -> 0.26 s (instrumented run), the
correction-surface window 6.32 -> 6.12 s and the final merge 11.03 -> 10.82 s (two interleaved warm
runs each); whole run 55.78 -> 55.45 s. 8qaw surfaces 2.95 -> 2.83 s.
Tried in the same window and not kept, as they measured nothing: the serial stable counting sorts
(surface group order, decay terms, fulls CSRs, delta-CC1/2 frame order) on all threads; delta-CC1/2
workers uncapped from 8; measuring two rounds of the edge search at once. The delta-CC1/2 window
(2.4 s) is bound by its longest chain of ranges, one serial accumulation, not by the worker count.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
On a canonical pass the post-refinement only measures: nothing in ScaleMergeAndSymmetry reads
result.post_refine or prepass_detector_geometry_ - RunAllPasses does, after RunPipeline returns. So
the pass now gathers the observations where it did (beside the guard's merge and the engine's host
ingest - the gather reads the outcomes, which the merge chain later re-indexes in place) and runs the
fit on a thread of its own, beside the merges. RunPipeline joins it when ScaleMergeAndSymmetry
returns and replays its log lines and notes there. The speculative geometry probe the fit may start
is started from that thread, on a copy of the run taken on the main thread before the fit, because
the merge chain goes on to change the run's members. The probe-only pass keeps the sequential path.
8a1a, GPU build, two interleaved warm runs each: the window after the image pass loses the fit
(post-refinement 7.6 s -> 2.4 s gather on the main thread), but the quality guard's merge of the
pre-pass frames (7.8 s) is what that window waits for, so nothing moves: tail 30.57 -> 30.59 s,
whole run 55.78 -> 55.79 s; 8qaw 104.08 -> 104.06 s (guard 13.8-14.1 s); myob/cytc within noise.
Peak RSS on 8qaw 24.07 -> 24.35 GB. The gain needs the guard off the critical path as well.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Full open and in-house battery on 9ced4c444 (run 20261007-2139_9ced4c_rc175-p1-baseline) against
the rc.173 battery, without the REFMAC check: R-free figure and numbers are kept from the rc.174
battery (20261005-2102_3770f4_rc174-final-refmac). No verdict changed from rc.174; 2ygz is new and
fails. Two sets lost quality at the 50 % pre-pass default (outlier-laden outer shell on 5jk4,
runaway cut on the in-house split myoglobin set); fraction 1 restores them, but nearby fractions do
too and a fixed cut gives equal statistics, so the pre-pass only tips two existing weaknesses (an
unmasked dead edge row; a noisy half-set CC1/2 on a multi-domain crystal). Time, zoomed time,
in-house time and resolution figures regenerated with paper_plots.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged scaled fulls the final merge hands back for the .hkl were gathered by one thread
pushing back over every full. Which fulls is now KeptIndices (the same list, in the same order),
and each record is made side by side.
The same records in the same order: p.mtz and the .hkl byte-identical (myob, thau, cytc, kdp,
cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a, 8tyy, CPU build myob). 8a1a, no other
load, interleaved: the P61 merge's statistics stage 3.94 -> 3.71 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
RotationScaleMerge::Ingest is now IngestHost (read the outcomes, sort, build every host array,
smooth the geometry and recompute the partiality) followed by IngestDevice (allocate the GPU's
arrays and stage them). The host half reads the outcomes and nothing else, as the post-refinement
and the quality guard's merge do, so on the canonical pass it runs on its own thread beside both;
the device half runs where the ingest used to, after the guard has handed the GPU back, so the two
engines never hold the card at once. The device feed now stages the recomputed partiality in the
same slices as every other field, straight from the ingest array, instead of in a pass of its own
after the rest; the values on the device are the ones they were.
8a1a, no other load, interleaved runs: the first merge (P1) ends at 10.9 s after the image pass
against 11.3 with only the guard overlapped and 12.7 on the rc175 base. p.mtz byte-identical on the
gate sets (myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a,
8tyy, CPU build myob); --model, stills, calibration, --mode scale (GPU and CUDA_VISIBLE_DEVICES="")
outputs identical; --no-merge / --mode azint / --write-process-h5 _process.h5 differ only in the
datasets that differ base-vs-base (azint sums, bkgEstimate, profiling times, dates, git).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
