The mapping is now built while the frames are read, and with every core it took them from the
readers - on a 16M EIGER the read went from 0.10 to 0.19 s. Built on AvailableCpus() minus the
readers it is back at 0.11 s. The mapping is per pixel, so the same whatever thread count builds it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The spot-width stage of the pre-scan copied every preprocessed frame back to the host (72 MB a
frame on a 16M detector, page-locked host mirrors per worker) for measurements that read a few
pixels about each spot. Now:
- MeasureSpotsOnTiles: the flux curves and the footprint read (2*24+1)^2 squares about each spot,
gathered on the card (ImagePreprocessorBufferGPU::GatherTiles) - no copy_image_to_host, no host
mirror. The flux curves are bounded at R_BKG_OUT = 20 px. The footprint keeps its arithmetic; a
per-iteration check that its window is inside the square, and a spot whose window walks off it is
measured again from scratch on an 80 px square, then on the whole frame. Where the squares would
be more pixels than the frame, the frame is read. Same pixels in the same order, so the same bits.
Harness over 34 rotation sets (~1500 frames): 0 differences; misses at 24 on a few spots of most
frames, at 54 on up to 80% of frames of a few sets, at 80 on none.
- The azimuthal mapping and the per-worker spot engines are built while the frames are read.
- The tiers no longer wait for each other: workers run on through the visit order, each tier's
answer is drawn from exactly its own frames, and frames measured past the tier the answer stops
at are thrown away - a frame is measured the same whichever tier it is read for.
- PreScanMemo keeps the spot measurement, the defective pixels and the background gain under the
pre-scan key plus the spot-finding and azimuthal settings, so a copy of the run that pre-scans
again on the same inputs reads no frames.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The budget is checked before the read, so a frame that will not be kept is read into the worker's
reused buffer instead of a fresh one paged in and let go (a 16 Mpx CBF sample kept 29 of 60 frames
and paged in all 60), and a kept frame borrows the worker's reader scratch for the read instead of
allocating one of its own. The budget can now be passed by at most a frame per worker.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The beam-stop projection (ShadowAccumulatorGPU) and the defective-pixel sums (HotPixelFinderGPU) are
now kept one per card, fed by workers pinned t % gpus like the other pre-scan workers, and folded
into the first card when they are read (cudaMemcpyPeer + an integer add / max-with-count kernel).
All of them are integer sums or a maximum with the first-value rule, so the fold is order-free and
the result is identical to one card's. With one card nothing is folded.
The pre-scan sample is read once and kept (compressed as stored; up to 2 GiB, a frame past that is
read again where it is wanted) for the spot measurement and the defective pixels, which read every
frame again each before - a CBF/marCCD/SMV frame was decoded on the host three times.
A beam-stop detection that finds no shadow leaves the mask as it was, so every probe copy of the run
repeated it on the same inputs. Its answer (shadow, ring centre, mean projection) is now kept in a
memo shared with the copies, keyed on the sampled frames, the geometry the rings are drawn in, the
polarization and the binary mask; up to 4 entries.
Debug TIMING lines for the pre-scan's parts (read and projection, beam stop, defective pixels, spot
measurement). New tests: two accumulators/finders folded match one fed every frame.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The pre-scan's spot measurement (spot width, bandwidth, powder rings) built
all of its GPU engines on card 0 from unpinned threads. Worker t now runs on
card t % gpus with pin_gpu(), as the image-loop workers do, which also binds
it to that card's NUMA node where binding is enabled. Every tier runs on
fresh threads, so each tier re-pins worker t to the card its engine was built
on, and the engines are freed on threads pinned the same way (as the first
pass's ReleaseSpotEngines does).
Worker count, width tiers (8/15/30/60 on a 60-frame sample) and the frames
read are unchanged. A frame's spots do not depend on the card that found them
(same model), and the per-frame results are stitched in sample order as
before, so the output does not depend on the number of cards.
The projection and defective-pixel passes are left on card 0: both fold
every frame into a single device accumulator (ShadowAccumulatorGPU,
HotPixelFinderGPU) that lives there.
Verified on one card: CI-tier battery p.hkl md5 and MTZ reflection data
identical to the base binary on every set.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The background ring's high-side clip, mean + 4*sqrt(mean), assumes counted photons. A CCD reports
ADU: on a Rigaku Saturn a background pixel scatters 4-5x its mean, so the clip sat at ~2 sigma, cut
the upper tail of every ring and added a positive offset to every reflection, compressing the
intensity statistics (a false 422 on one twinned I41 crystal).
No header carries a usable gain (SMV states none, every marCCD file the writer default of 1 ADU per
photon), so it is measured: background_gain::MeasureFrame takes the variance per unit mean of the
difference of pixels two columns apart, per level, clipped, on every fourth row of up to 16 of the
pre-scan's frames; Gain is the pair-weighted median over the levels. The clip becomes
mean + n*sqrt(max(g,1)*mean) (BraggIntegrationSettings::bkg_pixel_gain, CPU and GPU engines).
Only for SMV/marCCD input (ProcessConfig::counts_in_adu, set from the file format); HDF5 and CBF
keep g = 1, the identity. One log line reports the value. No new option.
Measured g: Saturn 944+ 4.7-5.2, Saturn 944 2.7, A200 0.68, ADSC ~0.3, marCCD ~0.3 - so the
ADSC/marCCD/A200 sets keep the photon clip exactly.
Checks (reflection data via gemmi and p.hkl, against the rc175 battery outputs / base build):
identical on 11if, 7qis, 8a1a, myob_x10sa, cytc_x10sa (photon counting), 4bwl, 5epe (ADSC,
marCCD) and 6cee (A200). Saturn sets, battery baseline -> this change:
3r6o I4122 R_meas 19.3% ISa 4.4 <|L|> .321 -> I41 (correct) 12.1% 8.1 .384
3mc4 R_meas 10.2 -> 9.8 %, ISa 12.1 -> 12.7, d_min 1.79 -> 1.75 A, <|L|> .360 -> .391
5uth 12.6 -> 11.8 %, ISa 10.4 -> 10.1, <|L|> .426 -> .483, offset vs |Fc|^2 0.21 -> 0.00
5vml 10.2 -> 9.6 %, ISa 11.0 -> 11.6, <|L|> .426 -> .475, offset 0.15 -> 0.05
5cc8 8.1 -> 7.9 %, ISa 10.8 -> 11.2, <|L|> .469 -> .494
3meb 14.5 -> 14.6 %, ISa 13.2 -> 13.2, <|L|> .442 -> .457
Space groups unchanged except 3r6o. Pre-scan cost: +0.5 s on a 9 Mpx ADSC sweep, within noise on
a 15 Mpx marCCD sweep, nothing on HDF5/CBF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This reverts commit f257d51ea.
The block-wise moving-shadow mask fixed its one target (a half-image
lysozyme set whose pin turns with the sample) but fired on 47 sets of
the full battery: on one small-molecule set the mask covered 51 % of
the detector and SHELXL R1 doubled (0.10 -> 0.20), on another the mask
grew from 8 to 21 %. The pre-scan cost rose by +87 s on the HDF5 sets
and +499 s on the CBF sets (the commit claimed +0.1 s / +1-5 s per set).
A narrowed version (regr2-mask) still costs +25-45 s battery-wide, about
+0.3 s per HDF5 set, for one difficult set - more than the owner's
budget allows. It is parked for rc176, where a moved shadow is to be
found by a far-end check of the early decisions instead.
The documentation paragraph the commit added goes with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pin, loop or capillary on the goniometer shadows one place for part of the rotation only. The
beam-stop mask is one for every frame and is read off the mean over the pre-scan sample, so a pixel
such hardware covers on fewer than about half the sampled frames is kept and integrated attenuated
on the frames it is covered on. Since the pre-scan samples only the first part of the sweep (part B,
5b9144410), a sweep whose moving shadow is short in the first half and long in the second loses it:
on the in-house half-image lysozyme sweep the mask fell from 32.8% to 21.6% of the detector, the
second half of the sweep was integrated through the shadow, ISa dropped 6.9 -> 4.1, R_meas 116 ->
240%, the cell refined a != b, and the attenuated intensities erased the a-axis screw evidence
(merged rc175: P4_1 2 2 instead of P4_1 2_1 2).
The sample is now also read in blocks of 10 consecutive frames (rotation only); each block's shadow
is found on its own with the rings about the whole sample's centre, and the blocks' pieces that
reach beyond the 30 px band the whole sample's edge can grow over (PENUMBRA_MAX_PX) are added whole.
A pixel shadowed on most frames of any longer stretch is shadowed on most frames of one block, so
nothing finer is needed; pieces within the band are a block redrawing a static stop's edge and are
left out, so a sweep without moving hardware keeps its mask bit for bit. Part B's principle is kept:
everything is still decided on the first part of the sweep.
Measured (GPU, against merged rc175 eac0833c8, 19 sets): half-image lysozyme P4_1 2_1 2, ISa 4.1 ->
9.7, R_meas 240 -> 113%, CC1/2 0.955 -> 0.979, R-free(model) 0.331 -> 0.282 (last night 6.9 / 116%).
p.mtz md5-identical on 14 of 19 (8v4o, 9rci, 5ky6, 6yqf, 6ze4, 5reo, 9qw8, insu_I low-ISa,
insu_H notwin, myob powder 1/2, myob split, nothing_1/2); changed: 5nw5 (a dim crescent at the stop
edge now masked: ISa 8.4 -> 8.9, CC1/2 0.937 -> 0.980, R-free 0.462 -> 0.429), 7atg (0.85% added at
the detector edge, statistics unchanged), 9min (0.03%, fail -> fail unchanged), 5jk4 (<0.01%,
unchanged). Cost: pre-scan +0.1 s on HDF5 sweeps, +1-5 s on large file-per-image (CBF, marCCD)
sweeps, where the blocks are re-read and their masks computed on the host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The pre-scan sample (beam-stop shadow, defective pixels, spot width, the background beam centre)
and the first pass's frames (the spread and wedge schemes, the validation frames, the prefetch) now
come from [start, PrepassEnd) - the part of the sweep the geometry pre-pass integrates - instead of
the whole sweep. Rugnux::PrepassEnd is the one place that range is defined, for the pre-pass and for
these decisions alike, so any change to the pre-pass range (such as a minimum rotation) moves both.
Stills keep the whole range.
With rugnux following a collection while it is written, nothing before pass 2 then waits for the
second part of the sweep: the pre-scan, both first passes and the pre-pass run while it is
collected, and pass 2 starts reading as soon as they are done. An offline run takes the same
decisions from the same frames, so the streamed result still equals the offline one by md5 (in-house
JUNGFRAU 4M thaumatin replayed at one data file per second, also at -N 8 and with the replay paused
and resumed): 2.4 s from the last data file to the result, against 8.0 s for the offline run.
Result-changing. p.mtz md5 at af3ea9cca -> here (with the 120 deg minimum of the previous commit),
GPU: myob de2ab820 -> 9bdd6916, cytc 3e5d2f91 -> 62e744a1, thau bc5e3de3 -> 3ec88c34,
kdp cf83ee5d -> cdc2c068, 8a1a 50ab659e -> 309ea9a8, 8qaw d3287922 -> c2849d52; CPU: myob 3f9d12d8 ->
18c7ac2f, cytc d30c9969 -> 40e17d97, thau 4cdbdd94 -> f14e52da, kdp f1d801f3 -> eb930714. Sweeps of
120 deg or less are unchanged by this commit (their early range is the whole sweep). Needs the
battery before it is merged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pixel that reads at most half of its own sector's level on average, by more
than the bound a hot pixel's excess has to clear, is now masked as dead - the
mirror of the hot rule. Nothing but a defect keeps a pixel dark frame after
frame, and every reflection whose disk covers it is integrated short.
The finder now also runs on CCD sweeps (no sensor depth): a dead CCD pixel
reads its offset, far below any ring with background, and the hot rule's
ten-fold ratio does not depend on Poisson counts either.
Found on 5jk4 (ADSC Q4, 2304x2304): the last row and the first two columns
read the offset (~20 ADU against ~60-100 around them) and were unmasked.
Reflections touching them merged to large negative intensities, and one
observation at y=2298 with sigma 10334 swamped the outer shell's half-set
CC1/2 (3.9 %) and the overall CC1/2 (0.786; 0.998 at rc.174). With the
6630 dead pixels masked the fall-off is clean: CC1/2 0.998, cut 0.96 A
(fit 1.01 A), at --prepass-fraction 0.5 and 1 alike. Counting-detector
sets with low background are untouched (myoglobin X06DA: 0 dead).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
leonarski_fandClaude Opus 5.5