Rugnux: Rotation processing recovers most of the speed lost since 1.0.0-rc.180; live processing in jfjoch_broker finishes sooner.
Rugnux: More reliable rotation processing and indexing, including small molecules and supercells.
Rugnux: More robust serial data and model refinement; molecular replacement handles multi-chain models.
jfjoch_broker (experimental): Live serial processing keeps the detector geometry fixed for the whole run.
jfjoch_broker: Pixel sum without an FPGA, a mask bit for inactive pixels, and several stability fixes.
jfjoch_broker: Rotation indexing and quick integration moved to rugnux live mode; their API settings are deprecated and ignored.
jfjoch_broker, Rugnux: Azimuthal integration can use logarithmic (log10) Q bins or a given number of bins, set from the API, the web interface and the Rugnux command line.
jfjoch_viewer: Build and display fixes; the README is rewritten.
* Rugnux: Rotation processing recovers most of the speed lost since 1.0.0-rc.180; live processing in jfjoch_broker finishes sooner.
* Rugnux: More reliable rotation processing and indexing, including small molecules and supercells.
* Rugnux: More robust serial data and model refinement; molecular replacement handles multi-chain models.
* jfjoch_broker (experimental): Live serial processing keeps the detector geometry fixed for the whole run.
* jfjoch_broker: Pixel sum without an FPGA, a mask bit for inactive pixels, and several stability fixes.
* jfjoch_broker: Rotation indexing and quick integration moved to rugnux live mode; their API settings are deprecated and ignored.
* jfjoch_broker, Rugnux: Azimuthal integration can use logarithmic (log10) Q bins or a given number of bins, set from the API, the web interface and the Rugnux command line.
* jfjoch_viewer: Build and display fixes; the README is rewritten.
The Wilson test keeps an improbable observation whose reflection has other
large observations: they confirm each other, which is how a genuinely strong
reflection survives. A spot of another lattice (satellite crystal, salt or ice
crystallite) that lands on a predicted reflection defeats this: that lattice
diffracts centrosymmetrically too, and half a turn later the rotation brings
-h and the spot's own Friedel mate to the same mirrored detector position, so
the artefact always arrives as a Friedel pair that vouches for itself.
Seen on a P212121 rotation set: one 1.04 A reflection measured three times -
h on one frame and -h half a turn later, both ~5 px off the prediction on the
same detector row, compact single-frame spots of 3000 and 10000 counts (500x
the shell mean), and a symmetry mate elsewhere that reads ~0. The pair
confirmed each other, the median of the three was then an artefact and the
median test deleted the true measurement, and the merged intensity alone took
the last-shell CC1/2 from ~0.18 to 0.80 (XDS: 0.08, both flagged as misfits).
An improbable observation of the exactly opposite indices is now neither a
confirmation nor company; a probable one still testifies against it. The
observation carries a key of its own unreduced indices with key(-h) = -key(h).
Same change on the device path (still bit-identical to the host).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
The cell e.s.d.s came from a d-spacing fit at the run's distance, so the
distance's uncertainty was left out, although a cell measured at a wrong distance
is scaled with it (correlation 0.93-0.95 on an atomic-resolution crystal, near 1
at low angle). The metric fit now takes the distance as one more parameter, each
|q| read as q + e d|q|/d(ln distance) at its observed centroid, and CELL_ESD_LSQ
is the metric's e.s.d. with the distance free: its uncertainty, as far as the
spacings measure it from how their residuals curve with 2theta, carried onto the
cell. The block estimate still holds the distance, which is common to every block,
and CELL_ESD stays the larger of the two. The log line names where the spacings
put the detector relative to the distance integrated at.
On that crystal CELL_ESD_LSQ on a goes from 1.4e-4 to 5.2e-4 A; CELL_ESD (the
block scatter, 5.1e-3 A) is unchanged. Nothing the run decides reads it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
BDiagonal took the median of an empty list when the dictionary describes none
of the model's atoms (a model of unknown residues), and the restrained stage
crashed. The diagonal is then empty, as the atoms are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
The detector-frame surfaces are held to a mean of one within each resolution
shell, because a factor that is a function of d alone is a null direction of
their fit. That mean was weighted by 1/sigma^2, and sigma is smallest where
little light arrives: on a detector part-shadowed by the pin the dark cells
carry the shell's mean, and their large factors go onto the lit cells as small
ones. How much of a ring is shadowed varies with resolution, so this bends the
merged intensities along resolution: REFMAC's fitted scale against the model
ran 1.48 / 0.80 / 1.16 in F (1.65 A / 2.7 A / inner) on a half-shadowed lysozyme
sweep, with R as good as XDS's within every shell and 0.02 worse over all, and
no shell-wise statistic can see it. Each observation now counts once.
REFMAC R-work / R-free on the depositor's flags (d_min of the baseline), model
as deposited, 31 sets chosen by the largest removed ramp and clamped cells,
largest R vs per-shell R gap, plus controls: 5 better by more than 0.0015, 23
the same, 3 worse.
half-shadowed lysozyme 0.3266 / 0.3306 -> 0.2996 / 0.3069 (XDS on the same
model 0.3036 / 0.3083)
soft X-ray insulin 0.2720 / 0.2841 -> 0.2423 / 0.2570
4p8r 0.2143 / 0.2518 -> 0.1873 / 0.2216 (depositor's data
0.2216)
6i3j -0.0029; insulin reference -0.0017; 2wnz +0.0025, 6jgh +0.0016, 5j23 +0.0015.
The ramp removed on the first falls from 186x to 2x. The merge's own hash test
records change as they must.
A mean over the ring still lowers its lit part by the shadow it averages in
(the new test reads 0.72 where 0.57 before and 1 would be exact); a median of the
counts did not do better on the half-shadowed sweep (R-free 0.3090), the only one
it was measured on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQs733XDnbQQyUbV1vs
Pass A's background-free centroid divided the disk's first moment by
net = I_sum - n*bkg whenever net > 0. An integer sum against a rational
background reaches zero to rounding (net ~ 1e-14), and any small net gives a
position off the disk: the first moment's noise is radius x sigma, so the
centroid's is radius x sigma/net. Below sigma the raw centroid is taken, as it
already was for net <= 0. Both engines; Metal shares the CPU code.
XDET/YDET up to 1e16 in the unmerged MTZ was the visible part: 67 of 68
rotation sets scanned had rows off the detector. Their I is right (profile fit
and box sum are centred on the prediction; z against symmetry mates has rms
0.92 vs 1.04 for the control).
The real harm was in post-refinement. Such a centroid can reach the strongest
20000 positions, where the held-out cost is a plain mean with a noise from the
fourth moment: one outlier inflates the noise and the cross-validation refuses
a fit that the paired change supports at -17 sd. Measured, --no-gpu, this commit
against its parent, 14 sets: 5 change their verdicts and so their geometry;
the 9 others have merged MTZs bitwise identical. On the half-image in-house
lysozyme: joint noise 6.8e-8 -> 2.4e-8, reject -> COMMIT, cell a 78.44 -> 78.35
(XDS 77.74), R_meas 56.0 -> 46.8 %, ISa 4.3 -> 4.7. Open-arm 8c3e: 1.74 -> 1.67 A.
Small-molecule L-alanine single sweep commits too, ISa 57.3 -> 55.6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQs733XDnbQQyUbV1vs
A Qt6 built with zstd looks for it through its FindWrapZSTD, which takes our zstd::libzstd alias
and then requires zstd_VERSION; the fetched zstd never sets it, so the viewer configure failed with
"Qt6Core could not be found because dependency WrapZSTD could not be found". The version is the
fetched one (1.5.7).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
ReanalyzeImage_i works on a copy of the current image's message, which still carried the previous
analysis. Analysis only sets the indexing and integration fields when it finds something, so after
a re-analysis that failed to index (algorithm None, stricter settings) the Inspector kept the old
cell, indexed-spot count and predicted reflections. They are now cleared before the analysis runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
README gets a line and a short paragraph in "Running rugnux": several rotation
sweeps of one crystal can be given at once (masters, or one frame of each
frame-file sweep), merged as one dataset by default, --sweeps separate for each
alone; marked experimental and not fully tested, pointing at RUGNUX_CLI's Inputs.
RUGNUX_FORMATS said rugnux does not combine sweeps in a run, which contradicts the
usage and RUGNUX_CLI. Its input paragraph now states the multi-sweep input per
format with the same experimental mark, keeps the downstream-merge pointer for
sweeps processed separately, and names the Bruker SFRM reader the page omitted
(a bullet beside the other one-frame-per-file formats, and the as-is note widened
to it).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The two never-destroyed FFTIndexerReuse and FFTPlanReuse objects move from the
JFJochReceiverRugnuxRotation constructor into the broker's main. In the broker nothing changes.
In jfjoch_test the first rotation receiver no longer keeps indexers and plans for the rest of the
process: with Catch2's random order, the GPU memory tests that ran after one found 37 plans kept and
the indexer memory already taken (CudaFFTPlan_SharedAndKept, IndexerThreadPool_ReleaseIndexersGivesTheCardBack,
IndexerThreadPool_FFTIndexerKeptAcrossPools, the three CI failures on 0d26ecd13).
On rc184 the statics move out of JFJochReceiverRugnux's constructor.
(ported from rc184-2 0df766328)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Since 0df766328 the broker keeps an FFTIndexerReuse for its whole life, so the broker's own indexing
pool (Lite and FPGA receivers) hands its GPU FFT indexers back to the reuse cache instead of freeing
them. The cache is keyed on the FFT settings and never shrank: every change of the indexing settings
through the API parked another 0.6-0.9 GB per worker on the card, until a pool build ran out of GPU
memory. (Before 0df766328 the same happened from the first live rotation run on.) The service now
releases the kept indexers when it rebuilds the pool; rugnux live keeps new ones at its next run.
(ported from rc184-2 8f47fb756)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
JFJochReceiverFPGA and JFJochReceiverRugnuxStills read three slot counters for each image through
GetStatus, which walks every slot under the buffer mutex, lists the image numbers and sorts them. With a
buffer sized for 2 kHz (thousands of slots) that is about a core of CPU and time on the mutex every
other buffer user needs. GetSlotStatus returns only the counters; GetStatus stays for the HTTP status.
On rc184 the per-image readers are JFJochReceiverFPGA's FrameTransformationThread and its GPUSink (the live-stills path).
(ported from rc184-2 4907014e4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Every image thread took the socket's mutex to add its message, and the thread that cut a batch then
serialised and sent the whole period's images while holding it, so all image threads waited for the
encode. Now the message is copied outside the lock without spots, reflections and image (the metadata
message never carries them), the lock only appends and swaps the batch out, and the batch is serialised
into its own buffer and sent with the lock released. Two batches cut close together can leave in either
order, which the documentation already allows; END stays last.
Two latent faults go with it: a disabled socket (no period) kept every image of the run and sent them
all at END - now it keeps nothing; and the batch buffer was a fixed 20 kB per image, too small for a 2D
azimuthal integration profile - it is now sized from the profile and ADU histogram lengths.
(ported from rc184-2 4147934e6)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
With modules x images at or above what the card's HBM holds for the
internal frame generator (64 module frames in the HLS simulation, 1024 on
the card), frame_generator refuses the run and sends nothing, and the
collection hung in Measuring with no error. The host now checks this:
PrepareAction refuses such a run at the start (a 32-bit EIGER frame
counts twice, as in frame_generator.cpp), and SetInternalGeneratorFrame
refuses an image that would land there, each with an error that says
how many module frames fit. The capacity comes from hbm_size_bytes of
the card's status (the mock keeps its 64).
The check keeps the last module frame free as well: load_calibration.cpp
accepts module_id == 2 * hbm_size_bytes / module size (">" where ">="
was meant), one past the memory. The firmware is frozen, so the host
never sends that index instead.
The HLS simulation keeps frame_generator's return code: a refused
configuration cancels the data collection rather than waiting for frames,
and HLSSimulatedDevice::FPGA_EndAction throws with the code.
rc184 has no MockAcquisitionDevice: its part is left out, and the test checks HLSSimulatedDevice only, as HLSSimulatedDevice_InternalGeneratorMemory in FPGAIntegrationTest.
(ported from rc184-2 25542a9af)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
pixel_sum existed only on the FPGA path, so a DECTRIS master file carried an all-zero
/entry/image/pixel_sum. The CPU and GPU image preprocessors now compute it in the pass that takes the
minimum and maximum, as a signed 64-bit sum, and MXAnalysisWithoutFPGA reports it: Lite, live stills,
live rotation and rugnux. On the simulated card it equals the card's own sum, for 16-bit and for
summed 32-bit frames.
The sum excludes error and saturated pixels and the device mask (detector and calibration). The hot
pixels and the beam-stop shadow a run finds on its own frames do not change it, though they stay masked
in the image and in the other statistics, so the number does not depend on what a run decided. Those
two have bits of their own (PixelMask::RunMaskBits); the byte-per-pixel mask keeps the difference, 1 for
the device mask and 2 for the run's alone, and the CPU preprocessor gets a packed mask of the second.
Live stills reports this sum of the assembled image in place of the card's, and with the run's mask out
of it the two agree on the test frames.
Frames a live rotation run does not reach stay 0 in the master file, as before, and have no dataset in
the data files. The reader now reads pixel_sum into the dataset and the per-image message; a file
without it opens as before.
(ported from rc184-2 15af14357)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The pixels rugnux measures as outside a CCD's active area shared bit 1 with the detector's own bad
pixels, so they could not be told from the device mask and pixel_sum left them out. They now have bit 11,
a run-determined bit like the shadow and the hot pixels: they stay masked in the image and the other
statistics, and pixel_sum counts them. The beam-stop finder still reads them as data.
A file written with this mask no longer shows them as bit 1. The jfjoch XDS plugin and DIALS mask any
bit, so they still mask them; Durin and Neggia, which ignore the bits from 8 up, no longer do.
(ported from rc184-2 e6baecda5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
DecideTwin's twinned refinement reads only the placed model and the data, never the
untwinned result, so RefineModel now starts it on a thread of its own as soon as the
untwinned arm has placed its engine and slots, on the next card where there is more
than one. DecideTwin touches the shared result only once the untwinned arm is done,
and decides nothing where that arm failed or was cancelled. The twinned arm logs to a
buffered logger that is replayed after the untwinned arm's lines, so the log reads as
before. The per-stage TIMING sums of the two arms are no longer separated.
The engines are made exactly as they would be alone: each arm's GPU/CPU decision uses
the same budget as before, and the arms overlap only where twice what the untwinned
arm holds of its busiest card fits that budget (OwnSlotPlaces now reports it);
otherwise the twinned arm waits for the untwinned one, as before. Trial outcomes do
not depend on the slots, so the result is unchanged.
Measured on one RTX 5080, -N 28, two runs each (Step 11 wall, s):
cytochrome C (P3x21, twin law -h,-k,l): 6.0 / 6.1 -> 4.5 / 4.6
thaumatin, lysozyme (no twin law): unchanged
Every output (model, maps, MTZ, report, refinement log lines) is identical to the
rc184-port binary on all three sets. With two or more cards the twinned arm gets a
card of its own, so the step should drop to about the longer arm (~3.3 s here).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
MakeMaps filled FWT at the reflections the data do not have only after the three map
transforms, and MissingReflections computed the atoms' Fcalc (CPU, anisotropic atoms
included) and the solvent mask's Fmask one after the other. Neither reads the other:
MissingReflections now runs beside the difference and 2mFo-DFc transforms and the
anomalous map, and inside it Fcalc runs beside Fmask. Both are serial-heavy (a
single-threaded FFTW transform each, the mask's flood fill), so the tail shortens to
about the mask alone.
Measured on one RTX 5080, -N 28, two runs each (Step 11 wall, s; with the twinned arm
beside the untwinned one):
thaumatin (anisotropic U chosen): 3.24 / 3.23 -> 3.04 / 3.08
lysozyme: 2.38 / 2.38 -> 2.25 / 2.28
cytochrome C (two arms): 4.53 / 4.58 -> 4.49 / 4.57
Every output (model, maps, MTZ, report, refinement log lines) is identical to the
rc184-port binary on all three sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Each shortlisted row's unfold test (the transform at its own length and at
its folded partner's, summed over up to 4096 spots in double) is independent
of the others, so the rows are spread over refine_threads with ParallelFor
and the answers applied afterwards in row order, logging included. With one
refine thread (stills) it is the serial loop as before. Results identical by
construction.
Rotation first pass, -N 28, 5950X + RTX 5080, machine shared with other
jobs (median of 2, s): Step 3 Indexing / Step 6 Indexing at the refined
geometry
thau 1000 images: 2.75 / 1.23 -> 2.46 / 1.06
cytc: 1.81 -> 1.65
lyso: 1.96 / 0.90 -> 1.73 / 0.85
_report.txt (minus run-specific lines) and .hkl identical on all three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
After the candidate cells are refined, each one is tested on its own - the
reference match, StepDownToReference where it does not match, the angle and
volume tests, and ComputeIndexedMask over the accumulated spots. That loop
ran on one thread per scheme; on a re-index with a reference cell (the
long-axis hypothesis) StepDownToReference and the mask took most of the
call. The candidates now go over refine_threads with ParallelFor, each into
its own slot, and are collected in candidate order before the ranking, so
the ranking sees the same list. Results identical by construction.
Rotation first pass, -N 28, 5950X + RTX 5080, machine shared with other
jobs (median of 2, s): Step 3 / Step 6, after the DeAlias commit
thau 1000 images: 2.46 / 1.06 -> 2.10 / 1.00
cytc: 1.65 -> 1.62
lyso: 1.73 / 0.85 -> 1.76 / 0.84 (noise)
_report.txt (minus run-specific lines) and .hkl identical on all three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The long-axis hypothesis (the coarse 1200 A search, then a re-index of each
cell it offers) was 1.10 s of thau's Step 3 and 0.36 s of its Step 6, almost
all of it CPU work after a few ms of FFT, while the rest of the machine sat
idle. Its searches need only the fed spots and the experiment, so they now
start ahead of it, on a copy of the experiment and a pool of their own
(FirstPassRun::long_axis_spec): the coarse search beside the file centre's
schemes (IndexSchemes), the re-index of each cell it offers beside the
beam-centre check. The hypothesis then asks the same questions, and
RotationIndexer's memo, keyed on every input an indexer reads, answers them.
Where the run moved in between (the axis-sign rescue, another centre), the
key differs and the hypothesis indexes as before; a search that failed
leaves no memo and the hypothesis meets the failure itself. The parameter
code the two share is factored out (LongAxisBound, LongAxisCoarseSettings,
LongAxisReindexSettings, LongAxisOffers). Results identical by construction.
Rotation first pass, -N 28, 5950X + RTX 5080, machine shared with other
jobs (median of 2 before, 4 after, s): Step 3 / Step 6, after the two
indexer commits
thau 1000 images: 2.10 / 1.00 -> 1.65 / 0.84 (long-axis line 1.16 -> 0.02 s)
cytc: 1.62 -> 1.50
lyso: 1.76 / 0.84 -> 1.40 / 0.65
a ~1000 A-axis crystal (private): wall 84 -> 70 s
_report.txt (minus run-specific lines) and .hkl identical on all four.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
An axis long enough for the hypothesis is rare, and whether a crystal has
one does not change between the passes of a run, yet every first pass -
the refined-geometry pass, the probes, the distance-check passes - ran the
coarse search again (thau: 0.36 s in Step 6 and 0.06-0.70 s in each later
pass, offering nothing in any of them). The first pass that asks now leaves
its answer on the run (Rugnux::long_axis_answer_, reset per RunAllPasses):
a later pass keeps its standing cell where that pass kept one, and
re-indexes straight at the cell it adopted otherwise, without the coarse
search. Only the long-axis path changes; within a pass both centres of the
beam-centre check are still asked.
Rotation first pass, -N 28, 5950X + RTX 5080, machine shared with other
jobs (median of 2, s): Step 3 / Step 6 / wall, after the speculation commit
thau 1000 images: 1.65 / 0.84 / 15.0 -> 1.58 / 0.67 / 13.9
cytc: 1.50 / / 20.7 -> 1.35 / / 19.5
lyso: 1.40 / 0.65 / 12.7 -> 1.30 / 0.48 / 11.5
_report.txt (minus run-specific lines) and .hkl identical to rc184-port on
these three and on a 13-set battery subset (lyso_x06da_ref,
thau_x10sa_16keV, thau_x10sa_injection, insu_I_x06da_ref, cytc_x10sa, 6toc,
4bwl, 5vml, kdp_x10sa_20keV, and the long-axis sets 8v4o, 7d98, 4mtk, 4p8r):
battery compare 0 changed, every report and .hkl equal.
On a crystal with a ~1000 A axis (private) the run ends on the same cell,
space group and resolution; its distance-check passes now re-index at the
carried cell instead of searching, which moves its merging statistics in
the fourth digit. Wall 70 -> 60 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The intermittent 1.1-2.7 s stall at the start of live refinement (TIMING
refine.engine) is the CUDA driver's lazy module load. Stack samples of the broker in
a stalled run show the engine's first kernel launch (phase_table_kernel in
RefinementGPUCard) inside cuLibraryLoadData for the whole stall, and per-thread
/proc samples show the process using no CPU meanwhile: it is waiting for the
executable's pages. On this machine the binaries are on a busy, nearly full HDD and
the page cache is under pressure, so pages of the executable touched for the first
time late in a run - the refinement's fatbin, and in the worst runs other code and
data too (refine.topology 0.015 -> 2.5 s, refine.placement 0.15 -> 2.3 s) - can take
seconds to come back. Reproduced on purpose: dropping the broker's unmapped pages
from the page cache with a 4 GB write to /data beside it gives refine.engine
0.45 / 0.85 s (0.02 s normally).
WarmUpModelValidation, which the broker runs when a model is uploaded, now also loads
the modules of the refinement engine's and the scale fit's kernels (the two
refinement TUs a validation does not launch) on every card. With that, the same
induced load gives refine.engine 0.020 / 0.020 / 0.021 s. Other cold pages can still
stall under such load; keeping the binaries on a local SSD avoids all of it.
Live thau replays, no load added: refine.engine 0.020 s in 3 of 3, refinement 2.9 s
(rc184-port 3.1-3.2 s, with the two previous levers in). The warm-up only loads
modules, so the results are unchanged (same R-work/R-free and evaluations live;
ModelValidation_WarmUpChangesNothing passes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The polarization plane (measured_plane in ShadowFinder) put the 18 M pixels of
a 16M detector into their sectors on one thread - an atan2, a ring position and
a push_back per pixel - and pooled them over a box with two serial pixel loops.
The pixels now go into their sectors by blocks in parallel, through the same
counting scatter the rings use (bin_by_ring, with the sector as the key), and
the pooling loops are ParallelChunks. A sector's median is an order statistic
of its values, so neither the order they come in nor the full sort bin_by_ring
does (nth_element before) can change it.
The pass after the pre-scan builds its azimuthal mapping at a geometry that is
final once the beam stop has decided the polarization plane: the shadow and
defective-pixel bits added after it do not enter what the geometry cache keeps
(AzimuthalIntegrationMapping applies the mask to its copy). So that mapping is
now built into the run's geometry cache there, beside the defective-pixel pass
(Rugnux::MappingsAhead), and the pass finds it in the cache - waiting under the
cache's lock where it is still being built. Offline the pre-scan's own spot
measurement has built it already and this is a cache hit; live (no spot
measurement) it was built in the gap before the first pass.
Thau, offline (-N 28 -e 1000 --model 5a47), "pre-scan beam stop" 1.33/1.16 s
-> 0.77/0.63 s; cytc 1.21/1.12 -> 0.79/0.59 s; lyso_x06da_ref 0.20/0.21 ->
0.13/0.14 s. The offline pre-scan step does not move: the spot measurement
beside it is the longer branch.
Live replay (thau, 1000 EIGER 16M frames at 100 Hz, idx-simplon harness, the
quiet runs): from the main line's last pre-scan frame read to its first pass
2.58-2.63 s -> 1.85-1.94 s (beam stop 1.25-1.31 -> 0.70; the gap after the
defective pixels 0.67-0.68 -> 0.44-0.51); the probe's beam stop 1.29-1.31 ->
0.80-0.82 s.
Outputs identical to the base binary (thau, cytc, lyso offline; MTZ data,
maps, models, blobs and the live status live).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
RugnuxHostBudget took the image buffer and the 16 GiB margin off the machine's
budget (75% of its memory). The 25% the budget leaves out is already the reserve
for the rest of the machine, and the margin keeps the broker's own part, so the
reserve was counted twice: with an 80 GiB image buffer on a 134.5 GB machine,
0.75 x 134.5 GB - (85.9 + 17.2) GB < 0 and rugnux planned against 1 byte. The
post-refinement then ran before the merge instead of beside it, the
all-observation P1 merge after this engine's instead of beside it, and
HostWorkerPool held one worker (molecular replacement and the null replicates
single-worker live).
Now the buffer and the margin come off the machine's total, capped at the
machine's budget: min(0.75 x total, total - buffer - margin), 1 byte only where
nothing is left. Here that is 31.5 GB. Chosen over MemAvailable because the
plan is decided from totals everywhere else (HostMemory.h), so the schedule
does not depend on what happens to be cached at the start of a collection; and
over a fixed floor because the floor would plan memory a misconfigured buffer
has already taken.
Scheduling only: what runs side by side, never what is computed. Live replay,
thau 1000 EIGER 16M frames at 100 Hz: the run now plans against 31.5 GB; the
post-refinement is no longer done before the merge, and "wait for the
all-observation merge" is 0.264-0.271 s -> 0.000 s. With the ISa bootstrap
change after this one, "Scaling and merging" is 1.88-1.94 s -> 1.56-1.62 s.
Merged MTZ, maps, models, blobs and every non-timing field of /rugnux/status
identical to the base binary's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
cudaHostAlloc and cudaFreeHost take the process's memory-map lock, 1-3 ms a call
here and 20-30 ms on a four-socket host (the comment above the pool). Thau
offline under nsys made 1417 cudaHostAlloc calls (4.76 s summed over threads)
and 288 cudaFreeHost (0.70 s). Counted by caller (gdb on cudaHostAlloc, with the
pool's kept bytes read at each call): 1107 of them from the image passes' engines
(BraggIntegrationEngineGPU 763, AdaptiveSpotFinderGPU 240, prediction, spot
extractor, decoder), 383 inside the canonical image pass, and every one a miss
while 135-830 MB sat in the pool - in other power-of-two classes (4 kB to
256 kB, set by each frame's reflection count), not in the one asked for.
None asked for more than 64 MB (the largest class used was the 64 MB ingest
stage), so CUDA_PINNED_POOL_MAX_BUFFER stays.
- The smallest class is now 256 kB (CUDA_PINNED_POOL_MIN_BUFFER), so the small
buffers of every engine share one class and one engine's are the next's.
- That class is cut from slabs of 64 (16 MB, one call): an image pass makes some
twenty per worker at once. A piece always goes back to the pool, and a slab to
the driver only when cuda_release_pinned_pool finds all its pieces back.
- A run lets the pool keep max(512 MB, host budget / 16) (cuda_set_pinned_pool_limit),
so the larger classes are not handed back to the driver mid-run for the next
step to take again (the 288 cudaFreeHost above).
- Live, the pool is released after the end message instead of beside it.
Thau, offline: 1417 -> 142 cudaHostAlloc calls (4.76 -> 1.53 s summed), 288 -> 0
cudaFreeHost; the image passes' engines 1107 -> 97 calls. Pinned memory held at
the end of the run 836 -> 970 MB. Wall here 15.18 -> 14.98 s with the other items
(the 5950X pays 1-3 ms a call); the server pays ten times that per call.
Outputs identical (thau, cytc, lyso offline; live replay).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The first pass's beam-centre check indexes again at the centre the background
puts the beam at (TryBeamCenter), and built that centre's azimuthal mapping on
its own path: 0.15 s on 32 threads for a 16M detector, before its spot finding
could start. The centre is known when the background-centre capture (started
at the end of the pre-scan) returns, well before the check reads it, so the
capture's thread now starts that mapping into the run's geometry cache
(MappingsAhead) and the check finds it there. The same experiment and centre
the check sets, so the same key; what the cache keeps does not depend on the
mask.
The mapping at the post-refined geometry was tried the same way and left out:
the geometry is committed 15 ms before the pre-pass ends, so there is nothing to
hide it behind, and live the pass after waited for it under the cache's lock
(0.25 -> 0.34 s setup). Offline the speculative probe at that geometry already
builds it first.
Offline, from the file-centre schemes' verdict to the measured-centre schemes'
(the mapping plus that pass's spot finding and indexing): thau 0.67/0.77 s ->
0.55/0.56/0.60 s; cytc 0.55/0.56 -> 0.44/0.44 s; lyso_x06da_ref 0.35/0.37 ->
0.35 s (a 2M detector). Outputs identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The bootstrap of the error model's b^2 (50 replicates) ran in line in every
full-statistics merge: 0.2-0.5 s a merge here, 36% of "rsm merge and
statistics" live. Nothing in the rest of the merge reads its interval - only
the report and, after the merge has returned, ErrorModelResolved - so it now
runs on a thread of its own from where it was (its pool, b^2 and thread count
taken by value; the pool is the merge engine's own, alive to the end of the
merge) and is taken, with its log line, at the end of the merge, before the
result is returned. Each replicate is seeded on its own, so the interval is the
same.
"rsm/m isa bootstrap", summed over the run's merges: thau 0.61/0.69 s -> 0.00;
cytc 0.66/0.67 -> 0.00; lyso_x06da_ref 0.33/0.33 -> 0.00; live (thau) 0.40/0.42
-> 0.00. "merge P1" (two merges): thau 0.53/0.80 -> 0.50/0.54 s, cytc
0.93/1.42 -> 0.86/0.89, lyso 0.77/0.78 -> 0.65/0.78. Outputs identical (the
report's ISa interval included).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
- Long-axis speculation: the re-indexes of the offered cells ran at once, one wide FFT pool each, beside
the measured-centre spot finding; with two offers on an 8 GB card the latter's allocations could fail
where they used to fit. They now run one after the other, each releasing its indexers when done, so no
more wide pools are alive at once than the hypothesis itself builds.
- Long-axis answer carried to later passes: where the beam-centre check asked at both centres, the run
recorded the adoption of whichever arm adopted, even if the other centre won. Each arm's adoption is
kept and only the one on the lattice the pass ends on is carried.
- Twinned arm beside the untwinned one: with the untwinned arm on the CPU (CPU-only build, or a model
too big for the card) nothing is held on a card and the memory gate let two full CPU refinements run at
once. It now waits in that case.
- Broker model warm-up: a CUDA error while loading the refinement modules skipped the whole validation
warm-up; it is now caught, cleared and logged.
- Live host budget: nine tenths of the machine, not all of it, is shared between the image buffer, the
broker's margin and rugnux, leaving a tenth to the system and other services on the host.
- First pass: the measured-centre speculation starts before the coarse long-axis search, since it is on
the critical path with two or more cards.
Results unchanged: thau, lyso and cytc reports, .hkl and refined models identical to rc184-port, also
with every pinned host buffer filled with 0xA5 on allocation (reused pool pieces carry no stale data
into results); the ~1000 A test sweep identical to the run without these fixes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
This reverts commit f51cc35c8, and with it the carried-answer selection of 2cd3da67b.
The answer is not the same in every pass. On 9rci (P1, c ~101 A) the first pass's coarse search at the
file geometry offers only a short cell and keeps the doubled ~200 A standing cell; the later passes, at
the refined geometry, offered the ~101 A lattice and the hypothesis adopted it. Asked once, the run kept
the doubled cell to the end (cell deviation 0.2% -> 98%, R-free 0.26 -> 0.40 in the full battery). Every
pass asks the hypothesis again, as before; the speculation that answers it from the memo stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
An azimuthal q range with no high limit reaches the detector's edge, which depends on the distance.
When the running geometry refinement moved the distance, the rebuilt mapping could have a different
number of bins from the start message and the run's profile. Adding the next frame's profile then
threw "Error combining two radial integration profiles" and cancelled the collection.
The q range is now frozen at its current value when the geometry moves. The replay tool adds up the
frames' profiles as the receiver does, so this path is exercised there; it ignored them before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit df861de7fb)
The running geometry refinement moved the distance during the measurement, so the radial profile's
binning and the frames merged depended on when the fit landed. It is removed, with the analysis threads'
rebuild and the frames held back from the merge: every frame is analysed and merged at the pre-pass's
geometry. Where the pre-pass could not refine the geometry, the header's distance stays for the whole
run, with a warning in the log and a note in the live status. The offline pipeline is not touched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d07e6e2aa9)
With a cell given, a pre-pass whose geometry fit did not reach its 200 frames now runs again on all the
frames held each time they have doubled, until the fit has its frames, half the image buffer is full or
the collection ends. A low hit rate, or a chip whose first frames are empty, then still gets a fitted
geometry where the buffer holds enough frames. The analysis starts when the pre-pass ends and catches up
from the held frames; the geometry stays fixed after that. Without a cell the pre-pass stops at the first
try, as there is nothing to fit. The replay tool logs when the analysis has caught up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ab54c514e5)
Rotation data are indexed in two passes, always: the online-like single pass
(--single-pass-rotation[=deg]) is gone with its min-angular-range argument,
and ProcessConfig::two_pass_rotation, which only that flag ever set false,
goes with it and with the conditions that read it.
(ported from rc184-2 5d24eac3c)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The interactive analysers now read a member copy of the worker's experiment
with rotation indexing off, so a rotation frame is indexed and predicted on
its own, like a still. They used to wait for the online rotation-indexer
counter, which is about to go; the rotation indexer itself needs the whole
sweep, which only a processing job reads, and those keep the flag. The copy
is refreshed wherever the analysers used to see curr_experiment change in
place (load, indexing settings, ROIs) and outlives the analysers.
(ported from rc184-2 ed86b9b29)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The live-stills settings function JFJochReceiverFPGA hands StillsGPUAnalysis
started from the broker's spot-finding settings, and IndexAndRefine gates
integration on their quick_integration - whose API default is false. With the
default, live stills integrated nothing and merged nothing. The analysis
settings now switch integration on themselves;
JFJochIntegrationTest_ZMQ_crystal_spot_and_index_GPUStills runs with
quick_integration off and checks that the live run reports a space group.
(ported from rc184-2 11eb76093)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The converter no longer passes indexing_settings.rotation_indexing and its two
angles into the experiment, so the receivers' IndexAndRefine never builds a
rotation indexer: rotation frames get the per-frame indexing stills get.
Output reports the old defaults (false, 20 deg, 0.5 deg). The receiver's
end-of-run rotation lattice (END message and scan result) goes; rugnux live
fills both for rotation runs, as it already did.
(ported from rc184-2 d279827a8)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
With --single-pass-rotation gone and the broker no longer asking for rotation
indexing, nothing reaches the counter-driven path any more: rugnux's rotation
run always forces the first pass's lattice (a failed first pass stops the run).
Removed: RotationIndexerCounter, the counter branch of
DetermineLatticeAndSymmetryRotation (and RotationAngle, its only user), the
RunIndexing fallback in FinalizeRotationIndexing, the uncalled
AddImageToRotationIndexer, and IndexingSettings' rotation-indexing minimum
angular range and stride with their entries in the rotation indexer's and
rugnux's memo keys (constants there, so no key changes meaning).
(ported from rc184-2 0d2a1c678)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Only rugnux runs the rotation indexer now, and it always passed false: the
candidate-cell refinements are bounded by iteration count, the post-search
fixed-point walk and tilt hold always run, and the memo key loses the flag.
IndexAndRefine keeps its own real_time, which bounds the per-image refinement
of the online receivers and live stills.
(ported from rc184-2 3d631d2f6)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
MXAnalysisAfterFPGA indexes each image (IndexAndRefine::IndexFrameOnly) and
no longer predicts or integrates; its Bragg prediction and CPU integration
engine go. Per-image reflections, integrated-reflection counts and B factors
are now left unset on this path - integration is rugnux's. jfjoch_fpga_test
loses -Q and the receiver-test helper its quick_integrate parameter. The two
processing tests that integrated now check that indexing happens and
integration does not (on rc184 they keep the HLS-simulated card they had).
(ported from rc184-2 68ec8a7f9)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The Lite receiver indexes each image and no longer integrates, whatever the
broker's quick_integration says: it calls the shared analysis with
quick_integration off. Per-image reflections, integrated-reflection counts and
B factors are left unset on this path, as on the FPGA one - integration is
rugnux's. jfjoch_lite_perf_test loses -Q. JFJochReceiverLite checks that
indexing happens and integration does not.
(ported from rc184-2 5f8e26e3d)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
indexing_settings.rotation_indexing, rotation_indexing_min_angular_range_deg,
rotation_indexing_angular_stride_deg and spot_finding_settings.quick_integration
are deprecated for one release: no longer required, accepted and ignored on
input, returned at their old defaults (false, 20, 0.5, false). The Lite and
FPGA workflows index each image and integrate none; rotation indexing and
integration are rugnux's. bragg_integration_settings now say they apply to
rugnux live.
Regenerated: broker/gen model, the frontend client, the Python client's docs
(docs/python_client) and broker/redoc-static.html (generator 7.20.0). The
frontend loses the "Rotation (3D) indexing settings" block and the "Quick MX
integration" toggle; etc/broker_local.json loses quick_integration. Docs:
JFJOCH_BROKER.md gains an "Online analysis" section (what the broker's own
analysis does and that rugnux fills the rest); HDF5.md, TOOLS.md and TESTS.md
follow. JFJochBrokerHttp_DeprecatedAnalysisSettings covers the API behaviour.
On rc184, where the FPGA workflow still has rugnux live for serial collections, the
"Online analysis" section says the broker analyses each image on its own there for rotation
collections.
(ported from rc184-2 ae4fa109e)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The sub-lattice hypothesis only asked the seven index-2 sub-lattices of the
standing cell, so a supercell of odd index could never come down to the true
cell. A small-molecule orthorhombic crystal with two axes below the FFT's
10 A floor was indexed in-plane on its (2,1,0) and (-1,2,0) rows - an index-5
triclinic supercell that indexes every spot - and merged in P1. It now asks
every sub-lattice of prime index 2, 3, 5 and 7 (7+13+31+57 frame counts),
with the dropped-node test generalised to 1/p translations and the chance
expectation to (p-1)/p of the nodes.
The nominee was also the first sub-lattice at the best frame count. On a
strong pattern several halvings tie at every frame, and the first one tested
can be a wrong one, whose dropped nodes hold half of the spots, so a doubled
monoclinic cell was kept and later read as C-centred. A tie is now broken on
the standing cell's spots on the nodes each candidate drops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 7f7d29b7010fb024826e9011bfcb7e084565fe5e)
An integrated rotation frame is fitted twice, each fit independent of the
other frames. The first fit is free and sets the frame's residual scale. The
second starts again from the sweep's geometry with a Gaussian prior on how far
the orientation and the beam centre move. The prior's two widths are measured
once per run, on the geometry pre-pass, as the rms of the seven-frame running
median of the frames' free shifts. This is a pass-level input, like the
pre-pass mosaicity. A beam centre that does not wander in the pre-pass is
therefore held near the sweep, and the data decide this per run without a
switch.
The orientation prior acts on the rotation away from the sweep orientation and
not on the angle-axis parameters. Applied to the parameters it was up to 2.4x
too tight, depending on how the crystal sits in the laboratory. Both fits use
XtalOptimizer's back-rotating form with the spindle given, so the beam-centre
component along the spindle, which on a single frame nearly trades against the
orientation, gets the same restraint it gets over a sweep. The frames the first
pass scores lattices on, the pre-pass and stills keep the free fit. The
residual scale counts 3N components and has no parameter correction (measured
as immaterial).
Squashes the sparse-eb prototype (2a54995fd, 8fb05679a, 51ae67467) onto
twin-subgroup-check, simplified as described above. On sparse small molecules
SHELXL R1 improved by a geometric mean of 3.943 -> 3.768 over 11 well-measured
sets. Protein REFMAC R-free and anomalous peaks stayed within noise. The
rotation-scale walk is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 045ae107d1c6b6ab6dc40773924426134dee69bf)
(cherry picked from commit 1f3c11f61c693e6eb5debbbb881245c69efe7472)
Two defects in the restrained per-frame refinement, both in what the geometry pre-pass measures.
The spindle form (XtalOptimizerOnSpindle) was applied to every integrated rotation frame, the
pre-pass's included, although the pre-pass is meant to keep the free per-frame fit. The pre-pass
takes the run's geometry decisions on per-frame fits - the beam-centre arm comparison, the metric-
symmetry check's held-out residuals, the post-refinement - and the spindle form changes how much of
a wrong metric a frame's orientation absorbs. The metric-symmetry check then lost its margin: a
tetragonal-promoted orthorhombic crystal (a and b 0.8% apart) kept a = b and ended in P1121, and a
monoclinic crystal was demoted to triclinic. With the spindle form switched off alone both decisions
return to the unrestrained ones; with the prior switched off alone they do not. The spindle form
now goes with the prior: only the frames refined under it use it, and the pre-pass is the
unrestrained one up to the integration of the pass after it.
A pre-pass frame whose fit failed (too few spots in tolerance) keeps the sweep's geometry and was
recorded as a zero shift. On sparse runs that is half the frames or more, and it pulled the running
median, and the prior with it, toward zero - on a run where one frame in sixty indexes, to the
1e-5 rad clamp. Only frames whose fit succeeded are recorded now.
Checked against the unrestrained build and 1f3c11f61, each with the first or last ten frames
dropped as well as on the whole sweep: the orthorhombic set is P212121 on all three frame sets
(1f3c11f61: P1121 on the whole sweep); the two-crystal I222 CCD set stays at ISa 15-19 on all three
(1f3c11f61: 8.1 / 18.1 / 8.1; unrestrained 13.4-14.4); the ice-ring monoclinic set is P2/P21 on all
three (unrestrained: P21, P21, P1). The small-molecule gains stay (kdp ISa 26-33 against 20-24
unrestrained, laspartic and lcystine R_meas below the unrestrained build's on every frame set).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 20f51f1cf8f05dc95cd8f845803a1f24f04abc2e)
The cell anchor (AnchorOnCells) let the crystals' own cells decide an operator once 90% of them fitted
the run's cell better as indexed, and then locked every crystal whose cell preferred its indexing. On a
P21 crystal with two pseudo-equal axes 0.2% apart that preference was close to 100% by construction: the
per-frame cells are refined under the monoclinic constraint in the indexing they were found in (alpha and
gamma held at 90) and put on an obtuse beta afterwards, so CellDistance(reindexed) - CellDistance(as
indexed) = 2 (beta_f - 90)(beta_run - 90) >= 0 for every frame. Measured: 95-100% "metric preference" on
all eleven pseudo-cubic operators, ~95% of the crystals locked in their random indexing, 135 of ~1600 left
to the intensities. The cells scatter by ~1.2% per axis, six times the 0.2% between the axes. A -z
reference could not overrule the anchor, and --mode scale -z did no reindexing against the reference.
Now each crystal's evidence is a log-likelihood of each indexing:
- CellMetric: its cell against the run's after each indexing, by shape (log lengths less their mean) and
by each angle's distance from 90 deg; an angle the refinement held at 90/120 deg is not measured, a cell
several crystals share exactly is the unrefined prior and says nothing. The per-parameter scatter is
fitted on the run by EM over the indexings, so an operator is told apart by the cells exactly as far as
their scatter allows - no METRIC_TIE / METRIC_DECIDED constants.
- Its intensities against all the other crystals: (n - 3)(zeta_right - zeta_wrong) atanh(CC), the Fisher-z
log-odds; zeta_wrong is the mean over the other indexings (the best of many noisy CCs reads as high as
the right one on weak crystals and weighed the intensities down to nothing).
The Brehm-Diederichs sweep makes the crystals consistent first; the cells - and a -z reference, with its
CC as a log-likelihood of the same form - then choose the run's indexing as a whole; then crystal by
crystal, sequentially as in the sweep (a simultaneous update drifted off the consistent answer on weak
data), each takes the indexing of highest summed log-likelihood. A crystal is decided by the data and its
cell, not by the reference: weighed in per crystal, a model reference cost accuracy (97% against 99.5%
with it fixing the frame), since it reads a weak crystal less clearly than the merge of all the others and
shares the crystal's noise with it. The per-image Resolve at integration time stays as the start; the
separate anchor step (AnchorReferenceReindex) is gone, and with it IntegrationOutcome::reference_cc.
The overall relabelling is restricted to operators that keep the merge's symmetry-equivalent reflections
together. Where the crystal's Laue group is not normal in the lattice's (monoclinic on a pseudo-tetragonal
lattice), the other coset representatives send crystals differing by the two-fold into different
indexings; the former "fewest moved" rule could take one and scramble a consistent answer (unit test:
243 of 300 crystals left consistent).
MergeStills (--mode scale, live checkpoints) runs the same resolution, with the reference where given.
ResolveReindexReferenceFree -> ResolveIndexingAmbiguity (it takes the reference). The live per-crystal
step (ReindexOnReference) and RelabelRunIndexing use the run's fitted cell model in place of the anchors.
Against model truth (per-frame CC with model Fc, frames the model decides at > 3 sigma), 4000-frame
subsets of pseudo-tetragonal P21 serial runs: given a right-shape cell 57% -> 99.5%; with the refined
geometry 99.7% / 98.4% (rc184 48-66%); --mode scale 64-66% -> 99.8-99.9%, and both halves of one run now
merge in the same indexing; with -z 80% -> 100%, 94.5% -> 99.1%, --mode scale -z (no reindexing before)
-> 99.9-100%; the run whose axes tie metrically stays 100%. Stills battery subset (7 sets, 5 with an
ambiguity): no regression; the pseudo-tetragonal P21 set R_split 0.170 -> 0.151, R-free 0.286 -> 0.284.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 790696641baab3954ffebb71b9a043f54e5d2dc6)
De novo, the stills consensus cell was a plain vote over the frames' cells,
and a frame's cell can be a supercell of the crystal's that indexes every
spot the true cell does: the centred conventional cell a pseudo-symmetric
metric suggests (a monoclinic cell with a ~ c refined as C-centred
orthorhombic, twice the volume), or face-diagonal rows from the FFT. On a
serial run of such a crystal two thirds of the frames came back as that
C-centred cell, it won the vote over the frames on the primitive cell, and
the run was merged in a cell with half its nodes empty (CC1/2 0.37).
The vote's winner is now asked, as the rotation first pass asks its
standing cell, which of its sub-lattices of prime index 2, 3, 5 and 7 drops
the fewest of the frames' indexed spots; that one is adopted where those
spots are no more than the frames' off-lattice spots put on the dropped
nodes by chance (the indexing tolerance sphere, (p-1)/p of the nodes) and
well short of what nodes as full as the kept ones would hold. The voting
frames then count with their sub-lattice cell and the vote is taken again.
On the run above: 119 of 33299 spots on the dropped (h+k odd) nodes against
123 by chance - adopted; on a run of the same protein in a tetragonal-
metric form, the best sub-lattice drops half the spots - kept, unchanged.
The frames' indexed spot indices are kept per frame for the test (stills
only). RelabelOnSublattice, which then puts the supercell frames on the
primitive cell, now logs the mean I/sigma of what it keeps and drops: the
dropped between-node reflections read 0.03 and -0.06 against 1.1 kept, so
it discards nothing real.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit a81851d36dc31b5e7d5cd09c408fc80c4bbb684d)
Three defects of the stills global geometry/cell fit, each moving the detector distance:
- The cell angles were held at the start cell (InitCellAngles, "constant in v1") while the lengths and the
distance were free, so a wrong angle went into the lengths and the distance: a monoclinic cell given with
beta 4 degrees off refined to lengths and a distance both ~4% long, and a cell given with beta held at 90
on a crystal at ~90.9 left the distance 1.3% short. The angles the crystal system leaves free (monoclinic
beta, triclinic all three) are now a refined block. The cell given is a Gaussian prior (1% on a length, 1
degree on an angle) weighed against the spots' ~1 px scatter, replacing the unused length regularizer;
the refined-vs-start deviation (scale, lengths, angles) and the distance move are logged. A cell move, not
only a distance move, now triggers the re-indexing refit.
- The module placement started from the file's geometry: on a run whose header distance was 3% short and
centre 6 px off, the module scan absorbed the centre into every module and the placement pushed two
second-ring modules 13 px outwards - a radius-dependent push the gauge does not hold, which trades with
the distance/cell-scale curvature - and the distance settled 0.8% long. The placement now starts from a
fit of the same frames with the modules held, each frame's lattice taken from it
(GeometryRefinerResult::lattices). Same data, right cell: the distance went from 1.0% long to
the value a reference model's R-free puts it at, the modules moved < 1.4 px.
- A cell given with the wrong shape still indexes the frames at a distance of its own and the fit
cannot leave it. With -C on stills the geometry refinement is now run both with the cell given and de
novo, from the file's geometry, and each is indexed on the same 96-frame sample with its own refined
cell and geometry. The cell given is kept where it puts >= 0.9x as many spots on the lattice, or where its
fit puts the detector within 2% of the de novo fit (a crystal form of the sample, possibly a minority one
the user asked for); otherwise the run is processed as without -C, with a warning. `rugnux -C <wrong
cell> file` now gives the result of `rugnux file`. Costs one more geometry-refinement sample on -C stills
runs (~+30-50 s on a 4000-frame run).
Docs: RUGNUX_CLI (-C, --refine-geometry), CPU_DATA_ANALYSIS_INDEXING 7.7, RUGNUX_MODEL_AND_REFERENCE.
Test: RefineGlobalGeometry_MonoclinicBeta.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit fe442ff33aeec7569de1fdf90076dba6b39a8e8f)
Fitted from the file's geometry, a right cell on weak frames settled as far off as the start (a start 3%
short ended 3% long), lost the comparison to the de novo cell and the run was processed as without -C. The
frames are now fitted de novo first, as without -C, and the cell given from the distance, centre and module
placement that fit found; the de novo fit, whose cell takes up the start's error, is the nearer start for any
cell of the sample. The same-detector rule counts only where the cell given was actually fitted. Where
nothing indexes de novo, the cell given is fitted from the file's geometry as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 610298396a0cb2169859922f554bcc652f78921d)
Started from the de novo fit's detector, a cell 5% too long with its beta 4 degrees off settled within 0.3%
of the de novo distance while putting 6% as many sampled spots on the lattice, so "same detector" did not
tell a minority crystal form from a cell of the wrong shape and kept the wrong one. The cell given is now
kept where it puts at least 0.9x as many sampled spots on the lattice as the de novo cell, and otherwise the
run is processed as without -C. Docs say that a minority form given with -C is dropped too where the de novo
cell describes the majority well enough to index materially more spots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 6e3e18cca5965481e032e09e2d956bb9d9766619)
A model of several chains is searched for as a whole and then, side by side, as one chain of each
molecule in it (chains with the same residue at every number they share are one molecule; the most
complete one is searched), with the PAE domains' machinery. The whole model's choice stands unless a
chain's search chose a placement better by the R-free margin (0.03), or the whole model chose none; the
chains that chose must agree on the space group. MR_SEARCH_MODEL names the winner ("chain A, one of 2
alike"). An experimental dimer from another crystal form, placed as one rigid unit where the target
crystal holds the two copies at another separation (tNCS), was placed 3.4 A off per chain (R-free
0.47); the chain search places both copies (R-free 0.28 before refinement, refined 0.27), and where the
dimer is right the whole model is kept (R-free 0.26 against the chain's 0.29). MR takes about twice
as long on such a model (0.55 -> 1.1 s).
With tNCS by half a lattice vector, the placement and its moves by (v + o)/2 are now told apart by a
paired per-reflection test of |F_obs - F_calc| on the weak reflections (3 standard errors, as the
equal-copies group comparison) instead of a 0.01 margin in R over them; UNDECIDED only where the best
is not ahead of every other by that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 690ee4784ddd8e046fec19edb558f28808d0a825)
Owner decision: the cell given (-C) is always the cell the run uses. The switch to the de novo cell and the
second, de novo geometry refinement (+20-110 s per -C stills run) are removed. What stays: the free cell
angles, -C as the prior of the global fit, and module placement from a held-modules fit of the same frames.
The frames' disagreement with -C is reported, never acted on, at almost no cost:
- the stills geometry trial (DecideStillsGeometry, already indexing a 96-frame sample with -C) also indexes
that sample de novo at the geometry it decided and warns, with both counts, where the de novo frames put
more than 1/0.9 times as many spots on the lattice as the cell given;
- RefineStillsGeometry warns where the frames refine the shape of the cell given (a length beyond the common
scale, which trades with the distance) more than three prior widths (3%) or an angle more than 3 degrees.
Docs: RUGNUX_CLI (-C), CPU_DATA_ANALYSIS_INDEXING 7.7, RUGNUX_MODEL_AND_REFERENCE.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 9b5531fe79012cef2f236b7eb29002f469ae8947)
Without the separate de novo geometry fit, a cell given fitted from a header distance 3% short on weak frames
settled 3% long, and the spot-count distance search, flat to 1-2% about the right distance, picked a peak 1.5%
off from which the refit could not recover (merge CC1/2 0.02). The stills geometry trial now also indexes its
96-frame sample de novo at the distance it decided; the distance at which the cell given has the volume of the
frames' de novo cell is a hypothesis of its own, taken where it puts >= 0.9x as many sampled spots on the
lattice and lies more than half the 1% fine step away. The same de novo cell, as a known cell, gives the
warning where it puts materially more spots on the lattice than the cell given (-C is still always used).
Cost: two trials on the 96-frame sample (about a second); nothing in live, which runs no geometry trial.
StillsTrial carries the consensus cell of a hypothesis indexed without a cell.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit 456ead9c802b2747437f74dcd06eed146cf260de)
A frame whose background region reads below zero - a detector module whose pedestal is off on that
frame - integrates reflections with a negative background variance, a sigma floored at one photon and a
net intensity of thousands (I/sigma ~8000). The unscaled merge the per-image scales are fitted against
weights such an observation ~100x above any real one, so it takes its reflection over; every frame that
sees that reflection and finds nothing there is then pulled towards a scale of 0 (the proportional model
error caps such an observation's normalised residual at ~3.3 sigma, so the Cauchy fit never rejects it
and its pull grows as G falls), lands under the 1/50 floor and is dropped, and the partiality refit
against the survivors drops more.
On a serial run whose detector shows this on one module every 324th frame, always on the same residue
of the frame number modulo 4, two same-arm quarters of the run (frames mod 4 = 0 and = 2) merged to
CC1/2 0.78 and 0.28. The collapsed frames were good frames (CC with the deposited model's Fcalc 0.29
against 0.34 for the kept ones, the same hand) whose CC with the scaling reference was 0.03: the
reference, not the frames, was wrong. Measured (frames mod 4 = 0 / = 2, run's geometry):
before: 446 / 1355 frames dropped, R_split 32.6 / 64.0%, <I^2>/<I>^2 2.08 / 2.74, <|L|> 0.475 / 0.788,
CC of the merge with the model Fcalc (d >= 2.5 A) 0.842 / 0.381
after: 446 / 412 dropped, R_split 32.6 / 35.9%, 2.08 / 2.17, 0.475 / 0.478, 0.842 / 0.816
The whole run: 9081 -> 3081 frames dropped, CC1/2 0.59 -> 0.955, R_split 18.6 -> 14.9%, Fcalc CC
0.879 -> 0.884 (d >= 2.5 A); two runs without the defect are unchanged.
AcceptReflection now refuses such an observation (a NaN variance, from older files, is kept), so it
leaves every stills scale fit, merge and ambiguity correlation, and the stills merge logs how many
observations on how many frames were left out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
(cherry picked from commit bd5c59915fd7698afcc0ef81722e562c78c5ee3a)
RUGNUX_MODEL_AND_REFERENCE.md still described the model as the per-crystal
indexing-ambiguity reference on stills given -C/-S; the code never uses
model intensities there (the ambiguity is resolved from the data and -z),
and the model only relabels the merged reflections afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The per-frame refinement prior log line added in 37c3661b3 used M_PI,
which MSVC does not define without _USE_MATH_DEFINES, breaking the
Windows viewer/rugnux build (C2065 'M_PI': undeclared identifier in
RugnuxImagePass.cpp). Use the portable PI from common/JFJochMath.h,
already included and used elsewhere in the same file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
A restrained rotation frame was pulled toward the sweep's lattice as the run carries it. Where that
lattice has a metric of its own - a free cell a few tenths of a degree off the symmetric one that the
per-frame fit holds it to (here a cubic cell refined slightly rhombohedral) - no orientation of the
held cell reaches it. The restraint's centre was then an orientation the fit cannot take, and with a
width narrower than the strain, every frame was pulled off its best fit along the strain axis. The misfit went into the intensities with a direction tied to
that axis, and the space-group search lost every cubic operator that moves it: F432 became R32, then
P42, against F432 on every frame subset before the restraint.
Now the restraint is centred on the sweep's orientation in the held metric - the free fit's lattice
turned to sit closest to the sweep's (least squares over the three vectors) - and the orientation
width is never narrower than the angle by which that best held orientation still misses the sweep
lattice's vectors. The pre-pass's recorded shifts are the same rotation with the strain left out.
Where the sweep's lattice already is in the held metric - the usual case - both are exactly what they
were.
Measured with --no-refine on the set and three frame subsets (all, first ten or twenty frames
dropped, last ten dropped): F432 on all four (432 contrast 0.86-0.92), against P42 / F432 / P42 before
this change. A two-arm test had shown the orientation prior and the beam prior both matter (either
off alone does not restore it consistently; a fixed orientation width of 0.25-0.45 deg does, wider
does not). The small-molecule and protein sets the restraint was introduced for are unchanged (kdp
ISa 32.8 vs 32.9, laspartic and lcystine R_meas identical, the orthorhombic two-crystal set and the
monoclinic ice-ring set unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
On a sweep too weak for the supercell probe to measure the lattice's own reflections, the doubled cell
was merged only where the added class stood clear of the six empty ones in at least two resolution
shells, each at z >= 3 and above twice the strongest empty class's z. Counted shell by shell, the
decision flipped on a small change of integration that left the evidence as strong as before: one
shell moving from z 3.5 to 2.9 while another rose from 3.4 to 4.8 (the summed z over the shells the
class lives in went up) took a doubled crystal back to its half cell.
The class's z is now pooled from the coarsest shell down (Stouffer's combined z over the resolved
shells, two at least - a weak sweep has what signal it has at low resolution), and so is the strongest
empty class's z; the doubled merge is made where some such pool gives the class a combined z of 5 and
twice the empty classes'. The decision then moves only as much as the evidence does. Over every
weak-sweep probe table in the recent battery logs it fires on all 29 runs of the doubled set (25
before) and on no other set. It no longer enters the doubled merge on the near-misses of a set whose
half cell is right (its doubled merge refused them anyway), nor on one sweep of a multi-sweep set the
battery scores as doubled against its reference.
The doubled merge then decides as before. On the set this was found on it is now entered again, but
under the current per-frame restraint its translation test refuses the doubled cell (the share one
pseudo-translation modulates reads 1.56 +- 0.79 against 0.91 +- 0.17 without the restraint), so the
result there stays the half cell; that is a separate matter of the doubled merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
A sweep whose isomorphism ratio leaves it between "the same intensities" and "another crystal" is
decided on the merge. It joined only where adding it lowered the held-out residual of the clearly
joined sweeps, or where it added at least 1% unique reflections while that residual was not worse
beyond twice its error. Same-crystal sweeps were left out on a median held-out change of +0.1-0.2%:
one at +0.12% +- 0.06% sat exactly on the 2 se line and fell off it on a small change of integration
that left its own merge as good as before (+0.11% +- 0.05%); another, whose bootstrap ratio error
went from 0.000 to 0.007 on near-identical data, dropped into the doubtful band, where adding fewer
than 1% new reflections it could only join by lowering the residual (+0.15% +- 0.03%).
Now a doubtful sweep also joins where the change, two standard errors up, is at most 1%
(MIN_HELD_OUT_DEGRADATION, one value for every run). A change that small is far below anything a
merge shows, and two standard errors up a sweep whose effect is not measured at all is not taken
for a harmless one: of the sweeps the joint-sweep battery arm left out as doubtful, the two
same-crystal sweeps join (both sets now join every sweep), and the small-molecule wedges with a few
reflections in common (+0.2% +- 4.4%, +21% +- 18%) stay out - admitting them, as a bare "change
below 1%" or "not significant" rule would, cost that set ISa 27 -> 23. The coverage path for a sweep
adding 1% or more and the exclusion of another crystal (ratio + 2 se below 0.97) are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The weak-sweep arm of the doubled-cell hypothesis decides on one translation's share of the added class
over all shells. That share was a straight-line fit across every added reflection, each in units of its
shell's lattice mean, so it had two defects:
- The class's rise with resolution passed for modulation: sin^2(pi h.eps) grows with |h| as the class
does, and the eps search found a direction to match. The class shuffled within its shells read a
share of 0.8-1.5 on a weak sweep and 1.0 on strong data - the test had no power.
- One added reflection the merge did not measure (merged at a thousand times its shell's mean, with a
sigma to match) set the eps, the share and its standard error; whether the doubled cell was adopted
came down to that one reflection's value.
The share is now read within the shells - x and y about each shell's own mean, one slope for all - and
each reflection is weighted by its spread about the shell mean plus its merged variance. The eps search
maximises that within-shell correlation. On the weak sweep the eps now comes out the same in every
perturbed run (dropping 0-4 leading frames), the share at 0.85-1.05 +- 0.13, and the shuffled class at
0.16-0.34; the per-shell shares of the probe arm are unchanged, and a strong doubled set still passes
them (weakest 0.83).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
With a cell given, the live stills pre-pass ran again on all the frames held each time
their number doubled, until half the image buffer was full. On a chip with a low hit rate
the sample grew to tens of thousands of frames; the last run on them took long enough for
the queue to fill, and every frame arriving until it ended was refused and not written.
The sample now grows to PREPASS_MAX_IMAGES (4 x PREPASS_IMAGES) or half of what the queue
holds, whichever is smaller, and frames that arrived during the previous run do not make the
next one larger than that: the last run is short and the other half of the queue absorbs the
frames arriving meanwhile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The negative-background-variance refusal lived only in AcceptReflection, which the rotation
merge does not call: its own admission test checked I, the correction and sigma only. Within
one rotation run the first-pass merge and the reindexing then saw a different population
from the final merge. The rotation merge now refuses such an observation too, and logs how
many it left out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The twinned arm runs beside the untwinned refinement and can share its card; an exception
in it (e.g. out of GPU memory) was rethrown after the untwinned refinement had finished,
failing the whole refinement and losing the twin arm's log. It is now caught in the twinned
arm and logged as "twin hypothesis not tested", and the untwinned result stands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The JFJOCH_BROKER.md example now parses and starts: one geometry per detector, EIGER trim
energies alongside the trim files, readout_time_ns, the required azim_int and image_format keys,
the pedestal RMS limit under image_format where it is read. module_placement entries are
described with all eight required fields. IMAGE_STREAM.md gives TCP protocol version 5, the
FILE_CLOSED ack, the 64-bit FIFO occupancy, the liveness-based END wait and the two optional tcp
timeouts; the metadata stream also drops reflections. SECURITY.md and the broker page list the
bearer-protected /rugnux/model.cif and /rugnux/map.ccp4. The etc/ examples drop the deprecated
numa_policy and a misplaced pedestal RMS limit.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The refinement pages no longer list TLS, anisotropic U, NCS restraints, twin refinement and
multi-domain twins as missing; water building is named as opt-in, the TLSMD cuts as 4/8/16 and the
Apple GPU's share (the model density only) as it is. HDF5.md: the detector depends_on target, the
reflection flag datasets, input_unit_cell and grid_scan under /entry/sample, which /entry/MX arrays
are linked and which copied into the master, the integrated-format /entry/detector split, the
detectorSpecific extras and the integer dtypes of the per-spot arrays. CBOR.md: the metadata message
carries every image field but spots/reflections/data, roi_integrals is keyed by ROI name, the START
roi_map, ROI names, Smargon position and the END space group are documented, scalars are stated to be
optional, user_data is a JSON string, and the zstd variants share one tag.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
-X is not consulted on rotation data and FFBIDX without a cell indexes nothing; the stills
integration radii, the fixed rotation min-pix, the SFRM/CCD imgCIF --gain inputs, the `he` alias
and the accepted ranges of the numeric options are stated. A CBF/SMV/marCCD sweep is named by any
of its frames, and --model needs no -C.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Image analysis: the beam-stop size rule, defective-pixel finding on every run (hot/unstable rules
under rotation only, CCD included), the adaptive threshold keeping the local SNR test, the live
stills analysis forcing it on the FPGA workflow, and what 0 does to each ice gate. Resolution cutoff
extends by a tenth of the kept range; time-dependent absorption bins per 10 degrees; outlier
rejection and joint multi-sweep merging described; the sub-lattice hypothesis of the rotation first
pass described. Stills keep the hands under -A; SWEEP_REFERENCE is the finest-resolution sweep;
ffbidx is vendored; calibrants other than ice are cells with reflection conditions. Writer: output
files are checked at START and PREFLIGHT; republishing is ZeroMQ-only. Deployment: packages for
el8/el9/Ubuntu, the installed frontend path, optional port. FFTW is fetched, not a host library;
jfjoch_portable_test and the four unlisted tools named; rugnux installs ACKNOWLEDGEMENT.md;
battery.py --gpulock is a `run` flag. The update_monomers.py path is given in full in the notices
(docs copy regenerated with update_version.sh's step).
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
- tcp_settings requires image_socket and pcie_devices items require blk (the
listed addr/block_device did not exist); dark_mask_settings no longer lists
a non-existent frame_threshold_keV.
- pcie_devices and the detector list use minItems instead of minLength.
- readout_time_ns is described in nanoseconds; detector hostnames are one per
module for JUNGFRAU and two per module for EIGER.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The test cancelled a 900-image live stills collection on the simulated device and required the
stop within 120 s. After a cancel the simulated FPGA still processes, at CPU speed, the frames
it has already taken into its memory, and that alone took 75-80 s on a workstation and over
300 s on the loaded CI machine, where the check failed. The receiver itself was idle, waiting
for the device.
The collection is now 30 images, still cancelled while its frames are held for the pre-pass
(20 of 30 held at the cancel), and the test checks that the stop returns and reports the
collection as cancelled, instead of timing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
GCC 8 (the RHEL/Rocky 8 system compiler) configured but failed minutes into
the build with "CUB requires at least C++17". The configure now refuses it and
names gcc-toolset-13.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Rewrite ten passages of the Rugnux pages in plain, direct sentences and rename the
headings written in the "What X does" style: Output files, Terminal output, Manual pages,
The three analysis tools (RUGNUX.md), Overview (RUGNUX_OVERVIEW.md), Input formats
(RUGNUX_FORMATS.md), Out of host memory (RUGNUX_INSTALL.md), Reference data
(RUGNUX_ADVANCED.md), The model and the space group (RUGNUX_MODEL_AND_REFERENCE.md), The
disposition column (RUGNUX_REPORT.md), with every link text and anchor that named them.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Rewrite the top-level README for someone who has never seen the project, ordered as a
code-and-software checklist: system requirements of the released binaries (OS and glibc
floors, driver and GPU generations, memory and disk measured on the demo, the platforms
tested), installation from the release page with a VERSION variable and the measured install
time, a demo that downloads five public datasets from their original repositories (Zenodo,
MX-RDR, IRRMC, SBGrid, CXIDB) with sizes, checksums and the exact commands, a 9ZLO
walk-through with the expected report keys and tolerances and the CPU-only time, the other
rotation sets and a SACLA serial example, instructions for own data, how to reproduce the
paper's battery results from the Zenodo record, building from source for developers, and
the licences. Every command was run as written with the released 1.0.0-rc.183 binary; the
numbers are from those runs.
The documentation's Quick start (RUGNUX.md), A first run in detail (RUGNUX_TUTORIAL.md) and
Installing Rugnux walk through the same 9ZLO dataset with the same commands and numbers, so
the README is the condensed version of them. The serial test-data page's euxfel83 row now
describes the 30 hit files the battery runs.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The usage message said "Give rugnux the data and nothing else: the defaults are meant to be
the answer"; it now says "Most datasets need no option beyond -o", in step with the
documentation's Quick start. The closing "Files" line of a run omitted <prefix>.ins, which
every merging run writes beside its .hkl, because the extension was missing from the list of
files counted as the run's own.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The Q bins can now be laid out in equal steps of log10(Q) between low and high Q,
dense at low Q and coarse at high Q, as small-angle work needs; such binning is
given by its number of bins. Linear binning takes either its spacing (as before)
or its number of bins, not both. Linear binning by spacing bins every pixel and
reports every bin centre exactly as before.
The Q axis (bin_to_q, written to CBOR and HDF5) holds each bin's centre on its
binning scale: the geometric mean of the edges for log10 bins. The ring background
of the Bragg radial correction and the ice-ring score read the bin centres instead
of assuming a constant spacing. A q rounding onto the far edge of the last bin now
stays in that bin instead of spilling into the next azimuthal row.
rugnux: --azim-q-binning linear|log10 and --azim-q-bins; --mode calibration
refuses log10 bins, as its ring search works in equal Q steps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
q_binning (linear|log10) and q_bins are read and returned; q_spacing is now
optional. Linear binning takes q_spacing or q_bins, not both (neither keeps the
0.01 1/A default); log10 needs q_bins and refuses q_spacing; a refused combination
is a 400 with the reason. GET returns the spacing or the bin count that was set,
and the binning. Clients that send only q_spacing work unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The azimuthal-integration panel offers linear binning by spacing, linear binning
by number of bins, and logarithmic (log10 Q) binning by number of bins.
A number field no longer rewrites the text being typed when the value it just
reported comes back from the parent: a spacing with more decimals than the field
displays (0.000875) or written in exponent form (8.7e-4) stays as typed and is sent
as typed, instead of the text being rounded to the displayed decimals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The low-Q headroom check and the detector-edge high Q are floored by the linear
spacing only where one was given, and by the smallest allowed Q otherwise, instead
of by a spacing log10 bins do not use. GetQSpacing_recipA says that under log10 it
means nothing about the bins. GetQBinEdges_recipA, which only the tests used, is
dropped: the tests state the edges from the documented formula. A test pins the
far-edge clamp of linear-by-spacing bins on a two-row setting. The broker docs note
that a GET carries q_bins and no q_spacing once bins are configured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The README and the Rugnux user pages state what a user gets: the install
and demo timings are given once, rounded, without the machine-by-machine
download and configure times; the memory figures are single upper bounds
instead of ranges; and the Phaser, SHELX and SHELXT walk-throughs describe
the behaviour without the numbers of one particular run. Version history
(package moves, older file layouts, former defaults), battery-internal
counts and corpus medians are removed from the user pages.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The data-analysis reference keeps the rationale for each decision but no
longer quotes battery arms, per-set counts and medians, or the history of
how a default came to be. The deployment, repository, release-contents
and writer pages describe the current packages and formats without the
renames and removals of earlier versions.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The paper's figures stand as the published ones. Where the README, Quick start, tutorial
and benchmarking page quote run times for the paper's datasets, they now say that the
paper's Extended Data Table 1 was measured with an earlier release that placed the model
but did not refine it, and the CPU-only time of the walk-through is given as minutes
rather than a figure measured on a different release. The demo table's R-free column says
what it compares, which is not the paper's REFMAC comparison. The battery counts match the
paper (207 rotation datasets, eleven serial datasets) with a note that the rotation
collection has grown since; the serial page lists the files and frame counts the still
battery actually reads for CXIDB 90, 152, 242, 103 and 213, which are those of the paper's
Extended Data Table 2; the 4NWV start model's shift is given per axis as in the paper;
the acknowledgements name all five AlphaFold start models of the timing benchmarks; the
hardware page names the Rugnux-live server of the paper's beamline test.
Co-Authored-By: Claude Fable <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Since the overlap test keeps integer sub-cells of an accepted cell (9b4acd6ab), a half-volume primitive
cell whose metric is C-centred monoclinic reaches the candidate selection. Its conventional C cell is
the true primitive cell, and IndexedFraction counted spots on that conventional cell ignoring the
centring, so it indexed exactly what the P cell indexes; the sub-cell rule, comparing primitive
volumes, then took the C label. On a monoclinic P crystal with no centring pseudo-symmetry (h+k odd as
strong as h+k even at every resolution) the first pass came out "monoclinic C", and with a
user-fixed P 1 21 1 the run was refused by the fixed-group lattice check.
A spot now counts only where its rounded conventional index is a node of the centred lattice (an
integer index on the primitive cell). A genuinely centred crystal is unaffected: its spots obey the
centring. Measured: 14 witness sets (I, F, R, C lattices, small molecules) give the same first-pass
lattices and merging results; the affected set merges in P 1 21 1 with and without -S.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The check that a user-fixed space group fits the indexed lattice refused every centring mismatch. A
primitive group on a centred lattice is not a frame conflict: with a fixed group prediction runs in
that group's centring, so the reflections the lattice centring would extinguish are integrated on the
conventional cell and the merge in the group's frame is well defined. The group is the user's choice,
so the run now warns - naming the centred lattice and cell the indexing found - and merges in it.
A centred group on a lattice that does not carry it, and a cell the group's operators do not leave
invariant, still stop the run: there the reflections are not in the group's frame (completeness above
100 % is the measured symptom).
Measured: a monoclinic P crystal whose first pass was labelled C (without the indexer fix) now merges in
P 1 21 1, 100 % complete; a genuinely C-centred monoclinic crystal run with -S P2 warns and merges in P2,
95 % complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
Cut the README to the essentials: what the programs are, requirements,
installation from the release archive, the 9ZLO demo with its run time,
terminal output and output files, the other example datasets as a table
with their sources, how to run your own data, reproducing the paper's
results from the Zenodo record, a short build-from-source pointer, the
licence and the citations. The FPGA cards serve JUNGFRAU and the PSI
EIGER; DECTRIS detectors need none. The Quick start cross-reference to
the README is adjusted to what the README now holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Short sentences throughout; the FPGA cards serve JUNGFRAU and the PSI
EIGER, DECTRIS detectors need none; requirements as one paragraph with
the tested OS and the install time; the command block before the format
list under "Your own data"; the build example without -march flags; the
paper-timing remark as one sentence here and in the Quick start. The
terminal summaries list the .ins beside the .hkl, as the program writes
them, in the README and in the tutorial.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The title is "Repository content" and the opening paragraph states what
the tools are for - crystallography processed fast, at the beamline and
without parameters - before Jungfraujoch and Rugnux are named. ARM
spelled out beside aarch64; the install examples name 1.0.0-rc.184 here,
in the Quick start and on the install page; two sentences on archive
size and download time dropped. Rotation and serial data are told apart
by the associated metadata; a model does not affect the intensities or
the lattice. The licence section says the software is free for industry
and academia and that modified distributions stay under the GPL, and a
short Acknowledgements section points at docs/ACKNOWLEDGEMENT.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README: new title, the introduction reduced to the mission paragraph, the stills example with a
real cubic cell and space group, the -A and --no-gpu lines dropped, and the GPUs and switch of the
acquisition server marked optional as the hardware page has them.
The model sentence in the README and the Quick start now agree with the code: the model does not
change the processing, it can settle the hand of the space group, and a different group it prefers
goes to <prefix>_mr.mtz. The pages that described -A as changing the counting basis and the error
model on rotation data now say what the code does: the rotation merge is the same with and without
it, and only FRIEDELS_LAW in the report changes; on stills it is what keeps the Bijvoet hands apart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The detector-frame surfaces are held to a mean of one within each resolution shell, because a factor that is a function of d alone is a null direction of their fit. That mean was weighted by 1/sigma^2, and sigma is smallest where little light arrives: on a detector part-shadowed by the pin the dark cells carry the shell's mean, and their large factors go onto the lit cells as small ones. How much of a ring is shadowed varies with resolution, so this bends the merged intensities along resolution: REFMAC's fitted scale against the model ran 1.48 / 0.80 / 1.16 in F (1.65 A / 2.7 A / inner) on a half-shadowed lysozyme sweep, with R as good as XDS's within every shell and 0.02 worse over all, and no shell-wise statistic can see it. Each observation now counts once. REFMAC R-work / R-free on the depositor's flags (d_min of the baseline), model as deposited, 31 sets chosen by the largest removed ramp and clamped cells, largest R vs per-shell R gap, plus controls: 5 better by more than 0.0015, 23 the same, 3 worse. half-shadowed lysozyme 0.3266 / 0.3306 -> 0.2996 / 0.3069 (XDS on the same model 0.3036 / 0.3083) soft X-ray insulin 0.2720 / 0.2841 -> 0.2423 / 0.2570 4p8r 0.2143 / 0.2518 -> 0.1873 / 0.2216 (depositor's data 0.2216) 6i3j -0.0029; insulin reference -0.0017; 2wnz +0.0025, 6jgh +0.0016, 5j23 +0.0015. The ramp removed on the first falls from 186x to 2x. The merge's own hash test records change as they must. A mean over the ring still lowers its lit part by the shadow it averages in (the new test reads 0.72 where 0.57 before and 1 would be exact); a median of the counts did not do better on the half-shadowed sweep (R-free 0.3090), the only one it was measured on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQs733XDnbQQyUbV1vsWith modules x images at or above what the card's HBM holds for the internal frame generator (64 module frames in the HLS simulation, 1024 on the card), frame_generator refuses the run and sends nothing, and the collection hung in Measuring with no error. The host now checks this: PrepareAction refuses such a run at the start (a 32-bit EIGER frame counts twice, as in frame_generator.cpp), and SetInternalGeneratorFrame refuses an image that would land there, each with an error that says how many module frames fit. The capacity comes from hbm_size_bytes of the card's status (the mock keeps its 64). The check keeps the last module frame free as well: load_calibration.cpp accepts module_id == 2 * hbm_size_bytes / module size (">" where ">=" was meant), one past the memory. The firmware is frozen, so the host never sends that index instead. The HLS simulation keeps frame_generator's return code: a refused configuration cancels the data collection rather than waiting for frames, and HLSSimulatedDevice::FPGA_EndAction throws with the code. rc184 has no MockAcquisitionDevice: its part is left out, and the test checks HLSSimulatedDevice only, as HLSSimulatedDevice_InternalGeneratorMemory in FPGAIntegrationTest. (ported from rc184-225542a9af) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAKA model of several chains is searched for as a whole and then, side by side, as one chain of each molecule in it (chains with the same residue at every number they share are one molecule; the most complete one is searched), with the PAE domains' machinery. The whole model's choice stands unless a chain's search chose a placement better by the R-free margin (0.03), or the whole model chose none; the chains that chose must agree on the space group. MR_SEARCH_MODEL names the winner ("chain A, one of 2 alike"). An experimental dimer from another crystal form, placed as one rigid unit where the target crystal holds the two copies at another separation (tNCS), was placed 3.4 A off per chain (R-free 0.47); the chain search places both copies (R-free 0.28 before refinement, refined 0.27), and where the dimer is right the whole model is kept (R-free 0.26 against the chain's 0.29). MR takes about twice as long on such a model (0.55 -> 1.1 s). With tNCS by half a lattice vector, the placement and its moves by (v + o)/2 are now told apart by a paired per-reflection test of |F_obs - F_calc| on the weak reflections (3 standard errors, as the equal-copies group comparison) instead of a 0.01 margin in R over them; UNDECIDED only where the best is not ahead of every other by that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK (cherry picked from commit 690ee4784ddd8e046fec19edb558f28808d0a825)A frame whose background region reads below zero - a detector module whose pedestal is off on that frame - integrates reflections with a negative background variance, a sigma floored at one photon and a net intensity of thousands (I/sigma ~8000). The unscaled merge the per-image scales are fitted against weights such an observation ~100x above any real one, so it takes its reflection over; every frame that sees that reflection and finds nothing there is then pulled towards a scale of 0 (the proportional model error caps such an observation's normalised residual at ~3.3 sigma, so the Cauchy fit never rejects it and its pull grows as G falls), lands under the 1/50 floor and is dropped, and the partiality refit against the survivors drops more. On a serial run whose detector shows this on one module every 324th frame, always on the same residue of the frame number modulo 4, two same-arm quarters of the run (frames mod 4 = 0 and = 2) merged to CC1/2 0.78 and 0.28. The collapsed frames were good frames (CC with the deposited model's Fcalc 0.29 against 0.34 for the kept ones, the same hand) whose CC with the scaling reference was 0.03: the reference, not the frames, was wrong. Measured (frames mod 4 = 0 / = 2, run's geometry): before: 446 / 1355 frames dropped, R_split 32.6 / 64.0%, <I^2>/<I>^2 2.08 / 2.74, <|L|> 0.475 / 0.788, CC of the merge with the model Fcalc (d >= 2.5 A) 0.842 / 0.381 after: 446 / 412 dropped, R_split 32.6 / 35.9%, 2.08 / 2.17, 0.475 / 0.478, 0.842 / 0.816 The whole run: 9081 -> 3081 frames dropped, CC1/2 0.59 -> 0.955, R_split 18.6 -> 14.9%, Fcalc CC 0.879 -> 0.884 (d >= 2.5 A); two runs without the defect are unchanged. AcceptReflection now refuses such an observation (a NaN variance, from older files, is kept), so it leaves every stills scale fit, merge and ambiguity correlation, and the stills merge logs how many observations on how many frames were left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK (cherry picked from commit bd5c59915fd7698afcc0ef81722e562c78c5ee3a)