RugnuxFarEnd.cpp used M_PI, which MSVC does not define without
_USE_MATH_DEFINES, and broke the Windows viewer build (C2065). The
ResultReport_SupercellCheck test still looked for the supercell warning's
wording from before the doubled-cell change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
TCPStreamPusher::SendImage is called from the receiver's frame-transformation
threads. It used a 128-deep per-connection queue and waited when it was full.
With images routed per file ((n / images_per_file) % writers), every writer
receives a whole file as a burst at the full frame rate (1000 images/file at
2 kHz fills 128 in ~64 ms), so the queue filled on every burst and all
transformation threads ended up waiting on one writer. They then stopped
returning FPGA frame buffers, and the FPGA dropped packets: partial images,
images without a background estimate, data loss on every writer. The ZeroMQ
pusher has an unbounded queue and never blocks, which is why it coped.
- The per-connection queue is unbounded: every element owns a distinct
image-buffer slot, so the bound is the image buffer's slot count. When slots
run out the receiver drops only the write (GetImageSlot() == nullptr), after
analysis. Absorbing one file burst therefore needs an image buffer of about
images_per_file slots (~20 GB for a 9M at 1000 images/file); below that,
images are still dropped, now as writer-queue-full instead of packet loss.
- SendImage never blocks. A dead or wedged writer is still caught in SendAll
(15 s silence / 60 s without progress), after which WriterThread returns the
queued slots unsent.
- The 200 ms fallback in StopDataCollectionThreads is gone: at END it closed
the connection of any writer whose queue stayed full for 200 ms - a normal
state for a slow but healthy writer - discarding up to 128 queued images and
never sending END. END now follows the whole backlog.
- A per-connection enqueue mutex makes "check active, enqueue" atomic against
"clear active, enqueue the end sentinel": an image could land behind the
sentinel when a connection was torn down mid-run, leaking its slot
(CheckIfBufferReturned then threw) or being sent as stale data in the next
run. New stress test SendImageRacingEnd_NoSlotLeak (verified to fail with
the race widened and the lock removed).
- TearDownConnection marks the connection broken first, so the destructor
returns a backlog instead of sending it.
Writer side:
- jfjoch_writer -b/--rcv_buffer sets the TCP receive socket buffer (was
hardcoded to the kernel default).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The model scale searched b_sol only over 10-80 A^2. That is the range
Fokine & Urzhumtsev report for models at ordinary B, but on low-resolution
entries whose deposited B is 200-400 A^2 the fit sat on the 80 A^2 ceiling
(and k_sol on its 0.1 floor): a solvent edge far sharper than the atoms it
lies against, reaching too deep in resolution. R-free came out 0.05-0.10
above REFMAC (0 cycles) and phenix.model_vs_data on the same MTZ, model
and flags.
The ceiling is now max(80, 2 x the model's mean atomic B) (SolventBMax),
for the reported fit and for the rigid-body placement's per-zone solvent
(CPU, GPU and the host fallback alike). The grid keeps eight b_sol values
across a wider box, so it costs no more points; the default box, and so
every model with mean B <= 40 A^2, is bit-identical to before.
Measured (10 open-arm sets with --model, GPU build, vs the same base):
high-B set, 7.0 A: R-free 0.462 -> 0.364, b_sol 80 -> 379 (REFMAC 0.366,
REFMAC's own b_sol ~371; model_vs_data 0.291);
the rigid-body step now commits
high-B set, 3.9 A: 0.410 -> 0.331, b_sol 80 -> 388, k_sol 0.10 -> 0.225
(REFMAC 0.328, model_vs_data 0.314); null 10.1 -> 14.2 sigma
B~200 set, 3.2 A: 0.408 -> 0.402, b_sol 80 -> 120; enantiomorph label
decision unchanged (27.2 -> 25.8 sigma)
7 other sets (mean B 16-57): bit-identical R, k_sol, b_sol, placement.
p.hkl md5 and the MTZ reflection data are identical to the base on all 10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A CUDA out-of-memory in an indexer worker was turned into an IndexerResult carrying only the
message string. The rotation indexer then returned no lattice, the first pass treated that as a
genuine negative and the rescues acted on it: in a stress run (two rugnux on one card) a run had
both first-pass schemes fail with out-of-memory, the axis-sign rescue compared 0 frames at the
flipped sign with the -1 "no lattice" sentinel, adopted the flipped sign, and the beam-centre rescue
then moved the centre - a failure of the machine used as evidence about the crystal. The CUDA
category was lost on the way, so the fail-fast check (IsFatalResourceError) never saw it.
- IndexerResult::error is now the std::exception_ptr the worker caught. RotationIndexer::RunIndexing
rethrows it before touching any state, so nothing is memoised for a failed run and the exception
keeps its category: rugnux stops with the CUDA message at once (verified with a forced indexer
failure: exit 1 after 1.9 s, "CUDA (GPU) error (...)"). FirstPass::indexer_error,
RotationIndexer::GetIndexerError and RequireLattice's machine-failure branch are gone.
- The beam-centre arm in RunAllPasses rethrows fatal resource errors like its sibling catches; it
had caught the out-of-memory and carried on at the file's centre.
- IndexAndRefine keeps per-frame indexer failures apart from frames that did not index (count +
first exception). The frame stays "not attempted" as before; the receiver logs the count and the
first error at the end of a collection (no cancel), rugnux's image pass and stills geometry
refinement rethrow it and stop. A rotation-indexer failure inside IndexAndRefine (online rotation
indexing, FinalizeRotationIndexing) is recorded the same way, so a live collection is not
cancelled by it.
- A failed device allocation or cuFFT plan now says how much was asked, how much of the card is
free, and - when the card is merely full - that another program is using the GPU or the data set
needs more than the card has, with the CPU remedy; previously "Failed to allocate device memory
(out of memory)" / "Failed to create CUFFT plan with cufftPlanMany".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Narrow, guarded: the doubled pass runs only where the added class grows with resolution and is resolved; the doubled
cell is adopted only where one pseudo-translation explains the added class in every shell and the point group does not
rise. Fixes the one remaining lattice failure (half cell -> deposited cell, R-free 0.574 -> 0.275); p.hkl identical on
the ci tier, the supercell gate set, 78 further supercell-candidate sets and all private sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A shell is read only where at least half of the class's reflections are clear of the lattice's by r3.
On a 701 A axis 26-43 of ~36000 per shell were clear; they read at the lattice's own strength, the
doubled merge (93% of predictions overlapping a neighbour) passed the one-translation test, and the
run was adopted doubled and failed. With the rule, of ~134 sets with probe logs only 9min and one
I-centred crystal reach the doubled pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
On strong data a class at one or two percent of the lattice rises significantly (0.009 to 0.059 at 4
standard errors on a ci-tier triclinic crystal, which was then adopted doubled and failed), and a half
cell missing a few percent of its intensity still describes the crystal. The doubled deposition ends
at 0.35-0.40, those classes at 0.06.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Exact: p.hkl md5 and MTZ data identical on every real set (34 open/inhouse incl. the ci tier, all 27 private);
both no-crystal controls now ~7 decades inside the bar instead of 0.3-0.9.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Exact: p.hkl md5 and MTZ reflection data identical on 18 sets. 4nwv R-free 0.545 -> 0.297 and cell in the depositor's
frame; written cells corrected on 9rcs, 9b22, 9qw8, myob_x06da_powder_1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
- ModelValidation reads the model and, where it carries non-given NCS operators
(_struct_ncs_oper 'generate' / MTRIX without the given flag), expands them
(gemmi expand_ncs, vendored assembly.cpp from GEMMI 0.7.5, MPL-2.0 as the rest
of gemmi_gph). Same in the battery's model_check.py and depdata_check.py.
An icosahedral-virus entry deposited as 3 of the 45 chains of its ASU was
scored against a fifteenth of its atoms: R-free 0.545 -> 0.297.
- The indexing the model chooses (AdoptModelFrame) now moves the cell with the
indices (reseat_integrated in the main path; CellInBasis in the forecast,
twin-file and stills/scale paths). A near-symmetry law (b within 2% of c, beta
a degree off 90, a~b in P1) relabelled the reflections only, so the files were
written with two axes swapped or beta flipped relative to the indices. Where
the law moved the cell, the validation is remade on the moved cell.
Measured on 18 open/in-house sets with models: p.hkl and MTZ reflection data
bit-identical on all; cell and model R unchanged where the law is exact; the
cell now matches the deposited one on the 6 pseudo-merohedral cases (orthorhombic
a/c swap, three monoclinic beta flips, a P1 alpha/beta swap), R-free within
+-0.001 there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Doubled-cell hypothesis: the run now always merges the doubled cell under <prefix>_doubled first and
decides with both merges in hand. Adopted only where (1) the probe's class grows significantly with
resolution (three standard errors between the 20-8 A shell and the finest measured one) and (2) in
the doubled merge one pseudo-translation - half a cell plus eps, fitted over the class - modulates at
least half of the added class in every shell (FitDoubledClassTranslation: A sin^2(pi h.eps) + B per
shell, share 1 - B/mean). Where adopted, every <prefix>_doubled file takes the plain name and the
probed cell's file of that name moves to <prefix>_halfcell. Replaces the HALF_CELL_BREAKS_DOWN bound.
Measured on the forced doubled merges: modulated share per shell 0.80-1.0 out to 1.9 A on the crystal
deposited in the doubled cell; on the ones deposited in the half cell it falls to 0.01-0.34 by
2.1-2.8 A (the rising class with twinning), or stays at -0.3-0.48 at every resolution (three others);
the one class it fully explains does not rise (eps 0.1 of a cell, flat at 0.1 of the lattice).
Space-group search: the trim of a screw zone's largest absence exempted a violation "at its row's
mean" on the de-modulated intensity, so under a deep pseudo-translation a reflection at 0.015 of its
row (de-modulated by 1/0.002) could never be trimmed and alone decided the row against the screw.
The exemption now reads the measured intensity, at both sites.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Where the first pass recognises two twin domains on their coincidence lattice (reticular
merohedry, index n = 2 or 3) and adopts the stronger domain's cell, the 1/n of that domain's
reflections that lie on the other domain's lattice too carry both domains' intensity. The step
now records the overlap as a lattice vector t of the standing cell (h . t divisible by n), the
rotation result carries it, and the per-image integration drops those reflections. t is a
real-space vector, so it holds through every later change of basis; it is read in sixths so a
centred conventional cell (F, I, C, R) is exact. Each unique reflection keeps its equivalents
that the other domain does not overlap; only those whose every equivalent is overlapped are lost.
The two-pass quality guard compared signal counts between a header pass that integrated
everything and a refined pass that left the overlap out, and sent the run back to the
composite lattice. A pass that left an overlap of index n out now has its counts taken up by
n/(n-1), and its axial rows are not compared with a pass that did not.
Nothing changes where the step does not fire: the field stays empty, the factor is 1.
Measured (merging the existing outputs with and without the class H.(111) = 0 mod 3, against
removing a random third): spinel-twinned F-cubic private set R_meas 25.5 -> 14.3% (random
26.7%), insu_I_x06da_low_isa 24.1 -> 16.6% (random 24.2%). With this change, end to end:
- insu_I_x06da_low_isa: I23, R_meas 24.0 -> 12.7%, ISa 5.2 -> 28.8, CC1/2 0.997 -> 1.000,
completeness 99.8 -> 95.7%, d 1.45 -> 1.47 A, model R-free 0.260 -> 0.246.
- private F-cubic twin: F432, R_meas 25.5 -> 6.3%, ISa 3.9 -> 20.4, completeness 100 -> 93%,
d 3.23 -> 3.77 A (XDS reference 4.0). 432 now passes the added-operator gate on the
all-observation merge (contrast 0.75, bound 0.72) instead of being refused there (0.62) and
rescued by the twin-immune zone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Supercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own
limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the
SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice
reflection predicted on the same frame lies within r3, and each resolution shell is read against the
median of the six other parity classes - which a single doubling leaves empty - as an excess in units
of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z
beside it ("Supercell probe to the limit" in the log).
Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its
noise and at more than twice the strongest of them, in at least three shells), the run integrates and
merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along
the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's
merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both
(DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT
(the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the
class reaches a quarter of the lattice's intensity in the finest shell both are measured in).
The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a
triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed.
Measured on forced doubled-cell merges (rc175 b530c2d base): the open-arm crystal deposited in the
doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell
(odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals
deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at
8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low
resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the
neighbour correlation, the separation from the nearest lattice reflection after the r3 filter -
separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call.
Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no
longer advises -C with the doubled cell, which the indexer reduces back to the half cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The line put the distinct-reflection count against a normal-approximation bar, which at a
handful of reflections reads the wrong way: a control the exact test rejected (5 vs 2 by
chance, tail 10^-2.70 against 10^-3) was printed 0.59 above the bar. It now prints the exact
binomial tail and the significance, both as log10, with the margin in decades and the noise as
the tail's change over one binomial standard deviation of the count. The tail is summed in logs
(ValidationEvidenceLog10Tail), so it stays finite for a real lattice; the decisions compare the
same tail in log10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The final no-crystal test (RejectChanceLattice) used the 0.001 binomial level of the comparisons
between candidate lattices. The lattice it tests is the winner of some fifty indexing attempts,
each fitted to an accumulated cloud that already holds the long-lived reflections it then re-finds
on the validation frames, so that null is optimistic by roughly the number of attempts. The two
in-house no-crystal controls passed it by one or two distinct reflections (tails 10^-2.06 and
10^-2.70 against 10^-3).
The test now asks 1e-9 (Z ~ 6); the rescue comparisons keep 0.001. With two reflections by chance
the controls would now need 15 distinct reflections instead of 4-5. Real lattices are tens of
decades past either level (weakest seen: 10^-83).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The projected histogram is sampled every 1/(2*max_length) of 1/d, so a row
of length L between max_length and 2*max_length folds to 2*max_length - L and
peaks there as sharply as a real row. At the default 500 A bound a 602 A axis
came back as ~400 A and a 701 A axis as ~299 A; the folded cell still indexed
59/60 and 37/60 validation frames (only the reflections whose index lands on
an integer of the folded period), so the long-axis rescue, gated on a poor
pass, never ran. The same fold also turned real >500 A lattice combinations
of shorter-axis cells into spurious shortlist rows.
Each shortlisted row is now asked once, by a direct Fourier sum over the spot
positions (no sampling, so no fold), whether the spots carry its own length or
its partner 2*max_length - L more strongly, and keeps the stronger. Partners
within 0.1 of an integer multiple are left alone (a harmonic, settled
elsewhere). Refinement accepts rows up to 2*max_length when one was unfolded.
Measured: a hexagonal P6122 c=602 A EIGER set and a P6522 c=701 A CCD set,
both indexed de novo on the folded cell before, now index the deposited cell
60/60 and merge to R-free 0.31 / 0.22 against the models (the same numbers as
-C with the deposited cell). Of 13 long-axis battery sets, 7 are bit-identical
(p.hkl md5) and 2 never unfold a row, 8v4o and 9yl4 have the same intensities (CC 1.0) without needing
the long-axis rescue, 4mtk refines a cell closer to the deposit at half the
wall time, 4nwv merges better (R_meas 0.58 -> 0.47, ISa 6.7 -> 8.4).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Stage A offered the b-unique two-fold (reference setting) on the lattice's holohedry alone, but held
its a- and c-unique siblings (pass-1 settings) to CellHostsRotations at 2e-3. On a cell refined free
~0.2 deg off orthogonal only the b-unique rung existed, so when the holohedral 222 was refused the
setting, not the data, chose the two-fold axis. Stage B already offers non-reference settings
without asking the cell, for the same reason; Stage A now does likewise where a holohedry is given.
With siblings of one order eligible together, ties were broken on the class CC alone, which a twin
law at fraction ~0.4 can win. A twin law cannot extinguish an axial row (every principal row maps to
itself or its Friedel mate), so ties now go first to the group whose pure two-fold axes carry the
more screw evidence on their own odd axial row - judged exactly as Stage B judges that zone - and
only then to the CC. Rows on 3-, 4- or 6-fold axes are not read.
Open arm: 6p8j (deposited P2_1, b = 152 A) was P 1 1 2 on the 88 A axis, now P 1 21 1 on the 152 A
axis (152 A odd row 158 nats, 46 A row 36, 88 A row refused; R-free 0.214). 9hnc and 5cc8 p.mtz
md5-identical (siblings offered but refused by added-operator R). On five passing P2_1 sets in an
orthorhombic metric the true axis carries all the evidence (offline replay), so no change there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The background ring's high-side clip, mean + 4*sqrt(mean), assumes counted photons. A CCD reports
ADU: on a Rigaku Saturn a background pixel scatters 4-5x its mean, so the clip sat at ~2 sigma, cut
the upper tail of every ring and added a positive offset to every reflection, compressing the
intensity statistics (a false 422 on one twinned I41 crystal).
No header carries a usable gain (SMV states none, every marCCD file the writer default of 1 ADU per
photon), so it is measured: background_gain::MeasureFrame takes the variance per unit mean of the
difference of pixels two columns apart, per level, clipped, on every fourth row of up to 16 of the
pre-scan's frames; Gain is the pair-weighted median over the levels. The clip becomes
mean + n*sqrt(max(g,1)*mean) (BraggIntegrationSettings::bkg_pixel_gain, CPU and GPU engines).
Only for SMV/marCCD input (ProcessConfig::counts_in_adu, set from the file format); HDF5 and CBF
keep g = 1, the identity. One log line reports the value. No new option.
Measured g: Saturn 944+ 4.7-5.2, Saturn 944 2.7, A200 0.68, ADSC ~0.3, marCCD ~0.3 - so the
ADSC/marCCD/A200 sets keep the photon clip exactly.
Checks (reflection data via gemmi and p.hkl, against the rc175 battery outputs / base build):
identical on 11if, 7qis, 8a1a, myob_x10sa, cytc_x10sa (photon counting), 4bwl, 5epe (ADSC,
marCCD) and 6cee (A200). Saturn sets, battery baseline -> this change:
3r6o I4122 R_meas 19.3% ISa 4.4 <|L|> .321 -> I41 (correct) 12.1% 8.1 .384
3mc4 R_meas 10.2 -> 9.8 %, ISa 12.1 -> 12.7, d_min 1.79 -> 1.75 A, <|L|> .360 -> .391
5uth 12.6 -> 11.8 %, ISa 10.4 -> 10.1, <|L|> .426 -> .483, offset vs |Fc|^2 0.21 -> 0.00
5vml 10.2 -> 9.6 %, ISa 11.0 -> 11.6, <|L|> .426 -> .475, offset 0.15 -> 0.05
5cc8 8.1 -> 7.9 %, ISa 10.8 -> 11.2, <|L|> .469 -> .494
3meb 14.5 -> 14.6 %, ISa 13.2 -> 13.2, <|L|> .442 -> .457
Space groups unchanged except 3r6o. Pre-scan cost: +0.5 s on a 9 Mpx ADSC sweep, within noise on
a 15 Mpx marCCD sweep, nothing on HDF5/CBF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ReadAttrStr and HDF5DataSet::ReadString read every string as fixed-length,
sizing the buffer by H5Tget_size. For a variable-length string (what h5py
writes for a str, UTF-8) that size is a pointer, so the read put a char*
into the buffer and returned garbage. transformation_type then never equalled
"rotation", and an EIGER master whose goniometer attributes were written that
way (depends_on, transformation_type, units on /entry/sample/transformations)
was processed as stills although it holds a valid rotation sweep.
All string reads now go through one helper, ReadHDF5String, which handles both
fixed- and variable-length strings in either character set; ReadElement<string>
already handled the variable-length case and now uses the same helper.
Test: JFJochReader_Goniometer_VariableLengthStrings reads the rotation axis
from a master with fixed-length ASCII and with variable-length UTF-8 string
attributes and depends_on dataset.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Near-perfect twins of a subgroup are undecidable from intensities, and a merge
in the higher group cannot be twin-refined. Where the adopted group may have
absorbed a twin law - a twin gate refused it and was overruled, or a group above
P1 was adopted on a merge that read twin-like before the search - the same
observations are also merged in a subgroup and written beside the answer, with
the twin law and its fraction in _pdbx_reflns_twin.
The subgroup is the one a decision named (the group a refused promotion was
adopted over, the indexed lattice's group a metric re-ask promoted from), else
the index-2 subgroup the search confirmed as a finalist whose lost two-fold
agrees worst with the intensities. It is merged from the same engine, to the
answer's resolution, without writing per-frame scales back; it carries the
answer's free set (flags read off the answer's merge by the adopted group's
asymmetric unit) and is written in the subgroup's reference and standard
setting (a C 1 1 2_1 description is reduced to P 1 2_1 1). The fraction is
Yeates' H on the pairs the law relates (1/2 - median H, a lower bound).
At most one file; other qualifying subgroups are named in the report
(TWIN_SUBGROUP_* keys). The answer's files are byte-identical.
IndexTwoTwinSubgroups and EstimateTwinLawFraction are new and tested; the
point-group ledger now carries each hypothesis's representative group.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Whether a twin law can exist was read off a hard-coded list of Laue classes
(4/mmm, 6/mmm, m-3m and rhombohedral -3m "holohedral", everything else
eligible). That is a property of the lattice, not of the class: a C-centred
orthorhombic or a monoclinic lattice with no special metric admits no twin law
for mmm or 2/m either, and a pseudo-merohedral one admits a law the class says
nothing about. On five pseudo-merohedral sets the report said "Twinning
INDICATED - refinement needs a twin law" where the lattice admits none.
TwinLawPossible asks gemmi's find_twin_laws (Le Page two-folds within 3 deg,
as RfreeFlags and ReindexAmbiguity already do), and AnalyzeTwinning takes the
cell to decide it; in P1 or without a cell a law stays possible, so the
pre-promotion analysis and the space-group search are unchanged. Twin-like
statistics (<|L|> below 0.44) above the false-operator bar where no law is
possible are now reported as NO_TWIN_LAW: not a twin in this group, so the
symmetry may be too high or something else narrowed the distribution.
Report-only: no reflection file changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The refined geometry is no longer written as a whole p_XDS.INP (and the
report's XDS_INPUT= key is gone). Instead the report carries one section,
XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the
result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the
detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with
X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated,
as verified with XDS IDXREF). No cell or space group: XDS determines those.
The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste.
The same keywords were logged by LogXDSGeometry at the start of every pass,
with the axis in rugnux's sense; that printing is removed, so the report
section is the one place they are stated. Docs updated (RUGNUX_REPORT,
RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL).
Output only; p.mtz md5-identical on 3 battery sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery
sets - most headers are a few pixels off. It now fires beyond three times
the tolerance; between one and three times the summary's Beam centre line
states the offset and recommends correcting the beamline value, without a
warning. The background-only branch (runs that refined no centre) uses the
same bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged observations already leave a run as the unmerged MTZ (and
the scaled fulls as the .hkl), and deposition does not come straight
from rugnux, so the _diffrn_refln block made the .cif up to ten times
larger for no consumer. The .cif is one data block again; the keys,
software rows, metadata, R_merge/R_pim/CC* and provenance stay.
ScaledFull loses the scale and frame it carried only for that block.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A rotation run now writes the geometry it integrated at as an XDS input
beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the
detector axes carrying the tilt, wavelength, the rotation axis the run
integrated at (ProcessResult::used_rotation_axis, after any sign rescue),
oscillation (with an adopted rotation scale), starting angle/frame, data
range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left
commented where the group is not in its number's reference setting),
FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the
master file as an x_??????.h5 template and a LIB= line to point at the
reader plugin; a CBF series gets its ???? template and its own frame
numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch
(2010) Acta Cryst. D66, 125-132.
The rotation axis goes out negated: rugnux turns the crystal by -phi about
its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with
XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis
indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the
CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged
(apart from the job list): 99.8% complete, ISa 23.3, direct beam within
0.15 px and cell within 0.07 A of rugnux's.
Output only; no result changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Overloads: the integrated reflections whose signal disk held a saturated
pixel are counted on the integration outcome before the merge, over the
whole range and over the reflections coarser than 4 A, where they
concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An
OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution
reflections, and recommends attenuation / a shorter exposure. Measured: a
strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an
ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%.
Overlaps: both integration engines count the signal disks a nearer
neighbour's prediction takes pixels from (the overlap ownership the default
--overlap exclude already builds), and those left under --overlap-minpk of
their own - BraggIntegrationCounts::overlapped / overlapped_major (log line
"Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning
fires from 1% in the second and recommends a longer distance, finer slicing
or a smaller beam; on 10 battery sets it fires on the two long-axis ones
(8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both;
documented in RUGNUX_REPORT.md.
Counters only; nothing reads them. p.mtz md5-identical to the rc175
battery on 10 sets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The merged .mtz and the unmerged .mtz carry, after the existing "From
Rugnux <version>" record, HISTORY records with the git commit and date,
the command line, the input file (absolute path) and a cheap identity of
it - size and modification time, the master file's for HDF5, nothing
hashed - and the collection date. MTZ allows 30 records of 80
characters; a longer value runs on over records starting with '+', cut
so that no record ends in a blank a reader would strip.
The mmCIF names the commit in _software.description and the command
line, input file and identity under _jfjoch_provenance (the local prefix
the other private items use; no PDBx item holds them), the first two as
text fields so any character survives.
OutputProvenance is filled once in the CLI and travels in ProcessConfig
to every writer of a reflection file. Reflection data are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the
entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on
_reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/
_diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a
loop naming Rugnux for 'data reduction' and for 'data scaling'.
The experiment as the input states it: _diffrn.ambient_temp from the
sample temperature, the source's general class (from NXmx
/entry/source/type, which the HDF5 reader now reads), facility and
beamline, the detector make/model in _diffrn_detector.type (where
.detector used to carry it) and its class in .detector where the name
settles it, and the collection date from /entry/start_time.
R_merge and R_pim are formed beside R_meas from the same per-reflection
sums (stills and rotation merges) and written with CC* overall and per
shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the
_reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as
three new columns of the shell tables. R_pim following Weiss (2001),
CC* following Karplus & Diederichs (2012).
A second data block, data_sample_unmerged, carries the scaled unmerged
observations of a rotation merge as _diffrn_refln (measured index,
frame, scaled I and sigma, pdbx_scale_value) - the same records as the
.hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now
carries the scale and the frame. The unmerged MTZ stays unscaled.
Reflection data are unchanged: the merged _refln rows, the .mtz columns
and the .hkl are written as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The pre-scan's beam-centre check and the post-refinement both measure where
the direct beam lands, but the report only carried the centre the run used.
Section 1 now sets the file's direct beam (FILE_DIRECT_BEAM) beside the
background's (BACKGROUND_DIRECT_BEAM, with its sigma), the shift of the
refined centre from the file's (BEAM_CENTRE_SHIFT) and the tolerance the
geometry asks for (BEAM_CENTRE_TOLERANCE: BeamCenterNeed_pxl, the number the
log's check prints, here on the run's own cell). A shift above the tolerance
is a BEAM_CENTRE_OFF warning naming the file's, refined and background
centres, the difference in px and mm and the value to correct the beamline
to; a run that refined no centre is warned on the background alone where
its fit resolves the difference (3 sigma). The summary gains a Beam centre
line.
The refined centre, not the background, is the trigger wherever one exists:
on a battery set where XDS's refined direct beam and rugnux's agree to 1 px,
the background fit is 12 px away from both. Report-only; no result changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Structure factors, maps and the null on the GPU (with gemmi's mask shrink step), the second
validation started on a forecast beside the first, change-of-basis candidates scored at the same
time, validation work spread over the visible cards, the merge's GPU engine freed before the
validation, a CUDA error failing the validation instead of restarting it on the CPU, and no
first-validation maps where the forecast one is kept. Output files md5-identical to the CPU path's
reflection files and placed model on 8sa8, 8xtg, 9ea5, 8tyy, 7qis and myob; maps within float
tolerance of the CPU path and bit-identical run to run and at -N 8. Validation time 8sa8 24.0 ->
7.4 s, 8xtg 18.4 -> 5.0 s, 9ea5 19.1 -> 3.0 s, 8tyy 47.7 -> 3.0 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two
threads that drive the merging tail were not: the quality guard took its card with set_gpu(1),
and the main engine runs on the pipeline's own thread, which goes on to everything else. On a
multi-socket server their pageable copies to the card went through a remote socket.
RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost,
IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous
device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding
covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads
the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use
std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not
squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses
pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask
themselves; the new test checks both that and that a pinned caller keeps its own mask.
No flag, no environment variable. On one NUMA node (this workstation) only the device setting
happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base =
rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa,
cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs),
-N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise,
e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for
multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ModelStructureFactorsGPU computes what compute_model_factors() and
map_from_coefficients() compute on the CPU - F_calc from the model's
density (IT92, Refmac-compatible blur, unblurred as prepare_asu_data()
does) and F_mask from the Refmac bulk-solvent mask, both on the
reflections prepare_asu_data(d_min) lists, in its order; and a map from
ASU coefficients on the grid get_size_for_hkl(coef, 0, 3.0) sizes - on a
device. Made once per cell, group, resolution and model, then evaluated
as often as the coordinates change, so refinement or MR can call it in a
loop. The device is an explicit parameter; every call leaves the calling
thread's current device as it found it.
Pieces:
- ModelDensityGPU: the rigid body's deterministic brick gather, moved
out of RigidBodyGPU.cu into a component of its own (ModelMaskGPU's
pattern); the rigid body uses it unchanged. MAX_BRICKS_PER_AXIS 8 ->
16, so fine grids with high-B atoms (lysozyme at 1.2 A, a 0.9 A P1
cell) are no longer refused; existing zones are gridded identically.
- One copy of the content is gridded and the symmetry composed in
reciprocal space (SymmetryComposition), operators applied on the fly;
the mask is ModelMaskGPU (every image of every atom, islands, shrink).
- Maps: gemmi's get_f_phi_on_grid() in ZYX order on the host (the
coefficients written are the same), in-place cuFFT c2r, transposed back
to XYZ on the device. One map at a time, in the engine's buffers.
Decided once, up front, per card, from its TOTAL memory: the engine's
bytes (16 N + cuFFT work + reflections, N the larger of the structure-
factor and map grids) must be at most half the card - the rigid body's
engines take at most a quarter beside it. Otherwise, or where the gather
cannot grid the cell, the CPU path runs, logged with needed vs total.
Anything to a resolution other than d_min (the null's 3.5 A fits) stays
on the CPU, so all replicates and the real model's side of the null are
computed the same way. A CUDA failure takes the existing path: the
validation restarts on the CPU.
Measured, model validation total per run (CPU path -> GPU), 16 GB card:
F432 215 A cubic, 1.30 A, 500^3 grid: 47.7 -> 15.6 s (two validations;
14.3 -> 2.4 and 33.4 -> 13.2, the rest of the second is writing the
three 0.5 GB maps); F_calc + F_mask 5.7 s -> 0.05 s
C2 1.11 A: 23.9 -> 13.6 s; P3_2 1.55 A: 18.2 -> 10.2 s;
P2_1 1.25 A: 13.4 -> 6.5 s; F4_132 328 A: 13.0 -> 5.5 s;
P6_5: 8.8 -> 4.2 s; P4_3 0.97 A: 4.6 -> 2.5 s; P1 0.92 A: 4.2 -> 2.2 s;
small P1: 3.2 -> 1.3 s; P6_1: 8.8 -> 6.2 s; lysozyme: 1.8 -> 1.4 s.
p.mtz md5-identical on all 13 sets. Against the CPU path: FC within
1e-4 of mean |F|, phases of the strong half within 0.003 deg, maps within
1e-4 (2mFo-DFc) and 7e-4 (mFo-DFc) of their rms; every logged R, CC,
FOM, k_sol and anomalous site list identical at the printed precision,
except where a rigid-body commit sat on an exact R-free tie (0.2155 ->
0.2155) and fell the other way (R-work 0.2127 vs 0.2129). The GPU result
is bit-identical run to run and with -N 8 (maps, map MTZ, placed model).
Peak device memory of the engine: 2.5 GB at 500^3 (process total peaked
at 14.4 GB with what the merge still holds).
Tests: ModelStructureFactorsGPU_MatchesCPU (five groups, 3.5 and 1.5 A:
same reflections, F_calc <= 1e-5 of mean |F|, F_mask 2e-7 rms, repeat
bit-identical), ModelStructureFactorsGPU_MapMatchesCPU (<= 5e-6 of rms).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The GPU bulk-solvent mask had no shrink step (SolventMasker::shrink(),
set_margin_around() with Refmac's r_shrink = 0.8 A), so every rigid-body
zone whose grid has an offset within 0.8 A was sent to the CPU. That is
not only fine grids: in an oblique setting (P2_1, beta ~141 deg) the
3.5 A zone's lattice-plane spacing is 0.69 A, and on such a set the real
fit and all nine null replicates ran their rigid bodies on the CPU.
The shrink is now two passes over the grid: mark the solvent points with
a macromolecule point among gemmi's near offsets, then turn into solvent
every macromolecule point with such an edge point at any stencil offset
(near or far, gemmi's split at the coarsest grid step). That is gemmi's
rule in both of its branches, and the stencil is built on the host in
double exactly as gemmi builds it. ModelMaskGPU::ShrinkIsNoOp() and
RigidBodyGPUEngine::MaskSupports() are gone; the rigid body no longer
refuses such zones.
Verified (ModelMaskGPU_ShrinkMatchesGemmi): the shrink alone on gemmi's
post-island mask is bit-identical, and the whole GPU mask equals gemmi's
put_mask_on_grid() bit for bit (0 differing points) on the five test
groups at 1.5 A and an oblique P2_1 cell at 3.5 A; repeats are identical.
Measured on the oblique-setting set of the open arm (P2_1, 14k atoms,
1.66 A): model validation 17.2 s -> 5.2 s with the GPU structure factors
of the next commit in place (the first validation's ten rigid bodies,
~12 s on the CPU, now take well under a second). p.mtz md5-identical;
the null moves from +73.1 to +86.1 sigma (the replicates are refined on
the GPU, which agrees with the CPU to rounding), verdict unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
--model validation (battery-only for users) was 18% of the battery's time. Every
number it produces is unchanged to the bit (p.mtz, maps, placed model and every
model-validation line of the report md5/diff-identical on 11 open sets); only
when and where the work runs changes:
- The bulk-solvent grid fit (FitModelScale, most of the CPU time) fits each
solvent pair on a copy of gemmi::Scaling's target that takes
|Fcalc + k_sol exp(-b_sol s^2) Fmask| once per pair instead of at every
solver evaluation; same expressions, same types (new test checks a grid
point against gemmi's own Scaling fit with ==).
- Fcalc density and the solvent mask are made on two threads; the model's
structure factors beside the GPU engine reservation.
- The indexing probe fits the relabellings concurrently.
- The null's replicates run beside the real model's placement (they start
from a snapshot of the model as read); one GPU engine per replicate plus
one for the real fit instead of a cap of 4 (engines are interchangeable
and deterministic).
- The 2mFo-DFc, mFo-DFc and anomalous maps are made and written
concurrently; the placed model is written beside the reflection files.
- A rigid-body zone whose solvent-mask grid needs gemmi's shrink is sent to
the CPU when the engines are reserved (ModelMaskGPU::ShrinkIsNoOp), instead
of failing on the GPU and validating everything again on the CPU - the
same CPU result, without the wasted first attempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Commands and configuration writes (initialize, disarm, threshold change) can
take minutes. With one shared connection a status poll waited behind them;
now commands/config writes go through one kept-alive client and status/config
reads through another, each with its own mutex and fixed timeouts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Error in detector operation" was raised whenever the wait-for-idle loop in
JFJochServices::Stop() ended in any state other than idle. For DECTRIS that
includes a single failed status request (connect/read timeout, non-200), which
GetState() silently turned into NOT_CONNECTED, and a cancel whose disarm
failed - none of them distinguishable in the log.
- DectrisDetectorWrapper::GetState() logs the failure reason before returning
NOT_CONNECTED.
- Stop() logs the state that ended the loop, whether the stop command failed,
and what the detector reports (DetectorWrapper::GetErrorDescription(); for
DECTRIS the SIMPLON state and the detector status "error" list).
- Cancel() logs why the stop command failed.
- A non-200 GET now reports the HTTP status and body (it printed "Success").
- DectrisSimplonClient keeps one keep-alive httplib::Client, serialised by a
mutex, instead of opening a new TCP connection per request.
- The wait-for-idle loop polls every 100 ms instead of every 5 ms.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first pass's last gate - is the lattice this crystal's at all - compared the validation spots
on the lattice with the median of five displaced-spindle-angle nulls, against the null's binomial
noise floored at one spot. On the two in-house no-crystal controls it is fed a handful of genuine
stray reflections from something rotating with the sample (the same detector positions, a pair
symmetric about the beam 198 frames apart, in both sweeps, offset by the 15-frame start difference),
plus one reflection near the spindle axis that stays in diffraction for ~46 frames. With the
decisions taken from the first part of the sweep the 60 validation frames are 3 deg apart instead
of 6, so that one reflection is counted on three validation frames, and the gate passed on
4 spots of 151 (null median 0) and 7 of 72, where the whole-sweep flow had seen 3 and 3.
Now the evidence unit is the distinct Miller index - a reflection on several validation frames
is one piece of evidence - and the test uses every displaced angle's count: with no crystal the
recorded angles are one of six exchangeable arrangements, so the gate is the binomial tail of the
recorded arrangement's share at p = 1/6, at the same 0.001 significance (SPOT_BUDGET_SIGNIFICANCE_Z).
No threshold is added; the median and the one-spot floor go from the gate. The comparisons between
lattices (ValidationEvidencePrefers) are unchanged.
no-crystal control 1: 4 distinct vs 2 over 5 wrong angles, tail 0.009 -> no lattice (was P 1)
no-crystal control 2: 5 distinct vs 2, tail 0.002 -> no lattice (was P 1)
p.mtz md5 identical to rc175 7c9a6c7d1 (GPU) on 5nw5, 5reo, 7atg, 8v4o, 9qw8, 9rci, 5ky6,
insu_I_x06da_low_isa, myob_x06da_powder_2; cytidine and myob_x06da_split pass. The weakest real
lattice in the open/in-house/private battery logs has 117 spots against a null of 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With a DECTRIS detector the broker starts the writers - and so learned the collection's start
message - only when the detector's own start message arrived, after /start had returned. A reader
started right after /start found no collection (404) and had to wait for, and race with, the
detector.
- /start registers the collection in LiveCollection (Expect) as it is accepted, so GET /live/events
is accepted from then on. The image pusher fills in the start message when it starts the writers
(Begin) - at once on the FPGA path, when the detector's start message arrives on the DECTRIS path.
- The event stream's first event is now `start`, carrying that start message (CBOR, base64) with
run number, file prefix, image count and images per file; it replaces `collection`. Until it is
known the stream only keeps alive. file and end follow as before.
- However the measurement thread ends, it ends a collection the pusher did not end (EndIfOpen), so a
cancel or a failure before the detector streamed reaches the reader as `end` with an error. The
wait is event-driven: it ends with the start message or with the measurement.
- GET /live/start.cbor is removed: the stream carries the start message, one route less.
- rugnux takes the start message from the stream (BrokerFeed::WaitForStart), so it can be started
right after /start; files reported before it attached the reader are replayed to it.
Tests: LiveCollection_ExpectedAtStart; BrokerFeed_StartFromEventStream (an event stream whose start
message comes late, then a file and the end).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
rugnux <path>/<prefix>_master.h5 http://<broker>:<port>/ - when the master file is not on disk yet -
processes the broker's current collection as it is written:
- the start message comes from GET /live/start.cbor; the given path has to end in the collection's
master file name, and the data files are read next to it;
- JFJochStartMessageReader builds the dataset from the start message (detector, geometry,
goniometer, mask) and reads the images straight from the data files; a read of an image whose file
the broker has not reported waits for it;
- BrokerFeed reads GET /live/events in a thread of its own and hands the reader each closed file. On
end, the run finishes with the images the collection delivered (a read past them returns no image).
A broken stream, another collection, or a refused follower makes every waiting read throw
LiveCollectionLost, which ends the run like a fatal resource error;
- the token is taken from JUNGFRAUJOCH_HTTP_TOKEN (the variable the viewer uses) and sent only in
the Authorization header; without it, and with no master file on disk, rugnux stops at once;
- rotation data and --mode mx only; plain http only (TLS is not linked into rugnux).
With the master file on disk, the URL is ignored and the run is offline as before. The streamed run
does not read the master file and is not compared with it: its result can differ from an offline run
of the same files in the last bits of the metadata.
Tests ([Live]): LiveCollection, the server-sent-event parser, the FileWriter callback, the TCP
pusher's closed-file reports with two real writers, and JFJochStartMessageReader against the
master the writer produced.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
So that rugnux can process a collection while it is written, the broker reports each data file once
a writer has closed it and renamed it into place:
- Writer: FileWriter takes a callback, called for every closed data file. StreamWriter uses it on TCP
to send an acknowledgement with ack_for = FILE_CLOSED (new TCPFrameType 10): file number and the
images the file holds. TCP protocol version 5. Files closed at END are reported before the END
acknowledgement.
- LiveCollection (image_pusher): the start message the writer of the master file received, the
closed files, the end. Fed by the TCP pusher (FILE_CLOSED acknowledgements; end once every writer
acknowledged END) and by the in-process HDF5 pusher (the FileWriter callback). The ZeroMQ pusher
has no back channel and offers none.
- Two routes, protected by the dataset's bearer tokens like the other dataset routes:
- GET /live/start.cbor: that start message;
- GET /live/events: server-sent events (collection, file, end, superseded) with the history first
and a keepalive every 10 s. One follower at a time; a second gets 409.
Documented in jfjoch_api.yaml; they add no schema, so the generated C++ model is unchanged.
- The HTTP server's thread pool is set explicitly (16, up to 64): an event stream holds a thread for
the whole collection.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The final master marks a finished collection for anyone watching the directory (rsync, autoprocessing,
a reader following the collection), so it must not appear before a data file it links to.
- FileWriter: at END every data file is closed (renamed into place) before the master is finalized
and renamed. A data file that fails to close does not cost the master; the failure is reported
after it. For NXmxIntegrated, file 0 is the master's own image dataset, so there a failure still
stops the master.
- With several writers, the master was written by the writer of the first socket as soon as it got
END, while the others could still be closing their data files. END now goes to every other writer
first:
- TCP: the connections are taken in reverse order, and each END waits for its acknowledgement,
which a writer sends only after closing its files;
- ZeroMQ, with a writer notification socket: END to every other socket, then their notifications
(the same ones Finalize collected before, now collected here), then END to the first socket.
Without a notification socket there is nothing to wait on and the order is all it changes.
Tests: TCPImageCommTest_MasterWriterGetsEndLast (three mock writers, the master's END after the
other two acknowledged), StreamWriterTest_ZMQ_MasterAfterAllDataFiles (two real writers, the second
started only after the collection was sent; the master's ctime is after every data file's). Both
fail without this change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Plugin 1.1.0 (2629f3821) passes a saturated pixel of a signed image (INTx_MAX) through with its
value so XDS flags it against OVERLOAD; only the error marker (INTx_MIN signed, UINTx_MAX unsigned)
becomes -1. The legacy test still expected INT16_MAX -> -1. It now checks the saturated value
explicitly and also that module-edge (bit 30) and chip-edge (bit 31) mask pixels read as -2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The fused adaptive spot finder's azimuthal profile (and the standalone AzIntEngineGPU) summed the
corrected pixel values - pixel times a float correction - with float atomicAdd, in shared memory
and then across blocks. Float addition is not associative, so the per-ring sums depended on the
order the threads and blocks happened to arrive in, and on the grid size chosen for the GPU. The
profile, and the per-image background estimate read from it, moved in their last bits between two
runs of the same command: on a thaumatin JUNGFRAU 4M sweep one bkg value in _plot.txt differed in
the 4th digit between runs. The raw ring sums that drive detection were already exact integers;
only the reported profile was affected.
Each corrected value (and its square) is now quantised to 2^-12 on its own and summed as a signed
64-bit integer, the convention the raw ring sums and the Bragg-integration profile already use.
Integer addition is associative, so the sums are the same whatever the order or grid.
The repeat test of the fused finder now also requires the profile and its std to repeat bit for
bit (it failed on the old kernel at the first repeat); the AzIntEngineGPU test re-runs 20 times.
Fused finder benchmark: 0.313 -> 0.324 ms/frame.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
